Yes. At work we recently finished a complete rewrite of the platform. The old codebase got abandoned and two new codebases got stood up. Previous stack was Phoenix LiveView and the new one is Phoenix API + Vue /w TypeScript. Zero code shared between the two. We took the opportunity to re-architect a lot of the core functionality and get rid of the tech debt that had been hounding us and killing our velocity for the past six years.

We finished the rewrite in two months. A little over 700k LoC total. 95% of it was planned, orchestrated and written by AI. We could have gotten it done faster, but we were well aware of the potential for slop and thus paid an absurd amount of attention to (and tokens for) code quality. For example, there is a 100% test coverage requirement in both server and client, combined with AI-driven review rules that say all tests must be non-vacuous, plus a ton of invariants enforced via deterministic checks. Everything from standard linting and formatting rules to AI-necessitated stuff like "comments cannot be more than five lines each" is in there. At the end of each week, we get together for ~30 minutes to review each bug that was fixed that week (there haven't been that many) and try to figure out if more deterministic checks can be added (or new ones extended) to prevent that class of bug from re-appearing. We actually have an Architecture epic with a dedicated Fable agent orchestrating all the initiatives and tickets in it and keeping track of improvements (and the occasional regression). So things like accessibility, usage of shared helpers, enforcement of common patterns, etc. are all kept track of.

The difference between the old codebase and the two new ones has been night and day. The new codebases, combined, are ~35% larger than what we had before, and yet there are way fewer bugs (we actually got suspicious at first and had to check to make sure our logging instrumentation was configured correctly). Performance is also stellar, features have been a lot easier to add and extend, and everything is a lot easier to find. When someone asks a question or has an issue we have a pretty good idea of what part of the codebase to look in and we're usually right.

It wasn't all unicorns and rainbows. Not everyone on the team was proficient with AI-driven development, so they had to be brought up to speed quickly and taught the tools and also what works and doesn't. Code reviews were also difficult and time-consuming. We had to deal with our anxieties about working in two new codebases none of us had authored much code in ourselves. What made all of it worse was that we were racing towards a deadline imposed by external factors so we all ended up working a lot of hours. It was essentially between getting it done this summer or waiting until next summer and we bit the bullet and did it. It absolutely would not have been possible without AI.

We're mid-way through a similar process at work. Rewriting a legacy app in a new language, with new architecture and new features.

And it's a mess.

We're at 10x loc (admittedly, the new programming language is more verbose than the old one), comments make no sense. Yes, we have ~100% coverage, but most of the tests are meaningless. The agent keeps removing our tests to replace them with tests that are easier to pass, breaking code invariants, removing all the engineered data structures and replacing them with stringly-typed code, etc.

And of course, given the number of LoC (and the fact that the agent rewrites so much code all the time), it's physically impossible that all of them were reviewed by a human being.

AI made it possible, insofar as upper management would never have greenlit the project without AI, but I can't escape the feeling that we're building on quicksands.

If your code base is script-kiddy garbage, your models are only going to keep writing more garbage. You should throw it all away. I expect that isn’t an option. Try instead instructing the model that the code you have is legacy garbage written in a hurry that needs siginificant refactoring, but that right now I need one new feature written to these new standards instead. Take it one small piece at a time. Don’t let context go beyond 100k. Good luck!

https://www.hanselman.com/blog/stringly-typed-vs-strongly-ty...

Reviews? They were the first casualty.

Can you be more specific? "Improve structure" can mean different things to different people.

We've ensured that agents strictly adhere to code architecture rules, using both agentic review rules and deterministic CI gates. Everything from file naming, location and namespacing to the "shape" of each file that shares the same role are consistent. We can trace functions of the same role across the same set of modules, with abstractions and code reuse where they make sense. Once someone learns the code in one part of the codebase, their understanding of that "spine" translates to other areas - the only differences relate to the specific business rules governing those areas.

>> And of course, given the number of LoC (and the fact that the agent rewrites so much code all the time), it's physically impossible that all of them were reviewed by a human being.

Code review was difficult at the start due to the volume of code. One insight we gained midway through though is that the value humans bring to code review is judgment and business context. So we created a code-atlas skill that creates an artifact for PR reviewers. That artifact highlights the most important parts of the code. For example, if a PR author has made the choice to use soft-deletes when the product uses hard-deletes for everything else, that is flagged by the code atlas, and a human reviewer can use that to ask questions about it. Meanwhile, things like boilerplate are de-emphasized; any problems associated with them are reliably found and flagged by AI reviewers anyway. The refreshing part about this is that this has almost completely eliminated bike-shedding: people no longer argue about module naming or whether a comment is worded properly.

Yeah, we did that, too.

But every time I end up, for some reason, digging up deep in the code, I realize that it's not nearly sufficient in our case.

2 months of 80 hour weeks is 4 months of regular weeks. Can you clarify how many hours your team was putting in? And how large was your team?

How?

I don't want to sound flippant, but if the point is to add human thought to the mix, that's a high review rate even when examining small tweaks to an existing, working product, even with substantial AI help to pre-filter major gotchas before you bother spending a lot of human effort on the review. That's only 20-30wpm, but a review isn't just scanning or reading code, especially if you're trying to figure out how a new system which doesn't run yet will fit together.

It's possible, I think, but humans cannot do repetitive mental work (like reviewing code) constantly without atrophy.

Can I review 130 SLoC/hour? Sure. Can I do it 80 hours a week? No. Not even a single 8-hour workday.

I could see a world potentially where they came up with a magic prompt allowing each proposed PR to be cohesive, shippable, well factored, and everything else you need to be able to actually review it at a higher level and be comfortable with the results, but I'm skeptical. That's a major innovation if they managed to do so even as a one-off, and that wasn't the thing they highlighted when talking about the project.

But then you need to watch for bugs coming from interaction with previous changes and in 700k loc that might be nontrivial. How do you know which states are reachable and which are not? That takes time.

It only takes a botched condition here (forgot a "not"? swapped "and"/"or"?), a swapped variable name there, code that looks ok, but isn't.

Also what kind of problems did you have with LiveView?

This is why my current view in terms of LLM productivity gains are:

- debugging and triaging 5x

- refactoring when you have tests in place 3x

- greenfield where you have zero tests is about 1.5x

not sure what you're point is here. It sounds similar to "we use to use blood letting and leeches and doctors didn't clean their hands and everything was fine so what are you getting at?"

Good tests have real benefits. The fact that people shipped without them in the past in no way suggests they aren't needed or have no point.

Are you getting older? A lot of people anchor their intuition of time and history to a certain year. There are probably still lots of people who think the 1990s is not that long ago even though it’s now over a quarter century since it ended. Maybe you mentally default to 2012 or so, when it might be true that most popular software shipped without automated tests (although manual QA was a lot more extensive in 1992).

But 20 years ago is now 2006, and unit tests were well established as a best practice. Perl had extensive automated tests in the late 1990s that everyone who ever compiled Perl would have noticed, since they were run by default and produced obvious output. Kent Beck’s “Test Driven Development: By Example” was released in 2002, and popularized both the name and practice.

I agree with the spirit of what you wrote, but my recollection of the timeline is different. The first decade of the 2000s was peak Crazy Agile Advocacy, but IIRC it wasn’t until the 2010s that unit testing really became almost universal practice. Much before that and it was still tangled up with XP, TDD and lots of other things that certainly weren’t universally accepted as good practices (notwithstanding the strident advocacy of a certain group of consultants/authors/speakers/bloggers and their fans).

I remember, back in the mid-2000s, when we had some consultants brought in to talk about different aspects of quality and testing. There were several working groups, each led by one of those external consultants, and one of them was about unit testing. This was in a relatively large software development organisation for the time, a few thousand people, and while some parts of the organisation had some form of automated testing operating by then, it definitely was not the case that the well-known products produced by the organisation all had a unit test suite. Other practices we’d consider routine today, such as peer code reviews, were also in their infancy during that period: some were doing them, many were not, and generally we had much less experience of how to do them effectively than we have today.

As an industry, I don’t think we really matured in how even the most ardent fans of unit testing were writing test suites until the 2010s either. In the 2000s, we still had lots of people mocking the entire universe and then writing unit tests that were 99% testing those mocks because of 100% test coverage requirements, and similar dogmatic nonsense.

By the 2020s, I think there was much more awareness of that automated testing is generally a good idea, but there are different kinds/levels of automated testing and finding a mix that suits each project’s specific needs is important. One of the great benefits from the more recent AI tools, particularly the agentic ones over the past year or so, has been that it has clearly demonstrated both the value of a good automated test strategy and how much of a waste of time vacuous tests are.

Not really, but I can see why some people think that.

We treat 100% test coverage as "required, but by itself not sufficient". It doesn't give us false confidence that everything will be perfect or anything like that. But it provides us with the discipline to make sure no corners are cut, and the bugs that are fixed don't come back.

One refreshing aspect was that during PR reviews we stopped debating whether something needed test coverage. Instead we focused on what was being tested and how.

I’d be curious to know what percentage of the time spent implementing tests would have otherwise gone to discussions about whether to implement them or not. ;)

Not really?

About 20 years ago, I was working on Firefox and we had millions of tests on CI. I was working on a host of other open source apps and they all had tests (most of them had no CI, of course).

However, what actually happens is that careless drivers often cripple or kill innocent bystanders in other vehicles as a result of their poor driving. That's why seatbelt laws and improved vehicle safety features are a good thing.

To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myself.

With Fable I can start with a general ask like "I'm trying to do X, can you investigate and tell me what the shape would look like" and have it poke around and think, ask me questions with single-choice or multiple-choice answers, then break the task into small chunks, each of which becomes a ticket.

With Astra, it's like pulling teeth. It often does not understand what I'm trying to do, takes things literally, does not go above and beyond (i.e. infer intent), and stops way too short of the actual goal. I have to constantly prod it and it's frankly exhausting.

I much prefer getting alerted when there's more than 1 approach to the problem and it's discovered mid-implementation.

I don't want to do the grunt work of writing code, but I do want to know the architecture and be responsible for the decisions.

Fable is also very good at pushing back when I propose something that will cost me. E.g. I'm working on a configuration layer above nix to manage my homelab fleet declaratively, and I tend to get into "config as new language", where Fable just goes - let's not do that.

Oh man! This also is a pet peeve of mine with Fable. I will look at what it's doing and say "Shouldn't it be done this way?" and then it will spend forever arguing with me that it should be done the way it wanted to do it. It seems to get stuck in a certain way of thinking and will insist its way is right until I can really prove it - or just go over to Astra.

Sam is admitting it now for two reasons:

1. He wants to make it sound like not IPO'ing is actually the rational thing to do 2. He wants to throw cold water on Anthropic's IPO in a few weeks

Number 1 is just spin. Number 2 makes it particularly nasty. Your opponent offers a fig leaf and you turn around and underhandedly try to sabotage them?

Sam doesn't have many redeeming qualities in my book. Just an atrocious, rotten person with a friendly personality.

Max Howell, creator of Homebrew, was famously rejected from Google for not being able to invert a binary tree on a whiteboard.

Well, I think the issue is that most people, even software developers, don't have the luxury of choosing. Everyone needs to put food on the table, and often times that involves working for companies one dislikes or even hates. The problem is now those companies are turning to AI.

Anthropic is structured as a Public Benefit Corporation with a strong charter. In addition, founders will have super-voting shares and so it won't be possible to push them out. Therefore the whole "beholden to investors" thing is not a concern.

I really hope I am wrong as my perspective is not anti-American, it's specifically anti-corruption and pro-democracy.

Golden shares, vetos, PBC, charter are all paper tigers , they only matter if/when the firm is self-sustaining business with no outside capital needed, the alternative to not listening to investors till then is crash and burn.

After that point, you will have to listen to the paying customers (sometimes but not always they are also users ) as they are ones now funding your organization.

Bottom line you are always listening to someone.

these are just words at a time and place.

sama showed that you can futz with it, and as long as you spend enough in court on judges, aint nobody gonna stop you

Although what keeps me up at night is the worry that it's easier to automate attack than it is to automate defense, and that containing these systems is a losing game. Could an optimally competent OpenAI succeed?

At some point I think we have to accept that turning a blind eye to their products hacking the world might actually be aligned with their commercial interests.

Yeah... don't do that. Don't make your job or career the center of your identity. You have to form and then internalize the belief that you are valuable not because you can offer value to others in a material sense, but rather in an intrinsic sense. You have experiences that nobody else has. You have meaning to others in your life, whether deep or not. Your thoughts and perspectives are unique to you. That is your real value.

"DeepSeek serves Claude instead of its own models and collects exchanges for model training"

Obviously. This is how they were able to score so high in benchmarks.