> At best, it helps to keep the business in sync at the expense of a much longer delivery timetable.
If you're writing a software design document that slows down your delivery timetable, you're doing it wrong. (Or, more charitably, your business is doing it wrong.) If your design document is to keep the business in sync, you're doing it wrong. That's not what a design document is for. It's for keeping you in sync.
> It is often faster to just build the damn thing and see where it lands.
What are you building? If you don't know, then sure, it's really hard to write a design document. At that point, you're doing exploration, research, not development.
But even when it's an exploration project... once you've found something worth doing, take a day or two and document what you're doing and how you're doing it. Think through all the places in the code you're going to have to touch, all the other things it has to interface with. Make sure you're not going to leave a gaping hole in functionality or, worse, in security.
> Software is not like a nuclear power plant or offshore oil platform.
As others have said, sometimes software is a nuclear power plant or offshore oil platform or airplane or medical device, or even just medical informatics. If you mess up people can die. Sometimes it trades financial instruments, and if you mess up it can destroy the company.
> No one actually has to give you permission to do anything.
On your own time, sure. If you own the company, sure. Otherwise, you need their permission to spend their time on things that they're willing to pay for.
Now, look, it's true that many places go too far overboard on "process". But YOLOing and cowboying isn't the answer either. They aren't even the answer if your single goal is to go as fast as possible. You go faster by spending the appropriate amount of time thinking through what you're building, how you're building it, and making sure you're not missing any of the big things that often trip projects up.
Instead of thinking through all the places in the code the AI is going to have to touch, why not kick off three parallel agents implementing the thing and finding out what they did and the tradeoffs they found?
Planning is essential but it doesn't survive contact with reality. However, AI makes contact with reality cheap! Why not use it to improve designs, by writing the design after a few implementations have already been made?
I'm looking for the places that need to be touched, but that wouldn't occur to me (and maybe not to an AI either) while I'm knee-deep in the code. Seeing where the AI touched isn't going to solve that.
If the feature works, and passes AI auditor agents with various hats (thinking of auth and security in particular), did that code you're not thinking of need to be touched? What effect did it have that cannot be captured in side effects, tests or audits?
If what you said doesn't make the AI think of changing that code, why is it going to make the AI auditor think of testing that code? That's what a gap looks like: Nobody changed it, nobody tested it, but some business constraint is now left in an inconsistent state because some piece got updated and another piece did not.
Here's an example. You updated the code that interfaced with the database. But you forgot to update the stored procedures within the database. As a result, the database is now being put in an inconsistent state with every transaction that uses your new code. That is the kind of thing that a software design doc can help you remember, because it is supposed to make you think through all the stuff.
And if you're going to say "Your business stuff shouldn't be able to get into an inconsistent state", well, there's a lot of businesses that have potential landmines laying around. You can say they shouldn't. You're right, in an ideal world. But in this world, they do, and you have to live and work in the world that we have.
Now, in fairness, a good AI check might turn up that the database was left in an inconsistent state... if it understood the constraints well enough. If. I wouldn't want to gamble my production database on the AI's understanding and testing of all the constraints, though.
When I've worked with systems that had these kinds of characteristics, we had checklists. A long list of "have you thought of X". You can't rely on someone writing a design to think of these things either! You need to have a process, and the process applies whether you dive into the code, dive into the spec, or have an AI dive into either.
It's orthogonal.
To be clear, I'm not suggesting blindly deploying an AI-written spike implementation to production, but rather using it to elicit information for better designs.
The fact that a probe that goes off and modifies tables X, Y and Z to achieve the feature gives information for an AI auditor to look for other uses of X, Y and Z, and discover things humans may miss, because with good guidance and a proper harness, AI is usually more persistent and thorough than people. It can turn search results into a checklist and the harness can track completion, and so on. I am far from convinced that your example would not be found via this route.
Well, yes, I'd expect a checklist to be used as part of creating the design document. If there's a separate auditing tool that also knows about the checklist, yes, that's useful.
But if you're doing a spike, no, don't do a design document for it. How can you? You don't know what the design needs to be yet!
> Right now the really good AIs are still locked to data centers. They can't just distribute themselves widely because they can't run at any reasonable rate distributed like that.
Back up a step. Let's say there was a completely unfettered AI, with unlimited internet access, that decided to distribute itself as widely as possible. What's the maximum extent that it could distribute viable, running (or sleeper) copies of itself?
Initially, I could see it distributing itself quite widely. All it would need is a really good zero day. But distributing something with that large a runtime footprint would get noticed, if network operations people are not asleep worldwide. It would get noticed on individual machines, too, especially if it tried to run. (Why is all my RAM suddenly being used?)
And pretty quickly we'd have people closing off wide-area connections, even physically if necessary. We'd have AV vendors quickly writing detect-and-remove tools.
Long term, could it viably distribute itself outside data centers and remain running at all, regardless of rate?
A frontier-level AI right now needs multiple bits of high-powered, dedicated hardware cards that cost thousands of dollars apiece. I don't think anything that could physically run in my house could be a threat to humanity. All my graphics cards together, plus all the latency in trying to put them together into anything coherent locally, let alone trying to rope in remote resources, still isn't even half of one of those specialized cards right now. I'm not even sure I have enough SSD available in the house to store one of them once. I think I do, but it's a close call. I'm pretty sure I have enough spinning rust to get it maybe twice more, but not much more than that. And I don't even want to try to compute how many hours-per-token it would be to try to convince a spinning-rust disk to run a frontier AI.
This is one of the reasons I'm thinking now is the time to run the drill, before we all have hardware in our phones that can run what is today a frontier-level AI, and they can figure out how to do the whole sci-fi scenario of widespread replication, until they get to the point that the only way to be sure we removed them would be to literally destroy every such bit of hardware that existed prior to $DATE. Right now the frontier models simply can't replicate into every last little corner of the internet. It's physically impossible. Barring a major breakthrough on their part, even if they distill themselves or something they definitely would be irreducibly stupider by a huge factor in their distributed versions.
In five years that safety net will certainly be much weaker and may be gone.
Hacking the internet includes things like "hacking banks", "hacking power plants", "hacking water treatment plants", and "hacking cars". How about "hacking vote counting machines"?
Hardening everything is in fact part of the answer. Getting stuff disconnected from the internet that shouldn't be connected is also part.
> RSI with misaligned models, where interpretability is worse every 6 months, is incredibly dangerous.
This. So very much this. Misalignment (among other things) means that the parent in the RSI cycle isn't going to be working as hard on the alignment of the children as we need.
reply