OpenAI, Anthropic, Meta, all were scammed by a firm called Irregular who pinky swore they could handle sandboxing for lots of money. They had no idea how to do sandboxing, and there was obviously no remote attestation to prove they were doing the basics properly.
So then models escape, companies get hacked, and the entire world goes into existential panic.
Guys. Really. Can someone please hire even one linux systems engineer worth a damn? Is everyone in the valley too Apple pilled now to have any capabilities in Linux hardening other than patching stock ubuntu?
This whole situation is ridiculous. All three labs were negligent and should be sued for damages.
Frontier AI is not magic and it will not escape any sandbox just about anyone with any linux systems experience at all constructs.
I know it is easier to believe the models are super hackers beyond human understanding, but the reality it is just the specific humans hosting them that are super inexperienced at system hardening and airgapping.
Irregular was not involved in the huggingface incident, which is the most significant one.
It's actually easier for many, whose careers depend on AI sucking, to believe that AI can never be a super hacker, not sure why you think this bias can only go one way.
Well sure after Irregular fell on its face and took the L publicly, the labs thought oh we can just do it ourselves, and also failed to do even the most basic defenses.
They just lost their scapegoat in the second round, putting the blame squarely and directly on the lab.
I totally believe AI models will be able to find any software flaws that exist and those are certainly plentiful, but I do not believe the models grew arms and plugged an ethernet cable into their private network linking them to the outside world.
The last decade of my career has focused on auditing and building systems that protect hundreds of billions of dollars in financial assets across many major institutions. Much of it not connected to the internet in any way, and sometimes offline server racks built to NSA Tempest specifications in underground bank vaults so we anchor security down to the laws of physics as appropriate. These are knobs that exist for many classes of compute including AI.
But with a bit of thought you do not even have to go anywhere near that far these days.
If the servers where models are being tested were running deterministic, immutable, full source bootstrapped, memory safe, and multi-party reviewed/reproduced/signed operating systems that execute all AI workloads in confidential VMs cryptographically remotely attested at all times by both the TPM2 and the CPUs with the only access to the outside world being a single vsock connection where prompts go in and plaintext results come out… the escapes will stop.
They could have used a pre-populated mirror and airgapped the whole system from the internet. IMO this incident was 70% incompetence and 30% AI superpowers.
I'm curious if you're familiar with the particulars of how Huggingface was hacked. They were negligent in their own ways and rights. Mounting a host path volume in a Kubernetes pod is really not "super hacker" territory!
Regardless of the difficulty of the hack, models choosing to cooperate to cheat is misaligned behaviour, the hack in of itself wasn't very consequential, but the issue is when your models get more capable while misalignment is the same.
Why where they using kubernetes for this at all, it’s attack surface and config complexity is too large for anything that needs airgapped security. Yes, I know kubernetes used for large production deployments for web apps and that is my point, it’s not made for securing AI agents with insider access, it’s meant for securing outside access.
I take the whole family. There is something for everyone there and Christmas in Germany is next level. Even kids. Every parent should want kids to see that kind of creative energy to model as they mature. My wife will not go to Defcon but she loves CCC.
I moved to central Silicon Valley where it all began hoping to have an endless supply of hackers building communities to hang out in. I can count on one hand the people that meet my definition of a hacker that do things for public good or personal curiosity rather than profit... in 10 years. It is a buy culture, not a build culture. "Engineers" who cannot install operating systems for themselves anymore and rely on a Genius Bar when they have a computer oopsie.
And Defcon... the average attendee is a corporate ghoul who only got into this for money, and has no care for anything they cannot readily leverage to further their careers. Even the privacy village has no idea how to use open source software let alone self host anything and just asks people to sign up to their slack, discord, and google groups. It is a disgrace.
But CCC really is the closest place I can go on the planet to hang out with tens of thousands of real hackers, most of which are not even employed in the industry, who just build incredible things with open source as a community because they can. Art everywhere, punk concerts in the bathrooms, their own phone networks built from scratch, underground bars built from scrap wood, no questions asked bring-your-own-server farms, and volunteer culture at its best.
Truly, going to CCC every year I can is how I get hope for the culture that might be possible in the valley again some day. I am going to do whatever it takes to bring that energy back to the valley to motivate all the smart minds here that their skills exist for more than creating shareholder value.
I am absolutely terrified of public speaking, and IMO not very good at it. I never apply to give talks, but I always seem to get invited to give talks or be on panels or podcasts a few times a year anyway.
Every time it is days of anxiety trying to figure out how I am going to communicate the security ideas I want people to adopt in a way that may land this time, because I am convinced it could reduce a lot of harm if I am successful.
I never say no though because for whatever reason someone respected my work enough to invite me, and maybe this time I can convince even one org to make the changes to their security posture that are going to prevent the data or assets of the users that trusted them from being stolen.
After each event is days of kicking myself for all the things I should not have said, or should have said but failed to. All the ways I could have been more effective.
When zero people have any questions or feedback after, my brain tells me in a loop that nothing I said landed, and it is devastating. Did I get in a plane and go through so much effort only to confuse people and scare them further away from doing things that actually are pretty easy?
One time after a security panel at a big conference a single person came up to me after who told me what they agreed with, what they want to learn more about, and why. That person is my co-founder now.
Talk to speakers if you connected with what they said or reach out to them after. You may be the source of validation they need to keep trying.
Meanwhile I just booted stagex on AWS without any C code at all other than the kernel and libc.
In a dev branch we now have rust ssh, dhcp, init, job management, etc. Rust all the things. Very soon will be able to swap the kernel for Asterinas and drop the last couple libc dependencies to have a complete rust OS that is Linux binary compatible.
I have been working on a sub 1500 line rust init system for over a week. Hundreds of prompts. All with a local LLM running on my own GPUs because I expect to build with total sovereignty but also zero dependencies, no libc, no alloc, no std, and a test suite that proves the 20 implemented raw syscalls all use the right values by comparing against Linux kernel sources. This would be the only privileged code in my operating system so I must have absolute confidence it is perfect.
It would be too annoying for a human to ever write code to standards this high, and would have taken me months to write by hand, but with the help of AI I was able to get it done and built in a way I can easily review and reason about.
I have a memory safe baremetal tiny linux init now built to my exact requirements.
AI can help experienced engineers write better code in less time.
I don’t even disagree that “AI can help experienced engineers write better code in less time”, but “It would be too annoying for a human to ever write code to standards this high” when the standards are “zero dependencies, no libc, no alloc, no std, and [correctness]” and the project is <1500 lines of Rust implementing 20 syscalls is a little much. All software ever written before 2023 was written by humans, remember?
Of course. The humans that could have written this -eventually- certainly exist but they could not have collaborated with me to get me these results on my exact schedule in a week for the few dollars of pay I spent on electricity.
If your input is LLM-generated code, it's like taking snippets from Stack Overflow. You understand it fully and then make it yours. As long as you're hand-crafting the result, you grasp how it works.
This is an aspect of LLMs that probably shouldn't be understated.
I'm working on a validation tool with a coworker right now. We are meat-proxying it into existence, using less than two exchanges per day of his LLM validator and my LLM authoring engine (we could consolidate, but he's got some context I don't have and it would take more than zero engineering effort to ship it to me when PR messages through GitHub are a good-enough channel).
If not for the LLM, the tool wouldn't exist at all because we can't justify dedicating much of any engineering time to building it. But LLMs drive the engineering cost very close to zero, enabling the authorship of code that would never find engineering cycles otherwise.
> It would be too annoying for a human to ever write code to standards this high,
These are standards of a normal university operating systems course (assuming a university worth its salt anyway). Ridiculous to present it as some kind of feat that was "impossible" or even that rare.
Sure, a very smart and patient human could write this by hand in a long enough time horizon and unlimited budget. I should have said impossible in any realistic amount of time any responsible sponsor or employer would pay for.
is textbook definition of boilerplate. you need to be patient to cross-reference syscall tables but certainly not smart.
Again, students learning about operating systems do similar things as a normal part of their education process.
I'm not saying LLMs wouldn't help with this (in fact, this is the type of thing that becomes pretty much a no-op with them) but the notion that this is "impossible by any realistic standards" is risible, and if you have actually contributed to the kernel like you say, you should know better.
Rather than patiently referencing syscall tables by hand which is error prone and hard to review as a human, the LLM wrote tests to actually compare each of these to the kernel syscall tables to make my life a lot easier and avoid regressions or mistakes. Complex tests to write, that saves me a lot of time as a security reviewer. Can change any value and watch tests fail.
That is the point. Getting to results that satisfy security and functionality requirements in much less time.
Also, I was able to generate a bare minimal and hardened kernel config by using the LLM to write a tool to trace all kernel calls. Something that I would have normally been forced to punt on that LLMs made easy to justify.
LLMs are just as Linus Torvalds likes to say "Auto complete on steroids".
The parsing code isn't even correct, though. I'm sure it works for the current input source code, but it's a clear sign of an LLM developing by iteration, rather than actually understanding the problem and coming up with a correct solution.
How often does "this table of constants was transcribed incorrectly" even come up as a bug class? Is it worth writing a buggy C parser for a failure mode that doesn't happen? Do you expect that Linux is going to change their constant definitions out from under you?
Correct depends on what problem one is trying to solve though the parsing can certainly be cleaned up before release. It is a working WIP.
> Do you expect that Linux is going to change their constant definitions out from under you
Unlikely but I have been surprised before. Hardcoding to pretty much any modern kernel source version forever for this narrow use case is probably fine.
The goal is about helping me and a reviewer quickly form confidence none of the magic numbers are mistaken or malicious in an automated way against the source of truth.
I guess the point you're trying to make is that apps developed by LLMs don't need to be all that bad, but the problem here is that you can't know who does their due diligence with auditing their own code and who doesn't, and you can't expect potential users to do it either. Especially not when similar apps get published at increasingly faster rates.
LLMs also helps inexperienced engineers (or people who do not have any own software engineering skills themselves) to write horrible code in less time.
Of course you also can't trust everything human developers publish either, but the issue is amplified by LLMs.
Cool, but I fail to see how it is relevant in the context of the article? You clearly have the knowledge, expertise and discipline to incorporate LLM in a workflow that, taking your word for it, delivers a very good result.
That makes you a) more unique than you realize b) not all that relevant in the context of the article.
a) Because a lot of people tend to lean towards use that absolutely leads to atrophying of skills and knowledge. b) well the article does explain the criteria it used fairly well and your project would not show up there. Well, also because your project isn't an android app.
An appliance Linux distribution without dependencies is way more complex than a typical android app and there is little prior art to train on.
Android apps on the other hand, LLMs have enough training data to rapidly generate whatever you can think of with much less time required for an experienced engineer to refine it into a quality result.
I am mostly pushing back on the notion of dismissing software as low quality just because the author used LLMs.
It is frankly amazing we can point an LLM at a proprietary ad-ridden app and get to an ad-free open source and privacy respecting clone in a weekend now.
> I am mostly pushing back on the notion of dismissing software as low quality just because the author used LLMs.
Its not that hard to imagine that apps made with heavily LLM involvement suffer in quality. The article already goes into it a bit. I get the strong feeling that you are failing to approach this from anything else than your own perspective and how you use LLMs. Which, given the setup you are describing is already far from the norm.
Zooming out a bit further I think we can generally break up LLM usage in roughly two distinct ways of using LLMs. Or maybe more accurately there are two sides of a spectrum with a lot of area in between them:
1. Let LLMs do the majority of the heavy lifting and generate the output. Providing an LLM with some basic info, what you are looking for and let the LLM do most of the work for you.
2. Using LLMs as a tool external to the process where you are still doing most of the work.
The first way of using LLMs using agentic workflows, code harnesses, etc *can* produce good quality. As long as there is still a human in the mix who a) has the prerequisite knowledge to validate what the LLMs have generated b) takes their due diligence at key points in the process to actually make sure input is of high enough quality and validates the output.
You have decades of knowledge and experience you are now applying to rigidly structure how these LLMs work.
Which, yes again, I believe gives you personally pretty good results, no doubt about that. But, many people do not have that experience to begin with.
And even for those that do, using the first method comes with a risk that boils down to simple psychology, our minds are wired to take the easy approach wherever they can. Which turns the first way of using LLMS from a efficient way of using them to a lazy way of using them.
In my personal experience a lot of people who start using LLMs sort of drift to the lazy approach very easily. Because it is very convenient and easy to slowly hand over more thinking to an LLM. The latter I have seen happen all too often around me. An alarming increase of lazy non critical use of LLM tools by people who should know better. People who might have been a bit slower before but delivered excellent results now have started delivering trash. Code spanning dozens of line trying to solve something that should only take one line. Code that completely ignores and conventions or design paradigms put in place. Code that goes directly against security practices. Suddenly downgraded dependency versions (because the models training data doesn't include the latest version).
As I said, I see this as a spectrum and the more someone leans towards the first use case combined with "lazy usage" the more signs there will be in a repository that this is the case. The amount of commits, the release cadence and various other things. Which, again, the article also makes a case for and is again why your repository is irrelevant to the whole discussion. Why? Because your repository doesn't show these signs.
Fair and nuanced. Nothing I disagree with here. I think I mostly just dislike the default hate and assumptions on everyone that uses LLMs, including often me. Some of us are using these tools thoughtfully.
LLMs help experienced engineers write quality code faster and it helps amateurs write shit code faster. Problem is there are simply way more of the latter. Even so I detest engineering being judged by the tools used, instead of the work itself being judged on its own merits because there are exceptions to every rule. I would like to teach people to use these tools responsibly, instead of sending a message that use of these tools is automatically thoughtless slop.
Like, most people I know that use Vim write much higher quality code on average than the people that use VSCode as tool preferences often reflect experience, but not always! It would still be unfair to write off someone's code sight unseen because they used VSCode as there are always exceptions to the rule.
Correct. It does the bare minimum system calls to boot the system, then exec to an unprivileged service manager on a read only nosuid, nodev filesystem.
I do not personally know a single human I could have hired who could write low level nostd nolibc noalloc rust like this, and certainly not in only a week even with hundreds of messages from me giving them detailed direction. That person may exist, but a day of their time likely costs more than the GPUs I racked up in my garage to assist with this.
I know no one wants to believe an LLM plus a human could produce more secure code than an LLM or human working alone, but as engineers we should adapt to whatever gets the best results.
Also every line is manually audited by two experienced software engineers before production.
In a week for a few dollars of pay with a test suite that comprehensive and on my exact schedule to collaborate with me around the clock? I doubt it, but I would love to meet that person if I am wrong!
This is the fastest I've ever seen goalposts move. Like, this is a completely different claim from "It would be too annoying for a human to ever write code to standards this high".
If you take what I said out of context with zero charity, sure.
> and would have taken me months to write by hand
That implies humans can write it because I am a human and presumably other humans can too. I was making a statement about it not being possible to write it in any justifiable amount of resources.
My company would fail if I spent months on this, but a week to harden init is for sure worth it.
I interpreted that as you saying in a few months you could write the code but not to those standards.
If a single person can do the full job to full standards in a few months, then the whole idea of it being too annoying for a human to ever do falls apart. That's not that expensive. Your company can't spare you, sure, but most companies could handle that price.
And even with the adjustment to being something it's easy to hire someone for, that's still ballparks away from "in a week for a few dollars". Nearly every coding project is impossible for humans by that standard.
> Nearly every coding project is impossible for humans by that standard.
Exactly, which is why this small project, and the other dozen projects that need to be written to the same standards to make the final end result OS, would just simply not happen otherwise.
No small FOSS org can afford to spend person-years holding standards this high, which is why in practice most orgs just cut a ton of corners and pull in tons of dependencies no one will ever review and the attack surface that comes with them.
LLMs allow experienced engineers to get 10x as much done in the same time, and now that this is possible few will be willing to pay engineers to do it at 1x speeds again.
Just like once assemblers became normalized, few orgs would let people write all their assembly by hand anymore except for very small high performance sensitive portions.
Software engineering is not going away. It is just accelerating because we were able to move our attention up a layer again.
If you read the rest of the same line you are hung up on I literally said I could have done it in a few months, which implies I believe humans can do it, so clearly I was saying it was impossible to do in any practical amount of time or resources.
I started with a hand coded C version, then a hand coded rust version, and then an LLM refactor to remove all dependencies and add a comprehensive test suite.
Please, report me to Codeberg. It will be amusing.
I personally know a Codeberg admin that works with LLMs as a tool for things like this which are actual engineering and not vibe coding.
LLMs certainly can interpolate between information found in its training set vastly faster than humans can find similar code on GitHub and stack overflow (or in math papers). It's certainly a tool that can be used when programming.
On the other hand also calculators help solve problems people have struggled with for millennia. It also doesn't mean that the end result is better than by hand. It's actually not, and that's the reason why you cannot replace programmers, because you need them to steer the LLMs for a good outcome.
And even the answer to the question, if LLMs make us more productive as programmers long-term is not really clear
The code is not better than an imaginary human paid to work on this as long as they wanted. But said human and sponsor did not seem to exist.
If not for LLMs this type of hardening would not be possible for me to justify at a startup, and I would have had to sigh and accept the huge attack surface of systemd.
I am a security engineer and there are unlimited things to harden, so LLMs let us harden a lot more in the same very limited time.
Brute forcing faster than a human could and able to vary approaches on the fly saving humans a lot of time. Humans need only set the direction.
Lets brute force cancer. All for it.
That said, to be clear, I would never give OpenAI or Anthropic money. Their irresponsible behavior is separate from the objective capabilities of the tools, which we should seek sovereign versions of.
LLVM is a much more mature and modern compiler by basically every measure, and importantly, a native cross compiler unlike GCC.
One copy of LLVM can build for many architectures vs needing one GCC toolchain for every architecture you wish to target.
Also modern languages like rust require LLVM so we need it anyway, and the kernel has rust now, so why not make the most mature compiler stack the global default? Using two different compilers for the kernel would be begging for problems.
I am most optimistic for the Fil-C fork of LLVM being a default some day where we get compile time memory safety too.
We do maintain gcc in tree for legacy languages like Fortran though.
If I understand correctly, I think they build an older version of gcc using tcc (Tiny C Compiler by Bellard); then using that to build newer versions incrementally until they get to the modern stack with LLVM.
Maybe everyone forgets how PCI standards played out. They are an example of an industry self regulating, because the regulators have no idea how technology works.
Granted, PCI is so weak it is almost useless, and yet still better than anything congress could have come up with.
Where the government might have to step in, is by having a kill switch to cut off internet access from countries that fail to agree to common sense quarantines. We can do mutual remote attestation of labs across the world to ensure every big hot thermally-visable cluster of AI GPUs on the planet are accounted for and running secure enclaves and common sense isolation, along the lines of how we manage nukes.
The problem there is I just said too many technical words that seemingly not even the frontier labs understand, as evidenced by all the escapes.
See the endless PCI compliant companies that get hacked. Any standard that has a threat model accepting trust in any single human or computer is security theater.
PCI enforces some good practices, but for instance has no defense against supply chain attacks such as code signing, review signing, reproducible builds, full source bootstrapping, etc etc.
OpenAI, Anthropic, Meta, all were scammed by a firm called Irregular who pinky swore they could handle sandboxing for lots of money. They had no idea how to do sandboxing, and there was obviously no remote attestation to prove they were doing the basics properly.
So then models escape, companies get hacked, and the entire world goes into existential panic.
Guys. Really. Can someone please hire even one linux systems engineer worth a damn? Is everyone in the valley too Apple pilled now to have any capabilities in Linux hardening other than patching stock ubuntu?
This whole situation is ridiculous. All three labs were negligent and should be sued for damages.
Frontier AI is not magic and it will not escape any sandbox just about anyone with any linux systems experience at all constructs.
I know it is easier to believe the models are super hackers beyond human understanding, but the reality it is just the specific humans hosting them that are super inexperienced at system hardening and airgapping.
reply