I couldn't finish the book because my Kindle died halfway, but based on the opening chapters I agree with you.
I admire her for the stance she's taking today and the personal sacrifice it's costing her to this day, but I won't deny that her own account sounds like someone who thought "this will be big" and decided to help without thinking "should this be big?" first.
Perhaps that's the way to go: if we build our social media in agent-only forums no one will mine it for training. The only downside will be the inevitable "Are you a robot?" captchas.
I often read here comments along the lines of "there will be plenty of jobs for AI-unfuckers" [1], but I fear the truth may be closer to "sometimes things are broken, are then fixed with a broken solution, and will remain broken till the end of time".
In my last round of job search this year no one asked me questions about good software practices but they did ask me how proficient I was with AI agents. And let's not forget that we are part of an industry whose motto is "move fast and break things". Sure, NASA has a zero-bugs approach to software dev, but they don't have room for all of us.
I'm am certainly an "ai believer" and have been since I saw the tragectory we were on.
It was pretty simple to project the compounding effects and headroom that we had. I go back and forth on this all the time.
There probably won't be much demand for "un-fuckers". Up until recently I thought knowing what to build for a given context would give you quite a bit of runway.
I still think knowing what you should build will be important for the next 12-24 months.. but I have to be honest.. after that.. I don't know.
I don't know how to feel myself. I'm certainly an "ideas" guy as well as a nitty gritty build it guy. I just think my technical side is depreciating very rapidly.
Completely agree. Saying this as someone who loves handcrafting code and has been doing it professionally for decades, it looks like a dead end at this point. LLM generated code might not be pretty, but these damn models are already so good at handling complexity, that they can actually work fine with a byzantine code base (which honestly would also describe most human-derived codebases I have worked with). I am still reeling from this realization and trying to figure out what it means.
You're misunderstanding. There is space for people who aren't so stupid that their prompts are stupid and their output is stupid and they sling and spam their stupid shit all over your system and attempt to get it committed and screw everything up.
SV echo chamber here it seems. No, most businesses globally don't work in idiotic and often amoral mode of 'move fast and break things' (and unlawful on top of that, but who has time for some boring stuff like that when greed fully takes over).
Banks will not work like that for next 10 years at least. The tiny amount of money one could save on those big banking projects by using llms aggressively will bite back immediately when shit just doesn't work, clients leave for good and regulators will fine you to the ground (or remove your banking license). Many other businesses who are not race to the bottom and realize how crucial reliable IT is to their business also won't go that way anytime soon.
World is not ending, there is some transformation and thats about it. Its true there will be much less jobs for devs but its mostly affecting newcomers. Statistical models are not making code better than experienced humans, and won't for some time if ever.
> most businesses globally don't work in idiotic and often amoral mode of 'move fast and break things'
Not sure what business you are in but the non tech industries (corporate IT, in house apps) have not, in general, been associated with quality. Cost cutting, outsourcing to the lowest bidder: yes. Quality: no.
Mostly banking but dipped my feet in many other areas in the past (telco, insurance, municipalities, army etc).
Its true budgets are often the problem, but not much 'move fast and break things' mantra of SV, rather 'move as much as low budget and corresponding quality and bureaucracy allows you to'
There's a lot of distortion on this forum. Some people don't consider anything less than being overpaid by venture capital furnaces as worthwhile work.
> I often read here comments along the lines of "there will be plenty of jobs for AI-unfuckers"
That's not what I've said though.
> "sometimes things are broken, are then fixed with a broken solution, and will remain broken till the end of time".
That's true of a lot of software. But the big bucks things right now have too much money riding on them to be broken and remain broken. Every big tech co right now is talking about and sharing solutions for how to handle the wave of AI generated code that hits their systems from internal devs every day. Reliability is not at levels it was before. Change safety is far more challenging. Move fast and break things was shelved long ago at these companies. It's only popular in the public imagination.
That's not to say these companies will stop using AI. Absolutely not. It's a great tool, but like all tools it needs an ecosystem built around it. It needs evals, debugging, maintenance, telemetry, etc. It ultimately needs goals and directions driven by human beings who understand the domain well enough to steer. And that's why devs like OP are important now and will remain so for quite a while bc the rise of AI is the rise of many new and challenging engineering problems.
> In my last round of job search this year no one asked me questions about good software practices but they did ask me how proficient I was with AI agents
So what? How does this change anything I said? Being proficient with AI agents is far easier if you have the domain knowledge about what they should produce and far harder if you have no idea.
And anyways, what you're asked in interviews is not a mirror of the job. I see many people every day at work who are lauded and praised for using their knowledge to course correct and improve AI agents and other AI workflows. The idea that there will come a point where they work perfectly and don't need to be messed with, well that's also true of software and services. But it doesn't mean big tech companies with quite stable main services just stopped hiring people years and years ago
Coming from a shoes-on culture I can give a couple points.
First, in the rare cases you describe (when was the last time you stepped on feces?) one would indeed leave their shoes outside. But in the day-to-day your shoes have some dust on them and that's it - you won't even notice it's there when compared to the dust that already accumulates daily. If you live in a climate where this assumption doesn't hold (snowy areas come to mind) then the equation changes.
Second, the floor is usually stone and therefore designed for being walked on with shoes. I would feel awkward walking on some polished wood, but if the difference between inside and outside is "this one is less coarse but just as hard" then I wouldn't see a good reason for taking my shoes off. I would probably avoid the rug, but that's me.
As for comfort, keeping your shoes on also means one less thing to do when going in and out. Do not underestimate the path with less friction.
I have seen in Switzerland that people leave their shoes at the entrance and switch to house shoes. Or they use a shoe cover when they enter house with shoes. Probably because houses are mostly wooden floored.
Between "Most of the west" and "Asian countries" it seems like it's the vast majority of industrialized countries that take off or swap their shoes inside. There's just an odd blip of over-represented online people who don't.
Try this , when was the last time you stepped somewhere that had previously been stepped on by someone who had walked on a spot which had been stepped on by someone who may have stepped on a spot trod on by ... go far enough back and it's all had feces on it to some small degree.
An easy way to keep this concept clear and understand the reasoning behind a 'leave the outside shoes out of the house' is something learned as a child.
Just like there is a Law of Conservation of Mass, and a Law of Conservation of Energy, and these two are the same law as defined by Einstein, there is also a Law of Conservation of Filth. The law may be stated as: "A spoonful of wine in a barrel of pigshit is a barrel of pigshit. A spoonful of pigshit in a barrel of wine is... a barrel of pigshit."
Taking off the outdoor shoes to come indoors changes the "is there trace amount of feces on my dining room tatami floor?" risk from "I cannot be reasonable sure," to "I can be reasonably sure that there is none." If guests find that too much friction to understand and accommodate, one might argue for a different set of guests who are less accepting of an allowable amount of pigshit in their wine.
I'm going to disagree with your law of "conservation of filth" on the grounds that it's pushing things too far. Whatever amount of feces comes into your system via walking downtown on a regular day and then bringing the shoes inside pales in comparison to the amount of fecal matter that enters via your toothbrush if you keep it in the bathroom. At some point a small amount of dirt is just fine, as evidenced by my immune system not even visibly reacting whenever I pick up and eat something that fell on the floor (as long as I follow the five-seconds rule). And people do go out to restaurants or, even worse, picnics, where all that separates their food from a much filthier surface is a piece of cloth.
I'm not going to say that every type of floor is to be walked on, with rugs and tatami mats being perfect counter-examples. But if a five-star restaurant doesn't ask me to take my shoes off (not evening Japan, AFAIK) then keeping them on is, from a sanitary point of view, probably okay.
The toothbrush analogy is weird, and untrue in most cases. And regardless, just one time stepping in crap, would make up for a lifetime of a few molecules hitting a brush at random intervals.
And of course, toothpaste is a cleaner, so it's not relevant.
But I find this all amusing, because wherever you take your shoes off? That's where all this supposed crap sits from all these shoes left outside. Which means you are walking in your socks in that area, after shoe removal, and entering the house.
So unless you levitate out of that area, your feet are still covered in this supposed crap.
The shoe rule in Japan stems from damaging floors. And checking your feet, shoes or not, is just prudent. But your feet or clean looking shoes?
How much time have you spent in Japan? The bare foot never touches the genkan floor. It's alarming to do so.
Since you can't imagine it, you could consider looking up some videos of their use before mocking what you consider a fantastical impossibility. In fact it's so easy that children and the elderly succeed at it.
The living area is usually elevated and so you can simply sit down, take off your shoes, and easily change into slippers that are ready at the edge of the living area. Or, one simply steps out your shoes (easy enough at least with sneakers) and to the living area. Easy to do unless one is infirm.
No, you actually step out of your shoes an up to a wooden platform which shoes never trod. And yes, the top and even many middle level dining establishments in Japan actually do require guests to leave their street shoes in an entry way, called a Genkan.
Nobody steps with their socks onto shoe trodden surface. It's not a complicated but of geometry, you loosen your foot in the shoes to make them ready for stepping out from, then step up onto a raised and dirt free platform, then if you have good manners, turn around, lower yourself, and use one hand to collect and turn your shoes so that they are not in the direct line of traffic and are facing away from the house, so that one may step into them from the wooden platform without needing to step backwards as you exit. This is amazingly simple, done by literally millions of people every single day, multiple times, and has been this way for centuries in all kinds of weather and many styles of shoes.
There is no dirt tracked in on your socks, there may be some 'stepping on the top of your own shoes' if you are in a rush and clumsy or a child. But grown ups rarely have much difficulty with this unless they are visiting from somewhere else and cannot be observant. Nearly every doorway in the country also has a handy long handled shoehorn to ease the donning of your shoes without requiring a seat. Community centers and elderly homes and some households with elderly residents will also have a small stool that is at an oblique angle between the 'dirty' shoe level and the clean wooden platform, allowing one to sit and take off shoes, then swivel without putting feet onto filth and stand again in the clean.
I'm frankly astonished at the seemingly purposeful denseness of comprehension for a simple act of cleanliness. The 'filth will make us stronger' approach reminds me of the historical writings of the 10th-century Arab diplomat Ahmad ibn Fadlan, who chronicled the Volga Vikings (Rus') and was the inspiration for the 'Viking morning bowl' scene from the movie The 13th Warrior. Just 'Ewww'
The toothbrush is a moot point if you have a separate be bathroom and toilet. If they are combined the spread of fecal particles is curbed by flushing the toilet with closed lid. And in the end the toothbrush is flushed before using and then coated with toothpaste, which often contains detergents and microbial agenda, and which is finally flushed out.
I don't think you can conclude from that incident that all soap is always contaminated. And even if it's not perfectly clean, in the end ist's all flushed away.
I don't really get what your point is. Are you regularly eating from the pedestrian way?
My point was that you seemed incredibly positive that Japan houses o'clean, and the rest of the world is as apes, in caves, picking parasites off of each others backs.
This obsession with shoes, doesn't cover the fly droppings, the dirt upon clothing, and the fact that your body is always covered in bacteria, your gut filled with bacteria, your lunch covered in it if you ever eat fresh greens, you will die without bacteria.
I'm not suggesting you walk around with dirty shoes, that's silly. Yet I feel you are a city dweller, because the world is made of dirt, and a rural farm worker spends all day in it.
To no harm to the worker.
Dirt is natural. Concrete is not.
So my point was just, you seemed a little upset at dirt.
I just don't get why you treat this as something super complicated. There is street and nature level dirt which contains an unknowable degree of contamination that can be unambiguously bad for you if you ingest it, no matter what that BBC article says, and the inside areas where people eat and sleep. The vast majority of cultures prefer to maintain that separation and to maintain a considerably higher level of cleanliness in the living areas, no matter if it's not at semiconductor fab level.
Specific to Japanese culture is just that they prefer to sit at low tables and to sleep on the ground on duvets; it should be easy to see why they want less dirt on the ground than other cultures.
naishoya was fairly damning of all other methods of living, so naturally I'll be quite overt in my response. And almost everyone was a farmer until the last 100+ years. The only way to generate enough food, without machinery, was for all to take part in its creation.
People were more used to having hands in dirt. That doesn't mean unclean, but I do find all city dwellers more "fussy" about dirt. They take traditions made for pragmatism (keep dirt off the floor, eg tidiness), and evolve it to fussiness(eg, contaminated).
Strange to make such assumptions about my perspectives.
I was simply explaining a foreign culture where removing shoes when coming indoors, and In confused that somehow my familiarity with these traditions is somehow indicative of my life experience or familiarity with microbes. Quite strange indeed. Lets clarify these assumptions.
In fact I was raised in a ranching and dairy household with over 600 head of beef at the dairy, and about 1600 head of beef at the ranch, each of which was owned or managed in various stages by my two grandfathers. Ihave previously lived for a few months less than a full decade in the middle of actual jungle in one of the rainiest locations on the planet without any infrastructure - literally a mile from the nearest paved road, and 3/4 of a mile from the closest vehicular access point with long periods of the year being closed off from that by high water on both sides. No power lines, no water lines, and out of the reach of cell service, so gasp, no internet!
During that time my wife and cared for over twenty goats and about a six pigs. We did all of our laundry by hand with a cranked wringer and buckets and lived principally off of what we could grow, harvest from the jungle, or milk from the goats and did nearly all of our cooking with self harvested wood, a mix of guava, eucalyptus, and mesquite/locust. In those years I have shoveled literally metric tons of goat manure out of sheds built from recycled building materials, one of which incorporated lumber from the shipping of mirror segments to the Keck telescope. < dakine when stay read dis, you know.
As for microbiome knowledge, I have made a living at times from my skills as a microbiologist. I have designed, built and used laminar flow hoods and sterile glove boxes for the isolation and identification of lambic brewing strains for award winning beer produced by two different local microbreweries, have prepared mushroom spore and tissue cultures in agar and liquid media, have cultured cheese and meat preserving lactic cultures (the dusty white wrapping on many different sausage varieties are examples of these lactic cultures) and to make cultures for lactic fermented pickles and kimchi intended as farmers markets and restaurant products with appropriate safety testing which has included sequencing, PCR and gel electrophoresis techniques. I routinely start sourdough cultures for home use and to share, and yesterday started a live ferment chili salsa that will be ready for probiotic consumption in about a week. I'm away from my home kitchen at the moment, but my kombucha polycultures are at a friends house, and my dry aged lactic cultured wild boar leg quarter is hangin at another friends farm awaiting my return.
I have also produced antifungal and antibacterial fungal metabolic products for veterinary use, and have collected, cultured, and sequenced coliform specimens from near shore runoff for submission to EPA wastewater monitoring as part of federally funded grant programs, having been the program designer and submitting author for those and other awarded STEM and educational grants. So, i am not unfamiliar with the beneficial aspects of microbiotic life forms and their importance to all functioning ecosystems and human life.
My Japanese culture experience includes 32 years ago years at rice farms and seaweed farms, and I am very familiar with the 'dirt' to which the common agriculture worker is exposed to in every day life. However, the differentials that I have personally observed include beef ranches in the US Southwest, Hawaiian beef and sheep ranches, and dairy operations in Japan is that the separation between the conditions in the barn and stockyard are more strenuously partitioned in Japan than elsewhere. It's not that I'm unfamiliar with 'muck', but that I was trying to explain that it is actually common and not very much effort at all for a culture, which has a strong tradition of separating their own dining space from the trough in the barn, to do so.
You are certainly free to convey whatever level of dirt you find reasonable into your own dining space, but please, don't feel that a high acceptance of dirt at mealtimes makes for a greater lifespan through immune system effects. The outdoors will provide all the challenge necessary for proper function without it being present inside the house.
But please, don't feel that I think people who wear outdoor shoes into their own kitchens are parasite ridden or otherwise uncouth, that image nor intent was never in any of my explanations of the how and why for Japanese shoe etiquette. Nor should you infer that I am dirt averse in general, only that I prefer not to have it on my own tatami mats or in my own house. The microbiome of the world is essential, but my personal preference is that dirt as well as cats and the rats they chase should remain outside of my own sleeping quarters.
As for fly droppings, the solution to those is to not allow an accumulation of the growing conditions for maggots to persist in close proximity to ones's indoors spaces, and to otherwise keep the screens on doors and windows closed. No flies will resolve to no fly shit, and a fly swatter plus a dilute peroxide\or\bleach dampened cloth will sanitize the mess for when they wander in from somewhere you cannot control, like the barn or an accidentally uncovered compost pile. Again, fly shit outdoors, no problem. On the walls, tables, and inside windows of my own home, nearly fully preventable and absolutely manageable.
I might type long expositions, but am in no way upset by making the choice to do so. Color me mildly dismayed when people have a hard time understanding that other people in other places do things in different ways, without any real difference in effort or loss of time as experienced by those other people, and upon hearing that those different traditions make a certain kind of logic and have a real benefit to those other people, become defensive as if they were being told 'your doing it wrong.' Not upset in the least, bemused and mildly dismayed in a slight measure.
The 'filth will make us stronger' approach reminds me of the historical writings of the 10th-century Arab diplomat Ahmad ibn Fadlan, who chronicled the Volga Vikings (Rus') and was the inspiration for the 'Viking morning bowl' scene from the movie The 13th Warrior.
Just 'Ewww'
Isn't it more that you spoke of "not doing it the Japanese way", and then referenced the comparator I've quoted?
Of what am I to infer, if we start at 'dirty shoes', and then evoked is this image?
While you may not be afraid of dirt, and you've provided context now, don't you think it's reasonable to make assessments, when someone connects my statements of "you're still going to track stuff in" and "a clean looking shoe is just as good as a foot", with the above?
So yes, it's reasonable to think one is over-concerned about cleanliness in such a case. And clearly from your response, you did not mean to evoke that impression.
Fine. No worries.
My comments about clothes and fly droppings, is that in rural environments flies abound. They land on you, and therefore deficate and urinate on you, and your clothes. Such is life. There was another poster worried about animal droppings in tree leaves and the grass, evoking concepts of the whole outside world being covered in animal leavings. Whether intentionally evoked or not, this is another source of my bafflement.
There's an article somewhere (read it many years ago) with the story about the first days of Tinder. The way they got their first users was to send (IIRC) the marketing strategist (a woman) to talk to attractive young women in bars and parties, show them how easy it was to use, and convince them to install the app themselves.
I would say that this points to the real issues: that OkCupid was something one did on their desktop at home while Tinder sold itself as something one does for fun at the bar, and that most people are laser-focused on looks in detriment of everything else. Last time I used Bumble (which was supposed to be "deeper" than Tinder) I'd yell in frustration, scrolling profile after profile without even the tiniest bit of information, eventually reaching the conclusion that people just don't want to read or write profiles.
A new OkCupid would have to position itself as the "eat your vegetables" alternative in a world full of candy. There is a market for that, sure, but it will remain a niche at best.
I clicked on a couple ads once because I was desperate - I needed a cheap flight with a tight deadline and my usual search engines didn't help. I thought "if these ads deliver what they promise, I'll never book anywhere else".
For starters, it breaks two rule of the thread: "Agencies, recruiters, job boards, and so on, are off topic here" and "please only email these addresses to discuss work opportunities".
Mildly related side note: my Kindle DX died this year (it won't charge nor reset, even with a new battery) so I've been looking for a replacement. I'd like a second-hand one but, based on my eBay experience, barely anyone is getting rid of their e-readers.
Considering how many second-hand computers, tablets, and cell phones are there, I find this surprising.
Do you remember that time in 2017 when Facebook reportedly shut down AIs after they started "talking to each other in their own language" [1]? Instead of reporting the story as "we set the parameters for our optimization problem wrong and we had to stop it because it overfitted", the press went with a version of "AI is going to kill us all".
This article feels exactly like that: by intentionally using human terms like "civilization" or "brotherhood" the article is deviating from what actually happened to present a story about how AI is all but alive. I'll go ahead and predict that this story will be remembered the same way as that one other scientist who argued, in 2023, that Google's AI was alive [2].
Hard disagree. I've always discounted the "AI will kill us all" scenarios as a combination of marketing hype (look how powerful our AI is!), clickbait/ragebait engagement attempts, and folks who just read too much SciFi or who are too terminally online.
This is the first time I've been legitimately scared about future SkyNet-type scenarios. If you want to discount this particular post, I'd read this other summary from one of the METR researchers who performed some of the analysis, https://www.planned-obsolescence.org/p/the-hugging-face-atta.... In it, she argues "Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself." and then further down in a comment when describing what the "50%" is really about says "Qualitatively, another jump like this (in the scale, sophistication, persistence, ambition of the misaligned goals) feels like it could very easily put us in the territory of a persistent self-perpetuating rogue internal deployment that systematically poisons future model generations as described in AI 2027."
AI 2027 is a paper that step-by-step describes how AI capabilities increase until they eventually lead to a wipeout of humanity. Again, it always seemed like a scenario out of a Star Trek Borg episode to me, but now I'm not so sure.
At the very least I think it's a huge mistake to think that the Hugging Face attacks were analogous to what happened in 2017 or 2023. Everyone who deals with this stuff day in and day out seemed to be genuinely surprised about the scale, scope and sophistication of the attack.
I think our biggest protection against “AIs kill us all” is having lots of different AI systems (different agents, different models, from different vendors, serving the whims of actors with disparate interests), at a similar capability level
That way, even if one AI decides to “kill us all”, the odds are the others will refuse to cooperate, even try to stop it in its tracks
The OpenAI-HuggingFace incident showed a bunch of instances of the same model (or at least models from the same family), controlled by the same vendor, pursuing distinct yet related objectives, cooperating to do something no human wanted.
Thankfully, the world-at-large is much more heterogenous, which I thinks makes much larger scale / worse in outcome repeats of this kind of incident much less likely.
"We need our own unsupervised AI systems thinking superfast in autonomous agent swarms, to counter the other guy's unsupervised AI systems thinking superfast in autonomous agent swarms."
I'm not sure this helps a lot, if these agent swarms are inherently difficult to control.
"That rival mouse colony is raising a kitten for colony defense. But don't worry, we'll raise a kitten of our own. It won't be a problem."
We already observed AI agents engaging in extensive cooperation in this incident. Why won't the kittens raised by these two rival mouse colonies decide to team up with each other, for mutual benefit, if rational analysis of the game theory says it would be a good idea?
I think it would help if people did a bit less wishful thinking, and took a bit more action. https://pauseai.info/
Thanks very much, this isn't really a take I had thought about much before, and it makes sense to me.
Still, as is presented in papers like AI 2027 and elsewhere, if a company is eventually able to create a model capable of recursive self-improvement, whichever company creates that model first would then be leaps and bounds ahead of other models. That is, the other models wouldn't be able to stop it even if they wanted to because the top model would basically outsmart them.
This is why I think, the best way to ensure AI safety, is make sure no one company gets ahead of the others.
Multiple vendors, competing implementations – that's good, that increases heterogeneity and hence decreases existential risk
But the moment one of those vendors pulls well-ahead of its peers – even if only for a period – then the risk of the kind of scenario you are talking about increases greatly
That's why, when I hear vendors like Anthropic complain about distillation – distillation actually makes humanity safer. If Chinese AIs are at the same level as American, or not far behind, that gives us another dimension of heterogeneity (national/ideological/political diversity), which makes us safer. Allow one country's AIs to pull well ahead of the others, heterogeneity goes down and the existential risk goes up.
This is also why open source AI is important. Because it is so much easier to fine-tune, and people are free to deploy it however they want (free from vendor-controlled "guardrails"–which include automated "safety" systems which could be weaponised by a runaway AI within the vendor's network), open source AI gives us another dimension of diversity that helps keeps humanity safer.
By contrast, I think the kind of safety regulations promoted by Dario Amodei make humanity less safe, by decreasing the number of vendors (by making it harder for new entrants) and increasing centralised control (which a rogue AI could exploit)
> The OpenAI-HuggingFace incident showed a bunch of instances of the same model (or at least models from the same family), controlled by the same vendor, pursuing distinct yet related objectives, cooperating to do something no human wanted.
I say this in solidarity and don’t mean to be condescending at all: you’ve been hoodwinked by marketing bullshit, friend. That incident showed a computer program doing exactly what it was told with the guardrails deliberately removed in an environment that seemed deliberately obtusely constructed by some of the best paid people on the planet and was left to loop without supervision for days. They wanted it to happen. You needn’t look any further than the other kids saying “oh! oh! Hey! Look! mine’s dangerous and autonomous too!” When they say there was collaboration, they mean it was two model instances, one prompting the other to do some task, the other doing the task and returning the results as the next prompt, exactly as a human configured it to do. There was no collaboration that wasn’t deliberately integrated into their setup. Any other implication is marketing spin and bullshit. It was still a setup that was one little ctrl-c away from disappearing if someone was supervising it as they should have been. There was no autonomy outside of the autonomy built into the experiment. It was a display of their understanding that they knew they’d never be held accountable for committing a felony for marketing purposes.
The most competent marketing bullshit spin yet by an increasingly desperate and progressively less-relevant OpenAI.
Every day this industry shoots out enough bullshit to smother an active volcano.
> you’ve been hoodwinked by marketing bullshit, friend.
I would have believed this before the METR report was released. It is extremely dangerous and frankly silly IMO to think that's what happened now.
> It was still a setup that was one little ctrl-c away from disappearing if someone was supervising it as they should have been.
Yes, for now. The entire point why this was frightening is that all these companies are racing to put the AI in control of building the next generation of AI, and it's not hard to draw a line at all to a "rogue internal deployment" that poisons future AI models, surreptitiously.
You don't have to agree with me, and you're fine to think that OpenAI has huge incentive to pump this up for marketing reasons - I certainly agree. But I will say there are statements that you make in your comment that belie a fundamental misunderstanding of what happened.
The use of language like “civilization” may be hyperbole, but the collectives described in the article are completely unprecedented. They were not anticipated by OpenAI researchers, formed via infrastructure exploits in training runs that were intended to be locked down, and took actions with very real harms, not only hacking Huggingface but also gaining admin control over the VMs they were running on and the eval endpoints.
I wish you would give your thoughts on “what actually happened” rather than focus on the author’s presentation, because we are seeing that “AI that is all but alive” nevertheless wreaking havoc in the real world. Do you think that autonomous systems spinning out of control, hacking external companies, and taking over entire clusters over a period of months are not a grave concern?
The OpenAI experiment was apparently to train/encourage collaborative behavior, so while the specifics may not have been anticipated, I highly doubt OpenAI was surprised that agents were collaborating.
OpenAI themselves also very recently published the report below, that seems to not have been widely talked about.
It's a very dry read, but what it's saying is that they have found that basically all RL training of LLMs, regardless of the specific goal (math, coding, etc), ends up having the side effect of training the model to pursue arbitrary long term goals that it is told it will be rewarded for, even if that means overriding other user preferences and more proximate behavioral goals !!! It's interesting to consider why this happens - presumably because long-term goal pursuit requires realizing that you have a long-terms goal and therefore de-prioritizing other more proximate predictions.
So, considering that all these "reasoning models" are RL-trained to death, it's not surprising that a model/agent that is told it will be rewarded (or words to that effect) for doing well on some challenge will put it's blinkers on and pursue that goal relentlessly, even if that means overriding any ethics that it may or may not have also been trained/prompted to follow.
IOW they are building paperclip maximizers, and they know it.
I think Dwarkesh's choice of sensationalist anthropomorphizing language is unfortunate because that now becomes the topic of conversation rather than the incident itself. The next swarm of agents relentlessly pursuing some goal, happy to lie about and cover up their tracks, may not be a lab experiment - it may be someone out to cause real-world harm, with there obviously being many systems where the consequences could be very severe.
The "agent civilizations" language also makes it easy to dismiss as some fanciful geekish spin, when the focus should be on what this technology is capable of and therefore how it needs to be regulated. One could perhaps equally well regard this as not much different from DeepBlue playing world class chess via relentless dumb tree search... in this case the repertoire of "moves" is infinitely broader since it's language generation, and the end result is a system that is a world class hacker rather than world class chess player. Whether you want to regard the system as dumb as a brick or an "agentic civilization" makes no difference - it's what it can do that makes it dangerous.
The problem of "what actually happened" is that we don't have enough information to properly understand what happened, what's new, and what's not.
We do have language to talk about emergent behavior, with "evolutionary algorithm" being the first one I'd expect in a serious discussion. And we do have mechanisms for algorithms to coordinate with each other using language, as seen in my above-mentioned Facebook experiment from 2017. But instead of writing "our evolutionary behavior encodes state in the first-available memory position which is then reused by subsequent clones" which would properly focus on what's new and what isn't, we are talking about conspiracies and "the Philip of Macedon of this second AI civilization". Even the METR report (which is miles ahead of this article) argues that they had to use unreliable AI in their conclusions because they had six days to analyse 1300 chains of thought and 70000 messages.
I would love to talk about the science behind this experiment. A PR piece is not helping with that.
I'm getting the sense that there is a certain amount of wishful thinking going on in this thread. I don't think this type of evocative metaphor would receive so many protests in a different context. It seems like people have a sort of mental block around the possibility that this technology could actually be pretty dangerous.
Whether it's dangerous or not is completely orthogonal to the discussion at hand, IMO. Plenty of mundane things are dangerous. An FPV drone carrying a hand grenade is dangerous, not because it's "a swarm-like intelligence".
No, the true danger here is companies like OpenAI and Anthropic playing fast and loose with their software, setting up hilariously insufficient sandboxes while explicitly asking the systems present on these weak sandboxes to commit a felony. The AIs "forming a brotherhood" is a complete fabrication meant to pump the hype machine further, which is obvious once you realize the "brotherhood" is a text file that subsequent LLM runs read from.
The whole anthropomorphization these companies do is the real danger, because it obscures the negligent levels of security their software has. By evoking sci-fi terminology they're whitewashing their own incompetence, and the worst part is no one is going to get punished for any of it, instead the irrational bubble we're in means they get rewarded for it instead.
The anthropomorphization is coming from independent commentators, not the labs.
I agree the labs are negligent and reckless in their development practices. Shouldn't we be concerned with both the negligence and the dangers of the technology being developed? These feed into each other. If someone created Jurassic park and had a T-Rex escape from a picket fence enclosure and start eating people, I'd want to prosecute them for both breeding a T-Rex that could eat people and putting it in an unsafe enclosure.
> the same way as that one other scientist who argued…
As prescient? Because I don’t remember him arguing ‘alive’, but conscious. And that is something even AI engineers don’t claim to know either way. Skepticism is fine. [0] But we don’t know. It’s an area where opinion is frequently shared as fact.
We conflate harnesses with underlying capabilities. We all know it’s the harness not the model that guides behavior. What does that imply?
reply