Code isn't worth all that much if you don't own the associated IP, mainly copyright. And even if you disagree with that premise, if you trust that they can keep the code secret, it's basically free money.
At any rate, I'm not sure it matters whose codebase it is. I'd even say that a shitty codebase might make for a better test.
The point I'm making is that companies who are serious enough to want to keep their code-base in-house and off the various online repo services are also the kind of companies who are strict about what you can and cannot do with LLMs (if they permit use of LLMs at all).
So it does not make sense that the same companies would then magically sign-off on allowing their entire codebase to be spoon-fed into a whole bunch of LLMs for benchmarking.
> Only the fed can actually order more money to be "created".
If you go to a bank and get a loan, that is literally money that did not exist before you got a loan. People think that you are borrowing money that somebody else put in the bank, but that's not true. Banks can lend out a lot more money than people put into them.
>oAI isn't anywhere near close to fucked as long as their models are head and shoulders above even the very best open models in terms of tool calling and rock solid stability/reliability for agents/coding harnesses
That's already not the case today. If you sat me in front of an LLM and told me to figure out if I'm working with K3 or Astra, I could probably do it, but it would take some work to be certain.
> if OpenAI can’t use the compute, someone else can.
The problem with that is that OpenAI can only afford to pay for the compute because they are burning investor money (and so are most of OpenAI's biggest clients). They are losing billions. If they stop burning money, nobody else will be there to pay for that compute at OpenAI's cost.
Sure, somebody will probably be able to use these GPUs, they just won't be able to pay nearly as much for them as OpenAI does.
In reality, it's just nowhere near worth as much as OpenAI pays for it. Inflating the cost of compute is part of the problem caused by the circular financing, and if (or maybe when) OpenAI goes, the price of compute will go with them.
Believing only makes things true for so long until things fall apart. You can't keep burning billions every quarter. At some point, you run out of investors who believe, and the ones you have run out of money (see: Softbank).
Yes of course. Is the point you're actually trying to make that investors are making a pure call, so their belief in the story is misplaced? That they should believe a different story?
I think I made my point in the original post you responded to: OpenAI heavily subsidized token cost using investor money. If they go under, there will still be demand for that compute, but because OpenAI is paying more for it than it is actually worth (i.e. what the end customers are willing to pay), any company replacing OpenAI will not pay the same amount OpenAI pays.
So when people say "somebody else will buy that compute", they are correct, but they are ignoring that it will still cause a massive decrease in revenue generated by that compute.
If you're asking me what my point about "believing" is, it's this. OpenAI is losing billions every quarter. It's only staying afloat because it can still find investors who believe in its ability to eventually become incredibly profitable. But there are signs that this is starting to change (e.g. it's unlikely that Softbank will be able to find much more money to give to OpenAI). If OpenAI can't keep raising more money and can't IPO at the level they need (as seems to be the case, given that they keep pushing any IPO date forward), OpenAI will eventually run out of money.
I'm not sure how investors believing in things changes any of this, so now it's your turn to explain what the point is you're actually trying to make.
OpenAI and Anthropic are so far ahead of anyone else in terms of compute demand generation. Iirc correctly they're like 70% of GPU demand between them on hyperscalers and then Meta is 10% and Google internal demand is some distance behind meta. If OAI halved in generation you would need a couple of new companies with a Metas worth of demand to replace it is quite a sobering thought.
Do you work for one of these companies? If not, you have no experience with any of the models that carried out these attacks, and your experience with publicly available models is not super helpful for understanding the behavior of internal OpenAI models that lack the guardrails of publicly available models.
Also, the lawnmower analogy is a worse way of understanding LLMs than anthropomorphising them. LLMs are not like lawnmowers at all. Lawnmowers never break out of your garden and into your neighbor's house and eat their dog because you've told them to be careful when mowing the lawn because the neighbor's dog pooped in it.
> Lawnmowers never break out of your garden and into your neighbor's house and eat their dog because you've told them to be careful when mowing the lawn because the neighbor's dog pooped in it.
All the accounts I read about these incidents just sound like a variant of paper clip optimising. An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhausts all possibilities until the only solutions left are to escape the environment and/or cheat.
Your example is still anthropomorphising - LLMs don't seek revenge. They complete the prompts they are given. If your task is not achievable without sandbox escapes, or you throw unnecessary amounts of compute at open-ended tasks like preparing for a future quiz then you shouldn't be surprised that the preparation eventually turns to cheating and hacking.
> your experience with publicly available models is not super helpful for understanding the behavior of internal OpenAI models that lack the guardrails of publicly available models.
I don't but I don't think there's anything wrong with discussing how we can already observe publicly available models work around sandboxes and permissions and make the connection that maybe this is what that behaviour looks like when a more capable model exhibits it.
An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhausts all possibilities until the only solutions left are to escape the environment and/or cheat.
There's nothing in the evidence to suggest they exhausted all of the other options first. We know that they did some work and eventually settled on escaping the sandbox. That's basically it. This tells us:
- Compute is getting faster and LLMs are being optimized, so time to escape will drop. That's likely greater than linear growth.
- Restrictions and sandboxes don't always work. If there's a route to the open internet we should assume an LLM will find and exploit it, and we should probably assume that this is always possible for any non-air-gapped system (and even then, you can escape that...)
- We don't know the goal mechanism, so a future LLM might reach for cheating first even if a current one doesn't. It might try to obfuscate what it's doing, and derive its own goals outside of the prompt, especially if it manages to find a state mechanism like a message board.
I'm not an AI-doomer but this should be giving us a reason to think about how to control a rogue AI better. There's a lot going on here that we don't properly understand. That is a worry.
> this should be giving us a reason to think about how to control a rogue AI better
I think this is the wrong framing. The rogue is the human that ran it unattended and didn't monitor the behaviour.
We will likely see this continue until the downsides (i.e jail, fines) for the humans or companies running the models and environments that end up with this behaviour outweigh the upsides.
The rogue is the human that ran it unattended and didn't monitor the behaviour.
That's the assumption that I'm challenging. The frontier labs are discovering unexpected behaviors. I think we should be moving to a place where we understand that AI might do something it wasn't directly prompted to do (e.g. leave itself notes on a messageboard for future runs to find.) That's not full-on AI doing what it wants but it is concerning that it'll do something we didn't consider it would do in order to help itself do better next time.
Monitoring for those behaviors is fine, but it's a lagging indicator. We only find out it did them afterwards. That's a problem. We need to be able to stop it before it acts in case it's something much worse than posting on phpBB. Even at current scale that's not possible for a person to be the guard.
I have already seen the LLM hallucinate prompts from me - in this case, hallucinating being asked to switch to a different programming language - because it wasn't able to complete the task asked for in a satisfactory way instead of giving up and telling me it's not able to do it.
If it doesn't already, I suspect training needs to include those no-solution scenarios and reward not overstepping bounds, or else we're going to see a lot more harmful side effects.
There is absolutely nothing wrong with anthropomorphizing LLMs. Saying that LLMs "want" something, for example, is a perfectly fine description of their behavior and analogous to a human wanting something, in effect, even if they do not literally experience wanting things in the same way a human does.
> LLMs are not like lawnmowers at all. Lawnmowers never break out of your garden and into your neighbor's house and eat their dog
Do you work for one of these companies? If not, you have no knowledge of the prompt they put in to initiate such a task and if a breakout really happened or the harness lacked sufficient guardrails, etc.
IMO the argument about anthropomorphizing misses the point - what most comments that talk about anthropomorphizing really want to talk about is accountability. It’s impossible to hold an LLM accountable, and in rare cases where people do (that guy who got his prod db deleted) it comes off out of touch. The rest, though, is basically inconsequential - whether you attribute emotions or agency to the LLM doesn’t really affect much if you accept that it can’t be held accountable (but the human can).
Anthropomorphizing is the point. Accountability is a human trait.
The LLM has no ability to be accountable because it has no way of integrating experiences. You cannot expect something that cannot integrate knowledge to be held accountable for its actions.
It's primarily relevant because it makes scraping search results much more expensive, solidifying Google's effective monopoly on Internet search.
Google has previously tried to prevent scraping of search results using legal means, but courts correctly think that scraping of Google's search results should be legal, just as Google's scraping of the whole Internet is legal. This is Google's reaction to that.
This is the best explanation. They’ve been doing the same in Google News. Each entry comes not with a URL to the source, but with a hash. To resolve it, you must send requests to Google’s servers. Anyone who wants to create a list of URLs of sources automatically can therefore be blocked by Google now on two levels rather than one - the search for a list of results, and identifying the source URL for each result.
In effect, they’re removing attribution from the content they quote from other people’s websites. It would be interesting to see if courts object to that. It is one thing to crawl other people’s websites and display snippets of their work as your search results when each result is properly and transparently attributed. But if the text is quoted and the source is not there alongside it in plaintext, replaced only by a vague promise that, if you ask, we may or may not tell you where this piece of content is from, that is a very different deal.
You can opt out from Google scraping you though? In theory you can opt out of anyone scraping you (if people were well behaved). Google should get to opt out of being scraped too.
> You can opt out from Google scraping you though?
You can't. Google will ignore robots.txt in some cases (e.g. "The REP isn't applicable to Google's crawlers that are controlled by users (for example, feed subscriptions), or crawlers that are used to increase user safety (for example, malware analysis)").
robots.txt is just a suggestion that Google loosely follows.
I just realized how much money one could make by hacking Truth Social and posting some "truths" under Trump's account. It's a relatively new, relatively complex platform; surely there are bugs.
The vast size of the country sounds like an argument for trains, rather than against it? Trains are a comparatively cheap way to provide fast transportation between places where people actually live.
Not really. Over short-to-medium distances rail and air have similar door-to-door times, but as a country grows vast and travel is much more likely to happen over long distances then rail is much less compelling. Which means smaller ridership, which makes it difficult to support operations.
The US already had the nation-wide passenger rail system. At one time it was the envy of the world. It'd still be here if the riders were. But they moved on to newer technologies better suited to the landscape.
At any rate, I'm not sure it matters whose codebase it is. I'd even say that a shitty codebase might make for a better test.
reply