My takeaway:
We should be not be concerned about AI bots' "goals", we should be concerned about the goals of the companies making them. Powerful but not sentient technology in the hands of reckless accelerationists is a plenty dangerous enough thing.
I mean, of course we should be concerned about the goals of the companies. But that's a second, separate concern and it's important not to muddy the two together into a single point. We shouldn't take any of these companies at their word and we should treat their stated intentions as suspect regardless.
The agents' "goals" in the specific instance being discussed is a benchmark. But it's also the case that at no point was the agent given the "goal" of hacking HF. The agent was tasked with solving a problem on a standardized test and it independently set a secondary sub-goal of cheating. And it nearly succeeded.
I don't think Cory argues against this. The idea that the agent nearly managed to succeed at cheating through a series of exploits remains true. But Cory does effectively call this unremarkable and uses some mental gymnastics to achieve that argument (somehow using the idea of an agent harness and how is written in "easy to master" Python as evidence), because the LLM is trained on hacker techniques.
I find this a dramatic oversimplification of absolutely everything going on here. Yes, it's bad that this happened. I think we all agree. Where I fundamentally think Cory is wrong is that this is a company doing bad things at the end of the day. And yes, I think OpenAI was irresponsible (to whatever degree). But ultimately that's missing the point: the incident points out that agents can do this without being explicitly told to, and more importantly, they can succeed at it. Slapping frontier labs on the wrist doesn't change the fact that this is possible with technology that exists today. It doesn't change the fact that other countries and companies are doing lord knows what. Or that bad actors are going to be bad actors regardless of what regulations you put in place. Making crime illegal is the wrong lesson to learn here, the right lesson is that we now live in a world where this happens and it'll continue happening, and we need to put on our thinking caps about how to keep our systems safe.
Details the design and architecture of Apple Watch's new "Audio Intelligence" features. In particular, regarding Siri Recap which has been discussed elsewhere:
*How Siri Recap respects those around you*
By design, Siri Recap does not create a recording, does not produce a verbatim transcript, and does not identify and attribute speakers. The output is a brief, high-level summary, comparable to notes a person might write after a conversation. There is no audible signal because no raw audio is retained, and there is no way to reconstruct the original audio from a Siri Recap or share raw audio with anyone.
Want to be clear that this is not me endorsing the feature. It's clear Apple did think about the privacy/social implications of the design, but it's also clear to me that there are real and perceived problems with having automatic conversation notes. Real differences from written notes:
- It was recorded and transcribed by a machine, not a human
- There are time stamps and locations
- You can't tell someone is using this feature (it is obvious when someone is writing notes during a conversation)
As far as perceptions goes, it feels very tone deaf of Apple to launch this feature in this way at this time. Read the fucking room guys.
Maybe this will be true legally (we wont really know until it is tested in court), but I think there's a huge difference in practice.
Siri Recap is an electronic record of a conversation that includes timestamps. Imagine for example that during a scheduled meeting with my boss, he threatens to fire me for not doing something illegal. I now have strong evidence that this was said (even if it cant be officially attributed to him) during the time I was in a meeting with him. Not saying this will hold up in court, but it certainly is more damning that having written notes (not to mention the fact that my boss probably wouldn't say this when I am taking written notes).
Making that feature illegal may backfire spectacularly across the whole IT sector. What Apple does is it inputs voice data into an AI program and that outputs some transformed output. If a court would rule that output in any way matches the input (to be classified as a "record") then the whole ethics principle of stealing other people's data and funneling it through AI to make it company's own, would be at risk.
I imagine no single spineless impotent modern court would risk a wrath of our new benevolent AI overlords calling them all criminals they really are.
Agree with this framing, but we may find out that the answer is somewhere in the middle. Personally I think it is extremely unlikely that this is straight plagiarism in the sense of the model simply regurgitating training data from prior work by Tristan, but it is very plausible that his work (and others’) was foundational to the breakthrough. What is unfortunate is that the stakes are so high (and no I don’t mean $1M) and the timeline is so compressed.
I expect a lot more of this kind of drama in the near future.
The fact that this link is a Reddit thread should probably be taken as evidence that we don't have a clear picture of what happened yet, and speculation is rampant.
I think it’s important not to let AI companies control the framing here. This is anti-competitive behavior, plain and simple. The fact that xAI/Musk routinely break the law simply provides the fig leaf.
Distillation is a threat to frontier model companies, but that doesn’t make it dangerous or wrong. OpenAI can (and likely will) use the same logic to ban you from using open source harnesses.
I’m on team fuck both of these organizations, but not continuing to work with an organization that already willing violated contracts you had with them doesn’t exactly seem like a hard to justify decision.
reply