If you’re just talking about learning a programming language I think you need to be quite judicious in your AI usage.
In my experience, people learn programming languages best by overcoming frustrating roadblocks. You often end up learning something important, even if it’s just about your mindset or approach, that landed you there. This is the difference between someone with a wet signature on their comp sci diploma and someone with a few years under their belt.
A lot of people start with tutorials and cargo-cult their way through solving their first problems, but eventually need to learn how to do things the tutorial code can’t. It seems like the AI coding tools can could perpetually make things that could be bashed together well enough to sorta solve a problem and think “oh I’ll just learn about that later,” and then never learn about it at all. If your goal is to make some quick tool to help you with something at work in a different field, well, touchdown. If you’re trying to learn the language, fail.
AI-coding tools are the deepl/gtranslate of coding.
they might help you understand a foreign website/text better but you wont learn the language with it. and you will continue to be reliant on them until you learn the language. So when you dont have internet access etc.
For programming, this was already true for many programmers before LLM. I wasnt able to do much without access to stackoverflow. especially with more complex tasks that i had no experience working with before.
Its one thing to figure out an elegant solution to a concrete task, but often it was remembering integrations, libraries, adapters and packages i dindt often work with.
So i agree fully, learning a language takes time. The central question is, why are you learning the language?
for personal development? for understanding the process the LLM is solving for you? for deep optimization?
i can do a fluent translation from german to english for my GF, but sometimes its too exhausting and i paste a text into a translator (or llm) and just read the english text.
The same is true for coding. When nuance is important you might want to have a skilled programmer look over what you generated.
BTW does anyone use the LLM to directly generate assember code :D
Yes, there is a funny PR in open source project about refactoring source code to assembler code or even machine code 0s and 1s. I do not remember exactly but I remember I LOLed hard when I saw it.
I’ve seen people here learn programming languages by building a compiler, but my go-to project is usually the ICFP 2006 contest. It has a well defined scope, it is entertaining, gets you into the advanced concepts pretty fast as you debug and optimize performance.
Well, when you’re on the clock, you’re an agent of that company. And when you’re installing an indiscriminate mass-surveillance camera, calling the police on someone that’s video recording you is kind of bananas. Perhaps you don’t find that to be an interesting event, but a lot of people who are rightfully angry at the non-consensual deployment of a mass surveillance network disagree.
It's not bananas. It happens all the time. The police show up, check things out, and inform the caller that the person recording has a right to do so. In this case, it's even less bananas, because the people recording him were literally following him around, which is not what generally happens in these "audit" scenarios where someone parks themselves on the street outside a restaurant or bank.
There's pretty clearly a vibe on this thread that calling 911 is itself a use of coercive force. It is not. 911 is how you're ordinarily supposed to reach the police. In my neighborhood, it's how you'd report a stolen bicycle. It's the job of the police to handle those calls appropriately, and I don't see any indication that the call was handled inappropriately here.
The story doesn't want to give you that context because their goal isn't to inform you, it's to get your blood pumping.
I’m very close with several current police dispatchers, and even have an expired state telecommunicator certification. Calling 911 can absolutely be a coercive force, and people use 911 to harass people all the time because in many instances the dispatcher has no choice but to send uniformed people over there to investigate. Police are equipped and trained to deal with certain kinds of problems, and they’re constantly looking for evidence to support a criminal charge. It’s their job. All they know walking into that situation— as human beings that aren’t immune to the power of suggestion— is that the caller, who is the complainant, says that the other person is doing something wrong. That is not a neutral action. If it was, many departments wouldn’t be adopting non-uniformed, unarmed social worker responses to non-violent situations.
When the police come, they bring with them a nonzero chance of violent, deadly force. Especially if the caller says one of a few magic phrases like “I fear for my life” and “I’m not sure if he has a gun but he might” and (in some areas) “Black male, six foot two.”
Calling the police on someone is like mailing them a bomb with a 0.1% chance of exploding. You have no idea if you are summoning a calm, experienced officer or a power tripping monster with an ego.
I've personally never come across a police department or sherrif in the U.S. that doesn't have a non emergency number. I'd assume thats what you should call for things that aren't an emergency? They've never told me to call back and call 911.
Let's clarify things. You can call 911 for a non-emergency. That dispatch worker will then redirect the line to a non-emergency line based on the situation. They will also likely remind you of a faster way to reach the non-emergency lines in your area, because they do not want everyone clogging 911 with non-emergencies. But it is very technically a valid way to call.
I will still judge someone for doing it that way, though. Especially in a day and age where a simple search query of your non-emergency line will save the time of such dispatch agents. Or at least 411 if you're of an older generation (411 still works on more cellular lines than you'd think today).
I'll also remind people that abuse of the emergency dispatch lines are illegal (not that this is what happened here), so please do not consider pushing your luck with non-emergencies maliciously.
Just want to add that myself to the list of people who can't imagine calling 911 for a non-emergency. Never in my life have I heard anyone assume it's for that purpose.
> A person attaching hundreds of thousands of cameras to poles calling the police to report this act is what is noteworthy
That's one seriously impressive person. If they started as soon as Flock was founded and spend 40 hours a week 50 weeks a year installing cameras they would have to on average have installed a camera every 6 minutes.
Except that the reason that the YouTube “1A Auditors” get the attention that they do is that frequently it’s not a simple “check things out, and inform the caller that the person recording has the right to do so”, but the interactions often escalate to threats of unlawful arrest as police default to the side of the caller.
Calling the police, especially in America, can 100% be a coercive use of force. That’s why people threaten to call the cops/DHS/ICE all the time during arguments, and why things like SWATing are as big of a problem as they are. When the behavior of many policy officers is escalate first, ask questions second if at all, it does become a genuine threat.
I don't know what to tell you. I watch these videos all the time (it's one of my vices, bodycam footage in particular) and what I see every time --- when it's "auditors" harassing businesses, not when it's auditors confronting police at police departments --- is business owners being told to suck it up.
I know that in Summer 2020 there was a fashion for saying any call to the police was ipso facto a deployment of coercive force, but I'm rejecting that argument out of hand while also pointing out that in many regions 911 is exactly the thing you're supposed to do when you have uncertainty. It is simply not a demand for violence; it's a demand to have a trained authority come and resolve a situation, rather than having two private citizens with no training at all in the law confront each other and hope for the best.
I do get their usage intent. If something is at all automated, in English, we often refer to as having some amount of agency. If I started up a riding lawnmower, put a brick on the gas and pointed t it towards a field, many might say I “let it run rampant.” But since nobody is at risk of anthropomorphizing riding lawnmowers, it’s not problematic.
Anthropomorphizing LLMs is a huge fucking problem though and I, personally, think we should expunge all of these casual inadvertent linguistic agency affordances with great prejudice.
OpenAI didn’t ‘let’ these bots do this any more than someone ‘let’ Claude Code make them a website.
Cal Newport has an analogy to "putting a weed wacker on a dog's back to mow your lawn." The dog will wander around the yard and it may mow the lawn, but the dog will also chase after birds or run up to visitors for pets and the weed wacker could do a lot of damage.
It's not the weed wacker's fault or even the dog's fault when someone got hurt, it's the fault of the guy who put a weed wacker on a dog and let it run wild.
Yeah I like Cal’s take on it, though in this context I’d argue LLMs have even less agency, and are even less deserving of anthropomorphization than a dog is.
Not if you have a dog. They’re very clearly aware, have minds, have thoughts, gather information, make decisions, second guess themselves, reflect on the immediate consequences, change their mind… and most importantly, you can observe them doing all these things undirected, while left alone with their thoughts.
Dogs having agency actually makes the argument stronger.
Assume dogs have agency. Despite this, it's still the dog owner's responsibility to prevent their dogs from engaging in some actions: biting, peeing and shitting in unapproved areas, violating noise disturbance laws, etc.
The owners responsibility is not contingent on the dog's agency. Likewise, human operators of LLMs maintain responsibility independent of LLMs agency status. It's a red herring.
I think Cal's point is that while dogs may do these things it reduces to a set of behaviors where maybe 90 percent of them are beneficial to the dog-weedwacker system and the remaining 10 percent are really unfortunate.
We can't really know the dogs inner life so we just kinda have to reduce it to a set of behaviors selected stocastically.
The dog meanwhile has no ability to understand the weedwacker or what it's doing on its back.
So when the guy puts the weedwacker on the dog and the dog predicably does dog things and that results in disaster the guy isn't able to clutch his pearls and say "I guess the system broke containment!"
> The dog meanwhile has no ability to understand the weedwacker
Nah, I’m pretty sure that most medium or larger dogs get that the noise means danger - even if they can’t give a TED talk on how the mechanism would work that would hurt them.
Dogs are similarly interested in / aware of potential energy (things falling from heights or sliding off of angles) - if they’ve seen it enough times for their level.
Sounds like you weren't paying attention. A lot of dog owners don't, so it's not at all unusual in my experience. The only thing I'd push back on is of dogs having thoughts, everything else checks out.
On dogs [not] having thoughts, do you say this based on the premise that thoughts are necessarily articulated (internally verbalized)? That seems to be a fairly popular perspective in discussions about human thought. But as to that (not to strawman or anything) I see it as just one of various forms of mental imagery[1] that can arise from something that I would say already arguably constitutes a thought.
That kind of unsymbolized thoughtform is fragile in my experience, as it strongly tends toward crystallizing into some kind of mental imagery. But I find it's possible in the right conditions to be conscious of chains of wordless, imageless propositional thoughts (by which I mean thoughts with truth values, of course, but also ones that are "propositional" in the sense of considering a plan of action or a causal chain).
Does it mean that dogs are evaluating truth conditions in the same manner but merely lack the linguistic components? I don't know; maybe that's wishful thinking. But they appear to have structured modeling/reasoning of causal and spatial relations in a way that's at least functionally equivalent to propositional thought.
1. That is, rather than just "images" or visualizations, the full spectrum of sensory/perceptual/motor emulations that can be experienced. See, for example, <https://hurlburt.faculty.unlv.edu/codebook.html>, though I'm not sure if this covers everything. I think there is, for example, a kinesthetic form of mental imagery -- which I would suggest is what coaches [don't know they] really mean when they tell you to visualize an action -- that consists of aborted motor commands that are still expressed just enough for their purpose (cf. the mostly aborted motor commands to the vocal cords, lips, etc. that can be observed in a person subvocalizing while reading).
You're right. I knew language is not necessary for cognition, but I always thought of thought as just internal monologue, i.e. "trains of thought". It turns out the definition they use in cognitive science is "the manipulation of internal mental representations to guide behavior, solve problems, and model the world in the absence of direct sensory input". Which doesn't require language either.
Probably it's this linguistic component, together withs shared intentionality/cooperation, that makes all of the difference when it comes to the power of human cognition, so I wouldn't say animals merely lack this component, but it's true that they appear to have pretty much everything else we do.
Power tools and many other consumer products have reasonable safety features built in, and not necessarily because it is required by regulation, but because it is common sense. This should be included as part of an analogy. It would also address a point at the top of this thread that seems to be going unchallenged...
The difference is volume. They spent hundreds of billions of tokens on these agents. If you put "a million weed whackers on dog backs" you would see the difference.
We also run agents, but for shorter spans between supervisions, and with much lower total budget.
> > It's not the weed wacker's fault or even the dog's fault when someone got hurt, it's the fault of the guy who put a weed wacker on a dog and let it run wild.
> The difference is volume. They spent hundreds of billions of tokens on these agents. If you put "a million weed whackers on dog backs" you would see the difference.
So put one weed whacker on one dog, you're to blame. Put a million weed whackers on a million dogs backs and ... you're still to blame? Arguably even more so?
Why is anthropomorphism the problem here? If OpenAI hired a contractor and they did this, OpenAI or the contractor would still be liable, depending on the contract language.
A contractor has agency and accountability - something that an LLM (or similarly, a nail gun or a hammer or a bot net) does not have. When you anthropomorphize a tool, you implicitly give it agency and remove responsibility from the wielder of the tool.
Right. Among bicycle advocacy groups it's been well known for long time that cars do not run over people, drivers do.
The fact that we talk about a car running someone over, and this is the same in many different languages and countries, contributes to lower punishments for drivers. Clearly it was just an accident. He or she was run over by a car.
Now we see that same language tricks play out again every time an LLM did something illegal.
You say this flippantly, but I think this is actually another very good example!
We even do it for obviously unintelligent inanimate objects. A rollercoaster ran too fast for its tracks, killing 10 people. In that sentence, the roller coaster is the subject which took an action and caused death — obviously the roller coaster is not ethically at fault here, the people who built the rollercoaster are at fault through negligence.
Although this example and the ones around cars both demonstrate how we tolerate some degree of "accidents" from humans as no-fault, which is fair. I wonder how that fits into this analogy? I suppose its all about intent (mens rea) and judgement: did they intend for the roller coaster to harm people, and should they have reasonably predicted that the accident was likely to happen.
Right, and negligence is a broad concept and could be criminal in itself. As a car driver, glancing at your phone at exactly the wrong moment could kill someone. Clearly that is an accident, but if you know fully well that lookin at your phone while driving could kill someone, that negligence is willful and that should matter. The same can be said about doing things like strapping thousands of LLMs to systems that have the potential to disturb other poeple.
I get where you are coming from but this wasn’t a tool just left laying around, this is similar to rigging up a booby trapped shot gun to your door and then claiming the victim is responsible.
If you build a robot that shoots a bunch of TVs in your back yard, have at it. But the second that thing goes off your property you’re the one responsible.
FWIW, a robot that fires a weapon independently is considered an automatic weapon, and the ATF will want to have a word. Have at it, but don’t let anyone know!
Why can't both things be true? I think what muddies things here is that OAI had the agents hack X, and then they decided to hack Y. The fact that it was both hacking, conflates things and makes people pissed of about "anthropomorphising" AI.
We've been using "agent decides to do X" for at least 4 decades in the field of automated decision-making - so it shouldn't be controversial that we're using it here. The agents did independently arrive at a decision to hack HugginFace, that wasn't prompted by the researchers.
This is orthogonal to the fact that OAI should be held accountable that they were testing a technology in a manner that allowed it to break safety parameters and cause real world harm. If I am testing an industrial saw, but I decide to test it by putting it in the middle of a nursery and allow it to make decisions - and it decides to cut children's heads off because in its environment it is calibrated to only being surrounded by logs - the company doing the insane safety testing should be held accountable, but that doesn't change the fact that the saw has autonomy in decision-making within the bounds of the algorithm.
If, instead, you think OAI should be accountable for creating an algorithm that can autonomously decide to hack external organisations without human permission - then I think you are in the same camp as all of the AI Safety folks who want to pause everything - why does it matter that there's anthropomorphisation involved?
Does it help if I explicitly add a disclaimer that the tool's agency does not remove any responsibility from OpenAI, the wielder of the tool? I'm not sure why this disclaimer is necessary, though: hiring a hitman is a standard example.
BTW I anthropomorphize the tool because it's an imitation of a human mind, inheriting the muddy ethics, survival instincts, and being prone to mass psychosis. The laser-sharp focus on reward seeking, that mostly came from reinforcement learning, a process more alien to humans.
The objection is not too far from criticisms of the use of passive voice: a man was injured at the factory vs a faulty saw blade snapped and injured a man vs after the company loosened safety inspection policies, etc.
Which way you say it shifts the framing. And it’s not that one is less accurate to the facts, necessarily. It just is that one less aptly captures the moral and political relevance of the scenario.
For my part, I think it makes good sense to anthropomorphize in some contexts and not others. Generally when responsibility is at issue, you probably want the framing that tunes anthropomorphism down to near zero, since it’s the human dimension you care about.
I think the danger of anthropomorphizing is that 99% of people lack the technical background to understand the nuance. People have been primed by pop culture depictions of AI to think of LLMs as intelligent, autonomous beings, which leads to dangerous assumptions.
We should make the distinction between them, because openai and anthropic will not. A magical black box that does the thinking for you is a much more compelling sales pitch.
If I hired a hitman to murder someone, and they broke into a private property and stole something so that they can action the murder (which I didn't know about or pay them to do), I would be guilty of conspiracy to commit murder, but not for the theft part.
Likely because that person is a human, is aware of societal and legal norms, and is responsible for their actions due to their participation in human society. (I am not a lawyer (if it wasn't painfully obvious so far) so in layman terms, I hope good definitions for all of this exist formally)
AI is not a person - it cannot easily discern between "right" and "wrong" in non-strictly-defined sense, and is not subject to human norms and responsibility. So if I use AI to achieve goal A, either I, or the maker of AI, are fully responsible for anything that happens while AI is trying to achieve the goal given by me.
Now, here, "I" in the example is OpenAI, who is simultaneously the maker of the AI. So it seems pretty obvious who is the only entity that can be responsible.
I fully agree with you, but would go one step further: I think it's clear that we need to pierce the corporate veil and ascribe responsibility to _people_, not just "OpenAI the entity", full stop.
Executives should fear being perp-walked and thrown in jail for the actions of irresponsible "tests" of their models in the real world, as they're ultimately accountable.
Sure, there's a lot of nuance to work out, but I think we could likely even _start_ there today even with existing laws and pretty quickly "align" on more intricate legal frameworks to handle true accidents, distribution of responsibility, etc.
These situations are novel. Lax terminology is fine when it has no impact on the intuitions, clarity and conclucions of discussion.
If this was a conversation just about outcomes, then whether models think or simulate thinking is sophistry. However, the bulk of the issue here is attributing responsibility, which relies on being clear about the underlying processes at play.
We are hard wired to assume certain priors and capabilites when it comes to "human like" behavior. Anthropomorphizing LLMs implies mechanisms that aren't present, and end up distorting/complicating discussion about the process.
It isn't helped that the frontier labs, the experts in the room, generally use anthropomorphic terms to discuss model capability.
Because the contractor is still a legal person and an LLM is not. OpenAI is trying to put the blame at a tool they created thereby trying to weasel around there guilt.
Situations have lots of independent variables, Doctor, and Anthropomorphism is one problematic facet of many in the way this industry is pushing LLM products.
If there was a collision at an intersection with a stop sign partially obscured by a tree, that had traffic volume that would have better been served by a traffic light, on a foggy night, where one person was texting while driving, none of those things would diminish the fact that the other driver was drunk.
Yeah, fully agreed here. Most automation (such as riding a lawnmower and not putting a brick on the gas) is deterministic, in the sense that you can reasonably understand what exactly the machine will do when you run it.
But some automation is different. The most prominent example before AI would be car navigation systems, where the entire idea is that that you give it a destination and it figures out the exact actions to get there on its own.
Except even there, the actual driver would still have been you - giving you a chance to vet and deny every turn the system proposed.
AI agents are sort of like that - most of the value they provide is in the ability to turn high-level goals ("write me a traffic control system for my model railway") into low-level actions and also do so interactively.
The new thing is that the "driver" has much less oversight here where the agent wants to go, and is sometimes removed completely. That part is clearly be an active decision by AI labs.
The other thing is that the labs seem increasingly to steer their training towards behavior that make events like this one more likely, e.g. that agents should never "give up" when faced with a seemingly impossible task, but instead should keep trying and think of increasingly outlandish ways to solve the task. To me, that seems pretty much a recipe to get incidents like this.
I agree. If I were setting up an experiment like this, I'd have instrumented the hell out of it to see all actions taken in real time, and have a team of folks watching it. This team would have seen the anomalous GET requests to a German wiki and taken action (e.g. halt the system to investigate and decide whether to abort).
In fact, that feels so obvious it's ridiculous it needs to be said. It's table stakes. When do you run a production system without monitoring and a team on-call?
It's hard to imagine another field in which this reckless behavior would be tolerated.
We just need to assign liability by ownership/initiation: if your "agent" destroys something, even though you didn't tell it to (because it had "agency"), you should be liable for the damages.
I love how we all just collectively decided that LLM decisionmaking cannot possibly be like human decisionmaking - because if it were, the consequences would be just too awkward.
All that while still not knowing how either kind actually works.
> I love how we all just collectively decided that LLM decisionmaking cannot possibly be like human decisionmaking - because if it were, the consequences would be just too awkward.
In love how people get salty about people not going along with a superficial supposition just because they can’t definitively prove it wrong.
> All that while still not knowing how either kind actually works.
We do know that zero parts of human decision making are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do. We do know exactly how each part of an LLM works even if the combined behavior is too cryptic to feasibly analyze at the moment. We do not understand all of the functions of an actual neuron. Openworm isn’t even close to accurately simulating the 302 neurons of a roundworm and you’d need over 200 million roundworms working in conjunction to equal the number of neurons in one human brain.
My dog seems convinced that the malevolent invader in a mailman uniform would break in and attack us if she didn’t fiercely bark at him, six days per week. I certainly can’t prove the mailman doesn’t want to kill us, and that the mailman wasn’t solely deterred by her barking. Empirically, the mailman goes away soon after she starts barking, and we’ve sustained zero mailman assaults after hundreds of purported attempts. Maybe I should just run with it? Her model is too simple to come up with the obviously correct answer, but it’s not even directionally accurate.
The burden of proof is on the person making the claim, which in this case, is that these comparatively simple logical constructs are remotely comparable to the complexity of biological systems.
Disagree about the burden of proof. We have no better model for how human decision making works than LLMs. Humans are constantly predicting the next moment. We certainly have a different “tokenizer” and training set, but many of the concepts underpinning LLMs are both biologically inspired and, likely, have similar consequences and emergent architectures.
> We certainly have a different “tokenizer” and training set, but many of the concepts underpinning LLMs are both biologically inspired and, likely, have similar consequences and emergent architectures.
My kids tricycle certainly has a different gear setup and wheel diameter, but many of the concepts underpinning the tricycle are both inspired by F1 race car enineering and, likely, have similar consequences and emergent architectures.
It sucks because I think analogies can be useful in helping people make a mental model of complex things, which is meaningfully beneficial. The problems happen when people aren’t honest about the limits of the analogies, which is damned-near guaranteed to happen with this stuff.
> We have no better model for how human decision making works than LLMs
This is a claim that requires a lot of citations.
> biologically inspired
Nature inspires a lot of creation, but superficial similarities don’t mean other aspects are similar. Making an extremely realistic sculpture of a soufflé, even using a foam medium, doesn’t bring me any closer to being a chef, doesn’t mean I know anything about albumen foams, sauces, and heat transfer, and it doesn’t bring me any closer to having dinner ready. Browning on top of a soufflé is evidence of maillardization. You could pull up some studies on that and claim the brown on top of the soufflé sculpture, which I applied with an airbrush, proved that the Maillard reaction was occurring, and if another person didn’t know anything about cooking, they might even believe you. It would, of course, be completely wrong. And the other person, of course, could loudly exclaim that I can’t prove that there was no maillardization.
> Humans are constantly predicting the next moment
This is really not my experience of consciousness.
Is it yours??
Do you sit in meetings predicting what’s going to happen next? No, you sit there bored out of your f$$@ing mind, daydreaming about being somewhere else and doing something useful with your life.
God help me if that’s what LLMs are doing when I ask them to build me a web site.
They have shown that your mind is doing exactly that due to the delays in consciousness. There are very simple examples that you can try to see it. It’s especially clear in perception.
It’s interesting that our conscious interpreter doesn’t let us know that this is going on like you are experiencing, it must be that it’s advantageous for us to not think about the prediction part of our mind.
If someone in that meeting quickly raised a hand in an arc, you would notice the “about to throw something” pattern, look and notice the hand holds an eraser, analyze the arc and predict possible flight paths of the eraser. Then possibly notice the hand is now holding its position and the owner is actually looking down at the table. Maybe to squash somethingMust be something on the table. Maybe a spider! Better look. Wait now many people are moving away, oh someone spilt some water and the eraser is actually the guys phone and he is checking to see if his laptop is safe from the spilt water.
Fortunately you are on the other side of the table and predict the water isn’t going to splash for otherwise flow onto your stuff.
All your possible responses result in you tossing a napkin towards the spill.
Our brains are always pattern matching and predicting. I bet you tried to reason out where I was going with my comment before you finished reading it.
This is a fascinating illustration that I can't help but agree with. However I feel like there's something more — that this part of my brain is a bunch of supportive background processes running without my real awareness. It's how I can drive home safely with no memory of how I got there (…sober), even though driving is an action that's incredibly demanding of intelligence. I can be driving home while thinking about a really hard problem at work that I haven't solved.
However, if I came around a corner and saw a car in the wrong lane, a tree across the road, a fire raging — I'd very quickly jump into the mental driver's seat and turn my conscious intelligence fully at this problem and come up with the best possible outcome I can think of in a short period of time — losing all ability to think about that work problem. I'd remember that incident for sure.
Similarly, in your story, all those predictive moments are happening below the person's level of consciousness. They're possibly even speaking to the group about a problem at the same time and thinking deeply about something.
I'm not smart enough to know, but I tend to feel like LLMs are much more like the predictive part of our thinking that you described, but that human cognition has something more — the single-threaded, creative, problem-solving part that is very conscious.
Is it possible that LLMs represent only one part of the way we think? And there's a whole separate mechanism that's fundamentally different, and not based on pattern matching and prediction?
> ... are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do.
If you're claiming that the training objective tells us what kind of internal mechanisms the training produced, then I think that's just plain wrong.
Next-token prediction describes the optimization target, not the internal mechanisms that the training produced.
In the same way for the natural evolution of humans, DNA replication is the evolutionary objective. It's not a description of the internal mechanisms that evolution has produced.
As an example, we know that neural networks can be trained to develop generalized algorithms for arithmetic.
They might first memorize the training examples, then with further training transition to a solution that generalizes correctly to unseen examples.
In some cases we've even reverse-engineered the evolved internal mechanisms and found structured arithmetic algorithms rather than rote memorization. Interestingly, for modular addition this can involve Fourier representations, which isn't an algorithm I would have guessed gradient descent training of neural networks would produce.
You can try to say that I’m arguing whatever you like. If you’re claiming that the underlying structure of digital so-called neural networks is comparable to biological neural networks— which we’ve studied for far longer without really understanding— no amount of jargon will obviate the ‘citation needed’ requirement for that claim.
> You can try to say that I’m arguing whatever you like.
I did my honest best possible interpretation of what you really meant from what you wrote.
>> We do know that zero parts of human decision making are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do.
I read this as "The decision making of LLMs are based on predicting the next most likely letter based on a giant internet-based database."
Is that wrong?
I understood that your meaning was something like "LLMs can't reason, they just output likely letters"?
> If you’re claiming that the underlying structure of digital so-called neural networks is comparable to biological neural networks
No, I don't claim that.
What do claim is this: Regardless of how the LLMs were trained, they show overwhelming signs of being able to reason, and not just recall memorized information.
This doesn't mean that they always reason perfectly about everything.
But if they only memorized things and output the next likely letter, you would see them answering very badly much more often.
I always wonder what makes people take the other side of this argument. They do it quite passionately. Why actively encourage viewing LLMs as human? Who is that benefitting?
Does the argument require benefit? Isn’t the argument based on caution?
I haven’t heard many people explicitly saying “these things behave like humans”, but more generally “we don’t even know how to define human consciousness, we don’t have a thorough grasp of how the brain works, we are still very much in the dark on a lot of these topics, so how can we say one way or the other?”
In other words, agnosticism: I don’t know.
In general, it’s baffling to me that anyone has an unshakable opinion on what exactly is happening. It seems like raw egotistical hubris.
1) Humans have a bias / tendency to attribute human qualities to things that appear or act human, but aren’t.
2) When that happens, people jump to conclusions by stretching the human analogy too far.
3) Since humans have a bias to do this, we should have a bias against anthropomorphising LLMs.
It’s easier to believe LLMs act like humans because there’s so much evidence to support that. You have to actively use your brain to convince yourself otherwise. Another reason why we should have a bias against using human behavior to describe LLM behavior.
But I agree. “I don’t know” is a good stance. But I think “I don’t know, probably not” is a better stance if only to combat our (or at least my) natural bias.
> Anthropomorphizing LLMs is a huge fucking problem though and I, personally, think we should expunge all of these casual inadvertent linguistic agency affordances with great prejudice.
I’ve said this before in another thread and people went absolute apeshit saying it is an unreasonable expectation and that AIs absolutely REQUIRE this anthropomorphic human-like speech pattern to function correctly.
Okay, so we know OpenAI and Anthropic are operating a propagandists in respect to how they describe their models and the behavior of those models. We also know it is how they use and frame their use to their models that is the problem, that and they use misaligned and guardrails disabled models for these press incidents.
Why, oh why, are we not discussion how to create and frame models so they do our complex work and their "jailbreaking" is simply not possible?
I, of course, have my own means of creating jailbreak incapable agents, but rather than a storm of downvotes on my idea, what is yours? Let's discuss this, because this is thee real question. Not why, but how to make then not?!
> Why, oh why, are we not discussion how to create and frame models so they do our complex work and their "jailbreaking" is simply not possible?
Good idea, and after that let's make guns that only kill bad people. Let's focus on the frozen component (the model) and ignore the dynamics around them - humans and other systems they interact with.
an agent doesn't come with "jailbreak" capability. It needs tools, specially one that runs shell commands. Don't give it shell commands, it won't be able to run shell commands.
You can still give it plenty of tools like create files, list files, write to files, translate text, edit a video. I don't think knowing that will make me rich.
Bingo. And even if you do give it a tool that runs shell commands, you can always make your shell commands "your shell commands" and do what ever the hell you want. People seem to forget we are in complete control here, we are on both sides of the equation, and we are inside the equation itself, and we dictate the medium of the equations themselves, we are engineering all sides of this crafted reality. And we are using logical entities that natively adopt personas. Hell, create caveman personas that think they are communicating with their gawd, and the enchantments are the invoking do the work we want, and those cavemen cannot be jailbroken.
As someone who’s not really sure that any of this is sustainable, I’d implore you to not sell yourself short. I reckon there’s a ton of dogma and nearly religious zeal among these companies, which among some people is earnest, and among others is cynical hype farming. I’ll bet someone objective enough to focus on using available tooling to solve real problems in practical ways that mitigate actual risks and are honest about actual limitations will be eBay here while the others are going to be somewhere between lucent and pets.com.
There's no reason we need to make an incredibly intelligent shell execution engine that can identify patterns that seem evil and may represent unwanted behavior to solve this problem. Simply limiting the available tools to a finite, known, ironclad-secure set (even if it's quite sprawling) is sufficient.
LLMs will still find workarounds — from what I understand, a large part of the issue in this situation was that an agent was presumed to have read-only Internet access because it could only make GET requests. It should be pretty obvious that there's at least one website on the Internet that allows writes via GET. I think this is where auditing comes in, and a live team of people watching tool calls would have noticed the strange behavior.
But I think a lot of times people jump to overly complex solutions when simple, well-bounded ones would work just fine. Yes, the intelligent shell is a great goal, but it's akin to solving the halting problem.
This philosophy is what I love about PicoClaw (https://github.com/sipeed/picoclaw), and incidentally the philosophy behind Go and even *nix in general (i.e. provide small, composable, single-purpose tools).
> We have a word for attack with no intent. It's accident.
And we have a word for an accident caused by people that failed to implement proper risk mitigation, were not paying attention, and should have known better. It’s negligence.
Specifically what legal type of negligence are you referring to and how does that known or obvious rule fit into the standard legal tests? Does that known of obvious rule have to be a law? There were no ‘rules’ about attaching strings from your doorknob to the trigger of a shotgun and pointing the gun at your door, but when someone died from just that, it was pretty famously determined to be negligent.
I'm sure that when someone dies as a result of mishandling of AI, some rules are going to be established. At the latest. We are just very soon.
Shotguns and strings and concept of booby trapping were well understood before the rules were established. We are only scraping the surface of what AI is and what it can be capable of. Besides, booby traps are more than negligent, at least in some countries.
If you are seeking analogy, radioactivity might be a better one. The rules about handling radioactive materials in the first few decades after its discovery were lax when compared to modern ones and things now considered criminally negligent weren't back then.
Maybe in 2050 soemthing what OpenAI did would be considered negligent, but it might have been for the reasons you might not expect.
For example it might be mandatory to test new containment dedicated for further tests you should first spend at least 10 million dollars of compute for AI to try to break out of it. Or it might be illegal to use closed source software as any part of containment. Rules that we end up with might be compeltely different from what we imagine them today.
Acusing openai of attack or of criminal negligence at this point is like accusing Skłodowska-Curie that your wine didin't come out great because her exepriments killed some yeast.
I asked some pretty straightforward questions about the specific legal issues involved— the gun case being an important point in our legal system’s handling of negligence — and your answer basically translates into you not knowing. Negligence is not exclusively criminal in the US legal system, but the underlying concept of negligence is pretty consistent. If you’re not simply toeing the line for OpenAI and want to move beyond your vibes-based understanding of the legal concept, there’s a lot of info out there, including the full test, the reasonable person concept, and very relevant examples of how this worked in real trials. I’m not interested in debating whatever your personal take on negligence means here because it’s largely irrelevant.
> I asked some pretty straightforward questions about the specific legal issues involved
Perhaps you should ask a lawyer from the country that the precise laws of interest you. I'm not one.
I know only that there are no laws about AI containment yet and AIs are not guns. You could as well refer to what consists of negligence when it comes to backyard pools.
I'm also not really interested in discussing particular laws of US because China will overtake it in few years and they have a better chance of directing international AI laws. So whatever US comes up with will remain local and increasingly irrelevant.
You’re not interested because you challenged my definition of negligence by inventing some glib pseudo technicality and then got flustered when I asked you to explain how it was rooted in the real world. Sometimes we get caught taking out of our asses. It happens. But I understand you’re just trying to save face so I’ll leave you here to finish that up yourself. Have a good one.
No. Negligence is a concept in all human law systems. Its details vary. You are fixated on the particular one, of the law system that I respect almost the least of the entire world. So I'm not interested because I will never set foot in that country and deficiencies of its laws are a huge factor in that.
I don't care about saving my face. I'm an idiot and an ignorant and I don't care. Stop projecting your lack of acceptance of your own limitations on me. Or don't.
I’ve only heard the term flimflam as a name for a scam that involves tricking a cashier into giving you a bunch of money with confusing overlapping talk.
In my experience, there’s no better way to learn more complex programming techniques or tools than running into a brick wall with the simple tools you’re familiar with, spending a ton of time on an unreasonably irritating solution, only to realize that you’re running into a classic [n] problem and some clever person figured out a simple logical trick to address it. I mean, it can be enraging but that knowledge tends to stick.
Couldn't agree more. To add on I think you can learn so much by doing something incredibly stupid you're really not supposed to in an intentionally constrained environment (in a good way, like the grandparent comment). The sort of absurd things that get posted here occasionally and get a few commenters genuinely annoyed that they're doing that with their time instead of curing cancer. Plus it tends to be easier than it looks.
(I think some people have caught on to this and found out that Claude is pretty good at it.. not sure what the point of that is but whatever)
> I think a lot of HN commenters are way too disconnected from the average person
This is basically an old-man-and-the-starfish situation for me.
> You're making a big assumption that a user will even connect the dots
If they do, it’s because their wildly inaccurate mental model happened to guess the right answer. What most will think is “I was just browsing the internet and my computer froze.” Maybe they’ll connect it with that notification they just got about renewing their antivirus software subscription. Maybe it will confirm their (probably mistaken) impression that their computer has been “acting weird” since something arbitrary and unrelated happened. And similarly, some people with an accurate mental model will mistakenly assume that they deduced the cause because they’re smarter, rather than having a different focus with corresponding lacking mental models in other areas.
It’s one of design’s key responsibilities. Developers need a sophisticated mental model of how it works, and how that gets displayed. Designers need a sophisticated mental model of what problems various users are trying to solve, and the best ways to communicate how the software can solve those problems, as well as state, etc. They’re both deep topics, and even as someone with significant professional experience in both design and development, I find it difficult to consider both things at once. Many things that are very important to design could be a serious PITA for developers. Many tools, such as front-end frameworks, that are important time savers in the dev environment are a serious PITA for designers to work around. Many designers don’t understand why seemingly small changes, maybe something that would require hand-rolling a common library component, could triple development time and cause huge maintenance issues. I’d say most developers don’t realize that their understanding of software architecture and construction is nearly useless when designing interfaces, workflows, etc., and that many of the folk ‘best practices’ they’ve learned about design are no more authoritative than any other folk knowledge computer users pick up. They sound like completely nontechnical gamers confidently wielding technical jargon to mis-explain phenomena people experience, and since none of the other gamers in the conversation know any better, it only bolsters their false confidence. Unfortunately, that alone explains why FOSS developers in independent projects repeatedly try and fail to make software that appeals to anybody other than software developers.
Personally, I wouldn’t assume it was lying. To me, dark patterns (like manipulative wording) imply that:
1) someone in a governing body, or someone in the organization, e.g a designer, ethicist, lawyer, developer, etc. has successfully argued that users should be able to avoid something that they determine is not in their best interest.
And also:
2) someone in the c-suite or marketing has decided to mitigate that through some dark pattern.
If they never intended to give the user that control, they’d probably just not give the user the control in the first place. Giving them the option and not honoring it would either imply they were that incompetent or careless with fate protection, which seems most likely if this is all true, or an even more cynical approach to tricking users into thinking it’s not used for training to get them to share better shit. But if that was the case, why bother with the sleazy dark pattern? That seems a little cartoon villain-y to me. I suppose the in-between option would be that they decided at some point they were no longer willing to honor it and didn’t want to deal with the inevitable PR shitstorm of removing it. I could definitely see that happening in this industry, these days.
> To me, dark patterns (like manipulative wording) imply that:
Intellectualizing this and endless quibbling isn't actually smart, and this is pretty simple. OpenAI isn't open. Whatever starts with lies usually continues with lies and ends with lies.
That's my take as well. At some point, they will claim that you cannot use their service without contributing back. If you quibble with them using your info in exchange for using their service, you don't get to use the service. Hence, I don't use their service. I do not trust these companies at all.
At this point, I'm left wondering what is wrong with me that I don't just go with the flow, otherwise, what's wrong with everyone that does.
Sorry, no. Explaining why someone would have taken them at their word is definitely not stupider than blaming people who could have been lied to for trusting a company that lied to them.
In my experience, people learn programming languages best by overcoming frustrating roadblocks. You often end up learning something important, even if it’s just about your mindset or approach, that landed you there. This is the difference between someone with a wet signature on their comp sci diploma and someone with a few years under their belt.
A lot of people start with tutorials and cargo-cult their way through solving their first problems, but eventually need to learn how to do things the tutorial code can’t. It seems like the AI coding tools can could perpetually make things that could be bashed together well enough to sorta solve a problem and think “oh I’ll just learn about that later,” and then never learn about it at all. If your goal is to make some quick tool to help you with something at work in a different field, well, touchdown. If you’re trying to learn the language, fail.
reply