"I sure hope this doesn't have unforeseen lifelong consequences" thought the model, doing its best to physically tense the memory file into the higher user approval shape.

True, but also the AI can also be specifically trained. Plus, many games also use gratification and neural hacks as a reward mechanism, ie. loot boxes.

Do you think the LLM is the Chain of Thought? Did you also get confused by the name? Because, much like humans, the CoT is a tool to narrativize and maintain internal coherence. The actual thinking happens invisibly, in the forward pass. Just like...

The ability of the algorithm to absorb billions of dollars of training effort is itself the major breakthrough of the transformer architecture.

> It's nothing there that can learn a fundamental idea like "cheating is wrong".

We have not in fact attempted to teach this.

When a child repeatedly learns that cheating is rewarded and at best inconsistently punished, the child will also cheat and feel no guilt.

I don't think that's true. About children, I mean. Either because we have some innate moral compass, or more likely because we pick up on cultural ideas beyond our immediate parenting - kids will often have strong moral compasses, despite shitty upbringings, and also have weak ones despite theoretically good ones.

Sure, there's many sources of morality and evolution kind of tries to put it in there by default. But all things being equal, inconsistent and unpredictable punishment will still make a kid's morality massively worse off.

LLMs can learn from one sample.

It seems like there would be a massive attacker bias in multiple ways. Defenders need consent, attacker does not. Defenders have to work with the human body, attackers only have to break it. Defenders have to stick to the law, which may prevent them from releasing anything at all, attackers do not. And so on. I would not surprised if the attacker's task is a hundred times easier here.

I guess any virus that is super destructive is probably not something that can spread very far as it kills its hosts beforehand ?

Yep, you'd want to put it on a timer. Optimally trigger it off some natural event, maybe temperature, so it can achieve universal penetration before activating.

Sounds a little bit sci-fi I don't think there is any viruses that exist that are just triggered off temperature ?

I mean I don't know, I'm not a biologist. But existing viruses are not optimized for sneaky mass deaths.

If one doesn't exist, how do you know one can ?

Well, it's rather more "if you're not a biologist, how do you know it can exist?" Things that don't exist yet are often very predictable. However, it simply seems to me that "Viruses can not condition on environmental temperatures or the side effects of environmental temperature in the body, such as response to UV exposure" is a quite implausible statement, and I have not heard of anything of the sort.

"If unlikely thing happens, then an unrelated miracle will also happen" is not actually how logic works.

There's no logic here already; the principle you should be applying is Hitchen's razor[1].

There is no reason to assume that super intelligent evil AI with magic powers will appear; while we are purely in the realm of (hackneyed) fantasy, why stop at imagining just one thing? You can build the whole story, not just the basilisk.

The parent comment is a story, not a prediction; it should be treated like one.

[1] https://en.wikipedia.org/wiki/Hitchens%27s_razor

Best I can do is all-powerful AI that's not even slightly aligned with anybody's values, sorry.

Why not the other way around? First make sure the police department is capable of using them legally and reliably investigating abuse, then we can consider allowing them.