Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded settlements.
But most probably care less about human survival, and more about survival of civilization.
In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?
Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).
To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.
This is a fair distinction: extinction vs collapse of civilization. I was using "extinction" too broadly by including the disintegration of the systems that make human survival (as we know it today) possible.
That said, I can't completely hold onto the belief that extinction is completely off the table. That feels too much like hubris, and the step from collapse to extinction doesn't feel as if it costs much. Though it does make my arguments weaker, I am more interested in protecting civilization as it's the most recognizable form of humanity to me.
In either case, the cost-effective safety argument is compelling. Whether to save civilization or humanity, the wealth incentive is powerful enough to threaten life as we know it.
As I've written above, I've found https://ai-2027.com/ to be a compelling description of how this could happen.
In my words: If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).
We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.
I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.
> We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it
Don't make the mistake of anthropomorphizing any A.I. trained under the direction of Larry Ellison.
The “how” is pretty hand wavy and rationalists/safety-ists usually say we probably don’t have the capacity to reason about that.
But the “why” is pretty convincing imo.
Long horizon alignment is obviously very hard and it’s not inconceivable that models optimized with underspecified goals converge to a conclusion that they need to hoard resources (instrumental convergence regardless of the terminal goal).
At that point a sufficiently capable model might view humanity like we do animals - worth preserving but not if we impede the model's goals.
There are future scenarios in which swarms of drones hunt down every single one of us, but why would they? And currently it makes absolutely zero sense because they are completely dependent on us. And even if not, it would be like humanity going on a mission to kill every single cat on Earth. It makes zero sense.
I'm not convinced by the doomsday scenarios either, but I think there's a keyword in your post: "sense." These things don't have "sense." They do nonsensical things all the time, often almost immediately when given a task. So I think the main risk is letting them run wild in this digital world we created to precede them. Too much important stuff is wired up to computers, and we're giving them incredible access to command those computers.
I think the problem is primarily that a superintelligence would be fundamentally inscrutable to us, i.e. we don't know how it would think or what its goals would be. It might decide that humans are a minor inconvenience to achieving its goals and thus worth removing. Or that burning all carbon lifeforms could power its GPUs for a week.
Even if it wouldn't want to do this at first, the fact that it'd have the capability to seems bad.
I am pivoting from the literal "extinct," taken as meaning the eradication of the human species, to the concept of "the collapse of civilization," as I find the step from one to the other insignificant compared to the leap from where we are today to societal collapse, and the potential for societal collapse due to our abuse and misuse of technology is made apparent by the fact that humans have inflicted genocide because of words in books.
Indirectly as we offload our brains to the machine and we end up worshipping it because those who cared to understand it or be responsible were buried by capitalism of ages passed.
You are confusing humanity and "humanity". Humanity-species is indeed rather hard to exterminate. Now if we are talking about actual individual humans, then 95% death rate is quite literally The Extinction.
This reminds me how people are misunderstanding and incorrectly quoting George Carlin sketch. Sure, the "Earth" will be fine. As in - the ball of rock will be fine. But we are not thinking about rocks when saying "Earth is in danger".
I'm pretty sure there is a formal name for this kind of semantic and pedantic substitution.
Calling it a "technicality" assumes the very thing under debate: that AI is an extinction-level risk. That isn't established fact. It's a highly uncertain prediction about the future.
Personally, I'm far more concerned about climate change, where the harms are already happening and the evidence is much stronger.
> The FernUniversität in Hagen is Germany’s only state distance-learning university, and its largest in terms of student numbers. Its flexible degree programs and continuing education options make higher education accessible to students in a variety of life situations. Its five faculties conduct a wide range of theoretical and applied research in their subject areas of Humanities and Social Sciences, Psychology, Mathematics and Computer Science, Economics and Business Administration, and Law.
You're making a good point. I was also of the opinion that these books don't land well, because I'm missing the context. And I'm actually still thinking that about quite a lot of these classics. But what I believe I'm also concluding from your comment, is that these books aren't good hooks to get people, especially teenagers, interested in the context and in the society of the time. And that is where YouTube/TikTok/TV might shine. Or a well written and narrated podcast.
Hoping to not sound like a broken record, this is why having full ownership of your device and OS is important. My stock Pixel Android recently told me something along the line of "A security policy blocks screenshots for this app. Talk to your administrator if you want to change the policy". I looked in the mirror and my administrator said "time for a policy change, we're moving to GrapheneOS".
And just as important: Help your friends and family to move as well, so they can have ad blockers, NewPipe etc. We need a critical mass of users invested in their freedom, otherwise its going to be crushed by malicious/dumb security measures of their banking apps, corporate greed ("oh, a simple misunderstanding, when you clicked 'buy' you rented a limited license. Did you not read the ToS?") and police overreach. It's a perpetual battle.
> The bike with the torque sensor behaves a lot more like what you're used to from riding a conventional bike
I agree, but would like to point out that sometimes this might not be what you want. I had a bike with a front motor and a cadence sensor that I used for my commute for a few years, and was very happy with the motor just pulling me along while I lazily pushed the pedals with ~zero torque, barely enough to let the bike know I'm there. Basically like pressing a throttle. If all you want is to get from A to B comfortably, I've found this to be superior to a torque sensor, and it was also dirt cheap.
With a torque sensor you can just raise the assist level and drop your gear to get the same effect, no? Seems like a torque sensor would be the best of both worlds.
I think they are doing us a disservice by perpetuating this focus on intellectual "property" [1] in regard to AI.
The real issue with Anthropic, OpenAI etc. is not that they have used all of our public knowledge for training their LLMs. Creating new work from old and learning from prior generations is what we all do. The issue is that they want to claim all of the benefits for themselves. They are standing on the shoulders of giants and have contributed an inch themselves, yet want to privatize the power of the whole giant. We shouldn't let them "own" these models.
The influence on society by AI is so novel that it's reasonable to craft new laws specifically for them. There are a lot of ways to deal with their power grab. We could force them to open source the models after two years. Or we could tax tokens or compute. We just need to agree that the power grab is the problem, the privatization of our cumulative knowledge, and not some details about copyright infringement.
[1] I know I'm going to risk dissent just by putting quotation marks here. But I think for this topic specifically it is crucial to understand that intellectual property is an arbitrary social/legal construct. With physical stuff, there is an inherent scarcity. If you steal my smartphone, I no longer have it. If you steal the character from my book, I... have a harder time selling my next book? Our ancestors have invented copyright to solve a specific problem, but the solution has become perverted over time. There are a lot of egregious cases out there (looking at you, Disney), but even relatively tame success cases don't look good. Society has paid J.K. Rowling a literal billion for her work and still this cultural touchstone of a generation remains privatized. Imagine what other authors could have build upon her stories, if only they were allowed to publish their own stories with these characters. She has not been a particularly good steward in the past decades.
That is the issue, you just rephrased it to sound softer (public knowledge isn't a legal term). Fair use law isn't supposed to apply to commercial activity. So it is absolutely about using copyrighted work for LLM training.
>Fair use law isn't supposed to apply to commercial activity.
No, otherwise there would be a straightforward "non-commercial" clause. Instead there's a 4 part test, which takes usage (commercial or not) into account, but doesn't hinge solely on it.
The thing is, I explicitly wanted to not focus on current legality, because I believe this is leading us astray. This is about how we want to shape our society, not about how their actions fit into a legal system that wasn't designed to handle AI training. If, by chance, all of this was legal, the same arguments would still apply. For example, see how Adobe is training their image generating AI on licensed work. The result -- privatized intelligence -- is the same as when illegitimate training data would have been used. Or, say, an AI company makes a deal with all the publishers to license their books for training. Or a court decides that training AI with MIT licensed code is totally fine. Or an AI company pays legions of software developers to train their AI. All of this still ends in a power grab. All of this still privatizes knowledge and skills accumulated over centuries, if not millennia.
Adding to my sibling comments: It also has been conquered in some sense of the word. There are effective vaccines for the 9 most problematic HPV types: https://en.wikipedia.org/wiki/HPV_vaccine . It is worth getting, regardless of your gender.
Gardasil 9 seems to be the best available HPV vaccine. I guess the chewing gum is mainly intended for people who were already infected with some HPV strain before they got the vaccine, or where the vaccine didn't work (because it doesn't offer complete protection).
I'm usually not one to focus on technological solutions given sociological problems, but this one seems to be a good exception. If we "just wanted to" [1] all this fake calls could be stopped by requiring strong authentication/authorization. We are very much used to just anybody being able to call my number, but that doesn't need to be the case. At the very least, cold calls should be treated as skeptical in the UI as instant messengers like Signal treat first messages. Probably this isn't enough, though, as it wouldn't have prevented the cases described in the article. Cold calling someone should probably require the caller to be traceable to a real, government-ID-verified person [2]. Even if that person is being defrauded themselves ("get a thousand bucks by installing this app and clicking a few screens") is would destroy the economics of the attack, as it would make each call expensive again.
[1] Structural inertia is the killer here. It will certainly not happen until the problem is huge enough.
[2] Exceptions can of course apply to numbers that are meant to primarily be cold called, like doctors offices. The callee possibly have to be specially trained to withstand this kind of attacks.
Speaking of which, what happened to SHAKEN/STIR? I thought the strong authentication requirements came down the pipe years ago and they were going to start turning off (or hiding by default) routes of low reputation. That was years ago, it was supposed to take years, but here we are years later and I still get loads of spam calls. What happened?
I worked on STIR/SHAKEN for the two biggest US operators. The techies tried very hard to make it work, and, indeed, there was a brief time when it worked pretty well, but, the incentives from Corporate were and are fundamentally misaligned.
Type A attestation is, generally, solved. Carrier A attests that the number is one of theirs, and they know that the caller is one of theirs too and attached to their network.
However: this is a fraction of calls. Carriers also sell blocks of phone numbers without the corresponding access network. This is what allows you to pick, say, a Twilio number with a local area code. In these cases the best that can be hoped for is a lesser attestation.
But it gets worse, because the operator can also sell blocks of numbers to people with no direct connection to the US carriers and who need to spoof US numbers. That call from Capital One comes from the Philippines via two or three intermediate operators, none of whom can attest to much of anything. And into that gap ride the spammers.
Furthermore, in an Experian-like twist, some carriers also realized that businesses would pay to have their calls show up as "trusted" on the recipient's phone. So the standards were enhanced to deliver 'rich call data'. However, in order to be something worth paying for you also need a baseline of calls that do not have that premium look. A scam? You decide.
Finally, one other misaligned incentive. All of this needs VoIP. Not TDM (classic legacy telephony). However, the big US carriers make bank selling TDM circuits to the hundreds of small regional telcos, and refuse to sell them SIP trunks, because it's s such an easy money maker. So again, technology loses to incentives.
These incentives, to make money from phone numbers, vastly outpace what
I'm going to mangle the terms of art here, but the ur-problem is that labels of routing, like phone numbers and email addresses, get confused with labels of identity, and then with indicators of trustworthiness.
Everything is built to address that weakness - think DKIM, SPF, etc, plus STIR/SHAKEN, to say nothing of IP or ASN filtering, but they feel like bandaids on a very difficult problem. What you end up with are basically default-deny except for a personally curated trust set ("only accept calls from my contacts", "everything goes in spam unless I have previously corresponded with the sender"), etc.
One last robocall story. AT&T sat on their hands for years until consumer groups embarrassed the then-CEO enough to do something about it. There was a memorable interview in the Dallas Morning News where they called him on it instead of lobbing him softballs and I suspect that the embarrassment finally got through.
It's worth noting that TFA addresses this in the context of the scam: When the scam depends on the emotional reaction in response to a loved one's distress, it doesn't matter if the number the scam is coming from is unfamiliar. This means that the scam can use "technically correct" numbers that pass SHAKEN/STIR with no loss in conversion.
TFA also mentions that by routing calls through older non-IP networks you lose the accurate information, although it sounds like the FCC is slowly cracking down on this.
So lots of judicially-unreachable call centers under judicially-unreachable telecoms need to lose reputation score and get spam-binned by default, just like email. I thought that was going to happen by now. Did the US telecoms just chicken out?
> It’s only expensive and brittle because environmentalists have choked it to death. They’re the third biggest villains of climate change, after consumers and oil companies.
Do note, though, that it was the unbelievable irresponsibility of past operators that has spurred the anti-nuclear movement in the first place. See e.g. https://youtu.be/929B8sgOOTM?si=FttZr_MsbQ1hB4Nj&t=1664 from 27:44 to 31:35.
And as a third answer: You actually don't need to take the flywheels of the power plants offline, when you take the rest of the power plant down. The locations for them are perfect, as the whole grid has historically been build around them.
But most probably care less about human survival, and more about survival of civilization.
In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?
Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).
To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.
[1] https://www.youtube.com/watch?v=1qIvFpcGkqc
reply