I have never worried about the singularity because I had a single conversation in 2007 with a guy who worked in a chip fab, and learned just the tiniest bit about the physical, mechanical process that goes into actually manufacturing a microprocessor, and how much of it requires direct intervention of human hands despite everyone involved in the process having wanted to automate it all since 1980

@glyph So I appreciate this notion that the paperclip optimizer cannot exist without coopting human fingers into bending paperclips, but OTOH there are a lot of people in this world who will be "reverse centaurs" and bend those fucking paperclips because they need to eat.

I also am not worried about a "singularity", but about something much more stupid.

I have never worried about the singularity because I had a single conversation in 2007 with a guy who worked in a chip fab, and learned just the tiniest bit about the physical, mechanical process that goes into actually manufacturing a microprocessor, and how much of it requires direct intervention of human hands despite everyone involved in the process having wanted to automate it all since 1980

@glyph I have never worried about the #Singularity because in the late nineties, on two separate occasions, I spoke at length with two (then and still today) fairly well-known Ayatollahs of the movement.

They both made certain to mention that they thought I was smart—perhaps even approaching as smart as they thought they were!—and so they were candid and enthusiastic with me. Neither had studied biology (ahem ahem obviously), and both genuinely were eager to share ideas with a PhD student in molecular biology.

When I pointed out that exponential growth always, but always, leads to system collapse in nature; when I pointed out that in a universe finite in space and time, this is the rigidly necessary requirement of arithmetic, they both responded nonverbally: with a dismissive wave of the hand.

In short, the entire movement is based on the belief that Technology is Magic.

At the time I had not reached the level of cynicism to see where it would go in thirty years.

@criffer @jwz yes, this is absolutely my point. The reverse centaurs are pressed into service by other humans, abused by other humans, and so it must be.

One of the other, more “reasonable” TESCREAL sects believes something more like; of course we won’t have a robot king in a robot body on a robot throne, but superintelligences will be so perfectly convincing that they will co-opt humans effortlessly by programming our brains, so it’s the same deal. But this is equally wishful thinking

@jwz @glyph We already have quasi-autonomous entities that while not being sentient act as if they had intelligence in the single-minded furtherance of maximising an output, to the detriment of all else.

They’re called corporations.

@criffer @jwz The reality of that outcome is a billionaire looking into their magic mirror, saying “mirror mirror on the wall, should I oppress and murder the poor” the mirror says “yes absolutely boss. You are so smart” and then they collapse onto a nearby fainting couch and say “oh no! Woe is me! My consciousness has been commandeered by a malevolent overmind!” before getting up and saying “oh well, nothing for it” before pulling the lever on their orphan-crushing machine

@criffer @jwz yes, this is absolutely my point. The reverse centaurs are pressed into service by other humans, abused by other humans, and so it must be.

One of the other, more “reasonable” TESCREAL sects believes something more like; of course we won’t have a robot king in a robot body on a robot throne, but superintelligences will be so perfectly convincing that they will co-opt humans effortlessly by programming our brains, so it’s the same deal. But this is equally wishful thinking

@glyph @criffer @jwz I am so here for this bullshit! SF thrives on dystopian paranoia and I am paranoia-ing the fuck out of their dystopia in the two novels I'm currently working on (when not recovering from eye surgery)

@cstross @criffer @jwz it’s a positive feedback loop! Assuming your two upcoming novels are “don’t create the orphan-crushing machine” and “okay you created it but *at least* don’t pull the lever on the orphan-crushing machine” we already know what silicon valley elite’s reaction will be

@glyph @criffer @jwz I guarantee you the silicon valley TESCREAL bros will LOATHE these two space operas, but hopefully won't quite be able to work out why. (It's much harder to argue with an idea that makes you unaccountably itchy and irritable than an obvious Ayn Rand style political soapbox lecture—which you can choose to skim past or otherwise ignore.)

@cstross @criffer @jwz I stridently hope so. (Can’t wait to read them myself.)

@glyph @criffer @jwz I believe that's a logic for a lot of people that make decisions that have huge swathing negative impacts on many people are of the type who go " I am clearly not a horrible person. Therefore, my actions clearly must not be having these horrific impacts on people because only a horrible person would do that."

@jameshowell @glyph

1997. I-Con SF convention on the SUNY Stony Brook campus.

We put Vernor Vinge and grey eminence of SF Frederik Pohl on a panel about the Singularity.

Before it started, Pohl asked "What is hell is the Singularity?"

I filled him in.

Right there, in front of Vinge:

"What a load of crap. Here's what's going to happen. We're going to burn through our resources, ruin the environment, civilization will collapse, and the survivors will despise us."

@glyph I have never worried about the #Singularity because in the late nineties, on two separate occasions, I spoke at length with two (then and still today) fairly well-known Ayatollahs of the movement.

They both made certain to mention that they thought I was smart—perhaps even approaching as smart as they thought they were!—and so they were candid and enthusiastic with me. Neither had studied biology (ahem ahem obviously), and both genuinely were eager to share ideas with a PhD student in molecular biology.

When I pointed out that exponential growth always, but always, leads to system collapse in nature; when I pointed out that in a universe finite in space and time, this is the rigidly necessary requirement of arithmetic, they both responded nonverbally: with a dismissive wave of the hand.

In short, the entire movement is based on the belief that Technology is Magic.

At the time I had not reached the level of cynicism to see where it would go in thirty years.