Sorry, when you say
Is stock ownership a good schelling point for the god designers? Maybe more than I was imagining.
I genuinely can't tell if you're being sarcastic. I agree the type of ownership discussed here is different from current legal property rights in many ways, but both your and simon's posts seem skeptical of owning galaxies using the common sense definition, not just the legal one. I agree all the us laws about property will be torn apart when ASI is invented. If you're not being sarcastic then I suspect we don't disagree that much.
I was under the impression you considered it more plausible than you do and I was using a rhetorical tool where I slightly cede ground before arguing against what I ceded. Probably unvirtuous on reflection. Still, though I don't think it's a likely they would honor stock holders, it is a possible. It does seem very unnatural as conspirators would likely have very different stock allocations and the biggest beneficiaries may well be those outside the conspiracy, a strange sort of altruism.
Anyway, my tweet was not worded particularly well.
Is stock ownership a good Schelling point for the god designers? Maybe more than I was imagining. You're assuming very fine-grained alignment ability and Anthropic/OAI employees willing to consciously defect against humanity. Why not defect against stockholders, too? It seems an odd Schelling point. Doing this requires a conspiracy to take over the world. The natural Schelling point is surely an equal share to all conspirators, no? And why would the board be involved? Have they been so far in alignment decisions? It will be technical people who understand how things work doing the actual defection.
And even granting you that, this sort of ownership isn't best characterized in the terms Dwarkesh was characterizing it. It is ownership in a sense, but Dwarkesh was talking about tax policy which is absurd in this scenario. The ownership Dwarkesh and Scott imagine is one embedded in a system of property rights and the existing political economy. The ownership you imagine is one of consumption granted to you by a god summoned correctly, which is less crazy. Though one still has to wonder how god will extrapolate your preferences such that "you" can consume a galaxy. [1]
It all comes down to initial preferences and the competitive equilibrium of the AIs in the end; contextualizing it as tax policy still reads as absurd and not giving the future its due. Notions of identity and polity will predictably break down to the point that the frames Scott and Dwarkesh are using look very unlikely to be fruitful, is my main claim. And if you assume the existing political economy remains in some form (which Scott and Dwarkesh do), you should expect a coup at some point that breaks property rights. They do happen from time to time and history will accelerate. They imagine the dynamics that maintain property rights will remain but those that cause their occasional dissolution will not, even in unprecedented time that resembles more a speciation event than any common historical process, and one in which history is happening thousands of times faster[2] than it ever has before
- ^
Galaxies are far away, so what does it even mean to control one? It means you get to decide what it will spend its computation computing after your nanobots or whatever convert it into something useful, which is a different sort of wealth. I suppose you might send copies of your brain state there. Brain state? What exactly are "you" again? If you can "solve alignment" and point an AI god at your preferences, you might be able to own a galaxy in some sense. However, a lot of optimization pressure will now be centred on "you" and "your preferences" and what comes out of that might not look very much like you expect owning a galaxy to look like
- ^
That is, there will be more history in a given year as the agents involved will be thinking vastly faster
1Sam would have to be 99.99999th percentile megalomaniacal - rather than just the already-priced-in 99.99th - to try this crazy thing that could very likely land him in prison.
Would this land him in prison? Would it even be a crime to add "work towards making Sam Altman god emperor" to the spec?
It's pretty bad.
Writing fiction on Substack being my hobby, I keep tabs on the rise of AI-written fiction. I came across this amusing example, a fiction story called Flagged as 97% AI, which is about a man unjustly accused of using AI to generate his thesis. It read off to me so I put it through Pangram. It flagged as 100% AI generated.
Being neither particularly intelligent, rational, or sociable I likely can't offer the best advice.
However, if the goal is to feel less hate towards people you consider intellectually inferior to yourself, maybe you're going about it backwards. Consider moves in the opposite direction. Find an environment where you are vastly outclassed, such that your contributions feel worthless, your efforts to improve yourself pointless, and you maybe even feel some bitter jealousy in your heart.
1
1We have an intense desire to feel superior. Those blessed with intellect should have some noblesse oblige. To despise those who lack your genetic and mimetic gifts lacks grace. It is their very inferiority that provides you with the pleasure of feeling superior. Schopenhauer decries the Malthusian ocean he floated atop of, the people that let him live his life of the mind. What a dick.
Reflections on Inkhaven
Would I recommend applying next year? Yes.
Would I go again? It was a really good experience. I do wonder if it is the type of thing that would never be quite the same the second time. Despite thinking this, I am tempted.
Did it make me a measurably stronger writer?
I don't think so. I am much prouder of the fiction I wrote before Inkhaven than what I produced during Inkhaven. A counterpoint to this: Ozy, Wales, Linch and Eneasz convinced me I should spend some time editing/revising my fiction - which I haven’t really been doing much. This could very well make me a much stronger fiction writer, but I haven’t really tried it yet. We will see what happens with my next short story.
Am I proud of what I wrote?
I am not sure. My first week, I wrote mostly terrible essays. I then started writing microfiction, and some of it still amuses me. I did get one longer bit of fiction done there, too: Lobsang’s Children. I am pretty happy with it, but it wasn’t a big hit.
These are my favourite of my short posts:
Harry Potter and the Rules of Quidditch
With Adjustable Chef's Tool Drawer
Most valuable thing about Inkhaven?
Inkhaven is nothing without its people.
Which residents impressed me most?
Linch struck me as the most likely to become a professional blogger. Vishal’s blog was great and he is insanely well read. Though I didn’t really talk to him, Screwtape impressed me. He takes teaching/learning rationality seriously. By revealed preference, this isn’t something I care much about. But it is a noble pursuit and I am glad at least one person is still carrying the torch. I had a brief conversation with GeneSmith and was in awe at how many paths to intelligence augmentation he has considered, including really crazy ideas like methods for expanding adult skulls so as to have more room for brain grafts. He struck me as insanely smart.
Biggest regrets?
I was in a bad mood at the start and psyched myself out of enjoying the first week. I started drinking in an attempt to improve my mood and this was a poor decision. I was consequently rather rude to a few people. Interestingly, my tweets were (for whatever reason) much better than usual during this period.
The snack bar was so tempting, I gained about 10 pounds. I just don’t have willpower in regards to snacks. At home, I eat bland food to stay at the high end of “normal” BMI. I am very annoyed I did this to myself and am considering an extreme fast. I would have paid extra to be forbidden snacks, as stupid as this sounds. Perhaps Screwtape is right about this whole rationality thing.
What is Lighthaven like?
An excerpt from a tweet I wrote about this:
I am not the most visual person, but I can relay impressions. I would say it feels a bit Chestertonian. The Rose Garden Inn as patient zero, infecting various houses around it. The Chestertonian bit, in my imagination, is the implicit mad scheme to slowly undermine various stifling norms of city planning. One house at a time, seeds of freedom spread. I imagine this spirit infecting first the neighborhood and then the city of Berkeley itself. At a key moment, the planning commissioner will be turned and the revolution completed.
There is an artificial lawn. And though you'd think artificial grass would be tacky, it does somehow work. Once again, materials scientists have outdone God. There are "onions," strange functional sculptures in which one can sit and have conversations. They look to me like large demonic hands transfigured from the artificial turf by some strange wizards, but in a pleasant way. There is a snack-bar, which is perhaps too generous with its offerings. I would recommend hiring Stephan J. Guyenet as a consultant to ruin everyone's good time.
The aesthetics, then, I will describe as Post-Bureaucratic-Apocalypse chic. Planning laws and an aesthetically dead, inhuman, architectural tradition preventing any new beauty being constructed, those with taste must take the old and put it to new purposes. And this Lighthaven has done.
Favourite Visiting Writers? I am mostly interested in fiction, so talked with Daystar, Ozy and Alexander Wales. They were all great. Ozy's feedback was particularly good. And they're just a very pleasant, happy person. Daystar seemed cool and I should have asked for more feedback from him, though I did send him this very personal, autobiographical work.
How well run was it? It was very well run. I have no complaints. Ben Pace did a great job. I expect next year it will be even better.
4
11
Sorry, I was claiming I was using this rhetorical trick. I saw your "share ownership seems like a fair schelling" comment and felt like I should mention that in my reply. My "maybe more than I was imagining" was just rhetoric on my part.