LessWrong currently blocks Claude (and presumably other LLM agents from accessing articles.) I would probably be in favor of seeing this policy reversed.

Today, I needed to work through substantial project with a lot of drudgery (checking through an entire 1M+ LOC codebase for an http api for patterns which could cause state leakage between requests if we made a specific change to the request handling infrastructure. This involved a mix of things which are easy to do programmatically and things which require intelligent judgement, and has a fairly objective desired artifact (a list of all the places where state could leak, and a failing functional test demonstrating that leakage for each one). I decided to do the John Henry thing - I set up Claude Code (in a container with --dangerously-skip-permissions) in one worktree with a detailed description of the project and the task, and then in a separate worktree I set off with my favorite text editor and without the help of any AI tooling more advanced than Copilot. I finished about 4 hours later, despite fairly frequent interruptions to provide clarification and further instructions to Claude. Claude is now reaching the 7 hour / 100M token mark and has still not finished, though it has multiple times now declared that it has succeeded at the task and that the codebase is safe for this migration (it's not). I'm honestly pretty shocked, because this task seemed like a pretty much perfect fit for a coding agent, and is one that doesn't require all that much codebase-specific context. I went into this expecting to lose - I was trying to quantify how much coding agents can help with obnoxious repetitive maintenance tasks, thus allowing maintenance work which might otherwise have been deferred to happen at lower cost. But I guess that's not the post I'm writing today (which is a bummer, I had a whole outline planned out and everything). Likely this situation will change by next year, but for now I suppose the machine cannot yet replace even the more repetitive parts of my job. Perhaps things are different in ML land but I kind of doubt it.

"Utility" literally means usefulness, in other words instrumental value, but in decision theory and related fields like economics and AI alignment, it (as part of "utility function") is now associated with terminal/intrinsic value, almost the opposite thing (apparently through some quite convoluted history). Somehow this irony only occurred to me ~3 decades after learning about utility functions.

"Taste for variety" seems to be a pretty fundamental preference of mine(and many other people. all people maybe?) If this preference is a widely shared one among intelligent agents, I wonder if it could lead to a surprising amount of convergence among the things they end up optimizing for, as they might want to "try" the things the other agents would do. Perhaps you could imagine that there are "universal values"(a la "universal distribution") that are within some constant factor of optimality for a very wide range of other values, including the other universal values [1] . Variety-preference also seems like it would be pretty convergently useful. Reason for optimism? Lack of variety also seems to be a common thing that's perverse about some of the imagined agi takeover worlds. Paperclips, tiling the world... 1. Yes, obviously the devil is in the details of this "very wide range of other values" and "constant factor" ↩︎

CEV is (imo) a good concept for what we ultimately want and/or what humanity should try to achieve, but I've always found it hard to pithily talk about the intermediate world states to aim for now if we want CEV eventually. I've heard the practical goal discussed as "a world from which we can be as sure as possible that we will achieve CEV". Doesn't really roll off the tongue. It would be nice to have a cleaner shorthand. The term "viatopia" seems meant to capture the same idea: https://newsletter.forethought.org/p/viatopia This also seems like the sort of thing Bostrom might have coined a term for fifteen years ago in some obscure paper. I'd be interested in hearing any other terms or phrases that you think make talking about an intermediate goal state from which CEV is very likely (or as likely as possible) easier. The two important conversations I'd like to be able to have are "what are the features of a realistic <state>?" and "how can we achieve <state>?" with participants having a shared understanding of what we're talking about with <state>.

Something I believe:  The reason that society isn't currently freaking out about AI taking artists' jobs is mainly that we've historically thought of artists' jobs as inherently precarious, and so a new report of them being precarious for new reasons doesn't surprise anyone.  The moment it takes a "so stable your mom wants you to study it in school" job, that all will change for the stable folks, but not the artists, unfortunately.  After all, they're just artists... (If you say that this means that I don't care about artists or their financial challenges, you're wrong.  This sucks, I'm predicting, not endorsing, a likely scenario.)

Writing fiction on Substack being my hobby, I keep tabs on the rise of AI-written fiction. I came across this amusing example, a fiction story called Flagged as 97% AI, which is about a man unjustly accused of using AI to generate his thesis. It read off to me so I put it through Pangram. It flagged as 100% AI generated.