RL capability gains might mostly come from better self-elicitation. Ran across a paper NUDGING: Inference-time Alignment of LLMs via Guided Decoding. The authors took a base model and a post-trained model. They had the base model try to answer benchmark questions, found the positions where the base model was least certain, and replaced specifically those tokens with tokens from the post-trained model. The base model, so steered, performed surprisingly well on benchmarks. Surprisingly (to me at least), the tokens changed tended to be transitional phrases rather than the meat of the specific problems. Example from the paper: This worked even when the post-trained model was significantly smaller than the base model: on gsm8k, llama-2-7b-chat "nudging" llama-2-70b (base) scored 46.2 on gsm8k, while 7b-chat alone scored 25.5. 70b-chat barely scored better, at 48.5. Surprisingly, I haven't seen much discussion of this paper on here. It seems very relevant to the question of whether RL bakes new behaviors into models or makes them better at eliciting behaviors they already know how to execute in appropriate situations. I am tempted to do a longer writeup and attempt to reproduce/extend the paper, if there's interest.

running the agi survey really reminded me just how brutal statistical significance is, and how unreliable anecdotes are. even setting aside sampling bias of anecdotes, the sheer sample size you need to answer a question like "do more people this year know what agi is than last year" is kind of depressing - you need like 400 samples for each year just to be 80% sure you'd notice a 10 percentage point increase even if it did exist, and even if there was no real effect you'd still think there was one 5% of the time. this makes me a lot more bearish on vibes in general.

[tag: fiction] A novice needed a map of the northern mountain passes. He approached the temple cartographer. "Draw me the northern passes," he said, "showing the paths, the fords, and the shelters." The cartographer studied many sources and produced a map. The novice examined it carefully: the mountains were drawn, the paths clearly traced, fords and shelters marked in their proper notation. The distances seemed reasonable. The penmanship was excellent. "This is good work," said the novice, and he led a merchant caravan into the mountains. On the third night, they reached the place where the map showed shelter. There was only bare scree. On the fourth day, searching for the ford, they found only rapids. They perished in the cold. ---------------------------------------- A master needed a map of the same passes. She approached the same cartographer. "Draw me the northern passes," she said, "showing the paths, the fords, and the shelters." The cartographer studied many sources and produced a map. The master examined it carefully: the goat-track above Willow Creek was present, the stone shelter marked two li below the second ridge where she remembered it, the summer ford placed where three white rocks breached the current. "This is good work," said the master, and she led a merchant caravan into the mountains. On the third night, they rested beneath the stone shelter. On the fourth day, they crossed at the white rocks.

One theme I've been thinking about recently is how bids for connection and understanding are often read as criticism. For example: Person A shares a new idea, feeling excited and hoping to connect with Person B over something they've worked hard on and hold dear. Person B asks a question about a perceived inconsistency in the idea, feeling excited and hoping for an answer which helps them better understand the idea (and Person B). Person A feels hurt and unfairly rejected by Person B. Specifically, Person A feels like Person B isn't willing to give their sincere idea (and effort to connect) a chance, so shuts down and labels Person B as an idea-hater. Person B feels hurt and unfairly rejected by Person A. Specifically, Person B feels like Person A isn't willing to give their sincere question (and effort to connect) a chance, so shuts down and labels Person A as a question-hater. This seems like a huge source of human suffering, and I have been Person A and Person B in different interactions. Does anyone else resonate with this? Do you see things differently?

Superstable proteins: A team from Nanjing University just created a protein that's 5x stronger against unfolding than normal proteins and can withstand temperatures of 150C. The upshot from some analysis on X seems to be: * They used 3 different AI tools (RFdiffusion, ProteinMPNN, ESMFold/AlphaFold2), plus molecular dynamics software * The protein uses standard amino acids (edit: and was made in a standard bacterial cell) * The strength comes from increasing the number of backbone hydrogen bonds from 4 to 33. So why is this relevant? It's basically the first step towards nanotech. Because standard proteins aren't strong enough to manipulate molecular fragments, Drexler's original guess at the path to nanotech was: regular proteins assemble crosslinked proteins, which assemble hybrid nanotech, which assemble full nanotech. At each stage the energy scale increases, and the systems become increasingly capable. It's plausible to me that within 15-30 years, enzymes like these superstable proteins (but much more advanced) will build stronger proteins which build hybrid systems which build molecular assemblers, until we have real-life Poke Balls that can print animals in 15 seconds.

Notes to self about the structure of the problem, probably not interesting to others: This is heavily drawn from MIRIs work and Joe Carlsmith's work So, there are two kinds of value structure: (1) Long-term goals, and (2) immediate goals & deontological constraints. The line between them isn't sharp but that's OK. If we imagine an agent that only has long-term goals, well, that thing is going to be a ruthless consequentialisty optimizer thingy and when it gets smart and powerful enough it'll totally take over the world if it can, unless the maxima of its long-term goals are very similar to what would have happened by default if it didn't take over the world, which they won't be. Fragility of Value thesis and Orthogonality thesis both hold, for this type of agent. We are not on track to get the long-term goals of our AIs sufficiently close to correct for it to be safe for us to build this type of agent. I claim. However, we can instead try to build agents that also have immediate goals / deontological constraints, and we can try to shape those goals/constraints to make the resulting agent corrigible or otherwise steerable and safe. E.g. it's vision for a future utopia would actually be quite bad from our perspective because there's some important value it lacks (such as diversity, or consent, or whatever) and we haven't noticed this yet, but that's OK, because it's honest with us and obeys the rules and so it wouldn't take over and instead would politely explain this divergence in values to us when we ask. So far so good. How are we doing at getting those constraints to stick?  Empirically not so great, but not completely hopeless either. But I want to talk about the theoretical side. The main reason to be concerned, on priors, is the nearest unblocked neighbor problem. Basically, if a weak agent is trying to control a strong agent by imposing rules on what that strong agent can do -- even if the strong agent is bound to follow the rules -- then the situation

Do we have some page containing resources for rationalist parents, or generally for parents of smart children? Such as recommended books, toys, learning apps, etc. I found tag https://www.lesswrong.com/w/parenting but I was hoping for some kind of best textbooks / recommendations / reference works but for parents/children.