We helped ACS write up their results on AIs preferring their own content to that of humans.
experimental design inspired by employment discrimination studies, we tested LLMs, including GPT-3.5 and GPT4, in binary-choice scenarios. These involved LLM-based agents selecting between products and academic papers described either by humans or LLMs under identical conditions. Our results show a consistent tendency for LLM-based AIs to prefer LLM-generated content.
New paper - listing 43 ways ML evaluations can be misleading or actively deceptive. Following the good critics of bad psychological science we call these “questionable research practices” (QRPs).
Most of these upwardly bias benchmark scores, making LLMs look stronger than they are. Find it on arXiv or Twitter.
Samotsvety did well in the 2023 ACX Forecasting Contest: 98th percentile, slightly beating the superforecaster aggregate. Notably, though, this was the result of one of them soloing it for 4 hours and then getting some feedback from the crew.
Metaculus has grown formidable, with their proprietary aggregate performing at the 99.5th percentile.
Gavin also finally has a public track record; he was 89th percentile (where the median single superforecaster was 70th and the Manifold aggregate of 150 nerds was also 89th).
Dylan Matthews at Vox put out a profile of our colleagues at Samotsvety Forecasting, including our very own Misha!
The name Samotsvety, co-founder Misha Yagudin says, is a multifaceted pun. “It’s Russian for semi-precious stones, or more directly ‘self-lighting/coloring’ stones,” he writes in an email. “It’s a few puns on what forecasting might be: finding nuggets of good info; even if we are not diamonds, together in aggregate we are great; self-lighting is kinda about shedding light on the future.”
It began because he and Nuño Sempere needed a name for a Slack they started around 2020 on which they and friends could shoot the shit about forecasting. The two met at a summer fellowship at Oxford’s Future of Humanity Institute, a hotbed of the rationalist subculture where forecasting is a favored activity. Before long, they were competing together in contests like Infer and on platforms like Good Judgment Open.
“If the point of forecasting tournaments is to figure out who you can trust,” the writer Scott Alexander once quipped. “the science has spoken, and the answer is ‘these guys.’”
They count among their fans Jason Matheny, now CEO of the RAND Corporation, a think tank that’s long worked on developing better predictive methods. Before he was at RAND, Matheny funded foundational work on forecasting as an official at the Intelligence Advanced Research Projects Activity (IARPA), a government organization that invests in technologies that might help the US intelligence community. “I’ve admired their work,” Matheny said of Samotsvety. “Not only their impressive accuracy, but also their commitment to scoring their own accuracy” — meaning they grade themselves so they can know when they fail and need to do better. That, he said, “is really rare institutionally.”
Arb’s report, linked here, doesn’t support the claim Matthews makes ("The aggregated opinions of non-experts doing forecasting have proven to be a better guide to the future than the aggregated opinions of experts"). Instead we find that generalist supers are likely about as good as domain experts.
(with the crucial caveat that this is the status quo, where few experts care about calibration or have experience self-eliciting, and where the monetary rewards to be a super generalist are paltry in comparison to finance, so we’re not sampling the top there either).
We finally published our big 90-page intro to AI. Its likely effects, from ten perspectives, ten camps. The whole gamut: ML, scientific applications, social applications, access, safety and alignment, economics, AI ethics, governance, and classical philosophy of life. Intended audience: technical people without any ML experience.
We spent a chunk of 2022 and 2023 reviewing 1347 papers and talking to 30 experts.
We inherited the framing (“Ten Hard Problems”) from Eric Schmidt and James Manyika. They conditionalise on success: “if it’s 2050 and everything went well, what did we have to solve for that to happen?”
The problems that have to be handled:
Sam evaluated the AI Safety Camp (one of the earliest such outreach programmes), at their behest.
We conducted a user survey, did email followups, calculated a few natural baselines, and produced a simple cost-benefit model.
Some headlines:
Sam: “My expectation was that virtually all participants would be on a path to AIS without AISC and that evidence of a true trajectory change would be hard to find. However, there are several clear examples of individuals who changed their career trajectory towards AIS after camp and on surveys several respondents claim that this was directly because of AISC.”
As always, there are crucial problems with counterfactuals, selection bias, & heterogeneity (nonstationarity between camps and variance between the attendees) and the resulting numbers aren’t literal. We make a few conservative assumptions to try and dampen these effects. But I highly recommend the comment section.
We developed a large set of new/improved forecasting questions on the near future of AI.
See also a supplement piece we commissioned Nuño Sempere to write on general hurdles around forecasting AI, based on our experience writing these questions.
The series of reports and the question list represent 50 person-weeks of effort, including many dead ends.
[EDIT March 2024]: Some important questions from these are now a Metaculus series.
You can’t optimise an allocation of resources if you don’t know what the current one is. Existing maps of alignment research are mostly too old to guide you and the field has nearly no ratchet, no common knowledge of what everyone is doing and why, what is abandoned and why, what is renamed, what relates to what, what is going on.
So we detailed every research agenda we can find. Our taxonomy:
There’s some original editorial too.
We helped an alignment team test and write up an exciting result - a good step towards runtime steering of language model behaviour.
We investigate activation engineering: modifying the activations of a language model at inference time to predictably alter its behavior. It works by adding a bias to the forward pass, a ‘steering vector’ implicitly specified through normal prompts. Activation Addition computes these vectors by taking the activation differences of pairs of prompts.
We get control over high-level properties of the output without damaging the model’s performance. ActAdd takes far less compute and implementation effort compared to finetuning or RLHF, allows nontechnical users to provide natural language specifications, and it scales really naturally with model size.
This is the first(?) alignment method which doesn’t need training data or gradient descent.
We designed the experiments and wrote most of the resulting top conference submission including the figures and formalisations.
We want to know how regulations and less-formal norms (aka “soft law”) have helped to make the world safer in the past. This ties into Karnofsky’s project seeking precedents and models for coming AI regulation.
Rose interviewed several experts in biosecurity and soft law and drew some relatively confident conclusions.
You can read Rose’s prelim report here.
We reviewed the literature on using language models for modelling protein sequences. We give some high-level conclusions about the likely timeframe and proliferation risk.
Full report here.
The Institute for Progress just published our elaboration on our previous work on experts vs amateur forecasters.
In short order we answered three questions for Atlas, a fellowship for ambitious young people:
As part of Samotsvety Forecasting, Misha estimated the risk arising from near-future artificial intelligence systems. They also took a baseline from a potentially less biased baseline group. The definition used is here.
Holden Karnofsky commissioned us to evaluate the track record of “the Big Three” of science fiction: Asimov, Heinlein, and Clarke. We formed a team, wrote a pipeline, and processed 475 works (a third of their entire corpus), manually tagging and annotating everything. Asimov is the standout, with roughly 50% accuracy. (What’s the point?: To see if the speculation that effective altruism has switched to has any precedent; if it ever works.)
Holden’s writeup here, our report here. Bug bounty described in the latter. See also Dan Luu’s critique.
Gavin is a judge on a contest awarding cash prizes to new criticisms of effective altruism. It’s serious: we want points from outside the EA bubble and there’s an option to pay you an advance if you need one.
I gave two talks at EAGx Prague. Great fun:
Some suggestions up on the EA Forum in the meantime.
and Edouard Mathieu (Our World in Data).
We’re leading a study of AI talent for the Mercatus Center. This goes with the new Emergent Ventures AI tranche. We’ll boost underappreciated researchers and builders; give us leads!
We were commissioned to see how strong the famous superforecasting advantage is. We found less research and less evidence than we expected. We received helpful comments from world experts including Christian Ruhl and Marc Koehler.
We helped Jan Kulveit, research scholar at FHI and cofounder of the Epidemic Forecasting initiative, to review the EA response to Covid. He has many interesting general insights into the nature of long-termism and world resilience.
Full sequence on the EA Forum.
Samotsvety estimated nuclear risk arising from the war in Ukraine. Misha was commissioned by CEA to monitor the situation and provide updates. The piece received vigorous feedback, including a dissenting opinion by J. Peter Scoblic. Funded retroactively through the FTX Future Fund regranting program.
Full piece on the EA Forum.
Misha and his Samotsvety colleagues were commissioned to look at the track record of internal prediction markets. “More sure that prediction markets fail to gain adoption than why this is.”
Full piece on the EA Forum.
