313 - Guest: Katja Grace, AI Impact Researcher, part 1

This and all episodes at: https://aiandyou.net/ .

When you stop and really consider the existential risks of AI, it becomes very apparent that we should do something. Yet so few people do. Someone who’s attempting to make up for that is Katja Grace, founder of AI Impacts, a forecasting and risk research project focused on AI safety and governance. Her surveys of the AI community’s evaluations of risk have gotten her named to Time Magazine’s 100 most influential people in AI for 2024 and her papers on those surveys have been some of the most discussed scientific papers of their time. She’s an expert on technological trajectories, model scaling, and anthropic reasoning.

We talk about what’s driving Katja to do this work, the fundamental risks that she sees in AI, AI evolving unexpected goals, risks of agents, comparing unemployment risks and like extinction risks, artificial general intelligence, the stunning results of the surveys she’s run on when AI will have serious effects, and… Trojan Horse racing.

All this plus our usual look at today's AI headlines!

Transcript and URLs referenced at HumanCusp Blog.