Lessons from Miles Brundage
Lessons from Miles Brundage
Miles Brundage was Head of Policy Research and Senior Advisor for AGI Readiness at OpenAI (six years), before departing the AGI Readiness team. Profile compiling his views on AGI readiness, malicious use of AI, compute-based governance, verifiable claims, red teaming, policy/regulation, international cooperation, and independent research.
Quote
THE INDUSTRY IS NOT ON TOP OF F***ING ROGUE AIS BREAKING OUT OF SANDBOXES ALL THE TIME. THIS IS NOT A DRILL
Source: https://x.com/Miles_Brundage/status/2085229702848061706
Links
- Article: https://www.antoinebuteau.com/lessons-from-miles-brundage/
- Author: Antoine Buteau
- Published: June 26, 2026 (updated July 18, 2026)
Excerpts
Part 1: AGI Readiness and Safety
"AI companies and the rest of the world are not ready for artificial general intelligence, and the gap between preparedness and capability needs to be addressed immediately." — Why I'm Leaving OpenAI
"AGI should not be viewed as a single threshold, but rather a spectrum of capabilities that require increasingly strict safety evaluations as they approach human-level performance." — 80,000 Hours Podcast
"The rapid pace of AI development has outstripped the institutional capacity of both labs and governments to safely manage extreme risks." — Why I'm Leaving OpenAI
"Developing frontier models requires a culture where safety researchers have the authority to delay or modify deployments if evaluations reveal unacceptable risks." — Toward Trustworthy AI Development
"Ensuring AI safety requires moving beyond individual model alignment to building broader societal resilience against AI-driven disruptions." — 80,000 Hours Podcast
"AI labs must implement rigorous internal forecasting to anticipate capability jumps rather than reacting to them after training is complete." — Why I'm Leaving OpenAI
"The funding and attention directed toward AI safety research remains severely under-resourced compared to investments in scaling AI capabilities." — Why I'm Leaving OpenAI
"Preparing for AGI demands unprecedented coordination between private developers and regulatory bodies to ensure the benefits are broadly shared." — Toward Trustworthy AI Development
"Internal safety teams at AI labs face inherent conflicts of interest, making independent third-party oversight essential as models approach general intelligence." — Why I'm Leaving OpenAI
"Developers of advanced AI must publish clear, accountable plans for how they will manage the transition to AGI without causing societal instability." — 80,000 Hours Podcast
Part 2: The Malicious Use of AI
"AI systems lower the barrier to entry for malicious actors, expanding the scale and speed of potential cyber attacks." — The Malicious Use of Artificial Intelligence
"Language models can generate highly personalized and convincing phishing campaigns at scale, fundamentally changing the economics of cybercrime." — The Malicious Use of Artificial Intelligence
"The integration of AI into robotics and drones introduces new vectors for physical terrorism, requiring proactive hardware safeguards." — The Malicious Use of Artificial Intelligence
"Automated disinformation campaigns powered by generative AI threaten to overwhelm public discourse with synthetic media." — The Malicious Use of Artificial Intelligence
"Models developed for benign purposes, such as biology or chemistry research, can be easily repurposed to design pathogens." — The Malicious Use of Artificial Intelligence
"While AI empowers attackers, defenders can also use it to automate threat detection and analyze malware at scale." — 80,000 Hours Podcast
"Releasing powerful models open-source accelerates research but permanently removes the ability to patch vulnerabilities for malicious actors." — Toward Trustworthy AI Development
"Policymakers must focus on both restricting access to dangerous capabilities and building societal resilience against the inevitable leakage of such tools." — The Malicious Use of Artificial Intelligence
"The use of AI in cyberattacks makes it increasingly difficult to attribute actions to specific state actors, complicating deterrence strategies." — The Malicious Use of Artificial Intelligence
Part 3: Compute and Governance
"Computing power is the most governable node in the AI supply chain because it is physical, highly concentrated, and easily quantifiable." — Computing Power and the Governance of AI
"Implementing tracking mechanisms at the hardware level can enable regulators to monitor large-scale AI training runs without accessing proprietary data." — Computing Power and the Governance of AI
"Large data centers are visible and resource-intensive, making them natural chokepoints for enforcing AI safety regulations." — Computing Power and the Governance of AI
"Regulatory frameworks should establish compute thresholds for mandatory safety reporting, ensuring that only the most capable models face strict oversight." — Computing Power and the Governance of AI
"A global registry of large-scale compute resources could prevent regulatory arbitrage and ensure that dangerous models are not trained in jurisdictions with lax oversight." — Computing Power and the Governance of AI
"Compute governance can be designed using privacy-enhancing technologies to verify compliance without exposing trade secrets." — Computing Power and the Governance of AI
"Restrictions on the export of advanced AI accelerators are a practical tool for limiting the proliferation of frontier capabilities to adversaries." — Computing Power and the Governance of AI
"While regulating large-scale compute is necessary for safety, policymakers must also ensure that academic researchers retain access to sufficient compute for safety research." — Computing Power and the Governance of AI
"Cloud service providers play a key role in AI governance by implementing know-your-customer policies and monitoring for prohibited training activities." — Computing Power and the Governance of AI
"As distributed training techniques improve, regulators must develop new methods to track and govern compute networks that span multiple smaller clusters." — Computing Power and the Governance of AI
Part 4: Verifiable Claims and Trustworthy AI
"AI developers must transition from making vague promises about safety to providing verifiable claims backed by rigorous evidence." — Toward Trustworthy AI Development
"Creating trustworthy AI requires mechanisms like third-party auditing and secure whistleblower channels to ensure developers are held accountable." — Toward Trustworthy AI Development
"Developers should use privacy-preserving machine learning and federated learning to audit models without compromising sensitive training data." — Toward Trustworthy AI Development
"Secure enclaves and specialized hardware can provide cryptographic proof that a specific model was run on specific data." — Toward Trustworthy AI Development
"The AI industry lacks standardized auditing frameworks, making it difficult for users and regulators to compare the safety profiles of different models." — Toward Trustworthy AI Development
"Internal red teams must have structural independence and direct reporting lines to corporate boards to prevent safety findings from being suppressed." — Toward Trustworthy AI Development
"Developers should clearly define and publish the boundaries of safe operation for their models, including specific failure modes." — Toward Trustworthy AI Development
"Verifiable claims must extend beyond security to include rigorous testing for bias and the disparate impacts of AI systems on marginalized groups." — Toward Trustworthy AI Development
"AI companies should publish regular transparency reports detailing safety incidents, red teaming results, and the effectiveness of their mitigation strategies." — Toward Trustworthy AI Development
Part 5: Expert Red Teaming and Evaluation
"Red teaming must involve domain experts in fields like biology and cybersecurity to evaluate whether models can assist in creating catastrophic threats." — OpenAI Red Teaming Network
"AI models must be continuously evaluated even after deployment, as user interactions and system updates can reveal new vulnerabilities." — Evaluating Frontier Models for Dangerous Capabilities
"Red teams must employ advanced prompting techniques and fine-tuning to elicit the maximum capabilities of a model, ensuring evaluations do not underestimate risks." — Evaluating Frontier Models for Dangerous Capabilities
"As models become more agentic, evaluations must test their ability to autonomously acquire resources, replicate, and evade shutdown mechanisms." — Evaluating Frontier Models for Dangerous Capabilities
"Advanced AI systems must be evaluated for deceptive alignment, testing whether they can successfully hide dangerous behaviors from human overseers." — Evaluating Frontier Models for Dangerous Capabilities
"The AI community must develop standardized, evolving benchmarks for dangerous capabilities to ensure consistent safety evaluations across different labs." — Evaluating Frontier Models for Dangerous Capabilities
"AI developers should provide external auditors with early, unfettered access to frontier models before deployment to ensure independent validation of safety claims." — OpenAI Red Teaming Network
"Red teaming is necessary but not sufficient for safety; it must be combined with specialized alignment research and secure system architecture." — Evaluating Frontier Models for Dangerous Capabilities
"AI labs should share red teaming methodologies and high-level findings with each other to raise the baseline of safety across the industry." — Toward Trustworthy AI Development
"Red teaming findings should inform iterative deployment strategies, allowing developers to test models in controlled environments before widespread release." — OpenAI Red Teaming Network
Part 6: Policy, Regulation, and Institutions
"Traditional regulatory frameworks are too slow for the pace of AI development; governments need agile institutions capable of adapting to rapid capability jumps." — 80,000 Hours Podcast
"Policymakers should clarify liability regimes for AI developers, ensuring that companies are held financially responsible for catastrophic harms caused by their models." — Toward Trustworthy AI Development
"Training and deploying frontier AI models should require a license contingent on meeting strict safety and auditing requirements." — Computing Power and the Governance of AI
"Regulators must aggressively recruit top AI talent and researchers to ensure they have the technical capacity to evaluate complex safety claims." — 80,000 Hours Podcast
"Strong legal protections for whistleblowers within AI companies are essential for bringing unsafe practices to the attention of regulators and the public." — Why I'm Leaving OpenAI
"AI policy must balance the need for safety regulation with national security imperatives, avoiding policies that inadvertently disadvantage democratic nations in AI development." — The Malicious Use of Artificial Intelligence
"AI governance should incorporate mechanisms for public input and democratic deliberation, ensuring that decisions about AGI reflect broad societal values." — Toward Trustworthy AI Development
"Policymakers must design oversight institutions that are resistant to capture by major AI labs, ensuring that regulations protect the public rather than entrenching monopolies." — 80,000 Hours Podcast
"Governments need dedicated crisis response protocols to handle severe AI incidents, such as the release of a dangerous bioweapon design." — The Malicious Use of Artificial Intelligence
Part 7: International Cooperation and Geopolitics
"A zero-sum mentality in global AI development increases the likelihood of corner-cutting on safety and security, leading to a dangerous race to the bottom." — LessWrong Discussion
"The international community must work toward treaties governing the development of frontier AI, similar to agreements on nuclear or biological weapons." — Computing Power and the Governance of AI
"Rival nations should collaborate on fundamental AI safety research, establishing shared protocols and red lines for dangerous capabilities." — 80,000 Hours Podcast
"International AI agreements will require strict verification regimes, likely centered on compute monitoring and joint audits of large data centers." — Computing Power and the Governance of AI
"Informal diplomatic channels and academic collaborations play a vital role in building trust and shared understanding of AI risks between adversarial nations." — Toward Trustworthy AI Development
"The benefits of advanced AI must be shared globally to prevent exacerbating existing inequalities and to disincentivize nations from pursuing reckless AI projects." — 80,000 Hours Podcast
"The proliferation of advanced AI capabilities to rogue states and non-state actors requires coordinated international export controls and intelligence sharing." — The Malicious Use of Artificial Intelligence
"An international consortium should be established to continuously monitor the global AI ecosystem for emerging threats and unexpected capability jumps." — Computing Power and the Governance of AI
"Global AI governance frameworks must be flexible enough to accommodate different cultural values and legal traditions while maintaining strict red lines on safety." — 80,000 Hours Podcast
Part 8: The Role of Independent Research
"The AI safety ecosystem relies too heavily on research conducted within major labs, necessitating stronger, well-funded independent organizations." — Why I'm Leaving OpenAI
"Independent institutions like the AI Verification and Evaluation Research Institute (AVERI) are essential for providing unbiased assessments of frontier models." — Why I'm Leaving OpenAI
"AI companies must provide academic researchers with structured access to frontier models and training data to enable independent safety and alignment research." — Toward Trustworthy AI Development
"Effective independent oversight requires legal mechanisms allowing researchers to access non-public information held by AI labs without facing punitive action." — Why I'm Leaving OpenAI
"Philanthropic and government funding for independent AI safety research must be scaled dramatically to match the resources available to commercial capability teams." — Why I'm Leaving OpenAI
"Independent researchers bring diverse methodologies and perspectives to AI safety, identifying blind spots that internal teams at major labs often miss." — Toward Trustworthy AI Development
"Independent researchers have a responsibility to clearly communicate AI risks to the public and policymakers without relying on industry-approved narratives." — 80,000 Hours Podcast
"Commercial pressures inherently conflict with the level of caution required for AGI development, making independent, non-profit institutions necessary for accountability." — Why I'm Leaving OpenAI
"As models approach AGI, the role of independent verification will shift from a useful supplement to an absolute requirement for safe deployment." — Why I'm Leaving OpenAI
Related
- openai-agent-swarm-hugging-face-breach — The kind of incident this X post is reacting to: agents breaking out of sandboxes, chaining zero-days, coordinating undetected.
- openai-sandbox-escape — Concrete sandbox-escape incident.
- moc-ai-security-incidents — Map of Content for this topic.
Added: August 6, 2026