With the rapid advancement of artificial intelligence, the conversation has shifted from what AI can do to what it should do. At the center of this dialogue is the critical need for governance and safety. To explore this, we’re speaking with Dominic Jainy, an IT professional with deep expertise in AI and machine learning. We’ll delve into OpenAI’s new “Head of Preparedness” role, examining the emerging risks that prompted its creation, from cybersecurity threats to societal influence. Our discussion will cover how to model these complex threats, the delicate balance between innovation and policy, and the unique qualifications needed to lead in this high-stakes environment.
Sam Altman stated that existing safety evaluations are “no longer enough” for upcoming AI. Can you walk us through what specific new risks, such as influencing human behavior, prompted this change and detail how this new Head of Preparedness will address them differently than before?
That statement really captures the heart of the issue. For years, safety evaluations were about capability—can the AI perform a task correctly and without obvious errors? Now, we’re in a completely different world. The new risks aren’t just about system malfunctions; they’re about intentional misuse of highly capable systems. When an AI can understand human psychology and craft persuasive arguments at a massive scale, the potential for influencing behavior in elections or markets becomes a national security-level concern. Similarly, an AI that can reason about complex systems could find novel cybersecurity exploits that no human has thought of. The Head of Preparedness role is a fundamental shift from a reactive to a proactive stance. Instead of just patching vulnerabilities after they’re found, this person is tasked with building a forward-looking framework to anticipate and model these threats before the technology is even released.
The new role involves building threat models and designing safeguards that scale. Using a high-risk area like cybersecurity, could you provide a step-by-step example of how you would model a potential threat and then develop a scalable safeguard to mitigate that specific risk?
Of course. Let’s take the high-risk area of cybersecurity. First, you’d model the threat: imagine a future AI that’s brilliant at writing code. A bad actor could use it to create a new kind of polymorphic malware, a virus that constantly rewrites its own code to evade detection. The threat model would map out how the AI could be prompted to do this, how quickly the malware could spread, and the potential impact on critical infrastructure. Then, you design a scalable safeguard. This isn’t just a simple content filter. It would be a sophisticated, multi-layered system built directly into the AI’s architecture. This safeguard would involve a “constitutional” principle—a core rule preventing the AI from generating code with malicious characteristics. It would also include real-time monitoring of user requests for patterns indicative of misuse, creating a dynamic defense that learns and adapts as attackers devise new strategies.
This position sits at the intersection of research, engineering, and policy. Could you share a hypothetical scenario where the Head of Preparedness might need to balance a research breakthrough against potential policy risks, and outline the decision-making process they would follow for its release?
This is where the job gets incredibly challenging. Imagine a research team develops a new model that can perfectly simulate complex biological processes—a massive breakthrough for developing cures for diseases. However, the same technology could be used by a rogue actor to model a deadly pathogen. The Head of Preparedness would have to step in long before a public release. Their process would involve convening an internal council of researchers, ethicists, and engineers to conduct intensive “red teaming,” actively trying to misuse the model to understand its full risk profile. They would then work with the engineering team to build robust safeguards, perhaps limiting the model’s ability to work with specific dangerous biological data. Finally, they’d engage with policymakers and external experts to decide on a release strategy. The outcome might be a limited release to a consortium of vetted medical research institutions under strict usage agreements, balancing the immense potential for good against the catastrophic potential for harm.
The article highlights that the role involves making difficult decisions under uncertainty and navigating “real edge cases.” Beyond technical knowledge, can you describe a past experience or a key metric you would use to identify a candidate who truly thrives and makes sound judgments under such pressure?
Technical expertise is just the ticket to entry for a role like this. The real differentiator is a demonstrated ability to think with clarity and principle when all the variables aren’t known. I wouldn’t look for someone who claims to have all the answers. Instead, I’d want someone with a background in a field accustomed to high-stakes uncertainty, like national security or emergency medicine. I would present them with a “real edge case”—a scenario where any decision carries significant risk. The key metric wouldn’t be what they decide, but how they reach that decision. I’d want to hear them articulate their decision-making framework, how they weigh competing values, and how they plan for contingencies if their initial judgment is wrong. A candidate who can remain steady and process-oriented under that kind of pressure is the one who can be trusted to navigate this landscape.
What is your forecast for the evolution of AI safety and governance over the next five years?
My forecast is one of formalization and integration. Right now, this Head of Preparedness role is a novel, high-profile position. In five years, I believe this kind of function will be a standard, non-negotiable part of every major AI lab’s organizational chart. We are moving away from the era of “move fast and break things.” The evolution will be a shift from reactive, post-launch safety patches to a proactive “safety-by-design” philosophy, where governance and risk mitigation are woven into the fabric of the research and development process from day one. I also anticipate much deeper, more structured collaboration between the private sector and government regulators to establish clear, enforceable standards for the safe development of increasingly powerful AI systems.