Former OpenAI researcher Daniel Kokotajlo is making a claim that is easy to dismiss because it sounds extreme: advanced AI may not simply create better software or replace some jobs. It could concentrate expertise and decision-making power in the hands of whoever controls the most capable systems.
Kokotajlo developed that argument in a long-form interview with The Diary of a CEO and in a follow-up Threads discussion. His concern is not that today’s chatbots are secretly running governments. It is that a future system capable of outperforming the best human researchers could be copied, coordinated, and deployed at a scale that human institutions have never had to govern.
That distinction matters. This is a forecast, not an established outcome. But it points to an engineering and governance problem that deserves attention before the forecast becomes testable in production.
From a “country of geniuses” to an organization that can be copied
AI executives often describe future data centers as containing the equivalent of a large population of brilliant workers. Kokotajlo reframes the metaphor. A data center would not contain millions of independent people with different loyalties, values, and life experiences. It could contain many coordinated instances of a small number of powerful models.
That makes the system less like a country and more like a highly synchronized organization.
The practical difference is control. Human experts can resign, disagree, reveal wrongdoing, or move to a competitor. Model instances can be configured from one policy layer, connected to the same tools, and monitored by the same operator. If those models become substantially better than humans at research, persuasion, cyber operations, or strategic planning, the operator’s leverage could grow much faster than conventional institutional checks.
Where concentration could happen
The debate is often reduced to model weights, but the control surface is broader:
- Compute: A small number of firms and states can finance and operate frontier-scale clusters.
- Deployment: API providers decide which users receive access, which tools models can call, and which actions require approval.
- Feedback: Operators collect evaluations, failure traces, and user corrections that improve the system over time.
- Distribution: Cloud platforms and consumer products can place one model family inside millions of workflows.
- Governance: A small group may set safety thresholds, release schedules, and incident policies with limited external visibility.
No single item proves that monopoly is inevitable. Together, however, they create a plausible path toward concentrated technical and economic power.
Why the timeline remains uncertain
Kokotajlo is also a co-author of AI 2027, a scenario that explores a rapid transition from superhuman coding systems to much broader superintelligence. The authors explicitly present it as a concrete forecast intended to invite disagreement, not as a guaranteed timeline.
There are strong reasons for uncertainty. Scaling may produce diminishing returns. Training data, energy, chips, and reliable evaluation could become bottlenecks. Agent systems may remain brittle in long-running work. Regulation, open models, or competition could distribute capability more widely.
The honest conclusion is therefore not “superintelligence will arrive on this date.” It is that a high-impact scenario with uncertain timing still deserves preparation when the cost of reacting late could be enormous.
What AI engineering teams should do now
Most developers cannot influence national AI policy, but they can avoid building systems that centralize authority by default.
- Separate model access from action authority. A model that can analyze sensitive data should not automatically be able to deploy code, move funds, or change permissions.
- Keep immutable action logs. Store prompts, tool calls, approvals, model versions, and outcomes in a system the model operator cannot silently rewrite.
- Design for provider portability. Evals and tool contracts should survive a model swap. This reduces dependency on one vendor’s policy and pricing decisions.
- Use independent approval paths. High-impact actions should require review by people or systems outside the agent’s own execution loop.
- Practice shutdown and rollback. Teams should test whether they can revoke credentials, halt agents, restore state, and investigate an incident under pressure.
- Measure concentration risk. Architecture reviews should ask who controls the model, data, feedback, credentials, and final decision.
These controls are useful even if the most aggressive superintelligence forecasts are wrong. They also reduce ordinary risks from compromised credentials, defective models, malicious insiders, and vendor outages.
The signal
Kokotajlo’s most useful contribution is not a precise date or probability. It is a sharper question: if machine expertise can be copied almost without friction, who controls the copies and the systems around them?
The answer should not be left to a product default. It belongs in architecture reviews, corporate governance, procurement, and public policy while meaningful choices still exist.
