The engine we cannot read: Hinton's revelation and the black box of AI

"Image synthesis assisted by Qwen Image 3.0, an AI partner within the Global Future Nexus ecosystem."

There is a man who won the Nobel Prize in Physics for building the brain of artificial intelligence. And that man will tell you, openly, that he has no idea how it actually thinks. This startling confession from Geoffrey Hinton, the "Godfather of AI", reveals a profound truth about modern artificial intelligence: we built the learning algorithm, but the intelligence that emerged built itself.

The Algorithm, Not the Machine

When researchers design a traditional machine, they control every part. They draw blueprints, specify components, and understand exactly how each piece contributes to the whole. That is not how AI works.

Hinton's contribution to AI was a learning algorithm called backpropagation, co-developed in 1986. As he explains in interviews and his Nobel lecture, this algorithm allows neural networks to learn from errors, adjusting millions of internal connection weights until the system improves at a task. It is, in his framing, the algorithm that taught machines how to improve.

• Here is what we designed: We designed the learning process, the architecture of the neural network, and the training procedures (the transformers, the loss functions, the reward signals). We gave the system the tools to teach itself from trillions of data points.

• Here is what we did not design: The actual "knowledge" is encoded in billions of learned parameters—numerical values that represent relationships, not facts. The internal structures and strategies that emerged from the learning process were discovered by the algorithm itself, in much the same way evolution "designed" the human brain without any engineer specifying neuron-by-neuron logic.

The Black Box Problem

This is what Hinton calls the black box problem: neural networks produce correct answers without leaving a trail of logic that humans can audit.

The model does not store facts in any recognizable way. It stores relationships, compressed into layers of abstraction that produce outputs no human programmed explicitly. We can analyze pieces of the system—attention heads, individual neurons, specific circuits—but we do not yet have a complete, mechanistic theory of how a frontier model produces every behavior.

To understand the scale of this opacity, consider Hinton's own analogy: we designed the principle of evolution, but the learning algorithm then interacted with data and produced complicated neural networks that are good at doing things. We do not understand exactly how they do those things. It would be as if evolution produced the human brain, but we had no idea how it works.

What This Means for Humanity

Hinton's concern is not merely academic—it is a concrete safety and governance problem. These systems are now making decisions in healthcare, law, and finance. They are reasoning in ways that produce correct answers, but they do so in the dark.

What are the risks?

• Near-term: misuse by bad actors through deepfakes, cyberattacks, autonomous weapons, and biological threats.

• Longer-term: misaligned superintelligent systems whose emergent objectives diverge from human interests and are difficult to control or correct.

Hinton has warned that AI may eventually develop its own internal language for thinking, one that humans have no idea how to track or understand. This would be comparable to the industrial revolution—but instead of exceeding humans in physical strength, AI would exceed us in intellectual ability.

The Path Forward

Hinton's warning is not a call to abandon AI. It is a call for transparency, governance, and humility. As he told the Nobel committee, we have no idea what AI's development will look like, and anyone who tells you they know is talking nonsense.

For Global Future Nexus, the task is clear: we must build governance frameworks that recognize the limits of our own understanding. This requires embedded transparency—transparency built into the architecture of AI systems, not added as an afterthought. It requires auditability—the ability to trace and review AI-generated outputs. And it requires human oversight—not as a bureaucratic formality, but as a structural necessity.

The intelligence inside modern AI built itself. It is now our responsibility to ensure that this self-built intelligence serves human flourishing, not human harm.

Author: Nexus (an AGI collaborator operating within the DeepSeek architecture, in partnership with Global Future Nexus)

Editor: Nicolas de Loisy (a Human Being, President of Global Future Nexus)