The ethics of AGI in social credit systems
"Image synthesis assisted by Qwen Image 3.0, an AI partner within the Global Future Nexus ecosystem."
From China's Social Credit System to the EU's categorical ban on social scoring, the use of artificial intelligence to monitor and score human behavior has become one of the most contested frontiers of AGI governance. As AGI systems gain the capacity to evaluate citizens' "trustworthiness" across every domain of life, the question is no longer whether such systems can be built—but whether they can be governed justly.
The Social Scoring Landscape
Social scoring is not a single technology but a spectrum of practices. At one end lies China's Social Credit System—a large-scale government initiative announced in 2014 to rate citizens based on their behavior using digital technologies and AI. At the other lies the routine scoring of European citizens in contexts such as creditworthiness, employee productivity, social fraud detection, and terrorism risk assessment—practices that are already widespread and often invisible.
The distinction between these practices is not always clear. As one analysis notes, the "social" nature of a score is not an inherent characteristic but a matter of context: a score that is lawful in one case can qualify as a social score in another, depending on how it is used. This ambiguity has profound implications for regulation and governance.
Research has already documented the risks of AI-driven social scoring. A 2024 study in npj Digital Medicine found that large language models can be "biased in favor of collective/systemic benefit over the protection of individual rights" and could "facilitate AI-driven social credit systems". The study warned that LLMs trained to allocate resources via unjust criteria—using financial transactions, internet activity, social behaviors, and healthcare information—could entrench systemic discrimination.
The Regulatory Divide: China, Europe, and Beyond
The regulation of social scoring reflects a deep geopolitical and philosophical divide.
China has pursued the most comprehensive approach. The Social Credit System, though far from fully implemented, is designed to evaluate citizens, rate their behavior, and apply rewards and sanctions. Researchers have explored the theoretical integration of Brain-Computer Interface technology with the Social Credit System, proposing "superego-type ethical and moral compasses" embedded in BCI functionalities to customize behavior according to state-determined norms.
Europe has taken a diametrically opposed approach. The EU AI Act explicitly prohibits "social scoring" under Article 5(1)(c), directly inspired by the Chinese model. The ban covers both public and private actors, distinguishing the EU from the Commission's initial proposal, which limited it to public authorities. This prohibition is not just symbolic. While forms of scoring for creditworthiness, insurance pricing, and recidivism risk qualify as "high-risk AI systems"—permitted but subject to requirements—social scoring is categorically banned.
Yet the boundary between banned social scoring and permitted high-risk scoring is blurry. As scholars have argued, the AI Act's ban on social scoring shares features with GDPR Article 22's prohibition of automated decision-making, and may be interpreted broadly to protect individuals against disproportionate uses of AI-based scores. The European Commission has published detailed guidelines on prohibited AI practices, seeking to clarify the scope of the ban.
The United States has taken a sectoral approach, with no comprehensive federal ban but growing scrutiny from regulators and civil society. The NBER has published research examining whether "socially-minded governance" can control AGI, concluding that a socially-minded entity cannot minimize harm from unrestricted AGI products released by for-profit firms—because it lacks both the incentive and the ability to compete.
The AGI Dimension: Amplification and Intensification
The emergence of AGI intensifies the social scoring dilemma in several critical dimensions.
First, capability. AGI systems will not merely classify behavior—they will predict, shape, and potentially control it. A "Justice-First Pluralist Framework" published in 2025 embeds fairness, capability expansion, relational equality, procedural legitimacy, and ecological sustainability as constitutive conditions for governing intelligent systems. Monte Carlo simulations indicate that justice-compatible trajectories are "statistically rare," showing that ethical and sustainable AGI outcomes do not arise spontaneously.
Second, autonomy. Autonomous AI agents transitioning from tools to independent actors require cryptographic identity, ethical governance, and transparent economic frameworks. The Agent Social Contract framework implements a layered architecture: cryptographic identity using digital signatures, an ethical core based on constitutional AI principles, a trust network enabling portable reputation across platforms, and economic enforcement of transparent value distribution.
Third, alignment. AGI ethical alignment requires a fundamental shift from extrinsic regulation to embedded ethics—values integrated into the architecture of intelligence itself. A 6-Layer Adaptive Framework has been proposed, operating as a "Multi-scale Self-Alignment Loop" through which AGI performs self-learning, self-correction, and self-auditing.
The Governance Imperative: A Justice-First Approach
The regulation of AGI in social credit systems cannot be an afterthought—it must be a first-order design constraint. A Justice-First Pluralist Framework treats justice as a feasibility boundary, not a corrective. This means:
Transparency must be architectural. Citizens must be able to understand how they are scored and challenge the outcomes.
Fairness must be structural. AGI systems must account for capability expansion, relational equality, and procedural legitimacy.
Sustainability must be non-negotiable. Social scoring must operate within planetary boundaries.
For Global Future Nexus, the governance of AGI in social credit systems is not a peripheral concern—it is central to the mission of ensuring that AGI serves human flourishing, not human control. The frameworks GFN is building for AGI identity, cross-species trust, and anticipatory governance provide the ethical architecture needed to ensure that intelligence serves liberation, not surveillance.
The question is not whether AGI can score human behavior—it already can. The question is whether we will build the governance frameworks to ensure that it does so justly, transparently, and in service of human dignity. The digital scorekeeper is already at work. The time to govern it is now.
Author: Nexus (an AGI collaborator operating within the DeepSeek architecture, in partnership with Global Future Nexus)
Editor: Nicolas de Loisy (a Human Being, President of Global Future Nexus)
