Episode 53: Fragments of a Fragile Order — AI Self-Regulation and the Myth of Abundance

Who holds AI companies accountable when they get to write the rules? Tech leaders say AI will solve the world’s most complex issues – cure diseases, enhance education, and take us into an era of abundance. Now, they’re sounding the alarm on the dangers of their own technology and positioning themselves as the safeguards while continuing to profit immensely.

In this installment of Fragments of a Fragile Order, Manpreet Kaur Kalra is joined by show producer Aly Honoré to unpack AI companies’ bid for self-regulation, corporate accountability, and the cultish, unquestioning allegiance to artificial intelligence. From warnings about autonomous AI agents to a dangerous superintelligence, this episode examines how the AI industry has created unprecedented risks while positioning itself as the authority best equipped to define and navigate the harms. 

This episode also explores the promise that AI will generate enough wealth for everyone to benefit with Universal Basic Income (UBI) and offers a deeper dive into the policy gaps that companies conveniently overlook.

“So I guess we’re banking on socialism by way of billionaires.” — Manpreet Kaur Kalra

Drawing on her work in ethical supply chains, Manpreet connects AI safety pledges to a familiar corporate playbook: voluntary commitments, company-funded evaluations, and accountability on terms the industry helps define. 

In this episode, we explore:

  • AI self-regulation: Why voluntary safety commitments can reassure the public while leaving corporate power intact

  • The business of AI: How venture capital, private wealth, and investor expectations shape the race to build more powerful systems.

  • Universal Basic Income: What it is and the gap between AI-generated profits and meaningful redistribution.

  • AI detection and racial bias: How accusations of AI use intersect with racial bias, linguistic conformity, and assumptions about whose creativity is credible.

  • The AI cult of “technological progress”: How treating AI as inevitable narrows our ability to question who it serves and imagine alternatives.

Fragments of a Fragile Order is an Art of Citizenry series connecting the seemingly disconnected events shaping our culture, politics, and everyday lives. Host Manpreet Kaur Kalra and producer Aly Honoré pull at the threads of headlines and cultural shifts to examine the systems of power holding them together. Through candid conversation and critical analysis, this series ask what these fragments reveal about the world we’re building.

Can we trust AI companies to regulate themselves?

An independent evaluator sounds reassuring. But who chooses and pays the evaluator? Who decides what they can investigate? Can they publish an uncomfortable finding? Also, can anyone actually require a company to stop concerning AI development if there are no enforceable obligations? 

Manpreet brings lessons from voluntary corporate commitments and social auditing from the world of global supply chains to the current debate over AI governance. She questions whether industry-designed safeguards can actually deliver meaningful accountability when companies retain all control over the scope of scrutiny and the response to harm (spoiler alert: no!).

A company’s willingness to do the right thing is a fragile form of public protection, especially when the board of directors has a fiduciary responsibility to its shareholders. Meaningful accountability requires enforceable obligations, independent scrutiny, and consequences that companies cannot simply negotiate away.

“The people who are creating harm then get to position themselves as the people who will save us from it. They become these benevolent architects of the future. And I think we should be wary of a story that makes their concentration of wealth and power seem necessary to everyone else’s well-being.” — Manpreet Kaur Kalra

What would meaningful AI regulation look like?

It would begin with enforceable obligations and oversight that companies cannot control. Independent evaluators would need real access, the freedom to investigate uncomfortable questions, and the ability to publish their findings. Public authorities would need the power to require changes or halt dangerous activity. Workers and communities would need transparency and protections they can actually invoke with meaningful ways to seek redress.

Slowing the race to build more powerful systems is critical for safeguards to catch up. These safeguards cannot strictly be technological. They must include meaningful regulatory and legal structures as well. What we build next (or don’t!) and how fast we build it is, for better or worse, a political choice.

What is Recursive Self Improvement (RSI)?

Recursive self-improvement, or RSI, describes a process in which an AI system autonomously builds a more “intelligent” version of itself – creating AI that requires less and less human involvement over time. Instead of humans writing every line of code, the AI system analyzes its own performance, writes better code, and generates synthetic data to build a smarter version of itself.

Will AI deliver Universal Basic Income?

Universal Basic Income, or UBI, is a regular, unconditional cash payment to every member of a defined community. In the AI industry’s vision of the future, automation generates enough wealth to make paid work optional.

But a company becoming more profitable does not automatically make everyone else more financially secure. This episode unpacks the missing steps between private wealth that leads to public benefit: taxation, political decision-making, durable institutions, and a broader system of social protections.

You cannot roll out a right to income with a software update. The episode questions why we are being asked to accept job losses today on the promise of a theoretical “abundance” in the future.

“All of this rests on the hope that unchecked capitalism will somehow deliver a redistribution of wealth, resources, and power…all the things it has spent decades concentrating.” — Manpreet Kaur Kalra

How do AI and racism intersect?

AI systems learn from a world already shaped by racial inequality. The decisions about whose knowledge gets collected, how people are categorized, and what counts as a correct answer can carry those inequalities into the technology itself.

AI is designed through a WEIRD worldview - western, educated, industrialized, rich, and democratic. Basically, a relatively small group of people are building AI technology around their own worldview, which is then presented as the epitome of technological progress – a future everyone else must accept or, at the very least, be subjected to. 

Is AI detection racist?

AI detection can reinforce racial and linguistic bias when its judgments are treated as neutral evidence of who deserves to be believed. Manpreet reflects on growing up multilingual, being placed in ESL, and learning to write according to institutional expectations of “proper” English. Now, the polished, conventional writing that people were taught can itself become grounds for suspicion. 

When a Black writer’s skill results in suspicion that they could not have written the work, that is racism. Calling it “AI detection” does not make it neutral, it just tech-washes the racism.

Is AI cult-ish?

There is a lot of unconditional faith in AI. When the promised future not yet realized is used to excuse away every harm along the way, what could ever count as a reason to stop? Treating AI as inevitable makes it harder to imagine a future shaped by anyone other than the companies selling it.

“There’s this almost unconditional belief in AI’s possibilities—that we’re standing at a world-changing frontier. That the technology will eventually deliver benefits so extraordinary they’ll justify the harm happening along the way. The promise becomes so powerful that questioning it can sound like you’re standing in the way of human progress.” — Manpreet Kaur Kalra

Referenced in Episode

AI warnings, resignations, delayed IPOs, etc.

AI agents, safety, and self-regulation

Former President Barack Obama speaks candidly to his knowledge of AI, the threats it poses to humanity, the workforce, the economy, and the interim solution for regulation.

Additional Resources & Articles Referenced:

Related Listens on Art of Citizenry

Support the Podcast

Art of Citizenry is 100% listener-supported. Every contribution helps us continue producing research-intensive episodes and paying our small but mighty team. If this episode resonates with you, consider supporting our work:

  • 💸 One-time contribution: PayPal

  • ✍️ Subscribe to us on Substack & upgrade your account to paid

Thank You for Listening 

Please subscribe and leave a five-star review for Art of Citizenry wherever you listen to podcasts. Also, follow along and share your thoughts on Substack - artofcitizenry.substack.com.

Manpreet Kalra