Algorithmic Austerity: When Algorithms Mistake Poverty for Fraud
What Governments Keep Getting Wrong About AI and Social Protection
The biggest AI bias problem isn’t facial recognition. It’s database joining. And it’s already costing lives.
Imagine a doctor prescribed to identify malingerers in a hospital ward. She gets rewarded every time she catches someone faking illness and penalized every time a faker slips through. Nobody tracks the patients she sends home sick. That's the incentive structure governments are building into AI for social protection.
In 2017, a woman in rural Jharkhand, India, walked three kilometers to her nearest ration shop and was turned away. Her biometric scan didn’t match. She was seventy years old, her fingertips worn from decades of agricultural labor, and the scanner couldn’t read them. She went home without food. She died eleven days later. Her name was Santoshi Kumari. A fact-finding team from the Right to Food Campaign documented her case and attributed her death directly to Aadhaar-linked authentication failure at the point of distribution. (Right to Food Campaign, 2017)
When governments deploy AI in social protection programs, the dominant political pressure runs toward fraud detection and budget control. The algorithm becomes an instrument of austerity dressed in the language of efficiency. The result is a pattern emerging across continents: systems sold as tools to reach the poor systematically exclude them. This is Algorithmic Austerity, and it’s one of the most consequential technology policy failures of our time.
The Evidence
Brazil’s story is the sharpest recent example, but it sits inside a larger global pattern.
The Brazilian Ministry of Development deployed an algorithmic system called Pente-Fino (”Fine-Tooth Comb”) to audit Bolsa Família, the world’s largest conditional cash transfer program, covering around 21 million families. The algorithm cross-references over 30 federal databases to flag “unipersonal” households suspected of splitting family units to claim double benefits. Local activists and the Defensoria Pública have documented that the system regularly flags legitimate poor individuals: young people who moved out of favelas, elderly people living alone, families with shared addresses in informal settlements. Many face benefit cuts without a meaningful appeals process. (Defensoria Pública da União, 2023)
Australia ran the same play on a larger scale. Between 2015 and 2019, the Department of Human Services’ Robodebt scheme auto-generated debt notices to roughly 470,000 welfare recipients by comparing annual tax averages against fortnightly payments, which systematically overstated income for anyone with irregular employment. The government clawed back approximately AUD 1.76 billion before courts intervened. A Royal Commission concluded in 2023 that the scheme was unlawful, caused serious psychological harm, and drove some recipients to suicide. The Commonwealth had to repay the money. (Royal Commission into the Robodebt Scheme, 2023)
The Netherlands took a different route to the same destination. Its SyRI (Systeem Risico Indicatie) system profiled welfare recipients using 17 datasets, generating fraud-risk scores concentrated in lower-income urban neighborhoods. A Dutch court struck it down in February 2020, finding the system violated Article 8 of the European Convention on Human Rights, the right to private life, because the government couldn’t explain how it worked or demonstrate it was proportionate. The ruling set a precedent that black-box risk scoring of benefit populations isn’t oversight; it’s surveillance. (Rechtbank Den Haag, 2020)
In the United States, Arkansas and Idaho deployed automated systems to calculate home care hours for people with cerebral palsy and other disabilities. Both states cut services significantly with no explanation provided to beneficiaries. Federal courts intervened in both cases. The rulings established that due process requires at minimum a human-readable explanation for any algorithmic decision affecting benefits. ([Ledgerwood v. Jegley, 8th Circuit, 2016; K.W. v. Armstrong, 9th Circuit, 2016])
These cases span different continents, income levels, and political systems. The common thread is the design choice: optimization for false positives (catching fraud) rather than false negatives (excluding legitimate beneficiaries), applied to populations with irregular data footprints.
The Real Problem: Informality Reads as Fraud
The dominant AI ethics discourse in Europe and North America focuses on biased facial recognition and discriminatory hiring algorithms. These are real problems, but they mostly affect the middle class. The systems doing the most harm to the most people are welfare fraud algorithms targeting people who are already poor. And they’re usually justified with fiscal responsibility language, which makes scrutiny rarer, not more frequent.
The deeper mechanism is what you might call the legibility trap, after James Scott’s work in Seeing Like a State. Formal databases were designed for formal-sector households: registered addresses, civil registration records, consistent tax filings, nuclear family structures. Poor families often don’t fit any of those categories. Multiple families share a single address in an informal settlement. Work is seasonal, informal, and doesn’t generate consistent tax records. Family structures are fluid across civil registration assumptions. Names appear differently across different government systems.
When an algorithm cross-references databases built for formal-sector lives, informality reads as fraud. The algorithm doesn’t know the difference between a woman living alone after her children moved to a different city and someone gaming the system. Shared addresses in a favela look identical to fraudulent double-claiming. This is biased database joining at scale, and it’s happening across every country that is rapidly digitalizing its social protection systems.
One Country Got It Right
Togo’s Novissi program during COVID-19 shows that the same data infrastructure can be designed in the opposite direction.
In April 2020, the government had two weeks to reach informal workers who had lost income and had no prior registration in any social system. The team, drawing on mobile phone metadata and machine learning models built partly with UC Berkeley researchers, used call data records, geographic patterns, and consumption proxies to identify and rank beneficiaries by estimated poverty. Critically, sparse formal records in a poor district were treated as a targeting signal, an indicator of informality and likely need, rather than a red flag triggering exclusion. The program reached approximately 60% of Togo’s adult population within weeks. (Aiken et al., Nature, 2022)
Same data. Opposite inference. Radically different outcome. The design choice was deliberate: the goal was inclusion, so data gaps were interpreted accordingly.
India’s Direct Benefit Transfer (DBT) system cut leakage in the LPG subsidy program from roughly 40% to under 5% by eliminating ghost beneficiaries through Aadhaar deduplication, which is a genuine and significant success. (Ministry of Petroleum & Natural Gas, 2016). The system’s error tolerance, though, was calibrated for convenience rather than consequence. A failed biometric scan is a minor inconvenience when you’re trying to buy cooking gas; it’s a crisis when it cuts your only food supply.
The lesson from comparing these cases is straightforward: the failure mode of algorithmic social protection is structural, not technical. It follows from the choice of optimization target, and that choice is political.
What Policymakers Should Do
Governments face genuine fiscal pressure to reduce leakage in social programs. AI can help with that. The question is how to do it without systematically excluding the people these programs exist to serve. Five things matter most.
1. Audit the false negative rate, not just the false positive rate.
Fraud detection audits measure how many fraudulent claims they catch. They almost never measure how many legitimate beneficiaries they exclude. Both are errors with real costs. Australia’s Robodebt scheme excluded hundreds of thousands of legitimate recipients while the government reported it as a fraud-reduction success. Every AI system deployed in social protection should report both error rates, disaggregated by geography, age, gender, and income level. If you can’t measure exclusion errors, you can’t manage them.
2. Treat informality as data, not noise.
Poor households have irregular data footprints because informal economies don’t generate formal records. Before any algorithm is used for eligibility or auditing, a data audit should map the gap between the population the system was designed to describe and the population it actually covers. Shared addresses, informal employment, alternative names across government systems, and migration patterns all need to be documented and built into the algorithm’s logic, either as inputs or as flags requiring human review. “Clean data” that misrepresents how poor people actually live is bad data, and building eligibility systems on top of it amplifies the error at scale.
3. Make “explain it to me” a legal right.
The Arkansas and Idaho court rulings established a minimum standard: any algorithmic decision cutting or reducing benefits must come with a human-readable explanation. This should be codified in procurement standards. When governments buy AI systems for social protection, the contract should require that any adverse decision be explainable in plain language to the affected person, and that a human decision-maker can review and override the algorithm within a defined timeframe. This is already law in the European Union under GDPR Article 22. It should be standard procurement conditionality everywhere.
4. Build appeals mechanisms with teeth before deployment, not after.
The default sequence is: deploy system, generate exclusions, receive complaints, build appeals process. This sequence means real people lose income during the gap. Brazil’s Pente-Fino has been generating exclusions since at least 2019; a meaningful appeals mechanism with civil society oversight still doesn’t exist at scale. The right sequence is: design appeals mechanism, conduct civil society testing with affected populations, then deploy. If the appeals mechanism can’t handle the expected volume of challenges, the system isn’t ready.
5. Set procurement conditionality on auditability.
Many of the systems causing harm, including versions of SyRI and early Robodebt iterations, were either proprietary systems or used decision logic the procuring government couldn’t fully access. No government should procure an AI system for social protection that it can’t audit independently. This means source code access or algorithmic documentation, regular third-party bias audits, and public reporting on exclusion rates. The World Bank’s procurement framework, which governs a significant share of social protection technology investment in low-income countries, should make this a standard condition. The IFC’s Environmental and Social Framework already applies similar conditionality to physical infrastructure. Digital infrastructure needs the same treatment.
The Bigger Risk
The global push to digitalize social protection is accelerating. The World Bank’s ASPIRE database counts over 3.4 billion people covered by at least one social protection program. A growing share of those programs are incorporating algorithmic eligibility, targeting, or fraud detection. Most of the countries deploying these systems have weaker data protection laws, fewer civil society watchdogs, and less administrative capacity to run appeals processes than Australia or the Netherlands. The Australian Robodebt scheme survived four years before courts stopped it. In a country with less judicial infrastructure, it might have survived indefinitely.
The policy window is now. Governments are currently designing and procuring these systems. International development institutions are funding them. The design choices being made in the next three to five years will determine the error rates that millions of people live with for the following decade.
The Togo case shows this can go well. The Australia case shows how badly it can go wrong. The difference was a single design choice: what do data gaps mean?
If you design an algorithm that treats informality as a red flag, the algorithm will punish the poor for being poor.
That’s a feature, not a bug.
Views are my own.