How algorithms became regulators – when private code functions as law without the accountability of law

Eric Loomis stood before a Wisconsin judge in 2013 having pleaded guilty to fleeing police during a traffic stop. The presentence report included a COMPAS risk assessment: a number from 1 to 10, generated by a proprietary algorithm, estimating his likelihood of reoffending. The number was high. His lawyers asked to examine the methodology. They were told it was proprietary — owned by a private company called Northpointe, not theirs to inspect.

The Wisconsin Supreme Court, ruling in 2016, held that Loomis had received due process even though neither he nor his lawyers could examine how his score was calculated — because he could review the inputs and the final number, even if not the logic connecting them. The court required that all future sentencing reports using COMPAS include a written warning listing the algorithm’s known problems. The warning was detailed and honest. Then the court upheld the algorithm’s use anyway. If the sentencing statute had been sealed from the defense, the Sixth Amendment crisis would have been obvious. Because it was an algorithm, the court called it a tool.

What, exactly, is the difference between a secret law and a proprietary rule that functions as law?

What makes something a regulation

The standard definition of regulation requires three things: binding effect, mandatory compliance, and consequences for non-compliance. By that definition, a law prohibiting lending discrimination is a regulation. But so, functionally, is an algorithm that automatically rejects loan applications from applicants in certain zip codes — an effect no less real for the absence of a signed statute.

Lawrence Lessig argued in 1999 that “code is law” — that the architecture of software shapes behavior as surely as legal rules do. Frank Pasquale extended the frame in 2015’s The Black Box Society, describing how opaque algorithmic systems govern credit, employment, and reputation without accountability. The question here isn’t whether code is law. It’s something more pointed: if an algorithm is already functioning as regulation — binding, mandatory, consequential — what accountability mechanisms should attach to that power, and why has nothing attached?

The functional test applies cleanly across the categories where algorithmic authority has become most consequential. A bail risk score is binding: the defendant cannot choose not to be scored, cannot substitute an alternative metric, cannot opt out of a system that determines whether they sleep in a cell or at home while awaiting trial. An algorithmic credit decision carries no individual decision-maker who can be examined, no discretionary judgment to second-guess. The algorithm generates a rule; the rule produces an outcome; the outcome is final unless the applicant can navigate a dispute process that doesn’t include access to the scoring logic itself. A content moderation removal is immediate: the post is gone. Whatever appeal mechanism exists for average users resolves against a content standard cited at a level of generality — “harassment,” “violence” — that provides no real basis for challenge.

Formal regulation carries four accountability criteria these systems lack. Transparency: in the United States, agency rulemaking under the Administrative Procedure Act (5 U.S.C. § 553) requires notice-and-comment — the rule must be stated publicly before it can bind anyone. Due process: the right to confront and contest the legal standard being applied to you is foundational to governance under law. Democratic legitimacy: the authority to make binding rules derives, in a constitutional democracy, from processes of authorization that trace back to elected bodies. Liability: mechanisms exist to assign responsibility when governance causes harm.

None of these apply to algorithmic systems as currently deployed. The objection usually raised at this point is that the Fifth Amendment, and constitutional accountability generally, governs state actors — not private companies. That’s correct as a matter of constitutional law. But the argument here isn’t about constitutional compliance. It’s about whether systems exercising regulatory power in the functional sense — binding, mandatory, consequential at population scale — should carry accountability mechanisms proportionate to that power. An administrative law frame fits better than a constitutional one. The question isn’t whether Meta is subject to the Bill of Rights. It’s whether systems with more governance reach than most governments should operate with less accountability than any government is allowed.

State action and its limits

The state-action doctrine holds that the constitutional protections in the Bill of Rights constrain government actors, not private companies. A private employer can restrict speech without implicating the First Amendment. A private platform can remove posts without triggering due process requirements. Legal scholars including Pasquale and Julie Cohen have argued for extending state-actor-equivalent accountability to private entities exercising what amounts to public power. The EU AI Act implicitly sidesteps the state-action problem by adopting a functional framework: it regulates AI systems based on what they do and to whom, not on whether a government deployed them. That approach — focusing on function rather than actor — is the right frame for algorithmic authority.

Liberty on a score: COMPAS and the sentencing machine

COMPAS — Correctional Offender Management Profiling for Alternative Sanctions — is a proprietary risk assessment tool developed by Northpointe, now Equivant. It produces a score from 1 to 10 estimating the likelihood that a defendant will reoffend. That score has been used in pretrial detention decisions, sentencing, and parole determinations across multiple US states.

In May 2016, ProPublica published “Machine Bias,” reporting by Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Using a public records request under Florida’s Sunshine Law, the team obtained COMPAS scores for more than 7,000 defendants in Broward County, Florida, arrested in 2013 and 2014, and tracked actual reoffending over two years. The finding that cut through: the false positive rate for Black defendants — people labeled high-risk who did not in fact reoffend — was 44.9 percent. For white defendants it was 23.5 percent. Nearly twice as likely to be falsely flagged as dangerous.

Equivant’s response was technically correct. They argued that COMPAS scores are calibrated: a score of 7 predicts the same probability of reoffending regardless of a defendant’s race. Both claims are mathematically true. They measure different things. The calibration claim is about predictive accuracy across groups. The false positive disparity is about what happens to people who don’t reoffend — and those people are disproportionately Black. An algorithm can pass a calibration test and still produce outcomes that fall hardest on one demographic. Choosing which fairness criterion matters is not a technical question. It’s a policy question that the algorithm made without anyone deciding it.

This isn’t a flaw in the math. It’s the math. Alexandra Chouldechova demonstrated in 2017, in a paper titled “Fair prediction with disparate impact: A study of bias in recidivism prediction instruments,” published in the journal Big Data, that when base rates of reoffending differ between groups, calibration and equal false positive rates are mathematically incompatible. You cannot have both simultaneously. Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan reached the same conclusion independently in “Inherent trade-offs in the fair determination of risk scores,” presented at the 8th Innovations in Theoretical Computer Science conference (ITCS) in 2017. Two research groups, working separately, proved the impossibility: the tension between ProPublica and Equivant wasn’t a disagreement that better methodology could resolve. It was a structural fact about the problem — one that makes the accountability argument more urgent, not less. If there is no purely technical answer to how fairness criteria should be balanced, then the choice must be made by someone. And that someone should be accountable for making it.

Back to Wisconsin. Eric Loomis had not been convicted of anything violent. He had fled police during a traffic stop. His COMPAS score came back high. He challenged its use at sentencing on due process grounds: the algorithm’s proprietary nature, he argued, prevented him from challenging the scientific validity of what the court was using to constrain his liberty.

The Wisconsin Supreme Court’s answer, in State v. Loomis, 881 N.W.2d 749 (Wis. 2016), was that due process was satisfied because the defendant could verify the accuracy of the inputs — his criminal history, his answers to the pre-sentencing questionnaire. He didn’t have the right to know how those inputs were weighted against each other, or what the scoring logic produced, or whether the tool had been validated on a Wisconsin population specifically. He just had the right to check whether the data entered about him was correct.

That’s like being told you failed a test fairly because your name was spelled correctly at the top of the paper.

The court did something notable, though. It required that all presentence reports relying on COMPAS include a written warning listing five specific limitations: that the proprietary nature of COMPAS prevents disclosure of how factors are weighted; that COMPAS scores are based on group data and identify high-risk groups, not necessarily high-risk individuals; that some studies have raised questions about whether COMPAS disproportionately classifies minority offenders as higher risk; that no cross-validation study for the Wisconsin population has been completed; and that risk assessment tools must be constantly monitored and re-normed for accuracy as populations change.

The court knew. It wrote down everything that might be wrong with the algorithm. Required that list appear in every single sentencing report involving this algorithm. Those are the court’s own findings about what is unknown, what might be wrong, what hasn’t been tested.

Then the court upheld the algorithm’s use anyway.

The US Supreme Court denied cert on June 26, 2017. The question of whether proprietary algorithmic tools can be used to constrain liberty without any mechanism to examine their methodology remains unanswered by the nation’s highest court.

The mathematical fairness impossibility

Alexandra Chouldechova (2017) and Kleinberg, Mullainathan, and Raghavan (2017) independently proved that when the base rate of the relevant outcome — in this context, reoffending — differs between demographic groups, it is mathematically impossible to satisfy both calibration (equal predictive accuracy across groups) and error-rate parity (equal false positive and false negative rates across groups) simultaneously. This is not a design problem or a data quality problem. It is a mathematical property of the situation. Equivant's defense of COMPAS had genuine technical validity on calibration grounds while leaving entirely untouched the accountability question: who decided which fairness criterion should govern, on what basis, and with what legitimacy? The impossibility result doesn't dissolve the fairness debate. It clarifies that the debate is ultimately political, not technical — which is precisely why technical proprietary protection is the wrong place to resolve it.

The invisible rule: credit, hiring, and economic life

Here is a comparison that the law has not adequately confronted. A discriminatory loan officer is, at least in principle, accountable. She can be deposed. Her emails can be subpoenaed. Notes she made about applicants can be reviewed. The reasoning behind her decisions can be examined in litigation, challenged against the Fair Housing Act or the Equal Credit Opportunity Act, and litigated to a verdict. A discriminatory algorithm producing the same racially disparate lending outcomes is not any of those things. There are no notes. No deposition. No individual decision-maker. The rule that generated the outcome is proprietary, and no plaintiff can compel its disclosure.

FICO scores, the dominant measure of creditworthiness in US consumer lending, are calculated on methodology that is not public. Consumers have some rights to dispute the accuracy of the underlying data — an address, a missed payment, a collection — but no right to contest how those inputs are weighted against each other. The weighting is the algorithm. The algorithm is proprietary. The algorithm is, in functional terms, the rule.

The Consumer Financial Protection Bureau has acknowledged that automated scoring models using alternative data — zip codes that correlate with race, spending patterns, social connections — can perpetuate and amplify the effects of decades of structured discrimination. Neighborhoods still show the outlines of redlining maps. An algorithm trained on historical data and using zip code as a feature doesn’t need to be designed to discriminate. It can absorb discrimination passively, as inherited signal. The CFPB has affirmed through Consumer Financial Protection Circular 2022-03 that existing adverse action notice requirements under ECOA apply to algorithmic lenders, including complex and opaque models. What it has not resolved is the disclosure problem: if the model is proprietary and the developer invokes trade secret protection, how does a plaintiff’s attorney actually conduct the less-discriminatory-alternatives analysis that the law nominally requires?

HireVue provided one illustration of how this plays out in employment. The company offered video-based pre-employment assessments, using AI to evaluate job candidates from recorded interviews. In November 2019, the Electronic Privacy Information Center filed a complaint with the Federal Trade Commission alleging, among other things, that HireVue falsely denied using facial recognition technology. In January 2021, following a third-party audit, HireVue dropped visual analysis from its assessments, acknowledging that it added no measurable predictive value. The technology assessed expressions, movement, presentation — none of it predictive of job performance, all of it creating multiple vectors for discrimination. It ran for years before anyone outside the company could establish that.

The iTutorGroup case produced the most concrete documentation. The EEOC filed suit against the online tutoring company alleging that its AI recruitment software automatically rejected female applicants aged 55 or older and male applicants aged 60 or older. The system didn’t deliberate. It applied a rule. More than 200 qualified applicants were rejected on age alone before the EEOC’s lawsuit forced the practice to stop. On August 9, 2023, iTutorGroup settled for $365,000 — the EEOC’s first resolution of a lawsuit specifically involving AI-based hiring discrimination. Spread across those 200-plus people, that works out to roughly $1,825 each for a lost employment opportunity.

The structural point is harder to litigate than any individual case. These systems produce systematic errors — consistent, patterned, identifiable in aggregate data — that no individual can effectively challenge, because the rule generating the error is never disclosed. If these were statutory rules, each one would be challengeable on its face: you’d know the rule, you’d know how it applied to you, you’d have a cause of action. As algorithms, the harm is real and documented in the aggregate. The rule generating it is invisible to the people it harms.

ECOA, Fair Housing Act, and the disclosure gap

The Equal Credit Opportunity Act and the Fair Housing Act both permit disparate impact claims — meaning a lending or housing practice can be unlawful even without discriminatory intent, if it produces discriminatory outcomes and a less discriminatory alternative was available. For algorithmic systems, the less-discriminatory-alternatives analysis requires understanding the model: what features it uses, what their relative weights are, what would change if certain features were excluded or reweighted. The CFPB has confirmed through Circular 2022-03 that existing adverse action notice requirements apply to algorithmic lenders regardless of model complexity. It has not resolved how disparate impact analysis is conducted when the model is proprietary and its developer asserts trade-secret protection. The practical result: the legal framework theoretically available to plaintiffs is practically unenforceable against systems whose rules are sealed.

The law of the feed

Meta’s family of apps — Facebook, Instagram, WhatsApp, Threads — reported 3.35 billion daily active people on average for December 2024, disclosed in Meta’s Q4 2024 earnings. For context: the population of the entire planet is roughly 8 billion. Meta’s daily active user base exceeds the population of every nation on earth.

YouTube does not officially disclose a global monthly active user count in its earnings reports; industry estimates and analytics firms cited approximately 2.5 billion monthly logged-in users in 2024. TikTok reported over 1 billion monthly active users as of late 2021, with third-party estimates placing the figure above 1.5 billion by 2024; no official ByteDance disclosure exists for the 2024 figure, as ByteDance is a private company. X — formerly Twitter — has not released audited user metrics since going private in 2022; estimates for 2024 from third-party analysts put monthly active users at approximately 600 million, though no official company report confirms this.

Don’t add these up. The overlap is enormous; most people who use TikTok also use Instagram. But hold the Meta figure alone, because it tells you something specific. A private company is making binding decisions — every day, billions of times — about what those 3.35 billion people can say, what they can see, whose content reaches them, whose doesn’t. Those decisions are made overwhelmingly by algorithmic systems. No stated rule. No individual adjudication. No legally enforceable appeal for any ordinary user.

The customary defense of this arrangement is that content moderation resembles editorial judgment — that a newspaper editor deciding what to publish is exercising the same kind of discretion as Meta’s content systems. The analogy doesn’t survive contact with scale. A newspaper’s editorial choices affect thousands of readers. It doesn’t claim to be public infrastructure. When billions of people’s elections, protests, health emergencies, and public records live on a platform, and when that platform’s algorithmic systems determine which voices are amplified and which are suppressed in those moments — that is regulatory power, not editorial discretion. The binding effect is real. The mandatory nature is real — you cannot participate in these public spheres at comparable scale through any competing platform that operates under different rules, because no alternative platform approaches this reach. The consequences are real.

Enforcement reach here surpasses anything any national speech law has operationalized. Not because governments lack authority, but because no government has ever needed to build enforcement infrastructure for speech across 3.35 billion daily users in every language on earth. These systems have. And their rules are not publicly stated.

The Meta Oversight Board was established to provide an independent check on content moderation decisions. It cannot access the algorithm. The Brennan Center for Justice published analysis documenting that the Oversight Board cannot do its job because it doesn’t have access to Meta’s algorithmic systems — the systems that make the overwhelming majority of decisions about what is amplified, what is demoted, and what is removed. The Board reviews a small selected fraction of decisions after the fact. Neither the Board nor the public can see the rule.

The consequences of this are not hypothetical. The UN Independent International Fact-Finding Mission on Myanmar documented, in its September 2018 report (UN Doc. A/HRC/39/CRP.2), how Facebook’s platform was used to spread anti-Rohingya hate speech in 2016 and 2017. The mission described Facebook as a “useful instrument” for spreading hatred and inflammatory content, and noted that Facebook’s role had been significant in the events leading up to atrocities against the Rohingya population. The algorithm amplified content it was not equipped to read — Facebook’s content moderation infrastructure did not cover Burmese at any meaningful scale during this period. No accountability mechanism existed to stop, challenge, or even document the pattern in real time. The company acknowledged it was too slow to act. Hundreds of thousands of people were displaced or killed in the intervening period.

At what point does the incapacity to moderate become indistinguishable from a decision not to moderate?

Section 230 and the immunity gap

Section 230 of the Communications Decency Act grants platforms broad immunity from liability for content posted by third parties and, importantly, for good-faith content moderation decisions. The result is a legal structure in which platforms are neither publishers (who bear liability for content) nor regulators (who bear accountability for their rules). They occupy a zone that was designed for an internet that no longer exists — small services hosting user content, not entities that make binding governance decisions for billions of people daily. Current efforts to reform or repeal Section 230 are animated by the same structural problem this article identifies: the mismatch between the power these systems exercise and the accountability mechanisms that currently attach to that power. Whether Section 230 reform is the right lever, and what unintended consequences it would create, is a question that deserves its own article.

Four things formal regulation requires

The accountability gap in algorithmic governance becomes clearest when you lay the four criteria of legitimate regulation against the systems we’ve been examining.

Start with transparency. The APA (5 U.S.C. § 553) notice-and-comment requirement exists because secret regulations are incompatible with the kind of governance that requires consent. You cannot meaningfully consent to or contest a rule you’ve never seen. A bail algorithm scores a defendant 8. That score has consequences — whether the person sleeps at home or in a cell while awaiting trial. The rule generating the 8 is not disclosed. It can be inferred, incompletely and imperfectly, by someone with access to large datasets and the statistical tools to reverse-engineer it. That is not what we mean by transparency. When a content moderation algorithm removes a post and the user receives a notification citing “violence” or “harassment,” the notification doesn’t describe which standard was applied, how its threshold was set, or at what level of confidence the algorithm triggered. The rule can only be known through accumulation of enforcement outcomes — the way medieval peasants inferred the law from which behaviors got them punished.

Transparency is the prerequisite for everything else. Without it, due process is theater. The right to confront and contest the legal standard applied to you is foundational — it is why defendants have the right to see the charges against them, why regulated parties have the right to participate in rulemaking. Eric Loomis’s lawyers asked for the methodology used to constrain their client’s liberty. The court said no. The iTutorGroup applicants who were auto-rejected never knew that a rule had been applied to them, let alone what the rule was. Facebook users who receive a removal notification face a content standard stated at a level of abstraction — “this post violates our Community Standards on violent speech” — that provides no real basis for challenge. Due process requires not just a chance to respond but a chance to respond to something specific.

Democratic legitimacy follows. Legislatures are elected. Agencies are delegated power by legislatures, supervised by elected officials, and subjected to the APA’s procedural requirements. Their decisions can be challenged in federal court. Private companies deploying algorithms that govern access to credit, employment, liberty, and expression for hundreds of millions or billions of people have no corresponding accountability structure. They are accountable to shareholders. Not to the populations they govern. The argument isn’t that private companies shouldn’t make product decisions — it’s that when product decisions become governance decisions, the accountability structure appropriate to product decisions is no longer adequate.

And then liability — the one that makes the other three enforceable. Tort law, product liability, administrative accountability: these exist to assign responsibility when governance causes harm. For algorithmic harm at population scale, none of them work adequately. Class certification under federal civil procedure requires demonstrating a common question of law or fact that affected all class members in the same way. When the rule is proprietary and the harm is distributed across millions of small individual injuries — the rejected loan, the false-positive bail score, the suppressed voice — the common-question requirement is structurally difficult to satisfy without access to the model itself. The harm in aggregate is enormous. The individual instances remain invisible to legal mechanisms designed for individual adjudication.

Return to the iTutorGroup figure: $365,000 to settle discrimination claims affecting more than 200 people. Roughly $1,825 per person for a lost employment opportunity. That number doesn’t change behavior. It doesn’t approximate the actual harm. It doesn’t create incentives to redesign the system. Scale that figure up: if an algorithm rejects 20,000 applicants on discriminatory grounds and each faces a $1,825 settlement value, the potential liability is $36.5 million. For a company whose revenue runs in the hundreds of millions, that’s a cost of doing business, not a structural constraint.

The four criteria — transparency, due process, democratic legitimacy, liability — don’t function independently. Transparency is the foundation for due process, which creates the path to democratic accountability, which gives liability meaningful teeth. Remove transparency and the rest dissolve. Algorithmic regulatory power currently has none of the four.

The response and its limits

The EU AI Act, Regulation 2024/1689, entered into force in August 2024. It is the most comprehensive regulatory framework built to address algorithmic authority so far.

What it does is substantial. The Act classifies AI systems by risk tier. Annex III explicitly lists, as high-risk systems requiring full compliance burden: credit scoring, employment hiring and performance evaluation, and — critically — administration of justice, including recidivism prediction and bail risk assessment. For these systems, the Act mandates conformity assessments before deployment, fundamental rights impact assessments, human oversight obligations, and transparency requirements to people affected by decisions. Article 86 establishes a right to explanation for individuals subject to decisions by most high-risk AI systems listed in Annex III — with a carve-out for systems under Annex III point 2 (critical infrastructure management systems, which don’t make individual decisions about people) — and the right applies where decisions produce legal effects or significantly adverse impacts on health, safety, or fundamental rights. Non-compliance penalties reach €35 million or 7 percent of global annual turnover, whichever is higher. The original compliance deadline for Annex III high-risk systems is August 2, 2026. On May 7, 2026, EU co-legislators reached political agreement under the Digital Omnibus package to extend that deadline to December 2, 2027; formal adoption of those amendments is pending, with the August 2026 deadline remaining legally binding in the interim.

Those are not symbolic requirements. COMPAS, if deployed in Europe under Annex III, would face mandatory conformity assessment and fundamental rights impact assessment before use. HireVue’s video-assessment system, if used for hiring decisions affecting EU-based workers, would require human oversight and explanation rights.

In the United States, progress is more piecemeal. New York City Local Law 144, effective July 2023, requires annual independent bias audits for any automated employment decision tool used in hiring or promotion affecting New York City-based positions, with public disclosure of results. Illinois Public Act 103-0804, amending the Illinois Human Rights Act, prohibits the use of AI in employment decisions in ways that result in unlawful discrimination, bars the use of zip codes as proxies for protected characteristics, and requires employer notification when AI is used in hiring or employment decisions; it took effect January 1, 2026. Colorado signed SB 26-189 on May 14, 2026, replacing its earlier AI legislation (SB 24-205 from 2024) with a framework focused on transparency and disclosure requirements for automated decision-making in consequential decisions; it takes effect January 1, 2027.

Real advances. Meaningful regulatory movement, on a problem that wasn’t seriously regulated five years ago.

But these frameworks share a structural limitation. They operate at the level of documentation, audit, and disclosure. They require that someone — an auditor, a regulator, a compliance team — assess whether the system is fair. They do not give the individual governed by the system the ability to inspect the rule or mount a meaningful challenge to its application in their specific case. The EU AI Act’s Article 86 right to explanation means that a person subject to an Annex III high-risk decision must receive an explanation of that decision. It does not mean the person can examine the model weights, the training data composition, or the algorithmic methodology. Explanation of a decision is not the same thing as auditability of the system.

The structural lag is a separate problem. The EU AI Act’s high-risk classification framework was built around AI systems as they existed roughly between 2020 and 2022. By the time Annex III compliance is required, the systems being regulated have gone through multiple generational changes. Regulations are written at legislative timescale — years. Algorithmic systems iterate at deployment timescale — months. This isn’t a failure of drafting. It’s a structural mismatch between the pace of lawmaking and the pace of the technology.

The GDPR’s experience should temper expectations. Article 22 of the General Data Protection Regulation, enacted in 2018, gave EU residents the right not to be subject to solely automated decisions with significant effects, along with associated explanation rights. The “solely automated” threshold was easy to circumvent: a nominal human review step — someone glancing at the algorithm’s output before clicking confirm — was held sufficient to remove the case from Article 22’s scope. The right to explanation was interpreted narrowly. Enforcement was inconsistent across member states. The EU AI Act is partly a response to Article 22’s ineffectiveness. The experience should inform how optimistically the Act’s own explanation rights are read.

GDPR Article 22 and learned skepticism

GDPR Article 22, which has been in force since 2018, guarantees EU residents the right not to be subject to purely automated consequential decisions and associated explanation rights. In practice, its impact has been limited. The "solely automated" threshold — meaning the protection could be bypassed by inserting any nominal human review — made the provision easy to circumvent. Explanation rights were interpreted narrowly: courts generally held that a general description of the system's logic satisfied the requirement, not the specific reasoning in the individual case. The EU AI Act's Article 86 right-to-explanation provision was drafted with Article 22's failures in mind. History suggests that implementation, enforcement, and judicial interpretation will determine whether the new provision achieves more than the old one did.

The accountability paradox

The legal mechanisms we have for challenging governance were built to handle individual acts by identifiable decision-makers operating under stated rules. A statute is challenged by one person in one case. An agency regulation is contested through notice-and-comment or APA litigation. An administrative decision is appealed to a reviewing tribunal. A discriminatory official can be deposed. These mechanisms assume a specific structure: a singular decision, a stated rule, a challenger who can identify the harm, a decision-maker who can be questioned.

Algorithmic regulatory power breaks every one of those assumptions simultaneously.

To mount a legal challenge to an algorithmic decision, you need three things: the rule that generated it (proprietary, cannot be compelled), a counterfactual showing what a different outcome would have required (impossible without model access), and a demonstration of systematic injustice (which requires population-level data that the affected individual almost certainly doesn’t have). The Wisconsin Supreme Court’s answer was to require written warnings. That is managing an accountability gap, not closing it. Calling a thing a problem in the presentence report doesn’t provide the mechanism to challenge the thing.

Democratic authorization is absent at every point where it might have been established. Meta’s Oversight Board cannot access the algorithm it nominally oversees. The CFPB cannot compel a lender to disclose model weights. The Wisconsin Supreme Court declined to order COMPAS opened. At each juncture — each moment where accountability could have been imposed — the proprietary interest was treated as a barrier that governance would not cross. Not because the law required that deference. Because no actor was willing to force the question.

Here is the paradox, stated plainly: the features that make algorithmic systems worth deploying at scale — consistency across millions of cases, speed, the elimination of human discretion that produces its own discriminatory failures — are exactly the features that make them illegible to the accountability mechanisms law has developed. A biased loan officer is accountable in ways a biased algorithm isn’t. A biased sentencing judge is accountable in ways a biased sentencing algorithm isn’t. This is not an argument for biased loan officers or biased judges. It is an observation about what the legal system has not yet figured out.

The frameworks being built now — EU AI Act, NYC Local Law 144, Colorado SB 26-189, CFPB Circular 2022-03, Illinois Public Act 103-0804 — operate primarily at the level of documentation and audit. They require attestation that someone assessed whether the system is fair. They do not provide the individual governed by the system with the ability to inspect the rule or challenge its application. The documentation gap is closing. The accountability gap remains structurally open.

Algorithmic systems can produce systematically discriminatory outcomes for years before any challenge is possible — because the challenge requires access the challenger isn’t given. iTutorGroup discriminated against hundreds of applicants for however long the system was deployed before the EEOC filed suit. HireVue ran facial analysis on job candidates for years before an audit found it added nothing. COMPAS has been used in sentencing across multiple states since the 2000s. The harms are retrospective. The challenge is retrospective. The people harmed during the period before challenge are not made whole by eventual resolution.

This is what algorithmic regulatory power without accountability looks like in practice. Not a sudden imposition. An accumulation. Decision by decision, court by court, regulatory carve-out by regulatory carve-out.

Return to Wisconsin

Read the warning the Wisconsin Supreme Court required in every COMPAS sentencing report. Not a summary — the actual language from the court’s opinion. The proprietary nature of COMPAS has been invoked to prevent disclosure of how factors are weighted or how risk scores are calculated. COMPAS scores are based on group data and can identify high-risk groups, not necessarily a particular high-risk individual. Some studies of COMPAS risk assessment scores have raised questions about whether they disproportionately classify minority offenders as higher risk. No cross-validation study for a Wisconsin population has yet been completed. Risk assessment tools must be constantly monitored and re-normed for accuracy due to changing populations.

The court wrote those sentences. Required them to appear in every single sentencing report involving this algorithm. Those are the court’s own findings about what is unknown, what might be wrong, what hasn’t been tested.

Then the court upheld the algorithm’s use anyway.

Hold that for a moment.

The distance between requiring a disclaimer and providing accountability is the distance between a warning label and a safety standard. Warning labels acknowledge the problem. Safety standards require it to be solved. We put warning labels on things we’ve decided to permit.

What the Wisconsin court produced was not a safeguard. It was a documented acknowledgment that the safeguard doesn’t exist. The warning tells the sentencing judge: this tool might discriminate, hasn’t been validated for your state, operates by rules you cannot inspect. It does not give the judge, or the defendant, or the appellate court, any mechanism to act on that knowledge.

The architecture described in this article — algorithmic systems governing billions of people’s access to credit, employment, liberty, and expression, carrying none of the accountability that any other form of governance would require — was not designed. It accumulated. Each court that called an algorithm a tool rather than a rule. Each regulator that accepted proprietary protection as a permanent answer rather than a temporary one. Each legislature that built disclosure frameworks without building access rights. Each settlement that distributed $1,825 per person and called it resolution.

Whether that accumulation was conscious or simply the path of least resistance doesn’t change what it produced.

Zastrzeżenie dotyczące Gen AI

Niektóre treści tej strony zostały wygenerowane i/lub edytowane przy pomocy generatywnej sztucznej inteligencji.

Media

Diagram of an algorithm for the Analytical Engine for the computation of Bernoulli numbers, from Sketch of The Analytical Engine Invented by Charles Babbage by Luigi Menabrea with notes by Ada Lovelace – Wikipedia

Kluczowe źródła i odniesienia

Julia Angwin, Jeff Larson, Surya Mattu, Lauren Kirchner, “Machine Bias,” ProPublica, May 23, 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

State v. Loomis, 881 N.W.2d 749 (Wis. 2016). Available at https://law.justia.com/cases/wisconsin/supreme-court/2016/2015ap000157-cr.html

Loomis v. Wisconsin, 137 S. Ct. 2290 (2017) (cert. denied June 26, 2017).

Alexandra Chouldechova, “Fair prediction with disparate impact: A study of bias in recidivism prediction instruments,” Big Data, vol. 5, no. 2, pp. 153–163, 2017. DOI: 10.1089/big.2016.0047. https://journals.sagepub.com/doi/abs/10.1089/big.2016.0047

Jon Kleinberg, Sendhil Mullainathan, Manish Raghavan, “Inherent trade-offs in the fair determination of risk scores,” Proceedings of the 8th Innovations in Theoretical Computer Science Conference (ITCS 2017), LIPIcs vol. 67, pp. 43:1–43:23. https://drops.dagstuhl.de/storage/00lipics/lipics-vol067-itcs2017/LIPIcs.ITCS.2017.43/LIPIcs.ITCS.2017.43.pdf

Lawrence Lessig, Code and Other Laws of Cyberspace. Basic Books, 1999.

Frank Pasquale, The Black Box Society. Harvard University Press, 2015.

Regulation (EU) 2024/1689 of the European Parliament and of the Council (EU AI Act), entered into force August 1, 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng

Report of the Detailed Findings of the Independent International Fact-Finding Mission on Myanmar, UN Doc. A/HRC/39/CRP.2, September 18, 2018. https://www.ohchr.org/sites/default/files/Documents/HRBodies/HRCouncil/FFM-Myanmar/A_HRC_39_CRP.2.pdf

EEOC v. iTutorGroup, Inc., settlement announced August 9, 2023. https://www.eeoc.gov/newsroom/itutorgroup-pay-365000-settle-eeoc-discriminatory-hiring-suit

EPIC, “In re HireVue,” FTC complaint, November 6, 2019. https://epic.org/documents/in-re-hirevue/

“HireVue, Facing FTC Complaint From EPIC, Halts Use of Facial Recognition,” EPIC, January 2021. https://epic.org/hirevue-facing-ftc-complaint-from-epic-halts-use-of-facial-recognition/

“Meta’s Oversight Board Needs Access to Facebook’s Algorithms to Do Its Job,” Brennan Center for Justice. https://www.brennancenter.org/our-work/analysis-opinion/metas-oversight-board-needs-access-facebooks-algorithms-do-its-job

Meta Platforms, Inc., Q4 2024 Earnings Release, Form 8-K, filed January 2025. https://www.sec.gov/Archives/edgar/data/0001326801/000132680125000014/meta-12312024xexhibit991.htm

Administrative Procedure Act, 5 U.S.C. § 553 (rulemaking requirements).

Consumer Financial Protection Bureau, Consumer Financial Protection Circular 2022-03: Adverse action notification requirements in connection with credit decisions based on complex algorithms (May 26, 2022). https://www.consumerfinance.gov/compliance/circulars/circular-2022-03-adverse-action-notification-requirements-in-connection-with-credit-decisions-based-on-complex-algorithms/

Illinois Public Act 103-0804, amending the Illinois Human Rights Act to address the use of artificial intelligence in employment decisions, signed August 9, 2024, effective January 1, 2026. https://www.ilga.gov/legislation/PublicActs/View/103-0804

Colorado SB 26-189, signed May 14, 2026, effective January 1, 2027. https://leg.colorado.gov/bills/sb26-189

NYC Local Law 144 of 2021, effective July 5, 2023 (automated employment decision tools).