Eric Loomis stood before a Wisconsin judge in 2013 having pleaded guilty to two lesser charges: attempting to flee a traffic officer and driving a car without its owner’s consent. The state tied the case to a drive-by shooting he denied taking part in. The presentence report included a COMPAS risk assessment, a score from 1 to 10 that a proprietary algorithm generated to estimate his likelihood of reoffending. The number was high. His lawyers asked to examine the methodology. They were told it was proprietary — owned by a private company called Northpointe, not theirs to inspect.
The Wisconsin Supreme Court, ruling in 2016, held that Loomis had received due process even though neither he nor his lawyers could examine how his score was calculated. He could review the inputs and the final number, if not the logic connecting them, and the score had not been the deciding factor in his sentence. The court required that all future sentencing reports using COMPAS include a written warning listing the algorithm’s known problems. The warning was detailed and honest, and the court upheld the algorithm’s use anyway. If the sentencing statute itself had been sealed from the defense, the due-process crisis would have been obvious; because the sealed rule was an algorithm, the court treated it as a tool.
What, exactly, is the difference between a secret law and a proprietary rule that functions as law?
What makes something a regulation
A useful working definition of regulation has three ingredients: binding effect, mandatory compliance, and consequences for non-compliance. By that definition, a law prohibiting lending discrimination is a regulation. But so, functionally, is an algorithm that automatically rejects loan applications from applicants in certain zip codes, an effect no less real for the absence of a signed statute.
Lawrence Lessig argued that “code is law”: the architecture of software shapes behavior as surely as legal rules do. Frank Pasquale has argued that opaque algorithmic systems now govern credit, employment, and reputation without accountability. If an algorithm is already functioning as regulation, binding, mandatory and consequential, what accountability should attach to that power, and why has nothing attached?
The functional test applies cleanly across the categories where algorithmic authority has become most consequential. A bail risk score is inescapable: the defendant cannot choose not to be scored, cannot substitute an alternative metric, cannot opt out of a number that weighs heavily on whether they sleep in a cell or at home while awaiting trial. An algorithmic credit decision carries no individual decision-maker who can be examined, no discretionary judgment to second-guess. The algorithm generates a rule; the rule produces an outcome; the outcome is final unless the applicant can work through a dispute process that never shows the scoring logic itself. A content moderation removal is immediate: the post is gone. Whatever appeal exists for average users resolves against a content standard as general as “harassment” or “violence,” which gives no real basis for challenge.
Formal regulation carries four accountability criteria these systems lack. Transparency: in the United States, the Administrative Procedure Act requires notice-and-comment for most binding agency rules: the rule must be published for comment before it can bind anyone. Due process: the right to confront and contest the legal standard being applied to you is foundational to governance under law. Democratic legitimacy: the authority to make binding rules derives, in a constitutional democracy, from processes of authorization that trace back to elected bodies. Liability: mechanisms exist to assign responsibility when governance causes harm.
Algorithmic systems as currently deployed meet none of these in full; at best they carry fragments of each, depending on the country and the sector. The usual objection is that the Fifth and Fourteenth Amendments, and constitutional accountability generally, bind the state, not private companies. That’s correct as a matter of constitutional law, but it answers a different question: whether systems exercising regulatory power in the functional sense, at population scale, should carry accountability in proportion to that power, and an administrative law frame fits that question better than a constitutional one. Nobody needs Meta to be subject to the Bill of Rights to ask why systems with more governance reach than most governments should operate with less accountability than any government is allowed.
State action and its limits
The state-action doctrine holds that the constitutional protections in the Bill of Rights constrain government actors, not private companies. A private employer can restrict speech without implicating the First Amendment. A private platform can remove posts without triggering due process requirements. Frank Pasquale argues that dominant platforms now exercise "functional sovereignty," regulating markets and behavior in ways once reserved for governments, and that power of that kind calls for public-law-style accountability. The EU AI Act implicitly sidesteps the state-action problem by adopting a functional framework: it regulates AI systems based on what they do and to whom, not on whether a government deployed them. Judging a system by its function rather than by who runs it is the right frame for algorithmic authority.Liberty on a score: COMPAS and the sentencing machine
COMPAS, short for Correctional Offender Management Profiling for Alternative Sanctions, is a proprietary risk assessment tool developed by Northpointe, now Equivant. It produces a score from 1 to 10 estimating the likelihood that a defendant will reoffend. That score has been used in pretrial detention decisions, sentencing, and parole determinations across multiple US states.
In 2016, reporters at ProPublica used Florida’s public records law to obtain COMPAS scores for more than 7,000 defendants in Broward County, arrested in 2013 and 2014, and tracked who was arrested again over the next two years. Among defendants who were not rearrested, Black defendants were nearly twice as likely as white defendants to have been labeled medium or high risk.
Northpointe answered that COMPAS scores are calibrated: a score of 7 predicts roughly the same probability of reoffending regardless of a defendant’s race. The data supported both claims, because they measure different things. The calibration claim is about predictive accuracy across groups. The false positive disparity is about what happens to people who don’t reoffend, and among those people the wrongly flagged are disproportionately Black. An algorithm can pass a calibration test and still produce outcomes that fall hardest on one demographic. Choosing which fairness criterion matters is a policy question, yet no public body ever made that choice for COMPAS.
The conflict is built into the math, and it has been proved: when base rates of reoffending differ between groups, a high-risk label that is equally predictive for each group cannot also give both groups equal false positive and false negative rates, unless the predictions are perfect. You cannot have all of these at once. No better methodology could have settled the dispute between ProPublica and Northpointe, and that makes the accountability argument more urgent. If there is no purely technical answer to how fairness criteria should be balanced, then someone has to make the choice, and whoever makes it should be accountable for it.
Eric Loomis had not been convicted of anything violent: he had pleaded guilty to fleeing an officer and driving a car without the owner’s consent, while shooting-related charges he denied were dismissed but “read in” at sentencing. His COMPAS score came back high. He challenged its use at sentencing on due process grounds: the algorithm’s proprietary nature, he argued, prevented him from challenging the scientific validity of what the court was using to constrain his liberty.
The Wisconsin Supreme Court’s answer, in State v. Loomis, was that due process was satisfied because the defendant could verify the accuracy of the inputs — his criminal history, his answers to the pre-sentencing questionnaire. He didn’t have the right to know how those inputs were weighted against each other, or how the scoring logic turned them into a number.
He had the right to check whether the data entered about him was correct, and to argue with the score that came out the other end. That could catch a wrong entry in his record, but not a flaw in how the record was turned into a risk score.
The court did something notable, though. It required that every presentence report relying on COMPAS carry a written advisement setting out the limits on its use and a list of cautions: that its workings are kept secret, that it identifies high-risk groups rather than high-risk individuals, that studies have questioned whether it classifies minority offenders as higher risk, that it had not been validated on a Wisconsin population, and that COMPAS was not developed for use at sentencing at all, but for corrections decisions about treatment, supervision and parole.
Having written down what might be wrong with the algorithm, the court upheld its use all the same, with a fence around it: a risk score may not be used to decide whether someone goes to prison or for how long, and may never be the determinative factor. Nothing in the ruling let the defendant see how his score was calculated.
The US Supreme Court declined to hear the case in 2017. Whether proprietary algorithmic tools can be used to constrain liberty without any way to examine their methodology remains a question the nation’s highest court has not answered.
The mathematical fairness impossibility
If the base rate of the relevant outcome (here, reoffending) differs between demographic groups, two mathematical proofs show that fairness criteria collide. A score cannot be calibrated (a given score means the same real-world risk in every group) and also give the people who reoffend, and the people who do not, the same average score in every group. Once scores are cut into high- and low-risk labels, those labels cannot be equally predictive for each group (the same share of those flagged go on to reoffend) and also have equal false positive and false negative rates. Neither pair can be had together unless the predictions are perfect, which in practice they never are, and no improvement in design or data gets around it: it is a mathematical property of the situation. Northpointe's defense of COMPAS held up on calibration and left the accountability question untouched: who decided which fairness criterion should govern, on what basis, and with what legitimacy? The impossibility result doesn't dissolve the fairness debate. It shows the choice is a political one, which is why proprietary protection is the wrong place to make it.The invisible rule: credit, hiring, and economic life
The law has not adequately confronted a simple comparison. A discriminatory loan officer is, at least in principle, accountable. She can be deposed, her emails subpoenaed, her notes on applicants reviewed. The reasoning behind her decisions can be examined in litigation, challenged against the Fair Housing Act or the Equal Credit Opportunity Act, and litigated to a verdict. A discriminatory algorithm producing the same racially disparate lending outcomes is not any of those things. The lender can still be made to produce witnesses and records about how the model was built and tested, but there is no individual decision-maker to depose about why this applicant was turned down. The rule that generated the outcome is proprietary, and a plaintiff can usually reach it only by winning a discovery fight, under a protective order, after persuading a court it is relevant at all.
FICO scores, the dominant measure of creditworthiness in US consumer lending, are calculated by a proprietary formula. FICO publishes five broad categories and their rough weights (payment history 35 percent, amounts owed 30, length of history 15, new credit 10, credit mix 10), but not the model itself. Consumers have some rights to dispute the accuracy of the underlying data, such as a missed payment or a collection, but no right to contest how those inputs are weighted against each other. That weighting is the proprietary part, and in functional terms it is the rule.
In 2023 the Consumer Financial Protection Bureau, together with the Justice Department, the EEOC and the FTC, warned that automated systems trained on historical data can perpetuate unlawful bias and automate discrimination. Neighborhoods still show the outlines of redlining maps. An algorithm trained on historical data and using zip code as a feature doesn’t need to be designed to discriminate. It can absorb discrimination passively, as inherited signal. The law’s response has since moved backwards. In 2025 the CFPB withdrew its guidance confirming that lenders using complex, opaque models must still give applicants specific reasons for a denial; the statutory duty remains, but the guidance is gone. Then a CFPB rule that took effect in 2026, and is now being challenged in court, declared that the Equal Credit Opportunity Act does not authorize disparate-impact claims at all. Outside housing, where the Fair Housing Act still allows them, that took away the main federal tool designed for exactly this problem: proving that a facially neutral rule produces discriminatory outcomes when a fairer alternative existed.
HireVue shows how this plays out in employment. The company sold video-based pre-employment assessments, using AI to evaluate job candidates from recorded interviews. A privacy group filed a complaint with the Federal Trade Commission alleging, among other things, that HireVue falsely denied using facial recognition technology. HireVue stopped analyzing candidates’ facial expressions in 2020, and later explained why: by the company’s own account, nonverbal signals, facial expressions among them, had added only about a quarter of one percent to predictive power for most roles, not worth it given the concerns about bias. For years, candidates had been scored partly on expressions and movement, and no one outside the company could test whether that was fair.
The iTutorGroup case is the bluntest example. The EEOC sued the online tutoring company for programming its application software to reject female applicants aged 55 or older and male applicants aged 60 or older automatically. More than 200 qualified US applicants were rejected on age alone. iTutorGroup agreed to pay $365,000 to settle, to be shared among the applicants the software had screened out: less than $1,825 each, on average, for a lost employment opportunity. The consent decree also required anti-discrimination training, a new policy and at least five years of EEOC monitoring.
The structural point is harder to litigate than any individual case. These systems produce systematic errors, patterned and visible in aggregate data, that individuals rarely manage to challenge, because the rule generating the error is not disclosed to them. If these were statutory rules, each one would be challengeable on its face: you’d know the rule, you’d know how it applied to you, you’d have a cause of action. With algorithms, the harm is real and documented in the aggregate, while the rule generating it stays invisible to the people it harms.
ECOA, Fair Housing Act, and the disclosure gap
Disparate-impact law lets a plaintiff challenge a practice that produces discriminatory outcomes without any proof of intent, if a less discriminatory alternative was available. The Supreme Court has confirmed that such claims are available under the Fair Housing Act. For lending under the Equal Credit Opportunity Act the door has narrowed: a CFPB rule that took effect in 2026 states that ECOA does not authorize disparate-impact liability, and the Bureau withdrew its guidance on algorithmic adverse-action notices in 2025. Where disparate-impact claims survive, the less-discriminatory-alternatives analysis requires understanding the model: what features it uses, how they are weighted, what would change if some were excluded or reweighted. When the model is proprietary and its developer asserts trade-secret protection, a framework that is theoretically available to plaintiffs can become very hard to use in practice.The law of the feed
Meta says its family of apps (Facebook, Instagram, Messenger, WhatsApp) averaged about 3.6 billion daily active people in June 2026. The whole planet holds a little over 8 billion. Meta’s daily audience is larger than the population of every nation on earth.
YouTube gives no overall count; it says its Shorts alone draw “billions” of monthly logged-in users. TikTok announced 1 billion monthly users in 2021. X, formerly Twitter, stopped reporting user numbers in public filings when it went private, and the roughly 600 million monthly users Elon Musk has claimed have never been independently verified.
Don’t add these up. The overlap is enormous; most people who use TikTok also use Instagram. The Meta figure alone tells you something specific. A private company is making binding decisions billions of times a day about what those 3.6 billion people can say, what they can see, whose content reaches them, whose doesn’t. Those decisions are made overwhelmingly by algorithmic systems. The written policies are public; the ranking and enforcement machinery that applies them is not. The European Union is the big exception: since 2024 the Digital Services Act has guaranteed users a reasoned decision, a free appeal and access to independent dispute bodies. India runs a government Grievance Appellate Committee whose rulings platforms must follow. In most other countries, an ordinary user has no legally enforceable appeal at all.
The customary defense of this arrangement is that content moderation resembles editorial judgment, that a newspaper editor deciding what to publish is exercising the same kind of discretion as Meta’s content systems. The analogy doesn’t survive contact with scale. Even the largest newspapers count their readers in millions, not billions, and none claims to be public infrastructure. When billions of people’s elections, protests, health emergencies and public records live on a platform, and that platform’s algorithms decide which voices are amplified and which are suppressed in those moments, that is regulatory power, with binding effect and real consequences. It is mandatory in practice too: you cannot take part in these public spheres at comparable scale through any competing platform with different rules, because no alternative approaches this reach.
No national speech law has ever been enforced at this reach. Governments have the authority, but no government has ever needed to build enforcement infrastructure for speech across 3.6 billion daily users in dozens of languages. These systems have, unevenly. Their policies are published, but the operational rules that decide each case, the thresholds, classifiers and ranking signals, are not.
Meta’s Oversight Board was established to provide an independent check on content moderation decisions. The Brennan Center for Justice has argued that the Board cannot do its job without access to Meta’s algorithmic systems, the systems that make the overwhelming majority of decisions about what is amplified, what is demoted, and what is removed. The Board reviews a small selected fraction of decisions after the fact. Neither the Board nor the public can see the machinery that applies the rules.
In 2016 and 2017, Facebook was used to spread anti-Rohingya hate speech in Myanmar. UN investigators called the platform “a useful instrument for those seeking to spread hate” and described the company’s response as slow and ineffective; Facebook had too few content moderators who understood the Myanmar language. The platform amplified content it was not equipped to read. No accountability mechanism existed to stop, challenge, or even document the pattern in real time. The company acknowledged it was too slow to act.
The killing was done by the military. Within a year of the campaign it launched in August 2017, nearly 725,000 Rohingya had fled to Bangladesh, and the UN investigators called an estimate of up to 10,000 deaths conservative. Facebook did not cause that campaign, but it carried the hate that preceded it, and when a platform cannot read the posts it amplifies, an inability to moderate starts to have the same effect as a decision not to.
Section 230 and the immunity gap
Section 230 of the Communications Decency Act grants platforms broad immunity from liability for content posted by third parties and for good-faith content moderation decisions. The result is a legal structure in which platforms carry little liability for what their users post and no public accountability for the rules they apply to it. That immunity was designed for an internet that no longer exists, one of small services hosting user content rather than entities that make binding governance decisions for billions of people daily. Current efforts to reform or repeal Section 230 are animated by the same structural problem this article identifies: the mismatch between the power these systems exercise and the accountability mechanisms that currently attach to that power. Whether Section 230 reform is the right lever, and what unintended consequences it would create, is a separate argument.Four things formal regulation requires
Lay the four criteria of legitimate regulation against the bail score, the credit score, the hiring filter and the feed.
Start with transparency. The APA’s notice-and-comment requirement exists because secret regulations are incompatible with the kind of governance that requires consent. You cannot meaningfully consent to or contest a rule you’ve never seen. A bail algorithm scores a defendant 8. That score weighs on whether the person sleeps at home or in a cell while awaiting trial. The rule generating the 8 is not disclosed. It can be inferred, incompletely and imperfectly, by someone with access to large datasets and the statistical tools to reverse-engineer it, which is a long way from the published rule that notice-and-comment demands. When a content moderation algorithm removes a post and the user receives a notification citing “violence” or “harassment,” the notification doesn’t describe which standard was applied, how its threshold was set, or at what level of confidence the algorithm triggered. The rule can only be inferred from the accumulated pattern of enforcement outcomes, one removal at a time.
Transparency is the prerequisite for everything else, because no one can confront and contest a legal standard they cannot see. That right to confront and contest is why defendants have the right to see the charges against them, and why regulated parties have the right to participate in rulemaking. Eric Loomis’s lawyers asked for the methodology used to constrain their client’s liberty. The court said no. The iTutorGroup applicants who were auto-rejected never knew that a rule had been applied to them, let alone what the rule was. Facebook users who receive a removal notification learn which Community Standard they broke and nothing about where its line was drawn. Due process requires a chance to respond to something specific.
Democratic legitimacy follows. Legislatures are elected. Agencies are delegated power by legislatures, supervised by elected officials, and subjected to the APA’s procedural requirements. Their decisions can be challenged in federal court. Private companies deploying algorithms that govern access to credit, employment, liberty, and expression for hundreds of millions or billions of people have no corresponding accountability structure. They are accountable to shareholders, not to the populations they govern. When a company’s product decisions become governance decisions, an accountability structure built for products is no longer adequate.
Then there is liability, which makes the other three enforceable. Tort law, product liability, administrative accountability: these exist to assign responsibility when governance causes harm. For algorithmic harm at population scale, none of them work adequately. Class certification under federal civil procedure requires showing questions of law or fact common to the whole class. When the rule is proprietary and the harm is spread across millions of small individual injuries, from rejected loans to false-positive bail scores to suppressed posts, the common-question requirement is structurally difficult to satisfy without access to the model itself. The harm in aggregate is enormous. The individual instances remain invisible to legal mechanisms designed for individual adjudication.
Return to the iTutorGroup settlement: less than $1,825 per person, on average, for a lost employment opportunity. That number doesn’t approximate the actual harm. The decree’s training and monitoring bind one company; for anyone else weighing the risk, the price on the record is the cash. Scale it up: if an algorithm rejects 20,000 applicants on discriminatory grounds and each settles at that same rate, the potential liability is under $36.5 million. For a company whose revenue runs in the hundreds of millions, that’s a cost of doing business.
The four depend on one another: without transparency, due process has nothing specific to contest, and without either, liability is hard to establish. Algorithmic regulatory power currently has none of the four in full.
The response and its limits
The EU AI Act entered into force in August 2024. It is the most comprehensive regulatory framework built to address algorithmic authority so far.
What it does is substantial. The Act sorts AI systems by risk tier. Its high-risk list, which carries the full compliance burden, explicitly names credit scoring, hiring and performance evaluation, and criminal justice: tools that assess a person’s risk of offending or reoffending, and AI that assists judges in applying the law. For these systems, the Act mandates conformity assessments before deployment, human oversight, and transparency to the people affected; public bodies and credit scorers must also carry out a fundamental rights impact assessment. People subject to decisions by most of these listed systems gain a right to explanation where a decision produces legal effects or significantly adverse impacts on their health, safety, or fundamental rights. Breaching the high-risk rules can cost up to €15 million or 3 percent of global annual turnover. Those obligations were originally due to apply in August 2026; a later amendment, already in force, pushed the high-risk requirements to December 2027. The explanation right kept its August 2026 date on paper, but it attaches to high-risk systems whose classification rules were deferred with the rest.
Those are not symbolic requirements. COMPAS, if deployed in Europe as one of the listed high-risk systems, would face mandatory conformity assessment and fundamental rights impact assessment before use. HireVue’s video-assessment system, if used for hiring decisions affecting EU-based workers, would require human oversight and explanation rights.
In the United States, progress is more piecemeal. New York City has required annual independent bias audits, with published results, for automated employment decision tools used in hiring or promotion for New York City-based positions since 2023. Since January 2026, Illinois has prohibited the use of AI in employment decisions in ways that result in unlawful discrimination, barred zip codes as proxies for protected characteristics, and required employers to give notice when AI is used in hiring or employment decisions. Colorado replaced its never-implemented AI law with a narrower one on automated decision-making in consequential decisions, which takes effect in January 2027: notice before use, a plain-language explanation after an adverse decision, correction of inaccurate data, and the right to ask for meaningful human reconsideration.
These are real advances on a problem that wasn’t seriously regulated five years ago, but they share a structural limitation. Apart from a few outright bans, such as the AI Act’s prohibition on predicting a person’s risk of committing a crime solely from profiling or personality traits, and Illinois’s ban on discriminatory use, they work mainly through design standards, documentation, audit and disclosure. They require that someone, whether an auditor, a regulator or a compliance team, assess whether the system is fair. At best they give the individual governed by the system an explanation, a correction of bad data, or a second look by a human. None of them lets that person inspect the rule itself. The AI Act’s right to explanation means that a person subject to a decision by most of the listed high-risk systems, where that decision has legal effects or significantly adverse impacts on their health, safety, or fundamental rights, must receive an explanation of it. It does not mean the person can examine the model weights, the training data composition, or the algorithmic methodology. The person learns why one decision went against them and gets no way to audit the system that made it.
The structural lag is a separate problem. The EU AI Act’s high-risk framework was built around AI systems as they existed roughly between 2020 and 2022. By the time high-risk compliance is required, the systems being regulated have gone through multiple generational changes. Regulations are written over years; algorithmic systems iterate over months. The mismatch lies in the pace of lawmaking set against the pace of the technology; better drafting would not have closed it.
The GDPR’s experience should temper expectations. Its Article 22, applicable since 2018, gave people in the EU the right not to be subject to solely automated decisions with significant effects, unless the decision is needed for a contract, authorized by law or explicitly consented to; companion articles added rights to meaningful information about the logic involved. On paper it is strong. EU guidance says a company cannot escape it by “fabricating human involvement,” and the EU’s top court has widened it: in 2023 it held that a credit score can itself count as an automated decision, and in 2025 that a person refused credit is entitled to an explanation of the procedure and principles actually applied to them, with trade-secret objections weighed by a court or regulator rather than accepted on the company’s say-so. In practice it took seven years and repeated litigation to get there. The AI Act’s own explanation rights deserve the same caution.
GDPR Article 22 and learned skepticism
GDPR Article 22, which has applied since 2018, gives people in the EU the right not to be subject to purely automated consequential decisions, with exceptions for contracts, legal authorization and explicit consent; the regulation's transparency articles add rights to meaningful information about the logic. In practice, its impact has been limited. The "solely automated" threshold invited a workaround, a thin layer of human sign-off, even though EU guidance says token involvement does not count. Explanation rights were contested for years until the EU Court of Justice ruled in 2025 that people are owed an explanation of the procedure and principles actually applied in their own case, not merely a generic description or the raw algorithm. The EU AI Act's Article 86 right-to-explanation provision was drafted with Article 22's failures in mind. If Article 22 is a guide, enforcement and the courts will decide whether the new provision achieves more than the old one did.The accountability paradox
The legal mechanisms we have for challenging governance were built to handle individual acts by identifiable decision-makers operating under stated rules. A statute is challenged by one person in one case. An agency regulation is contested through notice-and-comment or APA litigation. An administrative decision is appealed to a reviewing tribunal. A discriminatory official can be deposed. These mechanisms assume a specific structure: a singular decision, a stated rule, a challenger who can identify the harm, a decision-maker who can be questioned.
Algorithmic regulatory power breaks every one of those assumptions at once. To challenge an algorithmic decision in court, you typically need some mix of three things: the rule that generated it, which is proprietary and usually obtainable only through discovery under a protective order; a counterfactual showing what a different outcome would have required, hard to build without model access or a full explanation from the company; and, for a discrimination claim, evidence of a pattern, which requires population-level data that the affected individual almost certainly doesn’t have.
The Wisconsin Supreme Court’s answer was to require written warnings. A warning in the presentence report names the problem, and the defendant may still contest the score with other evidence, but nothing in it lets him examine the calculation.
Democratic authorization is absent at every point where it might have been established. Meta’s Oversight Board cannot access the algorithm it nominally oversees. The CFPB can demand a lender’s model in an investigation, but the borrower it scored is owed only a list of reasons. The Wisconsin Supreme Court declined to order COMPAS opened. At each moment where accountability could have been imposed, the proprietary interest was treated as a barrier that governance would not cross. No law required that deference, and none of them forced the question.
What makes algorithmic systems worth deploying at scale is consistency across millions of cases, speed, and the elimination of human discretion that produces its own discriminatory failures. Those are exactly the features that make them illegible to the accountability mechanisms law has developed. A biased loan officer or a biased sentencing judge is accountable in ways a biased algorithm isn’t, and the legal system has not yet worked out how to hold the algorithm to the same account.
The frameworks being built now, from the EU AI Act to the laws in NYC, Colorado and Illinois, audit the system and explain the decision. They do not open the model to the person it scores.
Algorithmic systems can produce systematically discriminatory outcomes for years before any challenge is possible, because the challenge requires access the challenger isn’t given. iTutorGroup discriminated against hundreds of applicants for however long the system was deployed before the EEOC filed suit. HireVue scored job candidates partly on facial expressions for years before dropping the feature as not worth its marginal value. COMPAS has been used in sentencing in multiple states for more than a decade. Where a challenge comes, it comes after the harm, and the people harmed in the meantime are not made whole by eventual resolution.
Return to Wisconsin
Read the cautions the Wisconsin Supreme Court required in every COMPAS presentence report, in the court’s own words: “The proprietary nature of COMPAS has been invoked to prevent disclosure of information relating to how factors are weighed or how risk scores are determined.” “Because COMPAS risk assessment scores are based on group data, they are able to identify groups of high-risk offenders — not a particular high-risk individual.” “Some studies of COMPAS risk assessment scores have raised questions about whether they disproportionately classify minority offenders as having a higher risk of recidivism.” “A COMPAS risk assessment compares defendants to a national sample, but no cross-validation study for a Wisconsin population has yet been completed. Risk assessment tools must be constantly monitored and re-normed for accuracy due to changing populations and sub-populations.” And: “COMPAS was not developed for use at sentencing, but was intended for use by the Department of Corrections in making determinations regarding treatment, supervision, and parole.”
These are the court’s own findings about what hasn’t been tested and what might be wrong. Having listed them, it still allowed the score at sentencing, within the limits it had set. The warning tells the sentencing judge that this tool might discriminate, hasn’t been validated for the state, wasn’t built for sentencing and works by rules no one in the courtroom can inspect.
The judge may discount the score, and the defendant may contest it with other evidence. But no one, not the judge, the defendant or the appellate court, gets a way to open the calculation itself.
Gen AI Disclaimer
Some contents of this page were generated and/or edited with the help of a Generative AI.
Media
Key Sources and References
Criminal justice and COMPAS
Julia Angwin, Jeff Larson, Surya Mattu, Lauren Kirchner, “Machine Bias,” ProPublica, May 23, 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
ProPublica, “How We Analyzed the COMPAS Recidivism Algorithm,” May 23, 2016. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
“ProPublica Responds to Company’s Critique of Machine Bias Story,” ProPublica, July 29, 2016. https://www.propublica.org/article/propublica-responds-to-companys-critique-of-machine-bias-story
equivant, “Who is equivant?” FAQ (Northpointe, CourtView and Constellation united as equivant on January 9, 2017), archived copy. https://web.archive.org/web/20221201174316/https://www.equivant.com/faq/
State v. Loomis, 2016 WI 68, 371 Wis. 2d 235, 881 N.W.2d 749 (Wis. July 13, 2016), Caselaw Access Project (Harvard Law School Library). https://case.law/caselaw/?reporter=wis-2d&volume=371&case=0235-01
Loomis v. Wisconsin, No. 16-6387, 137 S. Ct. 2290 (cert. denied June 26, 2017), Supreme Court docket. https://www.supremecourt.gov/search.aspx?filename=/docketfiles/16-6387.htm
Alexandra Chouldechova, “Fair prediction with disparate impact: A study of bias in recidivism prediction instruments,” Big Data, vol. 5, no. 2, pp. 153–163, 2017. DOI: 10.1089/big.2016.0047 (preprint on arXiv). https://arxiv.org/abs/1610.07524
Jon Kleinberg, Sendhil Mullainathan, Manish Raghavan, “Inherent trade-offs in the fair determination of risk scores,” Proceedings of the 8th Innovations in Theoretical Computer Science Conference (ITCS 2017), LIPIcs vol. 67, pp. 43:1–43:23. https://drops.dagstuhl.de/storage/00lipics/lipics-vol067-itcs2017/LIPIcs.ITCS.2017.43/LIPIcs.ITCS.2017.43.pdf
Law, regulation and accountability
Lawrence Lessig, Code and Other Laws of Cyberspace. Basic Books, 1999.
Frank Pasquale, The Black Box Society. Harvard University Press, 2015.
Frank Pasquale, “From Territorial to Functional Sovereignty: The Case of Amazon,” LPE Project, December 6, 2017. https://lpeproject.org/blog/from-territorial-to-functional-sovereignty-the-case-of-amazon/
Administrative Procedure Act, 5 U.S.C. § 551 (definitions) and § 553 (rulemaking), Cornell Legal Information Institute. https://www.law.cornell.edu/uscode/text/5/551 · https://www.law.cornell.edu/uscode/text/5/553
“State action,” Wex legal encyclopedia, Cornell Legal Information Institute. https://www.law.cornell.edu/wex/state_action
Federal Rules of Civil Procedure, Rule 23 (class actions), Cornell Legal Information Institute. https://www.law.cornell.edu/rules/frcp/rule_23
Credit and hiring
myFICO, “What’s in my FICO Scores?” https://www.myfico.com/credit-education/whats-in-your-credit-score
CFPB, DOJ Civil Rights Division, EEOC and FTC, “Joint Statement on Enforcement Efforts Against Discrimination and Bias in Automated Systems,” April 2023. https://www.ftc.gov/system/files/ftc_gov/pdf/EEOC-CRT-FTC-CFPB-AI-Joint-Statement%28final%29.pdf
Consumer Financial Protection Bureau, Consumer Financial Protection Circular 2022-03 (May 26, 2022) — withdrawn May 12, 2025 (“Interpretive Rules, Policy Statements, and Advisory Opinions; Withdrawal,” 90 FR 20084). https://www.federalregister.gov/documents/2025/05/12/2025-08286/interpretive-rules-policy-statements-and-advisory-opinions-withdrawal
Consumer Financial Protection Bureau, Equal Credit Opportunity Act (Regulation B), final rule, 91 FR 21620 (April 22, 2026; effective July 21, 2026). https://www.federalregister.gov/documents/2026/04/22/2026-07804/equal-credit-opportunity-act-regulation-b
National Fair Housing Alliance et al. v. Consumer Financial Protection Bureau et al. (D.D.C., filed May 27, 2026), challenge to the Regulation B final rule; summarized in Ballard Spahr, Consumer Finance Monitor, June 2, 2026. https://www.consumerfinancemonitor.com/2026/06/02/advocacy-groups-and-private-companies-file-lawsuit-against-revised-ecoa-rule/
Texas Department of Housing and Community Affairs v. Inclusive Communities Project, Inc., 576 U.S. 519 (2015). https://www.law.cornell.edu/supremecourt/text/13-1371
“HireVue drops facial monitoring amid A.I. algorithm audit,” Fortune, January 19, 2021. https://fortune.com/2021/01/19/hirevue-drops-facial-monitoring-amid-a-i-algorithm-audit/
EPIC, “In re HireVue,” FTC complaint, November 6, 2019. https://archive.epic.org/privacy/ftc/hirevue
EEOC, “iTutorGroup to Pay $365,000 to Settle EEOC Discriminatory Hiring Suit,” September 11, 2023. https://www.eeoc.gov/newsroom/itutorgroup-pay-365000-settle-eeoc-discriminatory-hiring-suit
Platforms and speech
Meta Platforms, Inc., Second Quarter 2026 Results, Exhibit 99.1 to Form 8-K, filed July 29, 2026. https://www.sec.gov/Archives/edgar/data/1326801/000162828026050596/meta-06302026xexhibit991.htm
United Nations, “Population” (Global Issues). https://www.un.org/en/global-issues/population
YouTube Official Blog, Press. https://blog.youtube/press/
TikTok Newsroom, “Thanks a billion!” September 27, 2021. https://newsroom.tiktok.com/1-billion-people-on-tiktok?lang=en
“Musk Says X Has 600 Million Monthly Active Users,” Bloomberg Law, 2024. https://news.bloomberglaw.com/tech-and-telecom-law/musk-says-x-has-600-million-monthly-active-users
Meta Transparency Center, Community Standards. https://transparency.meta.com/policies/community-standards/
European Commission, “Digital Services Act: Questions and answers” and “Out-of-court dispute settlement bodies under the Digital Services Act.” https://digital-strategy.ec.europa.eu/en/faqs/digital-services-act-questions-and-answers · https://digital-strategy.ec.europa.eu/en/policies/dsa-out-court-dispute-settlement
Grievance Appellate Committee, Government of India (appeals against intermediaries’ grievance-officer decisions under Rule 3A of the Information Technology (Intermediary Guidelines and Digital Media Ethics Code) Rules, 2021). https://gac.gov.in/
Faiza Patel and Julian Melendi, “Meta’s Oversight Board Needs Access to Facebook’s Algorithms to Do Its Job,” Brennan Center for Justice, June 17, 2024. https://www.brennancenter.org/our-work/analysis-opinion/metas-oversight-board-needs-access-facebooks-algorithms-do-its-job
Report of the Independent International Fact-Finding Mission on Myanmar, UN Doc. A/HRC/39/64, September 12, 2018; and Report of the Detailed Findings of the Independent International Fact-Finding Mission on Myanmar, UN Doc. A/HRC/39/CRP.2, September 17, 2018. https://www.ohchr.org/sites/default/files/Documents/HRBodies/HRCouncil/FFM-Myanmar/A_HRC_39_64.pdf · https://www.ohchr.org/sites/default/files/Documents/HRBodies/HRCouncil/FFM-Myanmar/A_HRC_39_CRP.2.pdf
Communications Decency Act, Section 230, 47 U.S.C. § 230, Cornell Legal Information Institute. https://www.law.cornell.edu/uscode/text/47/230
New rules: EU and US
Regulation (EU) 2024/1689 of the European Parliament and of the Council (EU AI Act), entered into force August 1, 2024; Annex III and Articles 5, 27, 86 and 99 via the AI Act Explorer. https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng · https://artificialintelligenceact.eu/annex/3/ · https://artificialintelligenceact.eu/article/5/ · https://artificialintelligenceact.eu/article/27/ · https://artificialintelligenceact.eu/article/86/ · https://artificialintelligenceact.eu/article/99/
Regulation (EU) 2026/1744 (Digital Omnibus on AI), published in the Official Journal July 24, 2026; summarized in Cooley, “Digital AI Omnibus Delays Key Deadlines, Introduces New Rules,” August 3, 2026. https://cdp.cooley.com/digital-ai-omnibus-delays-key-deadlines-introduces-new-rules/
Regulation (EU) 2016/679 (General Data Protection Regulation), Articles 13–15 and 22, EUR-Lex. https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng
Article 29 Data Protection Working Party, Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679 (WP251rev.01), endorsed by the EDPB. https://ec.europa.eu/newsroom/article29/items/612053
Court of Justice of the EU, C-634/21 SCHUFA Holding (Scoring), December 7, 2023, press release 186/23. https://curia.europa.eu/jcms/upload/docs/application/pdf/2023-12/cp230186en.pdf
Court of Justice of the EU, C-203/22 Dun & Bradstreet Austria, February 27, 2025, press release 22/25. https://curia.europa.eu/jcms/upload/docs/application/pdf/2025-02/cp250022en.pdf
NYC Local Law 144 of 2021 (automated employment decision tools), effective January 1, 2023 (§ 2); enforced by the NYC Department of Consumer and Worker Protection from July 5, 2023. https://legistar.council.nyc.gov/LegislationDetail.aspx?GUID=B051915D-A9AC-451E-81F8-6596032FA3F9&ID=4344524 · https://www.nyc.gov/site/dca/about/automated-employment-decision-tools.page
Illinois Public Act 103-0804, amending the Illinois Human Rights Act to address the use of artificial intelligence in employment decisions, signed August 9, 2024, effective January 1, 2026; summarized in Hunton Andrews Kurth, “Illinois Enacts New Law Regulating Employer Use of Artificial Intelligence,” 2024. https://www.hunton.com/hunton-employment-labor-perspectives/illinois-enacts-new-law-regulating-employer-use-of-artificial-intelligence
Colorado SB 26-189, Automated Decision-Making Technology, signed May 14, 2026, effective January 1, 2027; summarized in Taft, “Colorado Act Legislation: The AI Strikes Back,” May 27, 2026. https://leg.colorado.gov/bills/sb26-189 · https://www.taftlaw.com/news-events/law-bulletins/colorado-act-legislation-the-ai-strikes-back/
