We often hear about government misuse of face recognition technology (FRT) and how it can derail a person’s life through wrongful arrests and other harms. Despite mounting evidence, government agencies continue to push face recognition systems on communities across the United States. Key to this effort are the corporate makers and sellers who market this technology as reliable, accurate, and safe – often by pointing to their products’ scores on government-run performance tests.

All of this might tempt policymakers to believe that the safety and civil rights problems of facial recognition can be solved by mandating a certain performance score or grade. However, relying solely on test scores risks obscuring deeper problems with face recognition while overstating its effectiveness and real-life safety.

Many facial recognition systems are tested by the federal National Institute of Standards and Technology (NIST). In one of their tests, NIST uses companies’ algorithms to try and search for a face within a large “matching database” of faces. In broad strokes, this test appears to resemble how police use face recognition today, feeding an image of a single unknown person’s face into an algorithm that compares it against a large database of mugshot or driver’s license photos and generates suggested images, paired with numbers that represent estimates of how similar the images are.

These and other tests can reveal disturbing racial disparities. In their own groundbreaking research, computer scientists Dr. Joy Buolamwini and Dr. Timnit Gebru tested several prominent gender classification algorithms, and found that those systems were less likely to accurately classify the faces of women with darker complexions. Following that, the ACLU of Northern California performed its own test of Amazon’s facial recognition software, which falsely matched the faces of 28 members of Congress with faces in a mugshot database, with Congressmembers of color being misidentified at higher rates. Since then, additional testing by NIST and academic researchers indicates that these problems persist.

While testing of facial recognition for accuracy and fairness across race, sex, and other characteristics is critical, the tests do not take full account of practical realities. There is no laboratory test that represents the conditions and reality of how police use face recognition in real world-scenarios. For one, testing labs are not going to have access to the exact “matching database,” the particular digital library of faces on mugshots, licenses, and surveillance photos, that police in a specific community search through when they operate face recognition. And tests cannot account for the full range of low-quality images from surveillance cameras (and truly dubious sources) that police feed into these systems, or the trouble police have when visually reviewing and choosing from a set of possible matches produced by the technology.

Parks v. McCormac

On January 29, 2024, the ACLU and the ACLU of New Jersey filed an amicus brief in the U.S. District Court for the District of New Jersey in support of

In response to these real concerns, vendors routinely hold up their performance on tests in their marketing to government agencies as evidence of facial recognition’s reliability and accuracy. Lawmakers have also sought to legislate performance scores that set across-the-board accuracy or error-rate requirements for facial recognition algorithms used by police that would allow police to use FRT systems that clear these requirements. This approach would be misguided.

It is easy to be misled by performance scores. Imagine a requirement that police can only use systems that produce an overall true positive rate, a measure of how often the results returned by a FRT system include a match for the person depicted in the probe image when there is a matching image in the database, above 98 percent in testing. At first glance, that might sound like a pretty strong requirement — but a closer look reveals a very different story.

For one, police typically configure and customize facial recognition systems to return a list of multiple results, sometimes as many as hundreds of results. Think of this as a ‘digital lineup.’ In NIST testing, if at least one of the results returned is a match for the probe image, the search is considered successful and counted as part of the true positive rate metric. But even when this happens in practice — which certainly isn’t always the case — there is no guarantee that police will select the true match rather than one of the other results. True matches in testing might be crowded out by false matches in practice because of these police-created ‘digital lineups.’ This alone makes it difficult to choose one universal performance score that can be applied to many different FRT systems.

Let’s look at another metric called the false positive rate, which assesses how often a FRT search will return results when there is no matching image in the database. Breaking results down by race, the same algorithm that produces the 98 percent true positive rate overall can also produce a false positive rate for Black men several times the false positive rate for white men — and an even higher false positive rate for Black women. This example is not merely a hypothetical: in NIST testing, many algorithms have exhibited this pattern. (1) Other recent NIST testing also shows algorithms produced false positive rates tens or hundreds of times higher for females older than 65 born in West African countries than for males ages 20-35 born in Eastern European countries. (2)

By only considering the true positive rate, we miss these extreme disparities, which can lead to devastating consequences. Across the United States, police are arresting people based on false matches and harming people like Nijeer Parks, a Black man who police in New Jersey falsely arrested and held in jail for ten days because police trusted the results of face recognition, overlooking obvious exonerating evidence. Human mis-reliance on face recognition is already a problem; focusing on performance scores might make things worse.

Lawmakers should know that a facial recognition algorithm’s performance on a test cannot be easily or quickly generalized to make broad claims about whether a facial recognition algorithm is safe. Performance scores are not an easy fix to the harms that are resulting from the use of face recognition systems, and they of course don’t account for humans that will inevitably be in the loop.

As the ACLU explained in its recent public comment to the Biden Administration, the problems of facial recognition run deep and beyond the software itself. Facial recognition is dangerous if it’s inaccurate — a problem that testing aims to address — but also dangerous even if it could hypothetically be perfectly accurate. In such a world, governments could use face surveillance to precisely track us as we leave home, attend a protest, or take public transit to the doctor’s office. This is why policymakers in an expanding list of U.S. cities and counties have decided to prohibit government use of face recognition. And it’s why ACLU supports a federal moratorium on its use by law and immigration enforcement agencies.

Conversations about the shortcomings of performance scores are important, but instead of trying to find some magic number, policymakers should focus on how any use of facial recognition can expand discriminatory policing, massively expand the power of government, and create the conditions for authoritarian control of our private lives.

On October 24, 2022 at 2 a.m., 27-year-old Joshua Wesley called a crisis help line from his home in Washington County, Oregon, just west of Portland. He was having suicidal thoughts and knew that he needed professional help. But instead of receiving a mental health provider as specifically requested, he encountered a group of armed police officers at his door. This response not only deprived Wesley of the immediate psychiatric care that he needed, but it also led to him being arrested and seriously injured by the responding officer. He ultimately spent two weeks in the hospital, and six months in jail.

Wesley told us that he felt that he needed qualified professionals to console him, talk him down, and give him solutions. But the officers that showed up made the situation worse by simply trying tried to put him in handcuffs and cart him off.

Joining forces with the ACLU, Disability Rights Oregon, the ACLU of Oregon, and the law firm Shepherd Mullin, Wesley is a plaintiff in a recently filed lawsuit against Washington County and the local 911 dispatch center. The lawsuit asserts that the county’s emergency response system discriminates against people with mental health disabilities and exposes them to risk of serious harm, including injury, arrest, and incarceration. Wesley said that he joined the case because he believes strongly in helping out others facing similar struggles.

Washington County has a history of inappropriately responding to mental health crises. In 2022, police officers were dispatched to 100 percent of the calls coded as “behavioral health incidents” in Washington County. The county does have mobile crisis teams comprised exclusively of mental health clinicians, the sole non-police response available there. But, while the mobile crisis teams are intended to be available 24/7, in practice, they’re underfunded, not connected with the emergency dispatch system, and often unavailable — especially at night, when many mental health crises occur.

Police response to mental health crises can be dangerous and even deadly. Police officers are not qualified mental health professionals and should not be expected to assess and treat people in crisis. Beyond that, police presence may actually make mental health symptoms worse, triggering anxiety and paranoia. Most alarming of all, it is estimated that people with untreated mental illness are 16 times more likely than others to be killed by the police during an encounter.

911: Reimagining a System that Defaults to Dispatching Police

Emergency response systems must be revamped to equip 911 call-takers to dispatch non-police first responders.

That’s what nearly happened in Wesley’s case. Instead of being provided with the care he was seeking — on-site psychiatric assessment and treatment — he was placed under a “police officer hold,” a form of involuntary detention, and transported to a hospital via ambulance. Wesley was not treated or stabilized during transport and his symptoms worsened. At the hospital, Wesley was still suicidal and he attempted to take an officer’s firearm to use on himself. During the incident, the officer stabbed Wesley several times, resulting in serious injuries to his chest, stomach, and head.

The damage to Wesley’s body serves as a constant reminder of the incident. The scars left from the incident demonstrate that there could have been other ways to deal with the situation, Wesley told us.

Wesley then spent two weeks in the hospital recovering. During this time, his repeated requests for mental health assistance and therapy were denied. He remained handcuffed to his bed and kept under near-constant police surveillance. Wesley felt that the doctors stopped looking at him as a patient who needed help and treatment to heal, but rather, as a criminal.

After being released from the hospital, Wesley faced criminal charges arising from the altercation with the officer. He spent six months in jail, missing the birth of his first and only son. He also missed the holidays and time with his family at a time of great strife.

Ultimately, it took months for Wesley to receive the psychiatric help that he first sought in October.

When someone in Washington County experiences a physical health crisis, like a heart attack or a severe allergic reaction, they can call 911 and expect a response from a qualified medical professional, like an EMT or paramedic. The same cannot be said, however, for someone experiencing a mental health crisis.

The lawsuit explains how this discrepancy violates the Americans with Disabilities Act and Rehabilitation Act. Mental health crises demand a mental health response — not a police response — because they are, at their core, health emergencies.

Experts agree that mental health emergencies should be addressed by mental health professionals, not the police. As part of their recommended best practices, the Substance Abuse and Mental Health Services Administration (SAMHSA) proposes a three-tiered system that includes a crisis call center, mobile crisis teams, and stabilization centers for walk-ins and drop-offs. SAMHSA also noted that responding with police is “unacceptable and unsafe,” a view that the National Alliance on Mental Illness shares.

As a result of Washington County’s inappropriate response to mental health crises, it discriminates against people with mental health disabilities on a daily basis. . This lawsuit seeks to improve its mental healthcare system. Possible solutions include fully funding mobile crisis response teams that can bring care and support to the people who need it, when they need it.

Washington County isn’t the only jurisdiction with a system in need of reform. Justice Department investigations have found similar discrimination in Louisville and Minneapolis, stating that relying on police as mental health first responders causes “real harm in the form of trauma, injury, and death to people experiencing behavioral health issues.”

Wesley hopes that this case brings widespread attention to an issue that impacts many lives on a daily basis. People with mental health disabilities are harmed both because of a failed response to mental health crises , and because many people with mental health disabilities don’t want to call for help out of fear of an armed police response. Wesley sees a need nationwide for an important reckoning for how jurisdictions respond to mental health crises. Counties and other locales should be looking at their systems and asking, “Is our system for mental health crisis response fair? Is it safe? Is it right?”

How jurisdictions answer these questions could have a major impact on the care and support people with mental health disabilities receive while in crisis. We must not allow discriminatory practices that cause real harm and death to go unchecked.