About the Data

Police Scorecard Project Methodology

Overview

The California Police Scorecard utilizes data on a range of policing-related issues to evaluate how each police department interacts with, and the extent to which officers are held accountable to, the communities they serve. The indicators included in this scorecard were selected based on a review of the research literature, input from activists and experts in the field, and a review of existing publicly available datasets on policing in California. The scorecard is designed to help communities, researchers, police leaders and policy-makers take informed action to reduce police use of force and improve accountability and public safety in their jurisdictions.

How We Collected the Data

The California Police Scorecard integrates data on police deadly force, civilian complaints and arrests that were obtained from official databases including the California Department of Justice’s OpenJustice database, UCR and the California Monthly Arrests and Citation Register. Data on police use of force, use of force complaints, and police policy manuals were obtained directly from police agencies via public records requests. The full list and descriptions of each data source used in this project can be found in Appendix A below. We’ve also uploaded the source data used for the scorecard, including additional columns of data that could be the subject of future research, here. 

This project will be updated over time to include additional data on policing as these data are made available by law enforcement agencies, including data on police stops and searches, police personnel, civil asset forfeiture, issues occurring in police custody, response times, police militarization, police discipline and other issues. These data will be released and integrated into the scoring system in the coming months. If you have information or ideas to improve our database, email sam@thisisthemovement.org.

 

How We Calculated Agency Scores and Grades:

The scores or “grades” attributed to police agencies in our analysis should be interpreted as relative scores, showing how each agency compares to other agencies within the state across three areas of policing: police violence, police accountability, and an evaluation of each agency’s approach to policing (for example, the extent to which an agency focuses on arresting people for low-level offenses or solving serious crimes).

Here are the factors that were used to calculate each agency’s score:

Police Violence Score:

The police violence score takes into account how often police used higher levels of force against people including any time police used a firearm, taser, baton, OC spray, impact projectile or stranglehold on a civilian.[1] Since the vast majority of use of force incidents occurred during an arrest, consistent with previous research we benchmarked use of force incidents against the total number of arrests made[2] and then factored in the severity of force used and whether the people force was used against were armed into each police department’s police violence score.

Racial disparities in arrests and deadly force were also factored into each department’s police violence score. Since there are similar rates of illicit drug use (i.e. “offending rates”) among most racial and ethnic groups, disparities in which groups police arrest for drug possession suggest a biased approach to law enforcement. As such, we used data on racial disparities in drug possession arrests from the California Monthly Arrests and Citations Register as a proxy for estimating racial bias in arrests. Drug possession arrest rates between Black and White populations and between Latinx and White populations were converted to Standard Scores (z-scores), averaged together into one combined score, and then converted to percentile. To calculate a racial bias score for police use of deadly force, we calculated the baseline probability that police use deadly force against a White person during an arrest.[3] We then multiply the arrest-based probability by the total number of Black and Latinx people arrested by each police department[4], and calculate the log number of actual deadly force incidents for each community above the baseline expectation if the victims had been White. For each group, we used the most severe of the two disparities (either Black-White or Latinx-White, whichever had the largest disparity)[5] and converted that to a percentile. Then, the percentiles of racial bias in drug arrests and of racial bias in deadly force per arrest were averaged together to produce a combined percentile of racial bias in arrests and deadly force, which was included as one of the four factors determining each department’s police violence score. Each police department’s racial disparities and resulting racial bias scores are listed in Appendix D below. In summary, the following formula was used to calculate each department’s police violence score:

( Percentile Less Lethal Force Used per Arrest + Percentile Deadly Force Used per Arrest + Percentile Unarmed Civilians Killed or Seriously Injured + Percentile Racial Bias in Arrests and Deadly Force ) / 4

Police Accountability Score

The police accountability score evaluates the extent to which investigations into civilian complaints of police misconduct result in a sustained finding of misconduct against the officers involved, which is usually the first step to imposing disciplinary consequences. The score accounts for both how police departments compare with one another in terms of their civilian complaint sustain rate, the overall proportion of complaints sustained, and the severity of the misconduct allegations that were sustained against officers (for example, complaints alleging police committed a criminal offense were weighted more heavily than vs non-criminal allegations). The following formula was used to calculate this score:

( Percentile Civilian Complaints Sustained + Percent Criminal Complaints Sustained ) / 2

Approach to Police Score

The approach to policing score evaluates the extent to which police departments focus on arresting people for low-level offenses or focusing on solving serious crimes. The score accounts for the proportion of the most serious crimes - homicide - that an agency solves as well as how each agency compares in terms of its approach to the lowest level offenses, measured by misdemeanor arrest rates per population. Misdemeanor arrest rates per population have been cited in previous research as an effective measure of whether a department has adopted a “ broken windows” or “zero tolerance” policing approach; an approach that has not been proven to improve public safety but that instead often exposes communities, especially marginalized communities, to increased policing, arrest and incarceration (for additional reading, see here, here and here). The following formula was used to calculate this score:

( Percentile Misdemeanor Arrests per Population + Percent Homicides Cleared ) / 2

Review of Police Department Policies and Police Union Contracts

In addition to collecting data on police outcomes (use of force, accountability, arrests), this project also includes information on public policies that have been shown in the research literature to impact these outcomes. While the quality of a police department’s policies were not one of the factors used to calculate their score, this information was collected to provide greater insight into the ways in which policies and systems either encourage or discourage police violence and accountability.

Specifically, the full policy manual and police union contract of each police agency was obtained and reviewed to identify the extent to which they contained language that either restricted the use of force or that imposed barriers to investigating and holding police accountable for misconduct. Previous research has shown a significant link between the restrictiveness of a police department’s use of force policy and rates of police violence per arrest. Moreover, research shows the categories of policy provisions we identified within police union contracts to be associated with higher levels of police abuse and police killings of unarmed civilians. The full policy manuals for each of the agencies can be found here. For more information on the methodology used to review these policies, visit Campaign Zero’s Police Use of Force project and Police Union Contract project. We also reviewed the statewide Law Enforcement Officer Bill of Rights law and included the barriers to accountability within this law that impact every agency within the state.


Appendix A:

Data Sources:

Deadly Force Incidents

DEFINITION: ALL FIREARMS DISCHARGES AND ALL USE OF FORCE INCIDENTS RESULTING IN THE DEATH OR SERIOUS INJURY OF A CIVILIAN IN 2016 AND 2017

Government Code section 12525.2 mandates law enforcement agencies in California to report annually on all:

  1. incidents involving the shooting of a civilian by a police officer.
  2. incidents involving the shooting of a police officer by a civilian.
  3. incident in which the use of force by a police officer against a civilian results in serious bodily injury[6] or death.
  4. incidents in which use of force by a civilian against a police officer results in serious bodily injury or death.

 

The two years of data currently available under this reporting program, 2016 and 2017, were combined for the purposes of this analysis. In order to evaluate levels of police use of deadly force and ensure consistency across departments in terms of how deadly force was defined, we excluded the incidents in the database that either were not firearms discharges or that were not other uses of force that did not cause the death or serious injury of a civilian (for example, incidents involving the use of deadly force against a police officer but not a civilian and incidents where police caused an injury that was not categorized as serious). 38 of the 100 largest CA city police departments reported at least one incident that either solely involved the use of deadly force by a civilian against an officer or that involved a use of force that did not result in death or serious injury. In total, of the 1,489 deadly force incidents in the statewide database, 213 (14%) were excluded for these reasons (see Appendix B for the list of incidents excluded).

Importantly, deadly force as defined in this scorecard and as collected under California’s URSUS program is broader than fatal police shootings or the killings by police data commonly cited in media or crowdsourced databases. Deadly force includes all use of force incidents resulting in death or serious injury of a civilian or that involved a firearms discharge regardless of injury caused. In total, police shootings comprised 54% of the deadly force incidents included for the 100 city police departments included in this scorecard.

Since the URSUS program includes information about the circumstances of each deadly force incident - in particular which types of force were used in each incident - we were able to determine the proportion of each department’s police shootings where officers did not attempt to use another type of force prior to discharging their firearms.[7] 

Finally, in reviewing the URSUS data, we found that some incidents coded as "no confirmed weapon" were incidents where the person was alleged to be using a vehicle as a weapon when force was used against them. To address this, we compiled articles and DA statements for each incident to determine whether or not the person was in a moving vehicle and to confirm other details provided to URSUS. Through this process, we identified a total of 121 vehicle incidents, 56 of which were coded as "no confirmed weapon” and 31 of which were coded as “other weapon.” These incidents were reclassified in our analysis to “vehicle.”

Less Lethal Force

Definition: The Total Number of Uses of Tasers, Batons, Projectiles, Pepper Spray, Other Weapons and Strangleholds against Civilians in 2016 and 2017

To obtain information on serious police use of force that did not rise to the level of deadly force, we submitted public records requests to the 100 largest municipal police agencies in California asking for the total number of uses of force, by type of force used, in 2016 and 2017. 56 of these police departments sent us their data. Detailed data showing use of force by force type for each responsive department can be found here. Each department varied in how they categorized and reported the use of force. For example, some police departments required officers to report all weaponless strikes and control holds used against a civilian, while others only require these types of force to be reported when they cause injury. Similarly, some departments require officers to report whenever they display a firearm or taser, while others only require them to report when these weapons are discharged. Given these reporting inconsistencies, we limited our analysis to more serious forms of force since these were more consistently reported and categorized among the agencies we obtained data from. This included uses of “less lethal” force such as tasers, batons, impact projectiles, OC spray and a form of stranglehold called a carotid restraint.

Police departments also differed in how they counted uses of force when more than one officer used force during a single encounter. For example, in cases where two officers both used a taser on the same person, some departments count this as two uses of force while others count it as one use of force. We evaluated how each department counted use of force by examining their use of force reporting policies, use of force data, and by following-up directly with departments for clarification. Based on this information, we determined that most departments that sent their use of force data counted cases where multiple officers used the same type of force as multiple uses of that type of force while counting it as one use of force when one officer uses the same type of force more than once in the same incident.[8] 

Despite these efforts, we recognize there may still be differences that were unaccounted for within each department’s internal system for compiling and reporting uses of less lethal force. Moreover, we could not account for the degree to which force may have been systematically underreported by some agencies. This is a living project: we will continue to update the information on our site in response to feedback from law enforcement agencies and the public.

Civilian Complaints

Definition: The Total Number of Complaints, by Type of Complaint, reported by Civilians against Law Enforcement Personnel in 2016 and 2017

State and local law enforcement agencies provide the California Department of Justice with Citizens' Complaints Against Peace Officers (CCAPO) data through an annual summary. The information includes the number of criminal and non-criminal complaints reported by citizens, racial and identity profiling complaints and the findings from investigations into those complaints. In addition to these categories of complaints, we also submitted public records requests to the 100 largest municipal police agencies in California asking for the total number of use of force complaints and the number of those complaints that were sustained (i.e. upheld by investigators).

 

Arrests

Definition: The Total Number of Arrests Reported by Police in 2016

Data on the number and demographics of each agency’s arrests in 2016 were obtained from the California Monthly Arrests and Citation Register (MACR). This dataset classified arrests by whether the offense was a misdemeanor, felony or juvenile arrest. Since juvenile arrests were not disaggregated by felony or misdemeanor offense type, we excluded juvenile arrests (which comprised 5% of arrests statewide) from the analysis. In one case, San Francisco, we obtained data directly from the agency because they did not report data to the MACR in a format consistent with other departments (they did not disaggregate arrests of Hispanic individuals from White, Black, Asian and Other racial categories).

Homicides Reported and Homicides Solved

Definition: The Total Number of Criminal Homicides Reported and the Total Number Cleared by Arrest or Exceptional Means from 2013-2017

We obtained data on the number of criminal homicides reported and those that were cleared from 2013-2017 from the FBI Uniform Crime Report and Supplementary Homicide Report databases. Consistent with the definitions used by these databases, criminal homicides are classified as murder and non-negligent manslaughter but exclude suicides, accidents, “justifiable homicides” and deaths caused by negligence. Homicides were classified as “cleared” when they resulted in either an arrest or were cleared through “exceptional means” (These are cases in which there is sufficient evidence but an arrest is reportedly not possible, for example, if the person suspected has died). We recognize that homicides cleared by arrest or exceptional means is an imperfect measure of murders “solved,” for a variety of reasons (for example, not everyone arrested ends up being guilty of the offense). However, the Uniform Crime Report does not distinguish between these outcomes and, as such, we are limited by the data that are currently available.

We also included in the scorecard statistics on proportion of homicides cleared by race of the victim. Since the Uniform Crime Report did not disaggregate agency-level homicide clearance rates by race of victim, we relied on the Supplementary Homicide Report (SHR) database with enhanced case-level data from the Murder Accountability Project (MAP) to determine the percent of homicides “solved” by race of the victim for each jurisdiction. Importantly, since homicides tend to be reported to the SHR database an average of 5 months after each incident (depending on jurisdiction), fewer homicides are reported as solved in this dataset compared to the end-of-year Uniform Crime reports. According to MAP, 5-10% of homicides reported as unsolved at the time of reporting to the SHR were cleared later. As such, the SHR data should be interpreted as the proportion of homicides solved within a shorter time frame than the UCR, which is useful for evaluating police efficacy at solving homicides given the race of the victim.


Appendix B:

Use of Force Incidents Excluded from the Analysis of the 100 Largest CA City Police Departments:

Appendix C:

Deadly Force Incidents Benchmarked on Population, All Arrests and Violent Crime Arrests:


Appendix D:

Calculations of Racial Disparities and Racial Bias Scores for Each Police Department:


[1] 13 police departments reported that either they do not collect or possess data specifying the number of uses of less lethal force or that they would not provide us with these data despite being required to under the California Public Records Act. The remaining departments that did not provide us with these data indicated that they needed more time to compile and send these data, despite having several months to do so and being required by law to provide this information. These departments received an Incomplete (0%) for the less lethal force component of their police violence score. Upon receipt of these data, police department scores will be updated to reflect the data provided.

[2] For the purposes of comparison, deadly force rates for each department benchmarked on population and violent crime arrests are presented in Appendix C.

[3] In the case of departments where the number of White residents killed by police was zero, we replace the city-specific probability with a statewide probability.

[4] This can be seen as a Binomial distribution with a probability p of being killed while White given that an arrest was made with n independent experiments where n=the number of Black/Latinx residents who were arrested.

[5] The most severe disparity in police violence between either Black and White populations or Latinx and White populations in each jurisdiction was used to estimate their level of racial bias. This methodology prevents evaluations of racial bias against one marginalized group from being erased or offset due to more equitable treatment of another group, encouraging departments to address policing in communities where inequities are the most severe as a means to achieve more equitable outcomes overall.

[6] “Serious bodily injury” as defined by California’s use of force reporting system means a bodily injury that involves a substantial risk of death, unconsciousness, protracted and obvious disfigurement, or protracted loss or impairment of the function of a bodily member or organ.

[7] Police reports to the URSUS program identified 114 shootings statewide between 2016-2017 where police also used another type of force during the incident. The order of force used was reported in 47 of these incidents and we were able to obtain media reports, DA statements and other documents to determine the order of force used in the remaining 67 incidents. Links to these incidents and sources for each determination can be found here.

[8] There were, however, a few notable exceptions. Use of force reports from Los Angeles and San Jose police departments counted all uses of the same type of force during an incident as one use of force, regardless of how many officers used this type of force during the incident. While imperfect, we used these cities’ more conservative use of less lethal force counts to calculate their scores. Since both departments had relatively high levels of less lethal force even by this more conservative count, using the conservative count still provides useful insight into issues with less lethal force usage by these departments. Moreover, using this more conservative count did not change these departments’ overall grades or their grades for the police violence category of analysis, both of which were already F’s.