Validating Risk Need Assessment Tools: What You Need to Know

A Guide to Planning and Implementing Validation Studies for Criminal Justice Agencies

By: Genie Jackson, Research Team Lead, equivant Supervision + Pretrial

Risk Need Assessment (RNA) instruments combine two key components: Risk Scales that assess likelihood of future justice system contact and Need Scales that identify areas for intervention. Validation studies primarily examine how well the risk scales align with actual outcomes. For risk scales, validation specifically examines how accurately the calculated risk levels correspond with actual recidivism outcomes. This process answers two critical questions: 1) do individuals classified as High Risk recidivate at higher rates than the general population under supervision? and 2) do those classified as Low Risk demonstrate lower recidivism rates than average? Need scales, while not directly validated against recidivism outcomes, can be evaluated through complementary analyses including internal consistency testing, comparison with normative samples, and examination of their correlations with recidivism outcomes.

Risk instrument validation is the systematic process of evaluating whether a risk assessment tool accurately measures and predicts what it claims to measure and predict within a specific population and context.

This process involves:

1.Statistical verification of both discriminative and predictive accuracy:

• • Discriminative accuracy (measured through AUC values, which assess how well the tool distinguishes between people who do and do not recidivate)
  • Predictive accuracy (measured through Positive Predictive Value (PPV), which indicates how well the predicted probabilities match actual outcomes)

2. Testing the tool’s performance with the agency’s specific population:

• • For recent implementations: Compare performance against normative samples, recognizing that local populations often differ from even carefully selected norm groups
  • For revalidation studies: Compare current performance against both the original normative sample AND the agency’s previous validation results to identify any changes in predictive performance over time
  • Document any shifts in population characteristics, operational practices, or policy changes that may impact tool performance

3. Assessment of measurement quality and consistency:

• • Check if items within scales work together as intended using the agency’s own data

4. Evaluation of potential predictive bias across demographic groups

• • Examine whether the risk instruments are calibrated across demographic groups. In other words, does a particular risk level predict the same recidivism rate regardless of a person’s gender, race, or ethnicity?
  • Examine whether the tool’s discriminative ability (ability to rank-order risk) is similar across demographic groups. In other words, does the tool distinguish between those who do and those who do not experience a subsequent recidivism event within the study period equally well, regardless of gender, race, or ethnicity?

Practical Requirements for a Validation Study

Agencies and researchers work together to ensure that the study design and specified data can accurately answer the research questions. The general requirements of a validation study are listed here, although they may vary given an agency’s particular research questions. 

Assessment Data

• A sufficient sample of complete risk assessment records must be available. The required sample size will be determined by a power analysis which takes into account both the expected base rate and desired confidence level. This statistical approach ensures the sample is large enough to draw meaningful conclusions. 
• Complete risk assessment records include demographic information (limited to validation-relevant characteristics), assessment dates, responses and scores for all administered scales, and information that facilitates the identification of the study sample.
• Demographic information enables analysis of scale performance across groups. Comparisons between demographic subgroups will be limited to those whose sample sizes satisfy the minimum requirements established by the power analysis. This ensures adequate statistical power for meaningful group comparisons.

Outcomes Data

• Outcomes data consists of justice system contact records documenting new arrests, convictions, or technical violations that occur during a specified follow-up period after the initial assessment. This data must include the dates and types of subsequent events that meet the instrument’s operational definition of recidivism, with follow-up periods typically ranging from 1-3 years post-assessment. Outcomes data is often held by entities other than the agency, such as the State Police or the state’s Department of Justice. However, the process of obtaining outcomes data for a validation study must be facilitated by the agency. 

Population Context 

• Agencies assist in the study process by providing descriptions of the study population, details about any major changes in assessment practices during the study period or any significant policy changes that might affect outcomes.

Data Quality Checks

• The process for handling missing data should be agreed upon. Any data collection issues by the agency should be documented. Scoring should be validated.

Timeline

• Obtaining all the data required can be a lengthy process, often taking several months. Agencies should be prepared to work with researchers to obtain data as accurately and efficiently as possible, keeping in mind that analysis options will be limited until all the data has been received and verified for accuracy by the research team. 

A well-designed validation study requires careful planning, appropriate data collection, and close collaboration between agencies and researchers. While the process demands significant time and resources, validation studies provide essential evidence about how well risk scales identify individuals more or less likely to recidivate and how effectively need scales guide intervention decisions. This evidence enables agencies to make informed decisions about assessment practices and policies that support successful outcomes for justice-involved individuals. Regular revalidation studies help ensure that risk need assessment instruments continue to perform as intended as populations and contexts evolve over time.If it is time to validate or revalidate your agency’s risk need assessments, please contact us.