Skip to content

Calibrate review standards

Calibration compares independent human assessments of the same request. It helps a team find differences in interpretation and improve the rubric. Calibration is separate from ordinary reviews and does not inflate ordinary case totals.

Prepare the exercise

A manager opens Results → Calibration and prepares the exercise using the available workflow. Inspect the chosen evidence, rubric version, and participants before opening it.

The exercise preserves its agreed basis so the comparison is meaningful. Participants still need current source access and must satisfy the displayed eligibility and independence checks.

Review independently

Participants open their assigned calibration, read the shared evidence and rubric, and save or submit their own judgments.

Reviews remain blind before reveal. Do not share another participant's answers outside the app to coordinate a score. The calibration screen uses human submissions and has no AI-suggestion control.

Submitted calibration assessments are preserved; they are not silently replaced when someone changes their interpretation later.

Reveal and discuss

When the workflow permits reveal, compare the independent answers and evidence. A difference is a discussion prompt: ask whether it came from ambiguous criteria, different interpretations, or insufficient evidence.

Record resolution notes so the team can carry the learning into later reviews. An agreed interpretation does not rewrite the original submissions.

If the rubric needs improvement, publish a new version. Existing review and calibration history keeps its original basis.

If a participant is blocked

Use the exercise's available recovery, withdrawal, or cancellation actions. An exercise with missing or inaccessible evidence should not be represented as a completed agreement result. Ask a manager to check the exercise state and source permissions before continuing.