What is an Alignment Score?

An alignment score is a metric created during the Grader QA process. It is a percentage that shows how aligned the Original Grade on a ticket is with the Benchmark Grade created by the Benchmark Grader.

The goal of an alignment score is to see how accurately and aligned graders are grading towards your QA program standards.

An alignment score is not the percentage difference between the Original Grade on the ticket and the Benchmark Grade. The reason for this is that both the Original Grade and Benchmark Grade may have the same score, but obtained in different ways. (For example, the Original Grade and the Benchmark grade may both be 80%, but the specific Criterion where the agent lost points may not be the same - meaning there is misalignment in scoring.)

For this reason, we calculate alignment on the Criteria level, and then weigh that up to a Rubric Alignment Score.

Calculating the Criteria Alignment Score

Below is a brief example on how alignment is calculated on a Criteria level.

Let’s assume the below:

  • There is a Criteria on your rubric that is worth 15 points - “Did the agent provide the appropriate resolution”.
  • The Original Grader scores a 10 out of 15 on this question
  • The Benchmark Grader scores a 7 out of 15 on this question.

This would mean:

  • The Original Grader was “off” of the Benchmark grade by 3 points, or 20%.

    This is calculated by dividing the amount “off” (3 points) by the total points available on this Criteria (15 points)
  • Therefore, the Original Grader is 80% aligned for this Criteria
  • The Criteria Alignment Score for this criteria is 80%.

Calculating the Rubric Alignment Score

The above is calculated for each Criteria in a rubric, and then this Criteria Alignment Score is weighted based on the weight of that Criteria in the Rubric.

For Example, if the rubric had 80 total points, the example Criteria mentioned above is 15/80 = 18.75% of the entire rubric.

80% Alignment x 18.75% Weight = 15% Weighted Criteria Alignment Score.

15% is added with all the other Weighted Criteria Alignment Scores from the rubric to get the Rubric Alignment Score.

Please see this spreadsheet for a more detailed example on how these metrics are calculated.

Did this answer your question?