What is an Alignment Score?
An alignment score is a metric created during the Grader QA process. It is a percentage that shows how aligned the Original Score on a ticket is with the Benchmark Score created by the Benchmark Grader.
An alignment score is not the percentage difference between the Original Grade on the ticket and the Benchmark Grade.
The reason for this is that both the Original Grade and Benchmark Grade may have the same score, but obtained in different ways.
For example, an Original Score and a Benchmark Score could both have the same QA score of 80%. However, the criteria selected to add up to this 80% could differ across the two unique QA scores. This would mean that the scores are not truly aligned.
For this reason, we calculate alignment on the Criteria level, and then weigh that up to a Rubric Alignment Score.
Calculating the Criteria Alignment Score
As an example, let's assume this scenario:
There is a Criteria on your rubric that is worth 15 points - “Did the agent provide the appropriate resolution”.
The Original Grader scores a 10 out of 15 on this question
The Benchmark Grader scores a 7 out of 15 on this question.
Under these conditions:
The Original Grader was “off” of the Benchmark grade by 3 points, 20%.
This is calculated by dividing the amount “off” (3 points) by the total points available on this Criteria (15 points)
Therefore, the Original Grader is 80% aligned for this Criteria
This is calculated by subtracting the percentage "off" (20%) from a perfect alignment score (100%)
The Criteria Alignment Score for this criteria is 80%.
This step process is done for each criteria within the rubric.
Calculating the Rubric Alignment Score
In the case that a rubric has a max total point of 80, this is how you would calculate the Alignment score.
The above criteria would be ~18.75% (15/80) of the total score.
Multiply the Alignment by the weight calculated to receive how much this option contributes towards the alignment score.
The math for step two:
80% Alignment x 18.75% Weight = 15% Weighted Criteria Alignment Score.
15% is added with all the other Weighted Criteria Alignment Scores from the rubric to get the Alignment Score.