The new “Calibration Alignment” report is a heat map that displays how aligned graders are on the various calibrations they’ve completed.

Admins spearheading some of the most successful QA programs schedule regular calibration sessions with their graders to ensure they're all aligned with their grading. This increases the probability for agents to be graded properly and accurately no matter who happens to grade them.

In addition to consistent feedback, calibration reporting can help leaders uncover where potential grading discrepancies occur most frequently, which leads to healthy discussions around how to improve overall grading alignment:

Either way, this report gives leaders more ways to assess their overall QA process and determine actionable steps to optimize it. Understanding the WHY behind any misalignment could be the first step in improving QA's effectiveness for the team - as well as the added confidence in your QA metric and what it represents.

On the Reporting page, you'll now see a folder labeled Calibrations located on the left side of the screen. There are now two reports in this folder at this time:

Users who have View Calibrations set to "Own" or "Own + Final" on the Role Permissions page in User Roles can see the new calibration report. To summarize:

If View Calibrations is set to:

The Calibration report comes with 5 filters:

User Groups/Graders represent the rows in your heat map, while the Rubrics/Criteria serve as your columns.

The default view shows the most recently completed Calibration Ticket Review at the rubric level for all User Groups. Admins can select specific User Groups to refine the report.

Each percentage represents an Alignment Score. For example - The combined Individual Calibrations for all graders in the All QA Specialists user group are 76% aligned with the Final Calibrations for those calibrated tickets.

As you can see above, hovering over one tile with an Alignment Score shows a prompt with options to get more granular:

Above shows an example of the criteria view of the heat map. As we can see, the User Groups has an Alignment Score of 57% for the Merchant Growth rubric criteria, meaning their Individual Calibrations were only 57% aligned with the agreed-upon Final Calibration scores. Hovering over the tile again allows us to dive deeper and look at the grader level for this team.

We can now see the Alignment scores for the two graders on this team at the rubric criteria level. It appears that although both graders' Alignment Scores are fairly low for this criteria, One grader has a much lower Alignment Score for Merchant Growth. This could lead to further investigation into why that is the case. Hovering over the 44% tile will prompt the option to view the actual tickets that were calibrated during the calibration exercise to get a better understanding of where the discrepancy is occurring. Below is one of the tickets

Here is an example of seeing a Final Calibration at the Ticket level, comparing the Final Calibration score and response to one grader's Individual Calibration. If we look for our grader in question, we can see where he chose a different answer than the majority of his grading peers. Depending on how many points this criterion was worth relative to other criteria in the rubric, the Alignment score will factor that in and "ding" the grader more for misaligning on a criterion that holds more weight (or impacts the overall score more).

The Alignment Score is not just a metric for GraderQA - it is a more accurate way to gauge misalignment for Calibrations as well. For Calibrations specifically, it is a percentage that shows how aligned a grader's Individual Calibration for a ticket is with the Final Calibration created by the grading team, which is typically initiated and led by an Admin.

(Aside - This workflow is separate from Benchmark Calibrations, which are typically created by a Benchmark Calibrator. Most teams use the workflow referenced above)

An alignment score is NOT the percentage difference between the Individual Calibration Score for the ticket and the Final Calibration Score.

The reason for this is that both the Individual Calibration and Final Calibration may have the same score, but obtained in different ways.

For example, an Original Score and a Benchmark Score could both have the same QA score of 80%. However, the criteria selected to add up to this 80% could differ across the two unique QA scores. This would mean that the scores are not truly aligned.

For this reason, we calculate Alignment on the Criteria level, and then weigh that up to a Rubric Alignment Score. This is the core difference between the original Calibration Report and the new Calibration Report, which takes this logic into account for all calculations.

In the old report, Individual Calibrations were averaged separately from Final Calibrations and then compared. This means that if a grader only took part in 4 ticket calibrations, but 10 total calibrations were done on the same rubric, that grader's average Individual Calibrations would be compared to the average of all 10 Final Calibrations that were created for that rubric. In other words, the grader could potentially be impacted by calibrations he/she/they were never a part of.

In the new report, we first determine a grader's alignment at the ticket level, so that each Individual Calibration is compared to the right Final Calibration to generate an Alignment Score for each Ticket. Then we take the average of the Ticket Alignment Scores to get an overall Alignment for a Calibration Review.

Here's an example. Suppose a Calibration exercise has 3 tickets.

Admins can now view reporting in 2 ways:

As an example, let's assume this scenario:

Under these conditions:

This step process is done for each criterion within the rubric.

In the case that a rubric has a max total point of 80, this is how you would calculate the Alignment score.

The math for step two:

15% is added with all the other Weighted Criteria Alignment Scores from the rubric to get the Rubric Alignment Score.

Please see this spreadsheet for a more detailed example of how these metrics are calculated. Make a copy of the Google Sheet to test the calculations for yourself!

Yes! As of 9/10/2021, users can now export this report as a CSV file.

Select Export Chart in the top-right corner. In the following prompt, title your export and enter the email addresses of the recipients. The notification can be configured to send via email or Slack (if integrated with Maestro). Press Export Chart.

Go to the Exports tab (top of your screen) and the option to download the chart will appear.

The downloaded CSV chart (one file) will look exactly like the chart you created in the app. Your settings in the Calibration Alignment Report will dictate what columns (Rubrics/Criteria) and rows (User Groups/Individual Graders) of data will appear in your export. In the example below, the exported chart shows Alignment Scores for one group of Graders for one Rubric at the Criteria level.