What is Grader QA?
Grader QA is a workflow used to evaluate the accuracy of your grading team. It is also known as a “Grade the Grader” workflow.
In this workflow, a random sample of graded tickets are distributed for review to a Senior QA Grader (or Benchmark Grader), a person that sets the standard for how grading should be done.
The Benchmark Grader grades the ticket without seeing the original score. Through this process, an “alignment score” is created that shows how well aligned the Original Grader was with the Benchmark Grader.
Why is Grader QA important?
The value that a Quality program can deliver to the business is only as much as the confidence and trust in the data that the program generates. If the grading team is scoring tickets in a way that is not aligned with leadership's standards of Quality, then the insights it produces are not valuable or actionable.
When there is high trust in the data that the Quality program produces, it can provide actionable insights into the Support team, the Processes, and even your product or service. Leadership making decisions based on Quality insights is a sign of a high trust QA program.
How is Grader QA different than Team Calibrations?
In Team Calibrations, a ticket(s) is distributed to the grading team to all evaluate.
A key point for Team Calibrations is that multiple graders are grading the same ticket.
Typically, following this exercise, a Calibration Session is held that includes all the graders that graded the tickets. In this session, each grader discusses how and why they graded the way they did, and a Final Calibration score is determined. This Final Calibration score represents how graders should grade in similar situations going forward.
This can be a powerful collaboration process to build alignment within the team, foster consistent conversation on how to grade, and also create a forum for graders to consistently meet.
However, we also understand from our customers that there are some inherent drawbacks of this workflow. Below are the 3 primary drawbacks.
There can be a conflict-of-interest in the calibration. Because the graders are aware that they are partaking in a calibration process and session, there can be a conflict-of-interest to grade in a way they know will align with the correct standards. This can sometimes be more relevant with Graders at a BPO team.
It can be difficult to scale Team Calibrations. Given that these workflows generally include a Calibration Session, doing this with more than 5-10 graders can be very difficult. For this reason, if the grading team includes many graders, you may have to break up your graders into multiple / separate Calibration Sessions.
The metrics are sometimes unreliable. With the above scaling issues, the sample size - or number of tickets included in Team Calibrations - is done is very low. It may include 2 or 3 tickets per month.
How does Grader QA solve for some of these drawbacks?
Removal of Conflict-of-Interest: because Benchmark Graders are randomly reviewing already graded tickets, the Original Grader does not know which grades will be selected for review.
Scale: GraderQA does not require Calibration Sessions, since it’s just a comparison of the Original Grade and the Benchmark Grader. Therefore, the volume of tickets that can be evaluated can be much higher.
Additionally, MaestroQA has developed a specific Grader QA workflow that includes “Grader QA Assignment Automations” to streamline the distribution of these tickets, and an “Alignment Score” report to understand the results.
Trusted Alignment Metric: Given what’s mentioned in #2 above about scale, the Alignment Score metric that is created during Grader QA is much more reliable as it relates to understanding Alignment of the team (or BPO) with the Quality Grading Standards.
Enabling GraderQA
Now that you know why you should use GraderQA, here is how to enable it!
As a MaestroQA Admin on the Team and Professional packages:
Click on Settings
Click on Other Settings
Scroll down to Additional Feature Settings
Check "Enable Grader QA"
After that is enabled, check out the article below to learn how to start building out a Grader QA workflow!