What is GraderQA?
GraderQA is a workflow separate from AgentQA. The focus of GraderQA is to evaluate the performance of graders and understand how graders are grading against QA standards. To learn more about the GraderQA workflow, click here!
Who can submit a GraderQA Appeal?
There are two roles that can submit an Appeal:
Original Grader who created the initial score
Original Graders are only able to submit Appeals for their own GraderQA scores. Admins are able to submit Appeals for any GraderQA score.
Note: As of January 23rd, 2023, GraderQA Appeals are available to be turned on by request, and this can be done by emailing your team's CSM or reaching out to Maestro Support!
How can I submit a GraderQA Appeal?
Open a GraderQA score and then click on “Start Appeal.” This will populate an Appeal form that looks like the image below.
When assigning a reviewer, only Benchmark Graders will populate in the dropdown list.
Fill out all required fields and click on “Submit” to submit the Appeal to the selected GraderQA Appeal reviewer.
How do I conduct a GraderQA Appeal?
For the GraderQA reviewer, navigate to the Appeals page and select the “Grader QA Appeals” filter underneath the “Type” selector.
On the Appeals table, two new columns will be added:
Original Grader - shows who the Original Grader was on the ticket
Benchmark Grader - shows who the Benchmark Grader was on the ticket
The “Reviewer” column will reflect who is responsible for conducting the GraderQA Appeal.
Clicking on the Appeal itself will populate the Appeal page where a reviewer can only modify the GraderQA Score portion:
After making the Appeal decision, fill out the form below to approve or deny the Appeal.
What is the outcome of GraderQA Appeals?
When the score is updated on a GraderQA Appeal, if a different option is selected, the GraderQA Score is updated. As a result, the Alignment Score will update as well. To use the scenario shared above as an example, changing the option from “0” to “2” will result in 100% Criteria Alignment.
To learn more about how Alignment Scores are calculated, click here.
Is the AgentQA score updated?
AgentQA scores will not update when a GraderQA Appeal is approved or denied.
GraderQA Appeals only update the GraderQA scores, which may also update the Alignment Scores (depending on the change).
Is there multi-step support for GraderQA?
At this time, it is only possible for the Original Grader to submit an Appeal directly to the GraderQA Appeal Reviewer.