GraderQA is a workflow separate from AgentQA. The focus of GraderQA is to evaluate the performance of graders and understand how graders are grading against QA standards. To learn more about the GraderQA workflow, click here!

There are two roles that can submit an Appeal:

Original Graders are only able to submit Appeals for their own GraderQA scores. Admins are able to submit Appeals for any GraderQA score.

Open a GraderQA score and then click on “Start Appeal.” This will populate an Appeal form that looks like the image below.

When assigning a reviewer, only Benchmark Graders will populate in the dropdown list.

Fill out all required fields and click on “Submit” to submit the Appeal to the selected GraderQA Appeal reviewer.

For the GraderQA reviewer, navigate to the Appeals page and select the “Grader QA Appeals” filter underneath the “Type” selector.

On the Appeals table, two new columns will be added:

The “Reviewer” column will reflect who is responsible for conducting the GraderQA Appeal.

Clicking on the Appeal itself will populate the Appeal page where a reviewer can only modify the GraderQA Score portion:

After making the Appeal decision, fill out the form below to approve or deny the Appeal.

When the score is updated on a GraderQA Appeal, if a different option is selected, the GraderQA Score is updated. As a result, the Alignment Score will update as well. To use the scenario shared above as an example, changing the option from “0” to “2” will result in 100% Criteria Alignment.

To learn more about how Alignment Scores are calculated, click here.

AgentQA scores will not update when a GraderQA Appeal is approved or denied.

GraderQA Appeals only update the GraderQA scores, which may also update the Alignment Scores (depending on the change).

At this time, it is only possible for the Original Grader to submit an Appeal directly to the GraderQA Appeal Reviewer.