What is GraderQA?
Benchmark how accurate your graders are grading tickets by utilizing GraderQA. GraderQA utilizes a unique automation to help assign tickets to a designated grader who serves as the benchmark for how accurate QA scores are
Here is a quick list of terminology to use when discussing GraderQA:
- Agent QA: Process of Grading an Agent’s Performance
- Grader QA: Process of Grading a Grader’s Performance
- Original Grader: Original grader who graded the ticket
- Original Score: QA Score on ticket
- Benchmark Grader: User who conducts GraderQA
- Benchmark Score: The “correct” QA Score for the ticket
- Accuracy Score: How accurate the Original Score was relative to Benchmark Score
How to set up GraderQA?
Step 1: Set up users who will be the Benchmark Graders
In User Roles -> Role Permissions, users can now define what users can be the benchmark grader. This must be enabled first prior to setting up the automation.
Step 2: Setup Grader Groups for graders to review
Grader Groups function similarly to Agent Groups where the main purpose is to group together users to make creating automations easier.
This can be done by navigating to User Roles -> Grader Groups. There are no other uses for GraderQA outside of setting up the GraderQA Automation.
Note: Any user who QAed a ticket in the past is considered a grader.
Step 3: Create the GraderQA Automation
Go to Sharing Automations, and then click on “Create New Automation”.
Create a “Send Grader QA Assignments” automation, and fill out the Automation Form like you would with any other automation.
Step 4: GraderQA Workflow
After the automation runs, benchmark graders will receive their assignments, and they can freely check on these assignments from the Home page.
From here, benchmark graders can grade these tickets like they would during AgentQA.
During the GraderQA process, benchmark graders cannot change:
- The grading type
- Agent being graded
- Rubric being used
These will be grayed out like shown in the screenshot below.
Step 5: Viewing Results
After the benchmark grader creates a Benchmark Score, the Alignment Score is automatically calculated.
An alignment score will return a % value that signifies how different the Original Score is from the Benchmark Score.