Skip to main content
All CollectionsTroubleshooting and FAQsTips and Tricks
Setting Up a Peer Review Program
Setting Up a Peer Review Program
Matt avatar
Written by Matt
Updated over a month ago

Introduction

This guide will walk you through creating a peer review program in MaestroQA. The setup process is organized into three main phases: configuring permissions, setting up user access, and establishing the peer review framework.

Phase 1: Permission Configuration

Pre-requisites before starting Phase 1

  1. You are a MaestroQA Admin

  2. The ability to use either the Grader or Manager User Role as the dedicated Peer Reviewer role. For this walkthrough we will be using the Grader role

  3. A separate Peer Review Rubric. Do not use the same rubric you use for non-Peer Review QA.

Step 1: Restricting Grading, Calibration, and Coaching Permissions

  1. Go to User Roles > Role Permissions

  2. In the Role Permission table update the following permissions:

    • View Grades: Set to "Creator + Shared"

      • Peer reviewers can only see scores they have created or were shared/evaluated on
        ​

    • Create and Update Grades: Set to "Creator + Shared"

      • Allows peer reviewers to create QA scores with rubrics they have access to

      • Blocks peer reviewers from being able to modify scores they did not create
        ​

    • View Calibration: If peer reviewers will also participate in Calibrations, set this to:

      • "Own + Final": If you want the peer reviewer to be able to compare their calibration with the final calibration

      • "Own": If you only want the peer reviewer to only see the calibration they created
        ​

    • Create and Update Calibrations: This will depend on "View Calibration"

      • Our default recommendation is to set this to "Own" if your peer reviewers are intended to participate in Calibrations

      • Else you can set this to "No"
        ​

    • Edit Coaching Session Workspaces: Set to "None"

      • Prevent peer reviewers from editing Coaching Sessions

Step 2: Controlling Coaching Access

  1. Go to Settings > Coaching Templates

  2. Click Edit Access next to a disabled or active template

  3. Update Coach Access and select User Groups that cover these 3 buckets:

    1. Create Coaching Sessions

    2. Manage Coaches who are conducting Coaching Sessions

    3. Are Admins in MaestroQA

  4. Click Save Template at the bottom of the page

  5. Repeat steps 2 and 3 for all Templates

Step 3: Restricting Existing Rubric Access

Before making any user account changes:

  1. Navigate to the Rubrics page

  2. Find your peer review rubric and click the 👥 button

  3. In the access control menu under "Admin, Grader, Manager Permissions":

    • Switch from "Grader" to "User Group" for easier control

    • Only grant User Groups who can use the rubric to create QA Scores to have View and Edit access

  4. Click "Save Changes"

Phase 2: User Setup

Requirements for Phase 2:

  1. Have a rubric in mind you want your Peer reviewers to use

  2. Make sure each Peer Reviewer's Agent Profile is enabled via User Roles > Agents tab

Step 4: User Account Configuration

Set up Peer Reviewers as Graders:

  1. Link the Grader User Account to their own Agent Profile

  2. Provide the User Account access to the group of agents they are allowed to evaluate

    • Example: Use an agent group called "Peer Review Agent Group"

Step 5: Creating a Peer Reviewer User Group

Create a dedicated group for peer reviewers:

  1. Go to User Roles > User Groups

  2. Click Create Group

  3. Choose one of these methods to add peer reviewers:

    • Manual: Add each peer reviewer individually

    • Automatic: Set up automatic user addition based on rules

      • Example rule: "maestro_user_role equals grader"

  4. Name your group (e.g., "Peer Reviewers")

  5. Click "Save Group"

Phase 3: Peer Review Rubric and Dashboard Configuration

Step 6: Setting Up A Peer Review Rubric

Configure rubric access permissions:

  1. Navigate to the Rubrics page

  2. Find your peer review rubric and click the 👥 button

  3. In the access control menu under "Admin, Grader, Manager Permissions":

    • Switch from "Grader" to "User Group"

    • Find your Peer Reviewer User Group

    • Grant them "View and Edit" access

  4. Set up additional access for oversight:

    • Grant team leads/managers appropriate viewing permissions

    • Consider giving managers "View and Edit" if they need to modify scores

  5. Verify all access levels:

    • Peer reviewers can access the rubric

    • Team leads can view results

    • Other teams have appropriate access levels

  6. Save all changes

Step 7: Setting Up A Peer Review Homepage

After creating your Performance Dashboard, it is time to share it with your Peer Reviewers.

  1. Next to the Dashboard's name, click on the kebab menu and then click "Edit Dashboard"

  2. In the pop-up that appears, select the checkbox highlighted in the screenshot below.

    1. Doing this will prevent User Accounts that are linked to an Agent Profile from being able to see stats/metrics for other Agent Profiles

  3. Click "Save"

  4. Now on the dashboard itself, click on "Share" and then select the following User Groups:

    1. The Peer Review User Group created in Step 5

    2. Any separate User Groups containing the Peer Reviewers managers or other Maestro Admins

  5. Click "Save Changes" and you are all set!

Did this answer your question?