How to calibrate your customer service QA reviews

Regular conversation reviews are a great way to provide a constant stream of actionable feedback to your support reps and improve your support quality. However, the people performing the reviews must be aligned with each other, and with your company’s standards.

Imagine a group of referees preparing for the World Cup. The players (customer service agents) are hoping for consistent, fair, and unbiased refereeing. And the referees (reviewers) have guidelines and rulebooks (scorecards and process adherence) to follow.

Leading up to the World Cup, the referees will convene and discuss certain rule changes or dig deeper into a particular interpretation of a rule. These discussions will help ensure that they view the rules in the same way and deliver consistent team performance.

That’s exactly what quality assurance (QA) calibration sessions aim to do for your reviewers.

In this guide:

• What is calibration for QA?
• Different kinds of QA calibration sessions
• How to plan your QA calibration sessions
• Improve support quality with QA calibration

What is calibration for QA?

Customer service QA calibration is the process in which internal quality reviewers align their rating techniques to make sure that support reps receive the same level of feedback from all reviewers.

Essentially, QA calibrations help reviewers synchronize their assessments, provide consistent feedback to support reps, and eliminate bias from quality ratings. The goal is to have everyone on the same page. Support reps should receive the same quality of feedback regardless of who reviewed their customer interactions.

Customer service QA calibration is done by comparing how different reviewers evaluate the same support ticket during a calibration session. Usually, the aspects that require quality assurance calibration are the following:

• Rating scale: All reviewers should understand the scale and different ratings. The larger the rating scale, the more important the calibration.
• Failed vs non-applicable cases: When a ticket was handled correctly but a specific aspect of the conversation was missing, reviewers might rate it differently. It’s important for the reviewers to align on such cases and rate them objectively.
• Free-form feedback: QA reviews might include additional comments left to support reps. Reviewers should agree on the “length” of feedback included in each review, feedback techniques used in the comments, and the overall style and tone.

Different kinds of QA calibration sessions

1. Review separately, discuss together (blind sessions)

Most support teams opt for the “review first, then discuss” strategy in their quality assurance calibration. In this setup, all reviewers assess the same conversations without seeing others’ ratings. After everyone has cast their votes, the team of reviewers comes together to compare and discuss the results.

Pros: Pinpoints the discrepancies in your reviewers’ feedback. By doing ‘blind’ evaluations on the same tickets, you get a good understanding of where possible inconsistencies are lurking in your feedback.

Cons: Reviewers can get defensive on how they graded. The discussion session can become heated because everybody wants to prove why their response is right, instead of figuring out the correct solution together.

2. Review and discuss together (team sessions)

Calibrating feedback collectively with all reviewers is perfect for discussing and (re)defining quality standards in your support team. In this setup, you read over customer conversations for the first time together as a group. As a team, you decide how to score them. If you run into any conflicts or disagreements, usually either the leader of the QA program will make the final decision.

Pros: This strategy removes the feeling of being graded—and the possible stress that this can cause—from the sessions. By allowing reviewers to discuss tickets together, you strengthen the common understanding of your quality criteria among your reviewers.

Cons: You will not be able to measure the current discrepancies in your reviewers’ work. If you don’t have an understanding of how consistent and unbiased your reviewers are, it might be difficult to iron these differences out.

3. Review together with the support team (agent sessions)

Invite your support reps to meetings where you discuss the best way of handling specific situations. This engages them in the quality process so they have a more hands-on understanding of quality expectations, and see customer interactions from another perspective. This approach also allows your support reps to provide additional context to the situations when reviewers do not agree on the final rating.

Pro: This method is the most transparent way of handling quality calibrations. Discussing review scores together allows support reps and reviewers to learn from each other and it aligns the entire team around the same goals and QA standards.

Con: You will not have your reviewers’ original scores to compare to the agreed benchmark. Some customer service agents may feel uncomfortable defending their actions to a room of reviewers.

How to plan your QA calibration sessions

When planning your QA calibration sessions, consider the following:

1. How many conversations to calibrate?
   This number can vary for different teams, as the complexity of the tickets, the number of reviewers you have, as well as how much time you have allocated to the session will dictate the right amount for you. If you start off with five in your first session, then you can adjust for future sessions.
2. How often will you hold your sessions?
   For most teams, monthly sessions are the perfect amount. Keep in mind the more reviewers you have, the more often you should calibrate.
3. What is your calibration goal?
   To help keep your discussions on track, it is recommended to set a goal for your sessions. Are you wanting to update the rating scales on your scorecard? Would you prefer to discuss an internal process around escalations? Or maybe you want to use the session to align your reviewers with your QA standards. By having a clear goal for your sessions you can avoid the discussion going off-track or down a rabbit hole.

Once you have it figured out, it’s time to organize the sessions.

Choose a facilitator to be in charge

Holding regular customer service QA calibrations takes time and organizational effort. If you have dedicated facilitators, they can be responsible for making the process successful.

How to successfully engage facilitators in your QA calibration process:

• Let facilitators take the lead
  Sessions consist of dozens of mini-decisions, starting from which tickets to choose for review, to who has the final say in how to rate a ticket in case of a difference of opinions. Let your facilitators take charge to avoid any unnecessary quarrels about each aspect of the process.
• Rotate facilitators
  Instead of having the same person run the sessions, let all reviewers take turns in facilitating the process. This allows everyone in the QA team to take a fair share of the responsibility and helps them understand the complexity of aligning all reviewers toward the same rating style.
• Create facilitator guidelines
  Help your reviewers lead sessions successfully by having specific guidelines in place for the process. Think of different scenarios that could potentially happen, and provide tips on how to deal with different situations.

Allowing your reviewers to facilitate sessions helps to keep your team engaged and motivated. It increases the sense of shared responsibility for customer service quality and helps everybody work together toward the same goals.

Define your quality assurance calibration baseline

The quality assurance calibration baseline defines how much you allow your reviewers’ ratings to differ. It’s expressed as a percentage and usually falls around 5 percent. Having some bandwidth for differences removes minor fluctuations from the picture and helps you focus on the most important discrepancies.

• If the difference in your reviewers’ ratings is below the baseline, you can conclude that evaluations are done evenly. You’ve completed the calibration process and can be sure that your team receives consistent feedback.
• If the difference in your reviewers’ evaluations is above the baseline, you need to proceed with the calibration process and discuss the details with your team. That’s a clear indicator of discrepancies in the feedback your support reps receive.

Having a measurable baseline makes calibrations a lot easier for your team. You’ll know which differences in your reviewers’ ratings deserve attention, and which ones are fine to leave unnoticed.

Make use of the findings from your quality assurance calibrations

One aspect of calibration sessions that often gets overlooked is ensuring an adequate follow-up. This involves two key actions:

• Keep agents in the loop.
• Update your quality standards.

For example, perhaps in one of your sessions, you realize that one of your rating categories is unclear for your reviewers. The team makes the decision to expand on the definition of this category. You should also inform your customer service agents of this updated definition, as they likely also found the category unclear.

Keeping your documentation up to date with all of your calibration findings and discussion outcomes helps keep your support team moving along the same path.

Improve support quality with QA calibration

Like QA reviews, your calibration sessions should be a recurring activity. The benefits of regular sessions include:

• Keeping a relevant scorecard: Perhaps a rating category is poorly defined, or a 4-point rating scale does not make sense for one of your categories. Sessions bring all of your reviewers together, allowing you to discuss what you may need to update in your QA scorecard.
• Internal process updates: You may calibrate cases where you notice the customer service agent did everything perfectly, however, the customer experience was not up to standard. Your sessions can help bring to light processes in need of an update, while also helping to identify coaching opportunities for your team.
• Improve performance: By having these discussions regularly, you are able to achieve consistency and keep an up-to-date definition of what quality means for your team. This is crucial for making sure that your customer service agents are working toward the right goal.

On the contrary, when reviewers are misaligned, inconsistent or biased in their own grading criteria, your quality metrics become skewed and your support reps become frustrated. Customer service QA calibration sessions are an essential part of any sophisticated customer service quality program.