Skip to main content

What to do before using AI

AI can be a powerful tool to save time, but using it blindly can lead you to more pains than wins.

Matt avatar
Written by Matt
Updated this week

Overview

AI can seem like an easy way to quickly get answers and share instant findings with your team. This approach often backfires because it leads to:

  • Trusting fake insights that AI creates from bad or incomplete data

  • Not forming your own understanding of the problem

  • Getting analysis that sounds good but doesn't actually help

This workflow helps you build the right prompts for the right data and work more effectively with your support data overall.

Workflow

Step 1: Pick a Clear Starting Theme

Choose one specific area to focus on. Don't try to analyze everything at once.

Good starting themes:

  • Tickets that received DSAT survey responses

  • Interactions with high comment counts (over X comments)

  • Cases tagged with high handle time

  • Tickets escalated to management

  • Specific product or feature complaints

How to choose:

  • Pick something you can clearly define and filter for

  • Choose themes where you have enough data (at least 20-30 tickets)

  • Focus on areas that actually matter to your team's goals

Step 2: Find and Filter Your Tickets

Locate these tickets in your worksheet or performance dashboard and apply filters to get your dataset.

Check your filtered results:

  • Do you have at least 20-30 tickets? (Less than this won't give meaningful patterns)

  • Are the tickets actually related to your theme? (Skim through a few examples)

  • Do the tickets contain enough detail for analysis?

Evaluate if your data will work with AI:

AI cannot process or analyze:

  • Names of people in interactions (gets removed for privacy)

  • Timestamps or specific dates

  • Ticket metadata like tags or custom fields

  • Screen recordings or attachments

  • PDFs or non-text files

What this means:

  • If your analysis depends on these elements, you'll need a different approach.

    • For example:

      • If you want to analyze response times, you need timestamps that the LLM will not factor in when analyzing your conversation data

      • You will need to create a Standard Classifier to help flag long response time tickets and then use that as a filter for your Classifier and/or Worksheet

Step 3: Choose Your Analysis Path

Based on what you found in Step 2, decide how to proceed:

Use Standard Classifiers when you need to:

  • Count specific events or phrases

  • Track when agents use particular scripts or responses

  • Calculate metrics that require metadata (like time between comments)

  • Filter for tickets that meet complex criteria

  • Layer multiple conditions together

Proceed to AI analysis when you have:

  • Tickets with enough conversational content

  • A clear theme that doesn't depend on metadata

  • Questions about sentiment, themes, or patterns in conversations

  • Need for understanding context and meaning

If you don't have enough tickets or the right type of data:

  • Adjust your filters (broaden or narrow your criteria)

  • Pick a different theme

  • Combine with Standard Classifiers to get a better dataset

Step 4: Test Your Dataset with AI

Before writing a detailed prompt, do a quick test to see if your data works:

Simple test prompt:

  • "Look at these tickets and tell me what the main themes are."

What to look for in results:

  • Does the AI identify themes that make sense?

  • Are the examples it gives actually from your data?

  • Do the patterns match what you see when you skim the tickets?

If the test works: Proceed to building your structured prompt

If the test doesn't work: Go back to Step 2 and adjust your dataset or filters

Step 5: Iterate Based on Results

Your first prompt rarely gives you everything you need. Plan to refine:

When initial results are too broad:

  • Add more specific context about your business

  • Narrow your focus to one specific aspect

  • Ask for more examples of the most important theme

When results miss the point:

  • Check if your dataset actually contains what you're looking for

  • Clarify your goal - be more specific about what you want to learn

  • Adjust your filters and try again

When results look wrong:

  • Verify the examples the AI gives you (do those ticket IDs actually exist?)

  • Check if the AI is making assumptions about your business that aren't true

  • Consider if your theme was too vague to begin with

Remember: It's normal to go through this process 2-3 times before getting useful results. The workflow isn't about getting it perfect on the first try - it's about approaching the problem systematically.

Did this answer your question?