To prepare a gold standard for an AI ticket classifier, create a Label Studio project with text and a single category choice, import tasks, label them according to the shared guidelines, and export the result to JSON. Here, a gold standard means consistent human annotation: a language model’s response cannot be automatically treated as the correct label.
This guide is intended for a support manager and analyst who want to distinguish ticket reasons and then train or validate a classifier. The workflow is based on the official documentation, checked on November 4, 2026. The Label Studio interface and model training were not run for this article. The JSON and XML below are suggested training inputs, not an export from a live project.
Contents
- Prerequisites: what to prepare before import
- Step 1. Define categories based on support resolutions
- Step 2. Prepare JSON with stable identifiers
- Step 3. Set up text and single-category selection
- Step 4. Import tasks and label a test batch
- Step 5. Resolve disagreements before the full batch
- Step 6. Export JSON and find selected values
- Common mistakes and how to fix them
Prerequisites: what to prepare before import
You need an accessible Label Studio project with permission to change the configuration and import tasks. Server installation, access setup, and edition selection are outside the scope of this guide. For the first pass, use fictitious text; upload real tickets only in an approved environment with an allowed data set.
Choose the unit of annotation. In this example, it is one ticket as a whole, not a single sentence and not the customer’s full history. For each record, keep an internal ID, the source text, and the rules version. That way, after export, you can link the category to the correct record without matching texts by similarity.
The result of this step is a clear answer to the question: “What exactly is the person annotating?” If two messages in the same conversation require shared context, combine them first according to the agreed rule. Do not ask the annotator to guess missing context on their own.
Step 1. Define categories based on support resolutions
For this example, we will use “Payment,” “Delivery,” “Access,” and “Other.” This is a suggested scheme that you should adapt. Categories should support the next action: routing or cause analysis, not just reflect frequently used words.
| Category | When to choose | Example |
|---|---|---|
| Payment | Issue with a charge or refund | “The payment was charged twice” |
| Delivery | Issue with timing or receipt | “The package did not arrive” |
| Access | Cannot sign in or use the account | “The login code is not coming through” |
| Other | No rule applies | “I’d like to propose a partnership” |
Add a rule for overlaps. For example, “I can’t log in after payment” should be classified as “Access” if the goal is to restore sign-in rather than fix the payment. This is an editorial decision made by the process owner, not a Label Studio feature.
The result is an instruction set with positive and edge-case examples. If one text naturally requires several independent reasons, revisit the task definition: single choice is not suitable for a multi-label task.
Step 2. Prepare JSON with stable identifiers
Save the following training list to a file tickets.json in UTF-8:
[
{"data": {"ticket_id": "DEMO-A", "text": "The payment was charged twice"}},
{"data": {"ticket_id": "DEMO-B", "text": "The login code is not coming through"}},
{"data": {"ticket_id": "DEMO-C", "text": "The package did not arrive"}}
]In Label Studio import format the object keys in data are mapped to fields in the configuration. In our example, the text is in text; ticket_id is needed to link back to the source system later. Do not use the ticket text as the only identifier: identical wording can appear in different records.
The result of this step is a valid task list with non-empty texts and distinct IDs. Before importing, check that the file does not contain contact details, signatures, or attachments that the annotator does not need. For a separate data-cleaning task, the Presidio FAQ before LLMs will be useful; it does not replace access rules for the annotation system.
Step 3. Set up text and single-category selection
Open the project settings and the Labeling Interface section, then switch to Code mode. Suggested configuration:
<View>
<Text name="ticket" value="$text"/>
<Choices name="category" toName="ticket" choice="single" required="true">
<Choice value="Payment"/>
<Choice value="Delivery"/>
<Choice value="Access"/>
<Choice value="Other"/>
</Choices>
</View>$text points to the field in the input record. toName="ticket" links the choice control to the Text object, and name="category" sets the name of the control. The single type limits the user to one category; required forces a selection before saving. These parameters are described in the Choices documentation.
Save the configuration. Before working at scale, open a test task and make sure the source text is visible and the required set of categories is available. A blank text often means the field name and value do not match, not that there are no tasks.
The result is a saved configuration with verified mapping. If you modify an existing project, keep in mind the limits on changing labels already used in annotations; they are described in the labeling interface settings.
For a new scheme, it is safer to plan versioning and result migration in advance.
Step 4. Import tasks and label a test batch In the project, open Import, select tickets.json and complete the import using the on-screen prompts. Then open a task, choose a category, and save the annotation with the Submit button. The basic workflow is described in the labeling guide
For the training set, expect “Payment,” “Access,” and “Delivery.” These are the expected answers under the rules above, not a measured quality score for the annotators. Also check for missing responses: a required choice should prevent someone from accidentally saving an empty category.
The result is several completed tasks with the category saved. If the interface lets you select an answer but the export does not include it, analyze annotation saving separately: a visible label in an open task does not yet prove a saved record.
Step 5. Resolve disagreements before the large batch
The recommended workflow is to give one test set to two people independently and compare the answers outside the system if the necessary agreement features are unavailable in your edition. Do not assume that any plan automatically includes consensus, automatic assignment, and all quality metrics.
Review each type of discrepancy. Different interpretations of “Other” are fixed by clarifying the rules. Missing context is fixed by changing the input data. Two simultaneously correct categories are handled by revising the single-label classification or by a rule for choosing the primary cause.
The result is a consolidated version of the instructions and a list of records that need to be reviewed. Do not just fix the labels while leaving an ambiguous rule in place: the next annotator will repeat the same mistake.
Step 6. Export the JSON and find the selected values
Open Export and select JSON. Save the full original export: it is needed to verify the structure and the source of the annotations. In the export description the task data and annotation results are separated.
For this schema, look for the needed element inside annotations[].result[]: from_name must match category, to_name — ticket, type — choices, and the selected values are in value.choices. The structure is also shown in the official JSON format breakdown. Check the structure in the actual exported file from your version. Do not take the first result you find as the category: additional control elements may appear in the project.
When exporting through the Community Edition interface, canceled annotations may also end up in the result; the tab filters do not limit this kind of export. This is stated separately in the export documentation. So explicitly define which records count as completed and are allowed into the gold standard.
The result is a table of “ticket ID → category” with the link to the original annotation preserved. If a task has multiple annotations, first apply the agreed rule for selecting the final answer. A random first answer does not become the gold standard just because of its position in the array.
Typical errors and how to fix them
| Symptom | What to check | Fix |
|---|---|---|
| The task shows a blank text screen | The key data.text and $text |
Align the field name |
| Categories are selected, but no results are found | Saving and the structure of annotations |
Export the completed annotation and verify it |
| “Other” absorbs almost everything | Category boundaries | Add clear rules and examples |
| The same text gets different labels | Context and priorities | Identify the cause of the mismatch and update the instructions |
For training and evaluating the model, split the data before configuring the classifier and do not use the reference answers to tune the prompt. This instruction creates human annotation; Mistral batch annotation automation covers a different stage and requires separate error evaluation.
How Vladimir Lomtev’s AI studio helps build a gold standard for support tickets
Vladimir Lomtev’s AI studio works on data preparation, AI integration with business systems, and testing. In this task, the work may include describing categories and fields, linking the export to the source tickets, and comparing classifier answers with the agreed annotation.
The first step is to choose one routing process, describe the current workflow, the ticket sources, access restrictions, and the criterion for the correct category. The specific annotation scheme must be agreed before scaling; future model quality is not guaranteed by the tool choice alone.
Bottom line: what the final result should look like
The final result is source texts with stable IDs, a version of the rules, and agreed categories that are actually present in the export. Start with the training set, verify field mapping and annotation saving, then work through edge cases. After that, you can scale up the annotation and separately evaluate the AI classifier on a holdout set.
