To find semantic duplicates in the FAQ, convert the short questions into embeddings, generate high-similarity pairs, and compare the full answers along with the conditions under which they apply. Automatic search prepares a queue for the editor. The decision to merge depends on whether both questions can be answered by one live knowledge base article.
This guide is intended for a support manager, knowledge base editor, and technical specialist who can help run Python. The result is a CSV with merge candidates and a clear workflow for handling them. You do not need to set up a vector database or connect a generative model for this example.
The documentation and model card were checked on November 1, 2026. Below is an adapted, documented workflow and manually prepared examples. The model was not run on this data when preparing the article: numerical similarity scores, speed, and quality in Russian were not measured. The Python syntax and the structure of the sample CSV were checked separately; this is not a test of the model’s performance.
Contents
- 1. Determine which articles are truly duplicates
- 2. Export questions with stable identifiers
- 3. Set up the environment and choose a model
- 4. Generate candidate pairs into a separate CSV
- 5. Review pairs by answers and conditions
- 6. Update the knowledge base while preserving question variants
- Common mistakes and how to fix them
1. Determine which articles are truly duplicates
A semantic duplicate is two entries that express the same reader need and can use the same approved answer under the same conditions. Topic overlap alone is not enough. “How do I return an item?” and “How do I return a defective item?” may require different procedures.
Start with one area, such as an e-commerce FAQ about delivery. Mixing internal warehouse instructions, customer responses, and partner rules will make the review harder: a similar question may apply to different audiences.
Before exporting, align on three decisions with the knowledge base owner:
- which conditions change the answer: sales channel, contract, region, customer type, revision date;
- which source confirms the active rule;
- who is allowed to merge articles and change their links.
This is about editorial cleanup of an FAQ, not merging customers in a CRM or deleting legal documents. For broader data cleanup, there is a separate article on preparing data for AI.
Stage result: a limited set of articles and a definition of duplicates that accounts for both the answer and the scope.
2. Export questions with stable identifiers
One row should correspond to one article. Preserve the ID, question, full answer, source link, and conditions of use. Do not replace the ID with the row number in the editor: sorting will change the order.
For the minimal run, you only need id and question; leave the other fields in the original export for editorial comparison. Sample file faq.csv in UTF-8 with a comma delimiter:
id,question
F001,How can I find out where my order is?
F002,Where can I check the order status?
F003,Can I change the address after shipment?
F004,Can I change the address before shipment?
F005,How can I get closing documents?These are fictional questions, not customer inquiries. F001/F002 are intended as a possible duplicate. F003/F004 are a control pair: one word changes the situation. The actual order in which the model will return these pairs is not predicted here.
Remove empty rows. Check that IDs are unique. Keep a source copy with the full answers: the search file should not become the only copy of the database.
If the table was exported with semicolons, export it again with commas or change delimiter in file reading. Do not fix the CSV by replacing every comma in the text: that will damage the questions themselves. The standard csv module in Python supports tabular reading through DictReader and delimiter parameters.
Stage result: a file with non-empty questions and unique IDs that let you return to the full answer.
3. Set up the environment and choose a model
An embedding is a numerical representation of text that makes it possible to compare how close two wordings are. This example uses Sentence Transformers and the model sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. Its model card lists Russian; that is a reason to include it in a test, not proof of quality for your terminology. The stated parameters — 384 dimensions and a 128-token sequence limit — describe the model, but do not guarantee accurate duplicate detection. Model card.
That is why we compare short questions. We do not add long answers, reference data, and change history in full to the model input: an important difference may fall outside the text being processed. We will compare the full answers manually later.
Use a separate Python environment. Commands for the Linux/macOS terminal with python3:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U sentence-transformersThis is the documented way to install the library, not an installation reproduced here. The current instructions recommend Python 3.10+ and describe PyTorch dependencies; a suitable build is required for GPU use. The example below explicitly uses CPU. Installation documentation.
Initial loading by model name requires access to its repository. In a closed environment, agree on artifact loading and dependencies separately. The word “local” itself does not prove that there are no network requests anywhere in the environment.
Stage result: the environment can import the library and load the selected model. After a successful run, save the dependency versions and model revision so the result can be reproduced.
4. Generate candidate pairs into a separate CSV
The paraphrase_mining function returns triples: a similarity score and the indices of two source texts. It does not require a pre-labeled list of correct duplicates. It looks for close formulations; editorial equivalence of answers remains a separate task. Official paraphrase search example.
Save this adapted example as find_pairs.py next to faq.csv:
import csv
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import paraphrase_mining
with open("faq.csv", encoding="utf-8-sig", newline="") as f:
rows = list(csv.DictReader(f))
if not rows or not {"id", "question"} <= rows[0].keys():
raise ValueError("The file must include id and question columns and must not be empty")
ids = [r["id"].strip() for r in rows]
questions = [r["question"].strip() for r in rows]
if any(not x for x in ids + questions) or len(set(ids)) != len(ids):
raise ValueError("Check for empty values and duplicate IDs")
if len(rows) < 2:
raise ValueError("You need at least two cards")
model = SentenceTransformer(
"sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
device="cpu",
)
pairs = paraphrase_mining(
model, questions, batch_size=16,
query_chunk_size=128, corpus_chunk_size=1024,
top_k=20, max_pairs=10000,
)
with open("pairs.csv", "w", encoding="utf-8-sig", newline="") as f:
writer = csv.writer(f)
writer.writerow(["score", "id_a", "id_b", "question_a", "question_b"])
for score, i, j in pairs:
writer.writerow([
float(score), ids[i], ids[j], questions[i], questions[j]
])Run it from the same folder:
python find_pairs.pyThe parameters here are a starting configuration for a training example, not a measured optimum. batch_size refers to text encoding, while chunk sizes apply to comparison, top_k limits the number of neighbors per text in a chunk, max_pairs is the total number of returned pairs. Reducing chunk sizes lowers memory usage during comparison and can increase runtime. Function reference.
A sign of technical success: a readable pairs.csv file appeared; both ID columns point to the original cards, and the questions in the pairs match those IDs. This does not yet mean the duplicates were found correctly. The absence of a pair in the output under the limits top_k and max_pairs does not prove it is unique.
5. Review pairs by answers and conditions
Start with the highest similarity scores, but do not turn the first number you like into a deletion rule. A score is a measure of embedding closeness, not the probability that cards can be merged, and not the confidence of a legal or product expert.
For each pair, open both full cards. Check the answer, source, conditions, and effective date. Use this editorial framework:
| Situation | Decision | What to keep |
|---|---|---|
| One question, the same active rule | Choose the primary card | Keep the second wording as an alternate question; link the old ID to the primary one |
| Similar question, different conditions | Keep them separate | Condition in the title or explicit field |
| Same situation, conflicting answers | Escalate to the rule owner | Both sources until the conflict is resolved |
| Old and new versions of one rule | Apply the database versioning order | The effective date and archival link |
In the training set, F001/F002 can be merged only after confirming that both cards point to the same order-tracking method. F003/F004 should be reviewed separately until the process owner confirms the same address-change workflow before and after shipment. The model does not make that decision for the owner.
Do not merge a group based on the chain "A is close to B, B is close to C." Individual pairs do not establish that all cards in the group are interchangeable. Before merging each record, compare it with the selected primary answer.
Stage result: the reviewed pairs have decisions with reasons, and disputed rules are sent to the responsible owner without automatically averaging the answers.
6. Update the database while preserving question variants
After approval, move useful wording into the main card or into the alternate-question field supported by your system. If there is no such field, keep a separate mapping of "old ID → primary ID." Do not add made-up fields to the live system without changing its schema.
First check the links, bookmarks, and integrations that reference the old IDs. Then update search according to the rules of the platform you use. If the FAQ is part of RAG, changing the cards and updating the search index are different operations; the release order is explained in the RAG knowledge base update checklist.
Check the original question wording in the user interface. It should lead to the approved answer with the correct conditions. Separately test a question with an exception, for example, changing the address after shipment. A short knowledge base that loses an important exception is not better.
Stage result: repeated wording finds the primary card, while different procedures remain available separately.
Common mistakes and how to fix them
| Symptom | Possible cause | Action |
|---|---|---|
| The file is read as a single column | A different CSV delimiter | Specify the actual delimiter or repeat the export |
| Questions are highly similar but have different answers | The shared topic is stronger than the difference in conditions | Compare the full answers; clarify the condition in the question |
| Long cards look the same | An important ending may not have been included in the model input | Compare the short question; analyze long answers separately |
| The process is running out of memory | Encoding or comparison batches are too large | Reduce the relevant sizes; try again |
| Too few useful duplicates were found | The model is not suited to the terminology or the pair set is limited | Review the misses, expand the search; compare other models |
When switching models, use the same pre-reviewed set of pairs, including pairs that are similar by topic but different by rule. Count false positive merges and missed duplicates separately. The embedding model selection method will help you compare options without relying only on popularity.
Short answers to follow-up questions
Can I delete everything above the similarity threshold right away?
There is no universal, verified threshold for your database. Even a high score does not replace rule comparison. Start with manual review and labeled examples; treat deletion automation separately from candidate search.
Is it necessary to send the FAQ to a cloud chat?
No. The example uses a library and a downloadable model on CPU. It does not call cloud-based response generation. You need to verify download conditions, network settings, and data-processing permissions for your own environment.
Is this script suitable for a huge archive?
It shows the basic workflow. Output limits and chunk sizes require tuning; time and memory were not measured here on a large archive. Start with a limited topic-specific export, then choose the search architecture for your data volume.
How Vladimir Lomtev’s AI Studio helps clean up an FAQ database
If duplicate detection is needed within an active support operation, Vladimir Lomtev’s AI Studio can connect this task with data preparation and the corporate knowledge base: define the card structure and answer usage rules, build an MVP for finding similar records, integrate approved cards with an AI assistant, and test the responses against employee questions.
The first step is to choose one FAQ section, its owner, the sources of the rules, and the success criterion: identical questions return a single answer, while different conditions are preserved. Discuss the project.
Bottom line: search for similar questions and combine verified answers
Start with a topic-specific export, get candidate pairs, and use IDs to return to the full records. Merge records only after the rule has been verified, keep question variants, and test the search after each update. The practical result is a database with fewer duplicate entries and preserved differences that determine the correct answer.
