RAG, fine-tuning, and prompt engineering solve different deficits in an LLM application. Prompt engineering sets instructions and examples in the query context. RAG finds external data and adds it to that context. Fine-tuning changes the model weights on training examples so it can reproduce the desired behavior more consistently.
Choosing a method by trend or by a polished demo is risky. First, classify the error: the model is missing a rule, a current fact, a stable format, specialized behavior, or a deterministic action. One answer may require several layers, but they should be added one at a time and only after a baseline is established.
Short answer: start with prompt and eval. If the answer depends on changing or restricted sources, add RAG. If, after that, a repeatable behavior defect remains across a sufficient sample of examples, consider fine-tuning. Get exact values and actions through an API, rules, or structured output, not from the model's memory.
Key points in one minute
- Prompt engineering changes the instruction, not the model's knowledge or weights.
- RAG supplies retrieved facts at query time and can show sources.
- Fine-tuning changes the model using examples, but it does not replace a source of current facts.
- API and business rules are the fourth option for precise data and actions.
- Compare methods on the same frozen eval and the same base model.
- A combination makes sense when each layer removes a measured class of errors.
- Count not just launch, but also updates, review, inference, and rollback.
Contents
- Three ways to adapt an LLM
- Method DIAGNOSIS
- The error points to the method
- When prompt engineering is enough
- When RAG is needed
- When fine-tuning is justified
- When API or plain code is needed
- How to combine methods
- How to compare options fairly
- Cost and operations
- Risks and security
- Experiment plan
- Decision card
- FAQ
- How AI Sunrise chooses an adaptation stack
- Conclusion
Three ways to adapt an LLM
| Method | What changes | Where the data lives | Typical strength | Main limitation |
|---|---|---|---|---|
| prompt engineering | instructions and examples | in the query/context | quickly test a rule and format | limited window, instability |
| RAG | retrieved context | in an external source and index | fresh/private facts, provenance | retrieval and source quality |
| fine-tuning | model weights | in the training set and model artifact | repeatable behavior on a narrow task | data, training, reevaluation, and rollback |
The official OpenAI guide to model optimization puts eval before prompt and fine-tuning: first measurement, then instructions, and training only for some scenarios. This order is useful regardless of the provider you choose.
These methods do not form a maturity ladder. Fine-tuning is not “above” RAG, and a long prompt is not “simpler” than retrieval in every case. They are different places where the team changes the system.
Method DIAGNOSIS
DIAGNOSIS — seven questions before choosing:
- D — Deficit: what is missing — an instruction, a fact, behavior, or an action?
- I — Instability: how often do knowledge, policy, and format change?
- A — Auditability: does the user need to see the source and version?
- G — Guardrails: what is the cost of a wrong answer or disclosure?
- N — Notebook of evals: are there representative and negative cases?
- O — Operating cost: who updates the prompt, index, dataset, and model artifact?
- Z — Zero-fallback path: clarify, abstain, human review, or deterministic fallback?
The result of DIAGNOSIS is not a technology name, but a testable hypothesis: “the error is caused by the absence of a document in context; adding permission-aware retrieval should improve the retrieval/answer rubric without access leakage.”
The error points to the method
| Observed error | First hypothesis | What to check |
|---|---|---|
| the answer violates an explicit rule | prompt/policy | instruction clarity, conflict, and priority |
| does not know the new policy | RAG or API | source, version, ACL, retrieval |
| confuses the exact order status | API/tool | schema, auth, current state |
| the format is unstable | structured output, prompt | schema compliance in eval |
| the style diverges from the examples | prompt, then fine-tuning | blind style rubric |
| classification is repeatedly wrong | prompt/few-shot, then tuning | confusion matrix |
| the quote does not support the conclusion | RAG + answer policy | retrieval and grounding separately |
| exposes a restricted document | access architecture | filter before retrieval, logs, cache |
If tests do not localize the error, adding a new layer creates another variable. For example, fine-tuning will not fix a missing document, and RAG does not guarantee JSON-schema compliance.
When prompt engineering is enough
Prompt works when the model already has access to the needed data and the task is to explain role, goal, constraints, sequence, and format. Few-shot examples help show edge cases without changing the weights.
OpenAI prompt engineering guide recommends building instructions with clear sections, relevant context, and examples; the specific techniques depend on the model family. In practice, a prompt should have an owner, version, and regression eval just like code.
Signals that a prompt is not enough:
- the required facts do not fit or keep changing;
- provenance and document access are required;
- there are too many examples;
- after systematic optimization, a stable class of errors remains;
- inference context becomes expensive or slow by your measurements.
Do not try to fix a missing business rule with a prompt. If the decision can be expressed as a conditions table, keep the rule in configuration and pass the model only the result or the allowed options.
When RAG is needed
RAG is chosen when the answer depends on documents or knowledge outside the base model: a catalog, policies, a knowledge base, contracts, tickets, or technical documentation. The source can be updated and reindexed without training a new model, and the answer can be tied to the retrieved passage.
RAG requires ingestion, parsing, chunking, metadata, ACL, retrieval, reranking, answer policy, eval, and monitoring. A detailed production setup is described in the article on a business RAG system.
RAG is not needed for every external value. For balance, price, status, or calculation, it is better to call an authorized API. Search is useful where you need to find unstructured context, not replace a transactional system.
When fine-tuning is justified
Fine-tuning is considered for a repeatable narrow task where you have high-quality input/output examples, a stable rubric, and a proven residual defect after prompt/baseline. Possible goals include classification, a specific format, style, terminology, or a shorter instruction at high request volume—the last one is confirmed only through TCO and quality measurement.
In Google Cloud documentation on tuning tuning is described as adapting a model on your own data for specific tasks. Actual capabilities, supported models, and formats depend on the platform and change over time; record the provider/model snapshot in the solution.
Fine-tuning is a weak choice for precise, changing facts. A training example influences model behavior, but it does not provide a controlled mechanism to “update one item, show the source, revoke access.” That requires RAG, an API, or a rules database.
Before training, check dataset provenance, usage rights, PII, contradictions, leakage between train/validation/test, and labeling quality. Save the base model, dataset version, parameters, eval report, and rollback path.
When an API or regular code is needed
The fourth option is often more important than the three trendy terms:
- an exact field from a CRM/ERP — an authenticated API;
- a tax or limit calculation — a verifiable function;
- an allowed status — an enum and schema validation;
- a critical decision — a rule engine and human approval;
- search by SKU — keyword/exact match;
- record update — a bounded tool with idempotency and audit log.
An LLM can understand intent and prepare arguments, but the system checks them before action. This reduces the scope of uncertainty: the model works with language, and code works with invariants.
How to combine methods
A combination makes sense when the roles do not overlap:
- Prompt + API: the model extracts intent, and the API returns the current value.
- Prompt + RAG: the instruction sets the policy, and retrieval brings back allowed facts.
- Prompt + fine-tuning: a short runtime prompt controls a model with learned behavior.
- RAG + fine-tuning: search handles knowledge/provenance, and tuning handles a narrow style or task.
- All layers + tools: only for a scenario where each layer has passed a separate ablation.
Ablation means comparing the system without the new layer and with it on the same eval. If fine-tuning does not improve the target rubric or degrades critical cases, its complexity is not justified.
How to compare options fairly
A single frozen eval includes common, rare, ambiguous, unanswerable, adversarial, and unauthorized cases. For each one, define the expected outcome and critical failures.
The comparison records:
- provider, model, and snapshot;
- prompt/policy version;
- corpus/index/retrieval version;
- fine-tune dataset and model artifact;
- tool schemas and business rules;
- temperature/effort and other significant parameters;
- quality, latency, and cost under one definition.
Evaluate retrieval, grounding, task correctness, format, safety, and action correctness separately. An average score should not hide a critical leak or a wrong action.
Cost and operations
| Layer | Creation | Runtime | Modification | Primary Owner |
|---|---|---|---|---|
| prompt | design + eval | context tokens | version + regression | product/prompt owner |
| RAG | ingestion + retrieval + eval | search + context + generation | reindex + regression | knowledge/data owner |
| fine-tuning | dataset + training + eval | tuned-model inference | dataset + retrain + release | ML/model owner |
| API/rules | integration + tests | call/compute | schema/rule release | system/process owner |
Do not make the blanket claim that “method X is cheaper.” Cost depends on request volume, context, update frequency, review, infrastructure, and incidents. Use the CAPEX/OPEX model from the article on the cost of AI implementation and plug in your own measurements.
Risks and Security
A prompt can be exposed or bypassed, RAG can return a forbidden or poisoned document, fine-tuning can memorize an undesirable pattern, and a tool can perform a dangerous action. No method by itself is a guardrail.
OWASP LLM01:2025 notes that RAG and fine-tuning do not fully eliminate prompt injection. That is why permissions, allowlist tools, validation, approval, isolation, monitoring, and incident response remain external controls.
Experiment Plan
- Build a baseline on the original model and a short prompt.
- Label the errors using DIAGNOS.
- Fix explicit instructions and business rules.
- Add structured output or an API where determinism is required.
- If external facts are still missing, test RAG separately.
- If a stable behavioral gap remains, prepare a tuning dataset.
- Compare the base and tuned models on a closed test set.
- Test the combination only after separate tests.
- Run shadow/canary and production sampling.
- Decide
scale / revise / stopand save the decision record.
Decision Card
- business process and owner;
- observed error and baseline;
- chosen layer and rejected alternatives;
- evidence: eval, sample, logs;
- data, permissions, retention, and versions;
- acceptance and critical fails;
- runtime/change-cost assumptions;
- fallback and human review;
- monitoring and incident owner;
- decision review date.
This card prevents “permanent architecture”: if the model, sources, or volume change, the decision can be revisited using the same criteria.
Frequently Asked Questions
What should I choose for an internal knowledge base?
Usually you need prompt + RAG with document-level permissions, citations, and abstention. But first check source quality; for precise structured values, use an API.
Can fine-tuning load company knowledge?
Training data can influence responses, but fine-tuning does not replace a controlled source of current facts, versions, and permissions. For changing knowledge, use retrieval or an API.
What should I try first?
Baseline, eval, and a clear prompt. Then add the smallest layer that addresses the measured gap.
Can RAG and fine-tuning be used together?
Yes. RAG can handle facts and provenance, while tuning can handle stable narrow behavior. Prove the value of each layer with separate comparisons.
Does long context replace RAG?
Sometimes a small fixed document set can be passed in full. But access, selection, updates, latency, cost, and provenance still remain. Compare a long-context baseline with retrieval on your own data.
How do I know whether fine-tuning will pay off?
You need measured volume, defect, quality after tuning, dataset/training/release cost, and runtime cost. Without that data, ROI is Unknown.
How AI Dawn chooses an adaptation stack
AI Dawn can audit the process, collect eval and baseline results, configure prompts, RAG, integrations, and bounded tools, prepare a fine-tuning experiment, compare options, and organize launch, monitoring, training, and support.
The safe first step is to choose one process, describe the observed error, sources, constraints, and acceptance criteria. Then test the smallest change on a frozen eval. Discuss the task.
Conclusion
RAG, fine-tuning, and prompt engineering are not mutually exclusive products. Prompt controls the instruction, RAG brings in external facts, fine-tuning changes the model's stable behavior, and API and rules provide exact values and actions.
Use DIAGNOS: name the gap, variability, audit requirements, risk, eval, operating cost, and fallback. Start with the smallest change and add the next layer only when its contribution is visible in tests.