Introduction
If you work with contracts, financial reports, or medical documentation, you’ve likely wondered how to get an LLM to â€Å"speak†accurately and reliably. The answer isn’t a single prompt; it’s a promptâ€'chaining workflow that leverages Chain of Thought (CoT), ReAct, and Fewâ€'Shot learning. In this article I’ll walk you through each step with concrete examples, code snippets, and metrics to monitor accuracy.
1. When and How to Integrate Chain of Thought (CoT)
1.1 Why stepâ€'byâ€'step reasoning matters
- Contracts: many clauses depend on sequential conditions.
- Financial reports: evaluating a key indicator requires intermediate calculations.
- Medical documentation: diagnoses and treatments evolve in logical steps.
1.2 Adding CoT to the prompt
A simple prompt like â€Å"extract the expiration date†often yields incomplete results. Let’s add intermediate reasoning:
\nPrompt = "\n\nHere is the contract: [CONTENT] \n\n1. Read the entire document. 2. Identify all dates. 3. Select the expiration date. 4. Return the date in ISO format.\n";
The model, now guided, processes each step before giving the final answer.
2. ReAct: Dynamic interaction with the document
2.1 What ReAct is and why it’s useful
ReAct combines Reasoning and Act: the model can â€Å"read†parts of the text, ask clarifying questions, and then act. This is crucial when the document is long or ambiguous.
2.2 Practical implementation with the OpenAI API
import openai
messages = [
{"role": "system", "content": "You are a legal assistant. Use ReAct to extract data."},
{"role": "user", "content": "Here is the contract: [CONTENT]"}
]
response = openai.ChatCompletion.create(
model="gpt-4o-mini",
messages=messages,
temperature=0.0,
max_tokens=800
)
print(response.choices[0].message.content)
The model can return an action such as READ_SECTION followed by its result, enabling a feedback loop.
3. Fewâ€'Shot Learning: Optimal Strategies
3.1 How many examples and what granularity?
- Contracts: 3â€'5 examples of clauses with expiration dates.
- Financial reports: 4â€'6 examples of KPI calculations.
- Medical documentation: 5â€'7 examples of diagnoses and medications.
3.2 Example structure
\nExample 1:\n- Prompt: "\nContent: ...\n\nExecute CoT to extract the expiration date.\n"\n- Response: "\nStep 1: Identify the clause...\nStep 2: ...\nExpiration date: 2025â€'12â€'31."
Examples should be anchored (referencing points in the document) and concise.
4. Designing the Promptâ€'Chaining Workflow
4.1 Basic architecture
- Preâ€'processing: tokenization and chunking with
RecursiveCharacterTextSplitter. - CoT + ReAct: the main model that generates intermediate steps.
- Postâ€'processing: normalizing results (e.g., ISO dates, decimal numbers).
- Feedback loop: if the output fails to meet the
confidence threshold, trigger a new iteration.
4.2 Error handling
- Timeouts â†' fallback to a smaller model.
- Ambiguity â†' request clarification via ReAct.
- Unexpected value â†' recalculate with a new prompt.
5. Evaluation Metrics and Iteration
5.1 Key metrics
- Accuracy: percentage of correctly extracted fields.
- F1â€'Score: balance between precision and recall.
- Latency: average response time per document.
5.2 Iteration techniques
- A/B testing of prompts: variations of CoT, ReAct, and Fewâ€'Shot.
- Fineâ€'tuning on a domain dataset (e.g., 1 k contracts with ground truth).
- Active learning: ask the user to validate only ambiguous cases.
Conclusion
A promptâ€'chaining workflow that combines CoT, ReAct, and Fewâ€'Shot learning is no longer a futuristic ideaâ€"it’s a proven practice for extracting structured data from complex technical documents. Start with a handful of wellâ€'structured examples, insert stepâ€'byâ€'step reasoning, and leverage ReAct’s dynamic capabilities to handle ambiguity. Continuously monitor metrics and iterate: that’s how you build a robust, scalable system ready for the real world.
Practical Takeaways
- CoT is essential for documents with sequential logic.
- ReAct lets you â€Å"read†parts of the text dynamically.
- Fewâ€'Shot: 3â€'5 examples per domain, with a consistent format.
- An internal feedback loop can cut errors by 30â€'40% in a month.
- Target metrics: accuracy ≥ 92 % and latency ≤ 2 s are realistic goals.
Frequently Asked Questions
What’s the difference between Chain of Thought and ReAct?
Chain of Thought is a stepâ€'byâ€'step reasoning process where the model logically works through the answer. ReAct adds the ability to interact dynamically with the document, requesting actions like â€Å"read section X†before replying.
How many Fewâ€'Shot examples are ideal for a technical domain?
Between 3 and 6 wellâ€'structured examples are usually sufficient for complex domains. More examples can improve generalization but also increase prompt length and execution costs.