To set up RAG and retrieval, confirm the input format/device, choose output settings, then run a small batch. RAG sends only relevant chunks, saving tokens versus a full document.
Why it matters
Validate on a small sample before running the whole batch.
Chunks too large waste; too small lose context.
Step by step
- Confirm the model context and output budget.
- Paste the prompt/document and estimate tokens/usage.
- Above 80-90%, split, summarize or retrieve.
- Count system prompts/tool schemas too.
- Verify billing/hard limits with the official tokenizer.
Common mistakes
- Treating character count as token count.
- Ignoring output/system prompt usage.
- Assuming all models use the same tokenizer.
- Stuffing a long document instead of retrieving/summarizing.
Checklist
- Model context confirmed.
- Output budget reserved.
- System/tool prompts counted.
- Long text retrieved/split.
- Critical cases checked with the official tokenizer.
Key facts
- Chunks too large waste; too small lose context.
- Reranking/dedup saves.
FAQ
How do I set up RAG and retrieval?
To set up RAG and retrieval, confirm the input format/device, choose output settings, then run a small batch. RAG sends only relevant chunks, saving tokens versus a full document.
How long does it take?
Usually 1-3 minutes; for intermittent issues, run 30 seconds and retest elsewhere.
Is data uploaded?
Runs locally and does not upload video/audio; extensions may still read the page.
Source: RAG paper