Chunk text for RAG by estimated tokens or characters. Set the overlap, keep sentences intact, and export JSONL.
This tool runs entirely in your browser. What you enter is never sent anywhere.Your input is never sent anywhere.
DetailsText to split
0 characters · 0 words
Chunking settings
Split unit
Split mode
Chunk size (tokens)
Overlap (tokens)
Preferred break points
Model family for token estimates
ADVICE
Paste some text to start. A common starting point is a chunk size of 400 to 512 tokens with 10% to 20% overlap. Test it against your own queries.
How RAG Chunking Works: Chunk Size, Overlap, and Token Limits
A retrieval-augmented generation (RAG) system does not embed a whole document at once. It splits the text into chunks, turns each chunk into a vector, and stores the vectors in a database. This tool splits pasted text into chunks sized by estimated tokens or by characters, with an overlap you control.
Embedding models and LLMs set their input limits in tokens, not characters, so it is easier to plan chunk size in tokens. The token counts shown here are estimates based on average characters-per-token ratios for the model family you pick. They do not come from the model's own tokenizer.
Sentence-aware mode ends chunks at paragraph, sentence, or line breaks where it can. Fixed-size mode cuts at exact character positions. When you are done, copy the chunks or download them as JSONL with the chunk index, character offsets, estimated tokens, and text.
Where a Text Chunker Fits in Your RAG Pipeline
Preparing documents for a vector database
Split knowledge base articles, help center pages, or internal policies into chunks before embedding. Export JSONL and feed it to your ingestion script.
Testing chunk size and overlap
Try 256, 512, and 1,024 tokens with different overlaps and compare chunk counts and total tokens before you pay for embeddings.
Checking chunks against an input limit
Use the average and largest chunk estimates to see how close your chunks come to the token limit of your embedding model.
Comparing sentence-aware and fixed-size splits
Run the same text through both modes and see where each one cuts, especially in long paragraphs, lists, and code samples.
How to Split Text Into Chunks in 4 Steps
- Paste your text into the box, or click "Insert sample text".
- Choose a split unit (tokens or characters), then set the chunk size and overlap. The 10%, 15%, and 20% buttons set the overlap from your chunk size.
- Pick a split mode and the model family used for token estimates. In sentence-aware mode, choose which breaks to prefer.
- Review the chunks (overlap is highlighted in blue), then copy them or download JSONL or a plain text file.
Note: Your text is processed in your browser and is not sent to a server.
Token Estimates and Overlap: What to Know Before You Index
- Token counts are estimates from average characters-per-token ratios for each model family. They will not match a model's real tokenizer exactly.
- When you size chunks in tokens, the tool converts that size to characters using your text's average ratio and then splits by characters. Individual chunks can land above or below your target, especially in text that mixes prose, code, and tables. Check the Largest chunk figure.
- In sentence-aware mode, chunks and their overlap are built from whole segments. The actual overlap can be smaller than the value you set, because only segments that fit entirely are repeated. A single segment longer than the chunk size is cut at fixed positions.
- Sentence detection looks for a period, question mark, or exclamation mark followed by whitespace. Abbreviations such as "Dr. Smith" or "e.g. this" are treated as sentence ends, while decimals like 3.5 are not.
- More overlap means more chunks and more text to embed. Roughly, total text grows by 1 / (1 − overlap fraction): about 1.25x at 20% and 2x at 50%.
- The start_char and end_char offsets count JavaScript string positions, so an emoji counts as two characters.
Chunk Size and Overlap Starting Points by Document Type
These are starting points to test, not standards. The best values depend on your documents, your embedding model, and the questions users ask.
| Content type | Chunk size | Overlap | Note |
|---|---|---|---|
| FAQ / short Q&A | 200–400 tokens | 10–20% | Keep each question and answer together when you can. |
| General document search | 400–512 tokens | 10–20% | A common first setting to test. |
| Technical docs and manuals | 1,024–2,048 tokens | 10–15% | Longer chunks keep code samples and multi-step procedures together. |
| Chat history / agent memory | 256–512 tokens per turn | Little to none | Splitting by turn or topic usually works better than by size. |
| OpenAI file search default | 800 tokens | 400 tokens (50%) | Documented default for vector store files. Overlap can't exceed half the chunk size. |
The first four rows are rules of thumb. Measure retrieval quality on your own data before you settle on a value.
Why Chunking Choices Change Retrieval Quality
How chunk size, token limits, and overlap affect what your RAG system retrieves.
Chunk size decides what each search hit contains
A chunk that is too small can lose the context around a fact, so a retrieved piece may not answer the question on its own. A chunk that is too large can mix several topics, so the model receives extra text that has nothing to do with the query. Adjusting chunk size is a cheap change to test before you switch embedding models.
Why size chunks in tokens instead of characters
Embedding models and LLMs limit input in tokens. The same number of characters can produce very different token counts depending on language and content, and code or non-English text often uses more tokens per character than English prose. For Latin letters and digits, this tool assumes about 4 characters per token for GPT models, and lower ratios for other families.
Overlap is a trade-off, not a free fix
Overlap repeats the end of one chunk at the start of the next, so a sentence on a boundary is not lost. The cost is more chunks and more text to embed. A common starting range is 10% to 20%, though OpenAI's documented default for file search is 400 tokens of overlap on 800-token chunks. Test a few values on real questions instead of assuming more is better.
When sentence-aware splitting helps, and when fixed-size is simpler
Fixed-size cuts can end a chunk mid-sentence, so a definition or step may be split across two chunks. Sentence-aware mode closes chunks at paragraph, sentence, or line breaks when it can. For logs, minified code, or text with few natural breaks, fixed-size mode is simpler and behaves more predictably.
Common Chunking Mistakes and How to Fix Them
Sizing chunks in characters and running into the embedding input limit
A chunk that looks short in characters can still use more tokens than your embedding model accepts, especially with code or non-English text.
💡 See the solution▼
Setting overlap to zero
An explanation that straddles a chunk boundary gets cut in two, and neither chunk contains the full answer.
💡 See the solution▼
Using so much overlap that the index fills with near-duplicates
Very high overlap multiplies the number of chunks and can return several nearly identical chunks for one query.
💡 See the solution▼
Letting fixed-size splits cut through sentences
A definition or procedure step ends up half in one chunk and half in the next, so a retrieved chunk reads as incomplete.
💡 See the solution▼
Using one chunk size for every kind of document
A short FAQ entry and a long technical manual need different chunk sizes. One setting for both either splits short entries apart or lumps long sections together.
💡 See the solution▼
Token Counts, Overlap, and JSONL Export: FAQ
Q.What is chunking in RAG?
A. Chunking means splitting a long document into smaller pieces before you embed them and store them in a vector database. At query time, the system retrieves the chunks that best match the question and passes them to the language model. How you cut the text decides what each retrieved piece contains, so chunking affects answer quality.
Q.What chunk size should I use for RAG?
A. There is no single best value. Short Q&A content often works with a few hundred tokens, and general documents are commonly tried at 400 to 512 tokens. OpenAI's documented default for file search vector stores is 800 tokens per chunk with a 400-token overlap. Treat these as starting points, then test candidate sizes against real questions from your own data.
Q.How much chunk overlap should I use?
A. A common starting range is 10% to 20% of the chunk size. Overlap keeps a sentence or explanation that sits on a boundary available in both neighboring chunks. The cost is more chunks and more text to embed: with 50% overlap you store roughly twice the original text. OpenAI's file search default uses 400 tokens of overlap on 800-token chunks, so higher values are used in practice. Compare a few settings on your own queries.
Q.Should I split by tokens or by characters?
A. Embedding models and LLMs set their input limits in tokens, so sizing chunks in tokens is easier to check against those limits. Characters per token vary by language and content, and code or non-English text often uses more tokens per character than plain English prose. This tool lets you pick either unit. In token mode it converts your size to characters using the average ratio of your pasted text, so check the Largest chunk figure after splitting.
Q.How accurate are the token counts?
A. They are estimates. The tool applies average characters-per-token ratios for the model family you select, with separate ratios for Latin letters and digits, CJK characters, and other symbols. It does not run the model's actual tokenizer, so counts can differ from what your API reports. If a chunk needs to stay under a hard limit, leave some margin or confirm with the official tokenizer.
Q.Which split mode should I choose: sentence-aware or fixed-size?
A. Start with sentence-aware for prose such as articles, help pages, and policies. It closes chunks at the breaks you enable (paragraphs, sentences, line breaks, and optionally commas and semicolons), so chunks are less likely to end mid-sentence. Fixed-size cuts at exact character positions and suits text without natural breaks, such as logs or minified code. Because it ignores word boundaries, it can cut through a word.
Q.Is my text uploaded anywhere?
A. No. The splitting and the token estimates run in your browser with JavaScript, and the tool does not send your text to a server.
Q.What does the JSONL download contain?
A. One JSON object per line, saved as chunks.jsonl. Each object has chunk_index (starting at 0), start_char, end_char, estimated_tokens, and text. The offsets are positions in the text you pasted, counted as JavaScript string indexes. This format is easy to read line by line in an ingestion script.
Your feedback
makes this tool better.
Tell us which features you'd like, what should be fixed, or which new tool you'd like to see. An engineer reads every submission and uses it to guide development.