Text chunker
Description
The Text chunker transform splits a text field into smaller chunks and emits one output row per chunk, carrying the fields of the input row along with each.
Chunking is the first step of a retrieval pipeline: an embedding model has a bounded context, and retrieval works better on passages than on whole documents. A typical pipeline is Text chunker, then an embedding transform, then a vector store.
The transform has no dependency on an AI provider or a model. It is plain text processing and can be used on its own wherever a large document needs to be broken into passages.
Chunking strategies
| Strategy | Behaviour |
|---|---|
Character |
Split on a fixed character count, backing off to the nearest word boundary so words are not cut in half. Only a word longer than the chunk size is cut, and never inside a surrogate pair such as an emoji. Overlap carries the last whole words of each chunk into the next, which keeps a sentence that straddles a boundary retrievable from both sides. |
Paragraph |
Split on paragraph breaks, packing whole paragraphs up to the chunk size. A paragraph longer than the chunk size falls back to character splitting. By default a paragraph break is a blank line (any |
Structure |
Parse the document into a heading tree and emit one chunk per section, prefixing each with a breadcrumb of its heading path so an isolated chunk still says where it came from. A section larger than the chunk size falls back to character splitting within that section. |
Structure parsers
The Structure strategy selects a parser from the content type:
| Content type | Parsed structure |
|---|---|
Auto |
Infer from the content type field, or from the text itself: a pipeline or workflow when the XML root element is |
Plain |
Blank-line separated blocks. |
Markdown |
|
AsciiDoc |
|
Pipeline |
A Hop pipeline ( |
Workflow |
A Hop workflow ( |
Metadata |
A Hop metadata JSON document: an overview plus one section per entry. |
The Hop-native parsers make a pipeline, a workflow or a metadata file retrievable per transform, action or entry rather than as one opaque blob.
Options
| Option | Description |
|---|---|
Transform name |
Name of the transform, unique within the pipeline. |
Input field |
Field holding the text to split. |
Source document ID field |
Optional field holding the identifier of the source document. When empty, a row counter is used. |
Output chunk field |
Output field that receives the chunk text. |
Chunking strategy |
Character, Paragraph or Structure, as described above. |
Paragraph separator |
Optional literal separator for the Paragraph strategy, for example |
Content type |
Document format used by the Structure strategy. |
Content type field |
Optional field naming the content type per row, for a stream that mixes formats. |
Chunk size |
Maximum chunk size in characters. For Paragraph and Structure this is a ceiling that triggers the character fallback. |
Chunk overlap |
Maximum number of characters carried from the end of one chunk into the start of the next. The overlap starts at a word boundary, so it holds whole words and can be shorter than this value, or empty when no word starts in that span. Paragraph and Structure only apply it inside a paragraph or section that is split with the character fallback. |
Include metadata |
Add the chunk metadata fields below to the output. |
Chunk index field |
Output field for the position of the chunk within its document. |
Chunk start position field |
Output field for the character offset of the chunk in the source text. |
Document ID field |
Output field for the document identifier. |
Total chunks field |
Output field for the number of chunks the document produced. |
Notes
Chunk overlap never crosses a paragraph or section boundary. The Paragraph and Structure strategies split on natural boundaries, where an overlap would duplicate whole paragraphs or sections between neighbouring chunks. They only apply it between the pieces of a paragraph or section that is longer than the chunk size. Every chunk adds text the previous chunk did not hold: when an overlap would only repeat the previous chunk, the next chunk starts where the previous one ended.
The chunk start position is the character offset in the input text. For a section of the Structure strategy, the first chunk starts at the section heading and the chunks a long section is split into start where their own text is.
All strategies drop chunks that contain only whitespace. The chunk index and total chunk count cover only the chunks that are emitted. Text that yields no chunks produces a single row with an empty chunk and a total of 0, the same as empty text.
Chunk size must be a whole number greater than 0. Chunk overlap must be a whole number of 0 or more; leave it empty for no overlap. A value that is not a number, for example abc, or a negative overlap is reported in the pipeline checks and stops the transform when the pipeline starts.
When the input already has a field with the same name as an output field (names are matched without regard to case), the transform writes its values into that field instead of adding a new one. The field keeps its name and position but takes the transform’s type, storage and format. If the types differ, for example a String chunk_index in the input, the pipeline checks report a warning. Rename the output field to keep the input field as it is.
Keep Include metadata enabled when the chunks are going into a vector store: the document ID and chunk index are what let a later search merge neighbouring chunks back into a passage, and re-index a single document without touching the rest.