Text chunker transform Icon Text chunker

Description

The Text chunker transform splits a text field into smaller chunks and emits one output row per chunk, carrying the fields of the input row along with each.

Chunking is the first step of a retrieval pipeline: an embedding model has a bounded context, and retrieval works better on passages than on whole documents. A typical pipeline is Text chunker, then an embedding transform, then a vector store.

The transform has no dependency on an AI provider or a model. It is plain text processing and can be used on its own wherever a large document needs to be broken into passages.

Chunking strategies

Strategy Behaviour

Character

Split on a fixed character count, backing off to the nearest word boundary so words are not cut in half. Only a word longer than the chunk size is cut, and never inside a surrogate pair such as an emoji. Overlap carries the last whole words of each chunk into the next, which keeps a sentence that straddles a boundary retrievable from both sides.

Paragraph

Split on paragraph breaks, packing whole paragraphs up to the chunk size. A paragraph longer than the chunk size falls back to character splitting. By default a paragraph break is a blank line (any \LF, \CR or \CRLF line ending); the optional Paragraph separator option replaces it with a literal separator.

Structure

Parse the document into a heading tree and emit one chunk per section, prefixing each with a breadcrumb of its heading path so an isolated chunk still says where it came from. A section larger than the chunk size falls back to character splitting within that section.

Structure parsers

The Structure strategy selects a parser from the content type:

Content type Parsed structure

Auto

Infer from the content type field, or from the text itself: a pipeline or workflow when the XML root element is pipeline or workflow, metadata when the text is a JSON object, AsciiDoc when it starts with a = Title document title, otherwise Markdown or AsciiDoc by which has more headings, and on a tie by the first heading.

Plain

Blank-line separated blocks.

Markdown

heading levels. Lines in a fenced code block are not headings. A skipped level nests under the nearest heading above it, and the closing sequence is not part of the title.

AsciiDoc

= heading levels and two-line titles (a title underlined with =, -, ~, ^ or +). Lines in a delimited block (====, ----, ////, …​) are not headings.

Pipeline

A Hop pipeline (.hpl): one section per transform, plus the hops between them.

Workflow

A Hop workflow (.hwf): one section per action, plus the hops between them.

Metadata

A Hop metadata JSON document: an overview plus one section per entry.

The Hop-native parsers make a pipeline, a workflow or a metadata file retrievable per transform, action or entry rather than as one opaque blob.

Options

Option Description

Transform name

Name of the transform, unique within the pipeline.

Input field

Field holding the text to split.

Source document ID field

Optional field holding the identifier of the source document. When empty, a row counter is used.

Output chunk field

Output field that receives the chunk text.

Chunking strategy

Character, Paragraph or Structure, as described above.

Paragraph separator

Optional literal separator for the Paragraph strategy, for example <PARA>. The separator is matched as-is, not as a regular expression. When empty, paragraphs are separated by blank lines. Supports variables and the escape sequences \n , \r , \t and \\ so line breaks and tabs can be typed in the field, e.g. \n\n . Only used by the Paragraph strategy: the transform reports a warning in the pipeline checks when it is set for another strategy.

Content type

Document format used by the Structure strategy.

Content type field

Optional field naming the content type per row, for a stream that mixes formats.

Chunk size

Maximum chunk size in characters. For Paragraph and Structure this is a ceiling that triggers the character fallback.

Chunk overlap

Maximum number of characters carried from the end of one chunk into the start of the next. The overlap starts at a word boundary, so it holds whole words and can be shorter than this value, or empty when no word starts in that span. Paragraph and Structure only apply it inside a paragraph or section that is split with the character fallback.

Include metadata

Add the chunk metadata fields below to the output.

Chunk index field

Output field for the position of the chunk within its document.

Chunk start position field

Output field for the character offset of the chunk in the source text.

Document ID field

Output field for the document identifier.

Total chunks field

Output field for the number of chunks the document produced.

Notes

Chunk overlap never crosses a paragraph or section boundary. The Paragraph and Structure strategies split on natural boundaries, where an overlap would duplicate whole paragraphs or sections between neighbouring chunks. They only apply it between the pieces of a paragraph or section that is longer than the chunk size. Every chunk adds text the previous chunk did not hold: when an overlap would only repeat the previous chunk, the next chunk starts where the previous one ended.

The chunk start position is the character offset in the input text. For a section of the Structure strategy, the first chunk starts at the section heading and the chunks a long section is split into start where their own text is.

All strategies drop chunks that contain only whitespace. The chunk index and total chunk count cover only the chunks that are emitted. Text that yields no chunks produces a single row with an empty chunk and a total of 0, the same as empty text.

Chunk size must be a whole number greater than 0. Chunk overlap must be a whole number of 0 or more; leave it empty for no overlap. A value that is not a number, for example abc, or a negative overlap is reported in the pipeline checks and stops the transform when the pipeline starts.

When the input already has a field with the same name as an output field (names are matched without regard to case), the transform writes its values into that field instead of adding a new one. The field keeps its name and position but takes the transform’s type, storage and format. If the types differ, for example a String chunk_index in the input, the pipeline checks report a warning. Rename the output field to keep the input field as it is.

Keep Include metadata enabled when the chunks are going into a vector store: the document ID and chunk index are what let a later search merge neighbouring chunks back into a passage, and re-index a single document without touching the rest.