Apache Arrow Random Access File Data Stream

Overview

This implementation writes/reads the official Arrow IPC File Format (with footer). It supports random access and is ideal when you need to read the same file multiple times or seek within the data.

Configuration

Property Description

Name

Unique name of the Data Stream

File name

Path to the .arrow file (supports variables)

Batch size

Rows per batch when writing (default 10,000)

Compression

None, LZ4, Zstd

When to use

  • You want to archive or share the data

  • Downstream processes need random access

  • You prefer the standard Arrow File format over streaming