Apache Arrow Flight Data Stream
Overview
Apache Arrow Flight provides high-performance, low-latency RPC-based streaming using the Arrow columnar format over gRPC.
This implementation allows Hop to exchange data with Python (and other Arrow Flight clients) without writing intermediate files.
Configuration
The following properties are available when creating an Arrow Flight Data Stream:
| Property | Description |
|---|---|
Name |
The unique name of this Data Stream.
This name is used as the path on the Flight server ( |
Description |
Optional description of the stream. |
Static Schema |
The expected Arrow schema for this stream. When data is sent to the Hop Flight server, the incoming schema is checked against this static schema. If they do not match exactly, an error is thrown. |
Batch Size |
The number of rows that Apache Arrow will use per batch. (default: 500) |
Maximum Buffer Size |
The maximum number of rows that will be kept in the buffer on the Hop Flight server. Set it high enough to avoid losing rows. As a ballpark figure, take the throughput in rows/s and multiply that by 10 for the buffer size to avoid issues. (default: 10M) |
Hop Flight hostname |
The hostname on which the Hop Flight server can be reached. (default: localhost) |
Hop Flight port |
The port on which the Hop Flight server can be reached. (default: 33333) |
Use TLS |
Connect over TLS instead of plain gRPC. The server needs to be started with a certificate and a private key, see the |
Verify the server certificate |
Verify the certificate the Flight server presents. Only switch this off for testing with a self-signed certificate. (default: enabled) |
Trusted certificates file |
Optional PEM file with the certificate authority to trust when verifying the server certificate. Leave it empty to use the JVM trust store. |
Client certificate file / Client private key file |
Optional PEM files with the client certificate and its private key, for a server started with |
Username / Password |
The credentials to authenticate with, for a server started with |
| The TLS and credential fields are only used by Hop itself when it connects to the server. The hostname, port and Use TLS settings are also what the server advertises to clients in the endpoint it returns, so they need to describe how clients reach the server, not how it binds. |
| This is an IPC system, not a safe data queue. The purpose is to hand data over to the receiving party as soon as possible. That is why there is no blocking happening when we write data to the Flight server. Rows are kept in memory to avoid stalling the gRPC back-end system as this would cause data to get lost. The only time we wait is when we read data from the Flight server. That is why it’s recommended to start reading with one process before you write data with another. There is a time-out configured of 1 minute giving you plenty of time. |
Important Behavior
-
The stream name is simply the name of the Data Stream metadata element.
-
Schema validation is strict: the client must send data using exactly the same schema defined in the Static Schema field.
-
The Flight server must be started separately using the
hop arrowcommand. -
Transport security and authentication are off unless the server is started with the matching options. Without them the data travels in the clear and anything that can reach the port can read it.
Example reading with Python
Here is an example of reading from a Hop Flight server with Python. The stream name is FlightStream and the server was started on the default 0.0.0.0:33333:
import pyarrow.flight as flight
import pyarrow as pa
# Connect to your Flight server
client = flight.FlightClient("grpc://localhost:33333") # or "grpc://0.0.0.0:33333"
# On a server started with TLS and credentials, connect like this instead:
#
# with open("server.crt", "rb") as f:
# client = flight.FlightClient("grpc+tls://localhost:33333", tls_root_certs=f.read())
# token_pair = client.authenticate_basic_token(b"hop", b"s3cr3t")
# options = flight.FlightCallOptions(headers=[token_pair])
#
# and pass `options` to get_flight_info() and do_get().
# Define the stream name (must match what your server expects: the name of the Data Stream metadata element)
stream_name = "FlightStream"
# 1. Get FlightInfo (this gives you the schema and other metadata)
descriptor = flight.FlightDescriptor.for_path(stream_name)
flight_info = client.get_flight_info(descriptor)
print(f"Schema: {flight_info.schema}")
print(f"Descriptor: {flight_info.descriptor}")
print(f"Endpoints: {len(flight_info.endpoints)}")
# 2. Read the data using the first endpoint
reader = client.do_get(flight_info.endpoints[0].ticket)
# Option A: Read everything into one Table (simple)
table = reader.read_all()
print(f"✅ Read {len(table)} rows from stream '{stream_name}'")
# Show preview
print(table.to_pandas().head())