Arrow Flight

Arrow Flight

Arrow Flight for Data Transport

Arrow 129 Flight is an RPC framework for moving Arrow record batches over the network, built on gRPC. A client asks for data with a ticket and receives a stream of IPC messages that it can use without conversion, so a service can return millions of rows without the row-by-row serialization of JDBC, ODBC or a JSON API. This server, running in a thread inside the script, streams whatever columns the ticket names:

flight_demo.py: a Flight server and client for BookNest ordersPython

import threading, time
import pyarrow.flight as flight, pyarrow.parquet as pq
class OrdersServer(flight.FlightServerBase):
    def __init__(self, location):
        super().__init__(location)
        self.orders = pq.read_table("../parquet/orders.parquet")
    def do_get(self, context, ticket):                 # the ticket names the columns to stream
        columns = ticket.ticket.decode().split(",")
        return flight.RecordBatchStream(self.orders.select(columns))
server = OrdersServer("grpc://127.0.0.1:32815")         # any free local port
threading.Thread(target=server.serve, daemon=True).start()
client = flight.connect("grpc://127.0.0.1:32815")
start = time.perf_counter()
table = client.do_get(flight.Ticket(b"order_id,order_ts,channel,total")).read_all()
secs = time.perf_counter() - start
print(f"{table.num_rows:,} rows, {table.nbytes / 1e6:.1f} MB of Arrow data "
      f"in {secs * 1000:.0f} ms;", table.schema.names)
server.shutdown()

Output

100,000 rows, 4.0 MB of Arrow data in 51 ms; ['order_id', 'order_ts', 'channel', 'total']

A real service would also implement list_flights and get_flight_info for discovery and use TLS and authentication. Flight SQL layers SQL queries and metadata on top, and the ADBC (Arrow Database Connectivity) drivers give applications a JDBC-like API that returns Arrow; Dremio 249,796 , DuckDB 61,228 and others implement them.

MENU