Streaming Large Files with jq

jq 133,477 normally parses each input document completely, so a huge single array must fit in memory several times over. --stream instead emits [path, leaf] events as the parser meets them, plus a closing [path] at the end of each container; fromstream(1 | truncate_stream(inputs)) reassembles the top-level array's elements one at a time.

Slurping versus streaming a 37 MB array, and stopping earlyShell

echo '{"order_id":2,"items":[{"book_id":3,"qty":2}]}' | jq -c --stream .
jq -s . data/orders.jsonl > orders-array.json                 # a 37 MB single array
/usr/bin/time -f "slurp:  %M KB peak, %e s" jq -c '.[]' orders-array.json | wc -l
/usr/bin/time -f "stream: %M KB peak, %e s" \
  jq -cn --stream 'fromstream(1 | truncate_stream(inputs))' orders-array.json | wc -l
jq -cn 'first(inputs | select(.total > 200)) | {order_id, total}' data/orders.jsonl

Output

[["order_id"],2]
[["items",0,"book_id"],3]
[["items",0,"qty"],2]
[["items",0,"qty"]]
[["items",0]]
[["items"]]
slurp:  309592 KB peak, 5.61 s
100000
stream: 3868 KB peak, 30.97 s
100000
{"order_id":70829,"total":220.48}

Streaming cut peak memory from 302 MiB to under 4 MiB, about 80 times less, but ran five times slower on this shared 4-CPU host, so use it only when the document does not fit. Better still, convert such files once to JSON Lines, which jq reads record by record with no special flags. first(inputs | ...) and limit(n; ...) stop reading as soon as they have their answer.

MENU