Unoptimized observations

[amotl]

About

The program is not optimized in any way yet, other than using the orjson and orjsonl packages for high-performance JSON marshalling.

Problem

Processing an 11 GB JSONL/NDJSON file using Python 3.11, the CPU saturation looks reasonable, but the memory usage peaks at 75 GB,

and after 30 minutes of processing time, the program croaks.

Assertion failed: (p->curr_buf_pos == p->curr_buf_length), function jv_parser_next, file jv_parse.c, line 823.
Abort trap: 6

real	34m27.128s
user	8m5.955s
sys	18m25.432s

[amotl]

This patch adds streaming support for JSONL/NDJSON files, so the program does not fail any longer on large data inputs.

• Core: Added streaming support for JSONL/NDJSON files #8