Encounter `error: "Fastq record is too long"` when parsing Nanopore sequence data

[sagrudd]

What are the limits of the maximum sequence length within a record? I am imagining a workflow that should be regularly accommodating of reads over 100kb in length (and with recent ultra-long updates should occasionally expect multi Mb sequence reads.

What would be the most sustainable approach to working through this hurdle? Updating the buffer usize (and forking the project), reverting to bio::io::fastq? As a new to rust developer I'd welcome any comments as to e.g. how performance is going to suffer.

Would welcome some thoughts here - thanks!

[iskandr]

I have the same problem with PacBio reads.

Here's the culprit line that sets the buffer size to 68k: https://docs.rs/fastq/latest/src/fastq/lib.rs.html#133

[macklin-10x]

I suspect folks have moved on to other solutions, but the parsing provided by this crate is still extremely fast and may prove useful. We've forked it and are actively maintaining it here:
https://github.com/10XGenomics/fastq-rs

I just merged a PR there that exposes a method to use a custom buffer size to support exactly this use case.

cc @sagrudd @iskandr