Fix infostream_to_segments.py to parse compact segment descriptors

[jerinthomas1404]

The parser fails with RuntimeError: failed to parse the expected N number of sogments when InfoStream logs contain compact segment descriptors like _4(10.4.0):C8782/109 instead of the verbose form with the full :[diagnostics={...}]:[attributes={...}] :id=... trailer.

Two changes:

1. Make the verbose trailer optional in re_segment_desc by wrapping it in a non-capturing group (?:...)?. Group numbering (1-8) is preserved so all downstream consumers work unchanged.

This needs to be discussed with @mikemccand

2. Handle empty diagnostics in parse_seg_details() by defaulting to source = "flush" instead of raising. This is correct for the common case (writer starting from empty index) where the real flush/merge origin is derived from flush postings as segment and merge seg= events independently.

Fixes #596

[mikemccand]

Phew, good catch, thank you @jerinthomas1404 -- this is from recent optimizations to reduce the insane verbosity of IndexWriter InfoStream logs: apache/lucene#15885

That change logs the verbose details for a segment exactly once, when the segment is born, and skips it thereafter. We don't lose any information because those verbose details never change for a segment. The one place we do lose just a bit is when there is a pre-existing index when IndexWriter first inits -- in that case the verbose details (carryover from prior IndexWriter that created those segments) are lost -- I think that's the case leading to your 2nd question above. That's fine -- we really should extend this tool to stitch together N InfoStreams.

[mikemccand]

Thank you @jerinthomas1404!

[mikemccand]

Actually, from your response to my first comment on my last review it looks like Lucene should include the full verbose diagnostics of those initial segments? Let's fix the comment -- maybe change to something like "compact segment descriptors because the verbose version was already logged once for this segment" or so?

Or are you seeing pre-existing segments failing to have verbosity in a new IndexWriter?

[jerinthomas1404]

@mikemccand , you're correct, I have verified the upstream code and the init verbose log runs unconditionally.

I have rephrased the comment as below

Fallback when parsing a log that doesn't contain the verbose segment details
(e.g. truncated log missing the init/flush/merge birth line). We assume
"merge" since long-lived indices are dominated by merged segments

[mikemccand]

Looks great! Thank you @jerinthomas1404 -- I'll merge.