You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
describe and draft currently stub every nested column as "not summarised (nested)". The validation work in #113 gave nested columns real types, a schema tree (DataColumn), leaf resolution (leaf_index), and arrow navigation through structs and list offsets, and we now need to extend to describe and draft.
Lists descend into the individual values (one summary over the flattened
elements) and also capture the range of list lengths.
Structs just descend into the individual fields.
1. The shared engine: nested profiles
Both commands consume profile() (data-dict-parquet/src/profile.rs), so the
descent happens once, there.
Shape.ColumnProfile becomes a tree:
pubstructColumnProfile{// ... existing scalar summary (kind, null_count, distinct, min, max,// histogram, value_counts, examples) .../// Struct fields, one profile each, recursive. For `list(struct)` these/// are the element struct's fields.pubfields:Vec<ColumnProfile>,/// For a list column: the summary over its flattened elements.pubelement:Option<Box<ColumnProfile>>,/// For a list column: how long the lists run.publengths:Option<Lengths>,}pubstructLengths{pubmin:usize,pubmax:usize,/// Rows whose list is present but empty — distinct from null rows,/// which stay in `null_count`.pubempty_count:usize,}
Classification.classify() stops lumping groups into Unsupported("nested"): a plain group is a struct target with child targets, a
LIST group (or legacy repeated field) a list target wrapping an element target.
Maps stay unsummarised (Unsupported("map")), matching validation.
Scanning. The existing per-leaf scan already decodes arrow batches
(group_reader([leaf], ...)); for a nested leaf the batch column comes back as
the reconstructed nested wrapper, so each observer walks down with the same navigate() logic validate_data's D04 scan uses:
A struct field's array is row-aligned — the existing accumulators (histogram,
value counts, distinct sketch, min/max) apply unchanged.
A list's element array is the flattened values — again the existing
accumulators, which is exactly the "summary of individual values" we want.
The ListArray offsets give the lengths for free (Lengths above), and the
validity buffer splits null lists from empty ones.
Caveat to resolve during implementation: a null struct and a null field inside
a present struct both surface as a null in the field's child array. Simplest
honest reading: the struct's own null_count comes from its validity buffer,
and each field's missing counts nulls within present structs (child nulls
minus ancestor nulls). Decide whether that subtraction is worth it or whether
per-field missing just reports the conflated count with the struct's missing
alongside for context.
Perf note: the dictionary-preferring fast path (prefer_dictionary) reads a
leaf as a Dictionary array; whether that survives reconstruction through the
nested wrapper needs checking — if not, nested leaves just take the plain
path, and only flat columns keep the shortcut. Fine for a first cut.
2. describe
Text: nested blocks indent under their parent, each rendered exactly like a
top-level column. Mock:
JSON: ColumnDescription gains fields: [...] (structs), and element: {...} + lengths: {min, max, empty} (lists), omitted when absent
like every other key.
Also:
describe file.parquet addr.country — dotted-path column selection,
resolved with leaf_index. (Lists need no path segment; tags selects the
list, and its element summary comes with it.)
Fold in the find_leaf fix: profile.rs still resolves leaves by leaf name (profile.rs:223), which can profile the wrong column when a struct
field shares a top-level column's name. Same one-line fix as column_meta:
resolve through metadata::leaf_index. This is a correctness bug worth
landing even if nothing else here happens.
struct → type: struct + fields:, one drafted field per child.
Fields are reduced descriptors, so no constraints TODOs on them (the spec
bans constraints on fields); everything else — description TODO, examples,
enum-candidate swap, range for temporal fields — applies per field
unchanged.
list(scalar) → type: list(elem) from the element kind; examples (or range/values) drawn from the flattened element summary, per S07's
element-type rule. The enum-candidate heuristic runs on the elements and
suggests type: list(enum).
list(struct) → type: list(struct) + fields: from the element struct.
containers and constraints: only the required TODO (when the container
itself has no nulls — a row of empty lists still counts as present);
never the unique/primary-key stubs (S29 bans them on containers).
Mock:
- name: tagstype: list(string)# TODO: no missing values observed — uncomment or delete:# constraints: [required]description: > TODO: what does this column mean?examples: [a, b, zz]# TODO: only 3 distinct element values — if this is an enum, set# type: list(enum) and replace examples with:# values: [a, b, zz]
- name: addrtype: structfields:
- name: ziptype: stringdescription: > TODO: what does this column mean?examples: ['97201', '78701']
4. Sequencing
PR A (tiny, standalone): fix find_leaf via leaf_index; containers'
stub output gains a missing count where the footer proves it (same
all-leaves-zero rule column_meta uses).
PR B: engine descent + describe rendering (text, JSON, dotted-path
selection). Test fixtures via ArrowWriter (pattern already in crates/data-dict/tests/common).
Lengths presentation: is min–max plus an empty count enough, or do long
tails deserve a small histogram? (Sketch says min/max + empty; cheap to
extend later.)
Per-field missing semantics under a nullable struct (see caveat above).
Should describe's element block be nameable from the CLI (e.g. describe file 'tags[]'), or is selecting the list always enough?
describeanddraftcurrently stub every nested column as "not summarised (nested)". The validation work in #113 gave nested columns real types, a schema tree (DataColumn), leaf resolution (leaf_index), and arrow navigation through structs and list offsets, and we now need to extend todescribeanddraft.elements) and also capture the range of list lengths.
1. The shared engine: nested profiles
Both commands consume
profile()(data-dict-parquet/src/profile.rs), so thedescent happens once, there.
Shape.
ColumnProfilebecomes a tree:Classification.
classify()stops lumping groups intoUnsupported("nested"): a plain group is a struct target with child targets, aLIST group (or legacy repeated field) a list target wrapping an element target.
Maps stay unsummarised (
Unsupported("map")), matching validation.Scanning. The existing per-leaf scan already decodes arrow batches
(
group_reader([leaf], ...)); for a nested leaf the batch column comes back asthe reconstructed nested wrapper, so each observer walks down with the same
navigate()logicvalidate_data's D04 scan uses:value counts, distinct sketch, min/max) apply unchanged.
accumulators, which is exactly the "summary of individual values" we want.
The
ListArrayoffsets give the lengths for free (Lengthsabove), and thevalidity buffer splits null lists from empty ones.
Caveat to resolve during implementation: a null struct and a null field inside
a present struct both surface as a null in the field's child array. Simplest
honest reading: the struct's own
null_countcomes from its validity buffer,and each field's
missingcounts nulls within present structs (child nullsminus ancestor nulls). Decide whether that subtraction is worth it or whether
per-field missing just reports the conflated count with the struct's missing
alongside for context.
Perf note: the dictionary-preferring fast path (
prefer_dictionary) reads aleaf as a
Dictionaryarray; whether that survives reconstruction through thenested wrapper needs checking — if not, nested leaves just take the plain
path, and only flat columns keep the shortcut. Fine for a first cut.
2.
describeText: nested blocks indent under their parent, each rendered exactly like a
top-level column. Mock:
JSON:
ColumnDescriptiongainsfields: [...](structs), andelement: {...}+lengths: {min, max, empty}(lists), omitted when absentlike every other key.
Also:
describe file.parquet addr.country— dotted-path column selection,resolved with
leaf_index. (Lists need no path segment;tagsselects thelist, and its element summary comes with it.)
find_leaffix:profile.rsstill resolves leaves by leafname (
profile.rs:223), which can profile the wrong column when a structfield shares a top-level column's name. Same one-line fix as
column_meta:resolve through
metadata::leaf_index. This is a correctness bug worthlanding even if nothing else here happens.
3.
draft(#161)infer_columnrecurses over the profile tree:type: struct+fields:, one drafted field per child.Fields are reduced descriptors, so no constraints TODOs on them (the spec
bans
constraintson fields); everything else — description TODO, examples,enum-candidate swap,
rangefor temporal fields — applies per fieldunchanged.
type: list(elem)from the element kind;examples(orrange/values) drawn from the flattened element summary, per S07'selement-type rule. The enum-candidate heuristic runs on the elements and
suggests
type: list(enum).type: list(struct)+fields:from the element struct.requiredTODO (when the containeritself has no nulls — a row of empty lists still counts as present);
never the unique/primary-key stubs (S29 bans them on containers).
Mock:
4. Sequencing
find_leafvialeaf_index; containers'stub output gains a
missingcount where the footer proves it (sameall-leaves-zero rule
column_metauses).describerendering (text, JSON, dotted-pathselection). Test fixtures via
ArrowWriter(pattern already incrates/data-dict/tests/common).data-dict draft: generate a skeleton dictionary from parquet files #161 merges):draftfor nested columns on top of theextended profiles.
Open questions
min–maxplus an empty count enough, or do longtails deserve a small histogram? (Sketch says min/max + empty; cheap to
extend later.)
missingsemantics under a nullable struct (see caveat above).describe's element block be nameable from the CLI (e.g.describe file 'tags[]'), or is selecting the list always enough?