Skip to content

describe and draft for lists and structs #170

Description

@hadley

describe and draft currently stub every nested column as "not summarised (nested)". The validation work in #113 gave nested columns real types, a schema tree (DataColumn), leaf resolution (leaf_index), and arrow navigation through structs and list offsets, and we now need to extend to describe and draft.

  • Lists descend into the individual values (one summary over the flattened
    elements) and also capture the range of list lengths.
  • Structs just descend into the individual fields.

1. The shared engine: nested profiles

Both commands consume profile() (data-dict-parquet/src/profile.rs), so the
descent happens once, there.

Shape. ColumnProfile becomes a tree:

pub struct ColumnProfile {
    // ... existing scalar summary (kind, null_count, distinct, min, max,
    //     histogram, value_counts, examples) ...

    /// Struct fields, one profile each, recursive. For `list(struct)` these
    /// are the element struct's fields.
    pub fields: Vec<ColumnProfile>,
    /// For a list column: the summary over its flattened elements.
    pub element: Option<Box<ColumnProfile>>,
    /// For a list column: how long the lists run.
    pub lengths: Option<Lengths>,
}

pub struct Lengths {
    pub min: usize,
    pub max: usize,
    /// Rows whose list is present but empty — distinct from null rows,
    /// which stay in `null_count`.
    pub empty_count: usize,
}

Classification. classify() stops lumping groups into
Unsupported("nested"): a plain group is a struct target with child targets, a
LIST group (or legacy repeated field) a list target wrapping an element target.
Maps stay unsummarised (Unsupported("map")), matching validation.

Scanning. The existing per-leaf scan already decodes arrow batches
(group_reader([leaf], ...)); for a nested leaf the batch column comes back as
the reconstructed nested wrapper, so each observer walks down with the same
navigate() logic validate_data's D04 scan uses:

  • A struct field's array is row-aligned — the existing accumulators (histogram,
    value counts, distinct sketch, min/max) apply unchanged.
  • A list's element array is the flattened values — again the existing
    accumulators, which is exactly the "summary of individual values" we want.
    The ListArray offsets give the lengths for free (Lengths above), and the
    validity buffer splits null lists from empty ones.

Caveat to resolve during implementation: a null struct and a null field inside
a present struct both surface as a null in the field's child array. Simplest
honest reading: the struct's own null_count comes from its validity buffer,
and each field's missing counts nulls within present structs (child nulls
minus ancestor nulls). Decide whether that subtraction is worth it or whether
per-field missing just reports the conflated count with the struct's missing
alongside for context.

Perf note: the dictionary-preferring fast path (prefer_dictionary) reads a
leaf as a Dictionary array; whether that survives reconstruction through the
nested wrapper needs checking — if not, nested leaves just take the plain
path, and only flat columns keep the shortcut. Fine for a first cut.

2. describe

Text: nested blocks indent under their parent, each rendered exactly like a
top-level column. Mock:

addr — struct
  missing:  1

  addr.zip — string
    distinct:  2
    missing:   1

  addr.country — string
    US         1
    XX         1
    ---------
    missing:   1

tags — list(string)
  lengths  
    0        0  
    1 ▇▇▇    2
    2 ▇      1
    ---
    missing: 1

  contents — string
    a          2
    b          1
    zz         1
    ----------
    missing:   0

JSON: ColumnDescription gains fields: [...] (structs), and
element: {...} + lengths: {min, max, empty} (lists), omitted when absent
like every other key.

Also:

  • describe file.parquet addr.country — dotted-path column selection,
    resolved with leaf_index. (Lists need no path segment; tags selects the
    list, and its element summary comes with it.)
  • Fold in the find_leaf fix: profile.rs still resolves leaves by leaf
    name (profile.rs:223), which can profile the wrong column when a struct
    field shares a top-level column's name. Same one-line fix as column_meta:
    resolve through metadata::leaf_index. This is a correctness bug worth
    landing even if nothing else here happens.

3. draft (#161)

infer_column recurses over the profile tree:

  • structtype: struct + fields:, one drafted field per child.
    Fields are reduced descriptors, so no constraints TODOs on them (the spec
    bans constraints on fields); everything else — description TODO, examples,
    enum-candidate swap, range for temporal fields — applies per field
    unchanged.
  • list(scalar)type: list(elem) from the element kind; examples (or
    range/values) drawn from the flattened element summary, per S07's
    element-type rule. The enum-candidate heuristic runs on the elements and
    suggests type: list(enum).
  • list(struct)type: list(struct) + fields: from the element struct.
  • containers and constraints: only the required TODO (when the container
    itself has no nulls — a row of empty lists still counts as present);
    never the unique/primary-key stubs (S29 bans them on containers).

Mock:

- name: tags
  type: list(string)
  # TODO: no missing values observed — uncomment or delete:
  # constraints: [required]
  description: >
    TODO: what does this column mean?
  examples: [a, b, zz]
  # TODO: only 3 distinct element values — if this is an enum, set
  # type: list(enum) and replace examples with:
  # values: [a, b, zz]

- name: addr
  type: struct
  fields:
    - name: zip
      type: string
      description: >
        TODO: what does this column mean?
      examples: ['97201', '78701']

4. Sequencing

  1. PR A (tiny, standalone): fix find_leaf via leaf_index; containers'
    stub output gains a missing count where the footer proves it (same
    all-leaves-zero rule column_meta uses).
  2. PR B: engine descent + describe rendering (text, JSON, dotted-path
    selection). Test fixtures via ArrowWriter (pattern already in
    crates/data-dict/tests/common).
  3. PR C (after Add data-dict draft: generate a skeleton dictionary from parquet files #161 merges): draft for nested columns on top of the
    extended profiles.

Open questions

  • Lengths presentation: is min–max plus an empty count enough, or do long
    tails deserve a small histogram? (Sketch says min/max + empty; cheap to
    extend later.)
  • Per-field missing semantics under a nullable struct (see caveat above).
  • Should describe's element block be nameable from the CLI (e.g.
    describe file 'tags[]'), or is selecting the list always enough?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions