[Variant][Shredding] Support typed_access for timestamp_micro/timestamp_nano

[klion26]

Which issue does this PR close?

• Closes [Variant] [Shredding] Support typed_access for Timestamp(Microsecond, _) and Timestamp(Nanosecond, _) #8331 .

Rationale for this change

• Add typed_access for Timestamp(Micro, _) and Timestamp(Nano, -)

What changes are included in this PR?

• Extract some data gen logic in tests to simplify the test logic (commit 93090d5), but it based on some old code(before Add Arrow Variant Extension Type, remove Array impl for VariantArray and ShreddedVariantFieldArray #8392), rebase the master in the last commit
• Add typed_access for Timestamp(Micro, _) and Timestamp(Nano, _)
• Add test for typed_access for Timestamp(Micro, _) and Timestamp(Nano, _)

Are these changes tested?

Covered by existing and added tests

Are there any user-facing changes?

No

[klion26]

@alamb @scovich Please help review this when you're free, thanks.

[scovich]

LGTM, other than a panic that needs to be removed.

Your call whether to deal with timestamp precision widening in this PR or as a follow-on effort (maybe it depends on whether there's any controversy about the approach)

[scovich]

Can we return an error instead of panicking?

Also, AFAIK all the other timestamp precisions can be widened losslessly to microsecond; should we consider doing that instead of blowing up?

[alamb]

yes, let's not panic here

ANother potential approach is to return a not yet implemented error -- maybe by matching the supported types explicitly, so like

        DataType::Timestamp(TimeUnit::Microsecond, Some(_)) => {
                    generic_conversion_single_value!(
                        TimestampMicrosecondType,
                        as_primitive,
                        |v| DateTime::from_timestamp_micros(v).unwrap(),
                        typed_value,
                        index
                    )
                }

instead of

        DataType::Timestamp(timeunit, tz) => {
            match (timeunit, tz) {
                (TimeUnit::Microsecond, Some(_)) => {
                    generic_conversion_single_value!(
                        TimestampMicrosecondType,
                        as_primitive,
                        |v| DateTime::from_timestamp_micros(v).unwrap(),
                        typed_value,
                        index
                    )
                }

[scovich]

I'm always a fan of less nesting, when possible...

[klion26]

Thanks for pointing this out. Yes, TimeUnit::Second and TimeUnit::Millisecond can convert to TimeUnit::MicroSecond losslessly. Using panic here because this value is located in typed_value of the Variant, and from the spec of Variant, there are only Micro and Nano units for Timestamp(with/without timezone), I assumed that this is an error(this could not happen?), I can add the second and Millisecond units if this could happen.

[scovich]

Using panic here because this value is located in typed_value of the Variant, and from the spec of Variant, there are only Micro and Nano units for Timestamp(with/without timezone), I assumed that this is an error(this could not happen?)

Good point! There are definitely unit tests (parquet/tests/variant_integration.rs) that expect non-variant types in typed_value columns to produce an error (cases 127 and 137, specifically). So we should not widen on the read path, but rather reject.

Widening would seem to make sense on the shredding write path, tho?

[scovich]

See also #8435, which I noticed while trying to add struct support

[klion26]

IIUC, we need to widen to TimeUnit::MicroSecond when writing to a variant, as TimeUnit::MicroSecond is the minimum precision we support

[scovich]

Exactly. We have flexibility on the write path to decide whether/how we widen or convert values that don't exactly fit the variant type system, but readers expect the resulting types to always be valid variant shredding types.

[alamb]

Thank you @klion26 and @scovich

[alamb]

yes, let's not panic here

ANother potential approach is to return a not yet implemented error -- maybe by matching the supported types explicitly, so like

        DataType::Timestamp(TimeUnit::Microsecond, Some(_)) => {
                    generic_conversion_single_value!(
                        TimestampMicrosecondType,
                        as_primitive,
                        |v| DateTime::from_timestamp_micros(v).unwrap(),
                        typed_value,
                        index
                    )
                }

instead of

        DataType::Timestamp(timeunit, tz) => {
            match (timeunit, tz) {
                (TimeUnit::Microsecond, Some(_)) => {
                    generic_conversion_single_value!(
                        TimestampMicrosecondType,
                        as_primitive,
                        |v| DateTime::from_timestamp_micros(v).unwrap(),
                        typed_value,
                        index
                    )
                }

[alamb]

🎉

[klion26]

I'm rebasing the main to resolve the conflicts
rebased on main

[alamb]

Thank you @klion26 and @scovich

[alamb]

thank you for reducing the boilerplate

[alamb]

I merged up from main to resolve a conflict

[alamb]

@klion26 -- this PR seems to have accumulated conflicts -- I apologize for that. Can you please resolve the conflicts so I can merge it in?

[klion26]

@alamb I've rebased on main and resolved the conflicts. Please take another look, thanks.

[alamb]

Thank you @klion26 🙏 🙏