feat(go/adbc/driver/bigquery): add BIGQUERY:type field metadata

[serramatutu]

Motivation

The Type metadata key has two limitations which stems from BigQuery's API:

1. it says fields of type ARRAY<T> are just T with Repeated=true
2. it says STRUCT<...> fields are simply RECORD, and erases any information about the inner fields.

These limitations can cause problems when trying to parse the Type key or when using it verbatim against the warehouse in a statement, e.g a CREATE TABLE statement or a AS T cast.

Summary

This PR adds a new BIGQUERY:type key that formats the original SQL string as specified by BigQuery.

Most types remain unchanged as they come from gobigquery, and in those cases this key will contain the same value as Type.

However, arrays and structs get transformed to match the richer type string.

Testing

I ran a CREATE TABLE AS query against BigQuery. Here's the result for fields of different types

[1] Regular non-nested types are simply copied over from the value of Type

1

[2] An array of integers becomes ARRAY<INTEGER>, while Type remains INTEGER

2

[3] An array of structs becomes ARRAY<STRUCT<...>>

3

[4] A struct of arrays' inner types are ARRAY<...>

4

[5] A deeply nested struct also has the correct inner types

5

Related issues

• Proposal for standardization of metadata keys of schema fields containing the SQL type string #3449

[serramatutu]

Hmmm seeing some Python failures in CI, not sure how they're related?

[lidavidm]

Do we want to keep this? Should we rename it to something like BIGQUERY:simple_type?

[serramatutu]

I was wondering the same thing. I thought of keeping it as-is to avoid breaking changes to end users. But if the project maintainers are fine with a breaking change I can do it!

[lidavidm]

I would personally prefer we namespace all the properties now that we want to introduce this convention. Possibly we can keep the existing properties under their current name with a deprecation notice.

[serramatutu]

@lidavidm I pushed a new commit that does that. I called it BIGQUERY:raw_type since it's the "raw" unprocessed thing coming directly from the API. I think this is a bit more descriptive than simple_type.

[serramatutu]

I am not sure if this should be split across two PRs though. IMO these should be two separate changelog entries: one for standardizing the keys under BIGQUERY:... and another for adding the new rich type key.

If that's the case I can merge the last commit first in a separate PR, then rebase this one on top of that.

[felipecrv]

@lidavidm I think the convention in the BigQuery driver is that all the fields in the JSON response are copies as-is into the Arrow field metadata.

[lidavidm]

That's not a law of physics, though. We can change things.

If you want to defer this to a separate PR, that's fine by me. But I think they should be consistent.

[felipecrv]

@serramatutu can you move the last commit to a separate PR? So we can merge this one with just the BIGQUERY:type addition.

[serramatutu]

I just removed the last commit from this branch. I have it on a separate branch and I can open a followup PR after this one to standardize all keys.

[felipecrv]

Merging because the failing checks are Meson+PG and CMake specific.