Refactor Utils to support added type casting capabilities #5906
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Started looking at a PR, but we have some error with Basel. Currently, it is known issue and we are investigating it. |
AnuarTB
reviewed
May 17, 2023
AnuarTB
approved these changes
May 17, 2023
gbaned
approved these changes
May 24, 2023
|
@gbaned Updated with a simplified version by casting single types to a list to reduce two functions to one |
AnuarTB
approved these changes
May 24, 2023
gbaned
added
ready to pull
lego0901
reviewed
Jun 1, 2023
| @@ -101,36 +121,16 @@ def dict_to_example(instance: Dict[str, Any]) -> example_pb2.Example: | |||
| if isinstance(pyval, bytes): | |||
| pyval = pyval.decode(_DEFAULT_ENCODING) | |||
|
|
|||
| if pyval is None: | |||
There was a problem hiding this comment.
Actually, this case was not handled. So I revived if pyval is None conditional. Thanks!
copybara-service bot
pushed a commit
that referenced
this pull request
Jun 6, 2023
PiperOrigin-RevId: 536904657
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To facilitate @1025KB /
jyzhao'sTODO, separate functionality to support extended capabilities.Also enables more helpful error message to be clear on which key(s) are causing problems.
F-strings work in both supported Python versions on PyPi (3.8, 3.9)
Recommended expanded types handling:
Decimal - to int and float lists (BigQuery sometimes encodes numbers as Decimal)
Dictionaries - get to lists or list-of-lists
Datetimes - to UNIX / epoch time probably
Further discussion but possible and plausible types:
List-of-Lists - for Sequence Examples*
*This utils function could/would/should produce
SequenceExamples. In my estimation per PR 5689, I posit all TFXExampleGensshould produceSequenceExamplesas this is forward-looking (in terms of supporting NLP applications etc) and because aSequenceExamplewith emptysequence_featuresseems not only permissible but also is not world breaking withSchemaGen, unlike the situation currently.Passing a
SequenceExampletoSchemaGenwithout PR 5689 means there has to be some other piece of code used to makeTF Transformwork -- what average user should be burdened with figuring that out?