How writer manages writes of null values in Spark Dataframe?

[Il-Pela]

Background

While exploring the write feature of Cobrix. I'm facing this situation when i try to write a df that contains null values in certain columns i get as output in the positional files the bytes 0x00 for the length of the field.

Question

Is there a way (an option of the source(?)) to either output spaces (0x40) for alphanumeric fields or 0 (0xF0) for numeric fields in case of null cells?

[yruslan]

Hi @Il-Pela ,

Currently, there are no such options, but it can be added. Let's use this issue to track the implementation.

Writing null strings as spaces makes sense, a common mainframe convention. But would it really work for you if all null numeric values are written as 0? Zero and null are conceptually very different values.

As a workaround for now, you can use Spark's coalesce for numeric fields to replace nulls with zeros and string nulls with empty strings.

[Il-Pela]

Hi @yruslan , thanks for your reply.

I agree with you, not in all cases is desired to have nulls written as 0s.
Even for strings, if nulls are encountered during execution, it could be a symptom of poor data quality and/or errors during the Spark pipeline execution, and using spaces isn't always the solution (perhaps in non-production environments could be to speed up integration and other tests).

In general I think that it would be better to handle nulls at the cell level in the dataframe separately from the happy path (e.g., with a separate discard stream that writes to a table extra details to debug the reason why it ended there).

That's why I was thinking about an option to manage this behavior. The default for this option could be the actual behavior, while the other behaviors could be something like: substitute strings with spaces, substitute numeric with 0s, both.

I don't have much time at the moment, but if I can, I'd be happy to contribute.

[yruslan]

Yes, I agree. Going to add these 2 options soon.

[yruslan]

This is merged in master. New version is planned to be released next week.

[Il-Pela]

Amazing @yruslan !

I'll test it as soon as it will be available.
Thank you!