Use rapidjson for STAC serialisation#1095
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #1095 +/- ##
===========================================
- Coverage 83.59% 83.58% -0.02%
===========================================
Files 35 35
Lines 4322 4319 -3
Branches 546 545 -1
===========================================
- Hits 3613 3610 -3
Misses 507 507
Partials 202 202 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| "pygeofilter>=0.2.2", | ||
| "pyproj", | ||
| "pystac", | ||
| "python-rapidjson", |
There was a problem hiding this comment.
There are only 2 other calls to orjson, can we replace those as well so we don't have to have two high-performance JSON libraries?
(I think replacing the dumps(..) in logs with dumps(..).encode() and the dumps(..).decode("utf-8") in _filters.py with just dumps(..) should give the same wire-format as we have with orjson.)
| datetime_mode=DM_ISO8601, | ||
| uuid_mode=UM_CANONICAL, | ||
| default=lenient_json_fallback, | ||
| ) |
There was a problem hiding this comment.
Can't we tack on an .encode() here and leave __call__ below as it was to get this PR to work?
(I would happily get rid of structlog, but that's for another PR.)
There was a problem hiding this comment.
Even when I fix the structlog error by configuring it with WriteLoggerFactory instead of BytesLoggerFactory, I get a TypeError because of click.testing.BytesIOCopy - so structlog isn't necessarily the issue here.
There was a problem hiding this comment.
Oh, I was digging around in that code for fixing what turned out to be the multiple things closing the same buffer. I know there is one fix on Click in the 8.3.3 milestone that will improve things with testing, but I doubt that fix will help with this.
I don't know how much spare time you have, but if you want to go fix Click, it will probably save you some work if you do that after 8.3.3 is released.
Split out from #1065
NaN and Inf values are acceptable in STAC documents but orjson will serialise them to null so switch to using rapidjson instead.
📚 Documentation preview 📚: https://datacube-explorer--1095.org.readthedocs.build/en/1095/