Skip to content

Use rapidjson for STAC serialisation#1095

Merged
pjonsson merged 5 commits intodevelopfrom
stac_use_rapidjson
Apr 13, 2026
Merged

Use rapidjson for STAC serialisation#1095
pjonsson merged 5 commits intodevelopfrom
stac_use_rapidjson

Conversation

@Ariana-B
Copy link
Copy Markdown
Contributor

@Ariana-B Ariana-B commented Apr 8, 2026

Split out from #1065

NaN and Inf values are acceptable in STAC documents but orjson will serialise them to null so switch to using rapidjson instead.


📚 Documentation preview 📚: https://datacube-explorer--1095.org.readthedocs.build/en/1095/

@Ariana-B Ariana-B requested a review from pjonsson April 8, 2026 03:39
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 8, 2026

Codecov Report

❌ Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 83.58%. Comparing base (ac508cc) to head (01b1863).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
cubedash/logs.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1095      +/-   ##
===========================================
- Coverage    83.59%   83.58%   -0.02%     
===========================================
  Files           35       35              
  Lines         4322     4319       -3     
  Branches       546      545       -1     
===========================================
- Hits          3613     3610       -3     
  Misses         507      507              
  Partials       202      202              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread cubedash/_utils.py
Comment thread pyproject.toml
"pygeofilter>=0.2.2",
"pyproj",
"pystac",
"python-rapidjson",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are only 2 other calls to orjson, can we replace those as well so we don't have to have two high-performance JSON libraries?

(I think replacing the dumps(..) in logs with dumps(..).encode() and the dumps(..).decode("utf-8") in _filters.py with just dumps(..) should give the same wire-format as we have with orjson.)

Comment thread cubedash/logs.py Outdated
datetime_mode=DM_ISO8601,
uuid_mode=UM_CANONICAL,
default=lenient_json_fallback,
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we tack on an .encode() here and leave __call__ below as it was to get this PR to work?

(I would happily get rid of structlog, but that's for another PR.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even when I fix the structlog error by configuring it with WriteLoggerFactory instead of BytesLoggerFactory, I get a TypeError because of click.testing.BytesIOCopy - so structlog isn't necessarily the issue here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I was digging around in that code for fixing what turned out to be the multiple things closing the same buffer. I know there is one fix on Click in the 8.3.3 milestone that will improve things with testing, but I doubt that fix will help with this.

I don't know how much spare time you have, but if you want to go fix Click, it will probably save you some work if you do that after 8.3.3 is released.

@pjonsson pjonsson merged commit 8fccff0 into develop Apr 13, 2026
18 checks passed
@pjonsson pjonsson deleted the stac_use_rapidjson branch April 13, 2026 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants