Planetiler integration findings

[louwers]

This issue is meant to document issues with the Java library that @msbarry runs into. Peer issue to onthegomap/planetiler#1349.

Some of these may need to be split in separate issues.

Reduce size:

• Get rid of org.apache.orc:orc-core dependency #592
• Eliminate protobuf from Java encoder #564

Other issues with java tag: https://github.com/maplibre/maplibre-tile-spec/issues?q=is%3Aissue%20state%3Aopen%20label%3Ajava

[louwers]

@msbarry:

performance-wise I think the main culprits are var-int and hilbert encoding. I have some optimized versions in planetiler we could probably use instead:

https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/util/Hilbert.java
https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/util/VarInt.java

[louwers]

Quoting @msbarry on Slack so we have it documented here.

ok ran this over planet...
MVT:

• output: https://protomaps.dev/~msbarry/mvt.pmtiles (71.7gb)
• layerstats: https://protomaps.dev/~msbarry/mvt.pmtiles.layerstats.parquet
• weighted average tile size osm traffic stats: 74kb (50kb gzipped)
• largest tile: 1mb (611kb gzipped)
• total generation time: 1h25m (14m spent encoding tiles)
  MLT:
• output: https://protomaps.dev/~msbarry/mlt.pmtiles (66.5mb)
• layerstats: https://protomaps.dev/~msbarry/mlt.pmtiles.layerstats.parquet
• weighted average tile size: 72kb (43kb gzipped)
• largest tile: 1.1mb (658kb gzipped)
• total generation time: 2h7m (52m spent encoding tiles)
• looks like ~95% of mlt's are bigger than their equivalent mvt. must be more fixed overhead for very small ones?
• more details on layerstats format here: https://github.com/onthegomap/planetiler/tree/main/layerstats
• this is the largest tile due to poi layer: https://onthegomap.github.io/planetiler-demo/#14.5/35.70192/139.77905

feel free to poke around the layerstats to dig deeper. and thanks @brandon for letting me mooch off of idle cycles on his dev machine!

Reply @mactrem

@Mike Barry thanks for your work. By setting advanced to true it should clearly reduce the size, by using more advanced encodings. But even without the advanced encodings i would expect clear size advantages of MLT (about 25% on average on such optimized tiles with basic encodings and over 35% with advanced encodings as on previous benchmarks). We have to check the latest changes on the format and generator what causes the size gains. One problematic change for sure regarding the small tiles is the embedding of the metadata instead of using the proposed external global metadata dictionary

Link to Slack discussion: https://osmus.slack.com/archives/C05GNNSK3MK/p1760090783925019

[nyurik]

For varint optimization, if you expect a smaller value (e.g. for enums), you might be able to just treat it as a u8, and check that it is <128. For encoding, you might be able to use the same trick, i.e. do a special case for small value writing. Not certain if that will help though.

[msbarry]

I tried a planet build using the advanced encodings but had to kill it after 24 hours at 20% of the way through encoding tiles.

I ran over the U.S. instead:

• MVT: https://protomaps.dev/~msbarry/us-mvt.pmtiles 9.55GB, took 12m41s, max tile 610k (gzipped: 374k), weighted avg tile 26k (gzipped: 18k)
• MLT: https://protomaps.dev/~msbarry/us-mlt.pmtiles 8.78GB, took 18m20s, max tile 624k (gzipped: 383k), weighted avg tile 25k (gzipped: 15k)
• MLT advanced: https://protomaps.dev/~msbarry/us-mlt-advanced.pmtiles 9.68GB, took 14h6m56s, max tile 472k (gzipped: 401k), weighted avg tile 22k (gzipped: 17k)

layerstats output:
https://protomaps.dev/~msbarry/us-mvt.parquet
https://protomaps.dev/~msbarry/us-mlt.parquet
https://protomaps.dev/~msbarry/us-mlt-advanced.parquet

[msbarry]

You can use pmtiles tile <url> z x y to extract individual tiles from those archives. Here are some of the biggest z/x/y mlt tiles from the U.S. build:

1. 14/4824/6159 (383k) https://onthegomap.github.io/planetiler-demo/#14.5/40.72228/-73.99292 (poi:390k)
2. 14/4824/6158 (310k) https://onthegomap.github.io/planetiler-demo/#14.5/40.73893/-73.99292 (poi:278k)
3. 14/4825/6157 (292k) https://onthegomap.github.io/planetiler-demo/#14.5/40.75558/-73.97095 (poi:297k)
4. 14/4824/6157 (280k) https://onthegomap.github.io/planetiler-demo/#14.5/40.75558/-73.99292 (poi:285k)
5. 14/4771/6206 (265k) https://onthegomap.github.io/planetiler-demo/#14.5/39.93501/-75.15747 (building:243k)
6. 14/4770/6206 (256k) https://onthegomap.github.io/planetiler-demo/#14.5/39.93501/-75.17944 (building:264k)
7. 14/2620/6334 (252k) https://onthegomap.github.io/planetiler-demo/#14.5/37.74466/-122.42065 (building:202k)
8. 14/2619/6333 (248k) https://onthegomap.github.io/planetiler-demo/#14.5/37.76203/-122.44263 (building:175k)
9. 14/3411/6216 (237k) https://onthegomap.github.io/planetiler-demo/#14.5/39.76632/-105.04028 (landcover:171k)
10. 14/4771/6207 (230k) https://onthegomap.github.io/planetiler-demo/#14.5/39.91816/-75.15747 (building:236k)

and other tiles with large layers

• 9/137/211 (207k) https://onthegomap.github.io/planetiler-demo/#9.5/29.84018/-83.32031 (landcover:207k)
• 8/69/107 (113k) https://onthegomap.github.io/planetiler-demo/#8.5/27.68175/-82.26562 (landuse:113k)
• 13/1596/3157 (266k) https://onthegomap.github.io/planetiler-demo/#13.5/38.08269/-109.84131 (mountain_peak:266k)
• 5/5/11 (176k) https://onthegomap.github.io/planetiler-demo/#5.5/44.9512/-118.125 (park:176k)
• 10/256/423 (124k) https://onthegomap.github.io/planetiler-demo/#10.5/29.68794/-89.82422 (water:124k)

[mactrem]

i tested 5 of the large tiles provided above with the status in my research repo and get on average a size reduction on encoded tiles of about 42% with advanced encodings, 44% with additional sorting and 32% when using just a simple encoding. So we have to check where the compression regression comes from

[mactrem]

I’ve tested the five large tiles with the current state of this repository. The compression ratio has dropped significantly — in particular, the simple encodings are now about 4× larger. I’ll investigate what’s causing this regression.