Rustdoc JSON 2023 Roadmap

[aDotInTheVoid]

Foreign ID Lookup

This need's it's own issue at some point, but for the zulip chat is the best resource.

The problem we want to solve is that if I have an Id from one crate's json, but the Id is for a foreign type, then it won't be present in index, only paths. If you want to see the Item, you need to find the json for the crate that this Id is from. The problem is that in this JSON you need to use a different ID. To find the ID in the crate the item is local to, you need to find the path for the item from paths, and then look that up the the json for the crate the item's from.

This is cumbersome, slow, and unreliable ¹.

Idealy you should be able to just use the Id from one JSON in another. I'm not sure if this is possible, or if we'll need some translation scheme, potentially spiting Id's into two fields (crate id, and item id). How exactly this will work needs further design work.

Correctness.

Rustdoc JSON should never emit wrong, or invalid data (duh). There is still work to do on this, in particular:

• Rustdoc JSON produces dangling ID's when linking to foreign methods #106378
• Rustdoc JSON: Invalid output for core #106435

Once these are fixed, we should:

• Do a crater run with jsondoclint on the output running automaticly.

Performance

Rustdoc JSON has had essentially no performance work done to it ². Theirs probably alot of low hanging fruit.
The first thing to make this feasable is

• Add suport in rustc-perf (Add DocJson profile for benchmarking JSON rustdoc output rustc-perf#2156).
• Add DocJson profile for benchmarking JSON rustdoc output rustc-perf#2156

Once this lands and we can use @ bors try @ rust-timer queue, we can actually make some changes. It's hard to know what actually matters without data, but some ideas to explore:

• Don't allocate a String for each Id rustdoc-json: change item ID's repr from a string to an int #130078
• Don't use the default hasher when building index, but use one of the good ones without HashDos migrations (probably rustc's fxhash) rustdoc-json: Use FxHashMap from rustdoc_json_types #129124
• Be more clever about cloning items before inserting them into the index, by using the entry api rustdoc_json: reduce allocations #142335
• Avoid allocating a PathBuf in Span.

Internal Improvements

• Think about how to avoid hitting the recursion limit for more compex types (eg)
• Ensure we don't lose a FORMAT_VERSION bump to git race conditions. #94591
• Switch away from jsonpath_lib for jsondocck, as it can't do complex predicated, which is annoyed (issue): jsondocck: Replace jsonpath_lib with jsonpath-rust #138763
• Add suport for using variable to create jsonpath query in jsondocck. Would be usefull for $.index[$$some_id$$].inner.something, although the syntax for interpolation here needs work.
• Writeup docs for the testing settup, #100515: Start documenting tests/rustdoc-json rustc-dev-guide#2422

Footnotes

1. Especialy when the path isn't present in the public docs. ↩

2. Except for fixing an issue where a syscall was being done for every single key, punctuation and value in JSON. ↩