Provide mermaid diagram of relation between BIDS schema, specification and validators #626
Conversation
|
I would remove the legacy validator from that diagram. |
|
The preview on read the docs shows a mermaid syntax error. This does not mean there is a real syntax error but that's the kind of problem when I was trying to use some of the more advanced mermaid features with mkdocs. |
|
It's not rendering here: https://bids-website--626.org.readthedocs.build/en/626/standards/index.html |
|
|
FWIW: does render now. |
|
Looks like too much stuff going on. I can't read it at 100% zoom.
|
|
@candleindark -- would you be so kind to see if you could tame the drawing so it is of better fidelity without zooming? Meanwhile, I have added here use of panzoom plugin (https://playg0n.github.io/mkdocs-panzoom/) which should enable zoom'ing "Mermaid and D2 are included by default". Seems to work nicely. |
I think the problem stems having too many items in the graph rather the arrangement of the items in the graph, but I will make an attempt. It would be nice if there is some mermaid support that allows zooming in of a particular region of the diagram, like the behavior of the prezi app. I don't think this problem is particular to us though. Many linkml generated diagrams suffer the same problem in presentation (For example, https://concepts.datalad.org/s/temporal/unreleased/). |
|
I think if we can get any large mermaid diagram to display in full window as https://anvaka.github.io/panzoom/demo/index.html would be great. The example is a live demo of https://github.com/anvaka/panzoom which is the base for https://playg0n.github.io/mkdocs-panzoom/ |
|
@candleindark I already added zoom/dragging here -- check preview at https://bids-website--626.org.readthedocs.build/en/626/standards/index.html . Do you propose something instead/on top of it? |
In a way, yes. I do think the one I mentioned is more better, but it is not a Mkdocs extension. I don't know if it's worth the time/effort to get it working. |
Nevermind, I just found out some a feature of the mkdocs extension you are using. I think it is good enough. |
|
@yarikoptic I was able to make the items in the diagram appear a bit larger and sent a PR to your branch. |
yarikoptic
force-pushed
the
enh-schema-flow
branch
from
999d39e to
0d2e9b2
Compare
|
@effigies please check it out now : https://bids-website--626.org.readthedocs.build/en/626/standards/index.html . I also updated screenshot in original description |
we will forget to remove it in a year it useless complexifes the figure that is already WAY too complex in my opinion we have no intention of keeping maintaining the old validator, so let's not emphasize it the original blog post gradually explained things and complexified the figure incrementally: I think that the landing page for 'standards' should have a simpler version and we can put the more exhaustive version somewhere else |
|
Just a thought sparked by @Remi-Gau's comment: what if this also was a blog post that came with explanations and is tagged with an author and date? I haven't reviewed, but I feel like the editorial bar is lower for a blog post than a landing page. |
I like the idea and we can have the simplified version in the landing page and pointing to the blog post for anyone who wants the gory details |
|
Idea for a blog is neat but I do not think it is to materialize any time soon. So I am making this PR a candidate for merge - "perfect is enemy of the good" and I had a second look at the diagram and still find it useful to orient visitor in all the intricacies of dependencies, with URL hyperlinks coming quite handy. |
Metadata lifecycle inspired by the one we created for BIDS: see bids-standard/bids-website#626
…n and validators This is based of the mermaid diagrams in the blog post we have: https://bids-website.readthedocs.io/en/latest/blog/2024/11/13/bids-validator-2.html and the goal is to give a visual depiction of relations between the components mentioned on this page and elsewhere.
for more information, see https://pre-commit.ci
This adjustment will allow items in the mermaid diagram to appear bigger in the webpage because the page is more limited in the horizontal dimension. Additionally, the syntax of the mermaid diagram has been updated to the latest syntax, e.g. `flowchart` instead of `graph`. The intents of the lines in the diagram are those recommended/defaulted by the mermaid live editor at https://www.mermaidchart.com
yarikoptic
force-pushed
the
enh-schema-flow
branch
from
c1b89f6 to
753d3fd
Compare
* Design document with a diagram of metadata life cycle Metadata lifecycle inspired by the one we created for BIDS: see bids-standard/bids-website#626 * Remove duplication with now present vendorization issue * Address all questions and detail implementation more * Implement local schema serialization endpoints - Add new API endpoints to serialize JSONSchema at runtime - Update info endpoint to use local schema URL instead of GitHub - Use TypeAdapter for proper Pydantic schema generation - Add tests for schema endpoints - Add implementation documentation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * doc: Add mermaid diagram for updated schema architecture - Add visual representation of the new schema flow - Highlight key differences from the current approach - Show elimination of dependency on external schema repository 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Provide custom name for django testserver instance The default name django uses is "testserver" and then URL becomes http://testserver, but DJANGO itself does not consider it to be a valid URL because it lacks TLD. So whenever we add "schema_url" pointing to our server (in tests just "testserver") DJANGO starts raising validation errors. See encode/django-rest-framework#9705 for more information/rationale. * RF: refactor auto-AI-generated tests into parametrized ones to concetrate the testing logic and also to accent on the differences (parameters of the test) * Add GitHub schema comparison to test_schema_latest Extends test to compare local runtime-generated schema against static GitHub schema content. This test is expected to fail, demonstrating that vendorized schemas differ from static schemas due to runtime customizations. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Use TransitionalGenerateJsonSchema with proper type annotations Refactors schema generation to match dandischema approach and adds type safety. * Fix schema endpoints to use regular Dandiset/Asset models Changes from Published models to regular models to match GitHub static schemas: - Use Dandiset instead of PublishedDandiset - Use Asset instead of PublishedAsset - Update test parameterization accordingly Test now shows minimal vendorization difference (repository default value) demonstrating the expected runtime customization vs static schema difference. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Adjust test to handle vendorization differences Add logic to normalize vendorization differences before schema comparison: - Adjust repository default value from runtime config to match GitHub schema - Test now passes, confirming schemas are equivalent after vendorization adjustment - Demonstrates successful runtime schema generation with expected customizations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Add published schema endpoints and refactor to reduce code duplication Extends the schema API to support both regular and published models: New endpoints: - /api/schema/latest/published-dandiset/ - /api/schema/<version>/published-dandiset/ - /api/schema/latest/published-asset/ - /api/schema/<version>/published-asset/ Refactoring improvements: - Extract common logic into _schema_view_impl() helper function - Update URL patterns and view exports Testing enhancements: - Extend parametrized tests to cover published schema endpoints 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Fixup to mermaid diagram In d41f24e I committed a little rushed tune up by claude and it replaced "current state" diagram with the actually implemented, and then for implemented it just removed having jsonschema serializations altogether. Since we did merges into this branch, rebasing is "tricky", so I decided to just push a fixup commit. * Remove portion of the test checking for identity to released schemas * chore: remove unused import This resolve a complain from ruff * style: use `settings.DANDI_API_URL` to construct schema URL There are already consistent uses of `settings.DANDI_API_URL` for building URLs in the codebase. Check out #2386 (comment) detailed rationale behind this change. Co-authored-by: Mike VanDenburgh <[email protected]> * Don't re-implement get_schema_url in tests * Unify schema endpoints * Remove version query param from schema endpoint * Update wording Co-authored-by: Isaac To <[email protected]> * Add test against unsupported model values * Update schema endpoint swagger docs * Add endpoint for listing available models * Fix view import/export * feat: change endpoints related to serving schemas * doc: update design docs regarding endpoint changes. * docs: remove mention of `TypeAdapter` in `vendored-schema-1-implementation.md` Pydantic models can generate their own JSONSchema, and we do generate the JSONSchema directly from the models. TypeAdapter in Pydantic has other uses, but we are not using it here. * docs: consolidate vendor-configurable metadata models design docs into one file --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Mike VanDenburgh <[email protected]> Co-authored-by: Isaac To <[email protected]> Co-authored-by: Isaac To <[email protected]> Co-authored-by: Mike VanDenburgh <[email protected]> Co-authored-by: Jacob Nesbitt <[email protected]>
4 participants
This is based of the mermaid diagrams in the blog post we have: https://bids-website.readthedocs.io/en/latest/blog/2024/11/13/bids-validator-2.html and the goal is to give a visual depiction of relations between the components mentioned on this page and elsewhere.
Here is a screenshot of preview. I do agree that text is a bit small'ish, so we might want to prune it but it is quite usable and IMHO useful... May be there is a way to make it "full screen"?
version1: horizontal, harder to read, no zooming etc
version2: vertical
version3: removed bids-apps and stats-models:
TODOs