Skip to content

Rebuilding Sandcastle: Technical Implementation #12566

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jjspace opened this issue Apr 11, 2025 · 7 comments · Fixed by #12574
Open

Rebuilding Sandcastle: Technical Implementation #12566

jjspace opened this issue Apr 11, 2025 · 7 comments · Fixed by #12574

Comments

@jjspace
Copy link
Contributor

jjspace commented Apr 11, 2025

This issue will track the technical implementation of a new version of Sandcastle for CesiumJS. Design and features will primarily be worked out and discussed elsewhere. If you want to provide feedback on features you'd like to see please add them to our post on the community forum

High level plan

We plan to create a new workspace/package within our mono-repo to keep the new Sandcastle project separated but co-located with CesiumJS.

  • itwin-ui: We expect to use the newest version of iTwin/Bentley UI library for both components and styling as much as possible. We may contribute improvements or components back to it as necessary.
  • React: This is a given if we want to make use of iTwinUI.
  • Monaco: We'll want to balance robust feature set with usability for users who prefer a more minimal code editing experience. This should greatly increase the DX of the editor bringing familiarity for any already used to using VSCode
  • TypeScript: Unless there is a reason not to. This will be isolated to it's own package so it's separate from the rest of CesiumJS
  • Build tools: We'll likely get started with the Vite plugin for react, and if needed we can use esbuild like the CesiumJS library itself

As we build out the new version we plan to have a beta/dev deployment for people to test against. More info on this as build systems and CI get set up

CC @ggetz

@jdfwarrior
Copy link
Contributor

This will be a very welcomed update. Sandcastle has had the same look and feel for many years and could definitely use a refresh.

@ggetz
Copy link
Contributor

ggetz commented Apr 30, 2025

Some running TODO's which have come up in discussion like the forum and @jjspace's initial PR.

  • Re-evaluate search (PageFine?)
  • Re-evaluate gallery system management
  • Modernize (and probably document) Sandcastle API
  • Code formatting command
  • Consider Gist and GitHub integrations— re: saving, sharing, commenting, collaborating
  • How to handle CesiumJS versions
  • Monitor/update build system as needed. Sandcastle Reborn #12574 did a lot to stabilize but changes may be needed based on other discussions
    • re-evaluate/finalize the eslint config for the new package files. First time TS and React is in there

@jjspace
Copy link
Contributor Author

jjspace commented May 9, 2025

Build structure

A big complication with this process is the actual build configuration. This is more complicated because locally we want to point at all the locally built files instead of copying them into the vite build statically. I wanted to share this breakdown I've been referencing a lot during my work on #12574. This is also not helped by the fact that when we deploy to prod we copy everything to the top level domain and basically all routes need to be adjusted somewhat to account for that. This will need to be kept in mind as we discuss how we want to change the import process into Sandcastles.

Local cesium server: npm start
    localhost:8080/Apps/Sandcastle/index.html
                                  /standalone.html (CESIUM_BASE_URL = '../../../Build/CesiumUnminified)
                                  /templates/bucket.html (CESIUM_BASE_URL = '../../../Build/CesiumUnminified') - look at load-es6.js
                                            /bucket.css (Accessed by '../templates/bucket.css')
                                  /gallery/Example.html (CESIUM_BASE_URL = '../../../Build/CesiumUnminified')
                                  /gallery/development/Example.html (still run in bucket.html or have custom `<base>` in standalone)

                                  /images/... (Accessed by '../images/...')
                  /Apps/SampleData/... (Accessed by '../../SampleData/...')

                  /Build/CesiumUnminified/Cesium.js ('../../Build/CesiumUnminified/index.js' from '../load-es6.js')
                                         /index.js
                                         /Assets/...
                                         /Widgets/...
                  /Source/Widgets/*.css (Accessed by '../../../Source/Widgets/...')

Production (check gulpfile buildSancastle for when isProduction)
    sandcastle.cesium.com/index.html
                         /standalone.html (CESIUM_BASE_URL = 'CesiumUnminified/')
                         /templates/bucket.html (CESIUM_BASE_URL = '../CesiumUnminified/')
                                   /bucket.css (Accessed by '../templates/bucket.css')
                         /gallery/Example.html (CESIUM_BASE_URL = '../CesiumUnminified/')

                         /images/... (Accessed by '../images/...)
                         /SampleData/... (Accessed by '../SampleData/...')

                         /CesiumUnminified/Cesium.js (Accessed by '../CesiumUnminified/...')
                                          /Assets/...
                                          /Widgets/...

CI deployment
    ci-build.cesium.com/cesium/\[branch\]/Apps/Sandcastle/index.html
                                                         /templates/bucket.html (CESIUM_BASE_URL = '../../../Build/CesiumUnminified/')
                                                                   /bucket.css
                                                         /gallery/Example.html (CESIUM_BASE_URL = '../../../Build/CesiumUnminified/')

                                                         /images/... (Accessed by '../images/...')
                                         /Apps/SampleData/... (Accessed by '../../SampleData/...')
                                         /Build/CesiumUnminified/index.js
                                                                /Assets/...
                                                                /Widgets/...

Local vite dev: npm run dev
    localhost:5173/index.html
                  /templates/bucket.html (CESIUM_BASE_URL = '/Build/CesiumUnminified')
                            /bucket.css
                  /images/... (Accessed by '../images/...')
                  /SampleData/... (Accessed by '../../SampleData/...')
                  /Build/CesiumUnminified/Cesium.js
                                         /Assets/...
                                         /Widgets/...

Local vite build for server: npm run build-app -> Apps/Sandcastle2 -> npm start
    localhost:8080/Apps/Sandcastle2/index.html
                                   /templates/bucket.html (CESIUM_BASE_URL = '/Build/CesiumUnminified')
                                             /bucket.css
                                   /images/... (Accessed by '../images/...')
                  /Apps/SampleData/... (Accessed by '../../SampleData/...')
                  /Build/CesiumUnminified/Cesium.js (Accessed by '/Build/CesiumUnminified')
                                         /Assets/...
                                         /Widgets/...
                  /Source/Cesium.d.ts

There are basically 3 main ways a specific sandcastle can be loaded. They all need to work. More specifically routes to imported JS or CSS and sample data need to work for all of these routes regardless of the environment outlined above.

  • Directly by navigating to the gallery file sandcastle.cesium.com/gallery/3D Models.html
  • bucket.html which is the iframe used in the main sandcastle UI sandcastle.cesium.com/templates/bucket.html
    • bucket-requirejs.html which is the iframe location that uses load-cesium-es6.js to load the esm module instead of the iife. This is only used locally and gets replaced when deployed
    • Note that bucket.html is "nested" inside the templates directory which lets paths from it mimic those in the gallery directory, this is important for them to both work the same
  • standalone.html which is the full standalone page that is capable of loading a sandcastle from the URL string https://sandcastle.cesium.com/standalone.html#c=bY...

Gallery files

Another big topic we still need to settle on is the structure of an actual, saved, gallery file. Right now they're all html files that work on their own. The code is extracted from them and loaded into the code editors when they're loaded in Sandcastle. Metadata like the title, tags and description is included in the <head> tags.

Open questions:

  1. Should these remain as singular html files? or should they be broken apart into html + js files?
  2. Should every sandcastle example have a @import(bucket.css) statement or can this be moved to the bucket.html and gallery files themselves?
  3. Is there a better way we can extract information to avoid loading every single gallery item on every page load?
  4. How do we manage more metadata?
    • There's been discussion of at the very least longer descriptions
    • I also think more integrated "explainers" for lines or chunks of code could be nice but I don't know the best way to integrate those
  5. Is there anything more we should include in the wrapper for sandcastle examples that could help the application as a whole work better?
    • For example forcing the iframe window to have a cesiumViewer variable accessible so the surrounding app can access it

CC @ggetz there's more beyond these to talk about but hopefully this can help kickstart our discussions

@jjspace jjspace mentioned this issue May 13, 2025
6 tasks
@ggetz
Copy link
Contributor

ggetz commented May 13, 2025

@jjspace Re: Gallery files

Before we get into the technical aspects of the gallery file (like the JS/CSS/HTML breakdown, import styles, etc), I think it would be helpful to decouple what we'd like out of a gallery example and how to organize them from implementation itself.

To start what makes up a gallery example? I think defining this will help guide (3) and is directly involved with (4).

  • Currently, sandcastle examples consist of the following, correct?
    • The code
    • A thumbnail
    • A title
    • A description
    • Labels (or tags)
  • In addition to tags, of which there can be multiple, would it be helpful to organize the examples into categories (or even subcategories if appropriate)? That could help with organization and user discoverability because each category can be more intentional or curated than the current deluge of examples in each tag.
  • I do like the idea of having more digestible "explainer" content to augment a sandcastle. IMO this should be separate from the description as that should be more concise. The explainer could potentially be a text property. But maybe it could be something with a bit more expressiveness such as markdown.
  • Are there any other metadata fields which we would like to maintain?

Then going one level deeper, how do we ideally want to store that metadata (4)? I'm putting aside backwards compatibility for a moment to think about how it might make sense if starting from scratch.

  • I think it's a bit cumbersome to manage this all in an HTML file directly. Contributors often forget to update descriptions and tags because they're easy to miss.
  • Ideally, a more intentional approach to the metadata could help us avoid loading each entire sandcastle example on page load (3).
  • Maybe we can use yaml for capturing gallery example metadata. YAML is common in a lot of documentation tools and static site generators, so I don't think it should be a barrier to entry for contributors. For instance:
title: Adjust 3D Tileset Height
thumbnail: "./3d-tiles-adjust-height.jpg"
description: Adjust the relative height of a tileset to better align it with terrain.
labels: 
   - 3D Tiles
   - Tutorial
category: Working with 3D Tiles
context: Each instance of a 3D tileset has a modelMatrix property that, when set, modifies the entire tileset's position, rotation, and scale. This transformations are applied in worldspace before the tileset's root tile transform.

Or, if we want more formatting, that yaml can be ported to a markdown file's front matter:

---
title: Adjust 3D Tileset Height
thumbnail: "./3d-tiles-adjust-height.jpg"
description: Adjust the relative height of a tileset to better align it with terrain.
labels: 
   - 3D Tiles
   - Tutorial
category: Working with 3D Tiles
---

Each instance of a 3D tileset has a `modelMatrix` property that, when set, modifies the entire tileset's position, rotation, and scale. This transformations are applied in worldspace before the tileset's [root tile transform](https://github.com/CesiumGS/3d-tiles/tree/main/specification#transforms).

For (5), I'm having trouble picturing a use case, but it sounds like you have some in mind. Could you share what you were thinking about?

Let me know your thoughts on all that, and then we can talk through the details of how we'd like to store the code and what this may mean for backwards compatability.

@jjspace
Copy link
Contributor Author

jjspace commented May 15, 2025

@ggetz sorry my response got a little lost but here's my thoughts

To start what makes up a gallery example?

Correct, that is the current list, even if not all of those are displayed promenantly (description).

I think it could be helpful to include a general category but I'm not sure how different it would be than tags. It would definitely help to categorize in whatever UI we pick but I could also see it being an issue that certain sandcastles should belong to multiple categories and then we're right back to tags.
Maybe we just pick a set of tags that are "important" that we can use to categorize in the UI but still only store them as tags?

how do we ideally want to store that metadata?

I think it's a bit cumbersome to manage this all in an HTML file directly. Contributors often forget to update descriptions and tags because they're easy to miss.

I agree it may get cumbersome to all be in the same html file. I also think the more we do in there the harder it may make our lives to try and parse/process it when we want to load examples. That said, a separate file may end up being even easier to miss and forget to update but I think that's going to be true for any solution we come up with.

Ideally, a more intentional approach to the metadata could help us avoid loading each entire sandcastle example on page load (3).

I agree, even if we stuck to a single file I would want to pre-process it more so we can load only the metadata on page load instead of the full example.

Maybe we can use yaml for capturing gallery example metadata.

I do really like the idea of using YAML, it's common and easy to use and can be pretty extensible. Definitely easier to write and read than JSON. The main question will be where does it live.

This is all pointing to having one or more "sidecar files" for a given sandcastle.
We have also discussed adding a larger "explainer" type block of text or MD so that needs considered as well. Maybe that could be where we use Frontmatter for the metadata and the rest of the file for any explainer if it exists? If a sandcastle doesn't need that maybe it's just an adjacent [sandcastle name].yaml that has the data?

One big question I think we need to answer is still: Should people be able to load gallery examples directly? If the answer is "yes" then I think we should keep what we can inside the html file itself. If the answer is "no" then we can do more with the "save format". Maybe even separating into dedicated JS and HTML files. or creating subdirectories for each sandcastle and use the file structure itself to encapsulate the metadata sidecar files and the thumbnail and the code etc.

For (5), I'm having trouble picturing a use case, but it sounds like you have some in mind. Could you share what you were thinking about?

The primary usecase I like is just having access to the active Viewer from the console in the outer window context. It's also really useful if we want to add any sort of helper buttons to the Sandcastle UI itself to add code like "Save the current view" which I've found very helpful. Without a consistent way to reach into the iframe or for the iframe to reach out it's impossible to do things like that.
Interestingly I found that Babylon's playground requires that the code creates a createScene function in order to run. Maybe we should do the same with viewer?

@ggetz
Copy link
Contributor

ggetz commented May 16, 2025

Categories

Maybe we just pick a set of tags that are "important" that we can use to categorize in the UI but still only store them as tags?

I agree this is valid from an implementation standpoint. From a usability perspective, just displaying the tags as "categories" is a bit overwhelming due to the number of them, and I think we would like to be more intentional about which ones we present and in which order.

I agree it may get cumbersome to all be in the same html file. I also think the more we do in there the harder it may make our lives to try and parse/process it when we want to load examples. That said, a separate file may end up being even easier to miss and forget to update but I think that's going to be true for any solution we come up with.

One big question I think we need to answer is still: Should people be able to load gallery examples directly?

Good point. Just for clarity here, you're talking about the ability to take the HTML file and load it outside of the context of Sandcastle as more of a standalone example, correct? Or are you looking more to address "saving" a sandcastle to a file and being able to bring it back in?

Overall, I'm personally favoring a gallery example being a directory of files for the flexibility and simplicity from the point of view of the user. Theoretically, this wouldn't prevent the example from working as a standalone example. And we could save as a zip file if the user wants to download, while an example could be brought back in by unzipping in browser (for example).

  • [Optional] README.md for the explainer text so we can use markup, link to other resources, etc. This also gives us the benefit of having the explainer info be discoverable and browsable in the GitHub repo as well as Sandcastle.
  • [Optional] Thumbnail image
  • HTML
  • JS
  • [Optional] Standalone CSS
  • yml (or frontmatter in the README.md) specifying all the needed metadata:
    • title (string),
    • description (string),
    • tags (string[]),
    • html (string, relative path),
    • js (string, relative path),
    • css (string, relative path, optional)
    • thumbnail (string, relative path, optional),

Potentially, we could include things like version info, author, last updated, or search embeddings if they prove useful.

I really am leaning towards the yml data, as it will be easy to generate a gallery list with all the needed info for displaying the example in the Sandcastle UI without having to load to content itself.

@ggetz
Copy link
Contributor

ggetz commented May 16, 2025

Talking offline, with @jjspace– To facilitate different directory structures, we'll consider path aliasing for things like SampleData. Also we should probably move (or duplicate) images and othered referenced data to the SampleData directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants