Architecture

There are three major parts to the architecture:

• Express.js backend: The project backend is contained in the boneset-api/ directory. The primary purpose of that folder is to house server.js, which acts as our app's server using Express.js. As described below, we are in the process of moving all bone data into this folder to live alongside the server scripts.

• HTML/CSS/JavaScript frontend: The frontend in the templates/ directory consists of HTML, CSS, and JavaScript files for the user-facing portion of the app. Some portions of the HTML files use HTMX to dynamically load in content from the backend. Otherwise, data from the backend is fetched through templates/js/api.js.

• Python scripts for data extraction: The bone data from our app is sourced from PowerPoint presentations created by our client, Professor Brian Elliott. The data_extraction/ directory contains Python scripts used to automate extraction of data from those slideshows into formats usable by our app. Our internal development team uses these scripts to obtain and clean up the data in our app.

The backend and the frontend are each their own distinct Node.js projects with their own package.json files. The root directory also contains a package.json that contains scripts to run the backend and frontend concurrently.

Data Migration

We are transitioning the architecture for how our data is stored. The old model for the app involves storing the bone data in a separate branch in the repository and fetching it from GitHub while running the app. Under this old model, our main codebase is stored in the main branch, and the bone data is stored in the data branch. The code in the main branch fetches the data by directly fetching from the content URL, e.g. https://raw.githubusercontent.com/oss-slu/DigitalBonesBox/data/data/.

The old model also included a cache on the server to store data to avoid constantly fetching the data from GitHub. This had only been partially implemented.

The new model involves storing the data directly on the server. The same files that are hosted in the data Git branch in the old model will be moved into a subfolder of the boneset-api/ directory that houses our server.

The reasons we are moving the data are:

• More secure: Attacks on GitHub will not impact our live app
• Faster and more reliable: Failure to fetch data from GitHub for any reason will not impact our live app. Continually fetching data from GitHub would also have resulted in rate limiting and would have been slower than storing the data locally; a plan was therefore in place to cache the data on the server once initially fetched. But storing the data on the server in the first place is simpler and less prone to errors.
• Faster to develop: If someone is developing a feature and notices that a change in the data is necessary for it to appear correctly, they would currently have to submit the change as a pull request into the data branch and wait for the entire process of reviewing and merging before they can even see the change reflected in their local development environment. But after we move the data, they will be able to update the data locally and see the change reflected immediately in their local test server.

Data Flow

Overall

How data flows from the original PowerPoint files, through data extraction and uploading to the database, through the web app.

Old model

New model

Backend

How the app backend caches and serves data.

Old model

New model

Data Structure

How the bone data is structured.