Skip to content

Add registry-watcher image and separate docker-compose services #2269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
/target/
/.rustwide
/.rustwide-docker
/Justfile
/LICENSE
/README.md
/docker-compose.yml
/docs/
/ignored
**/target
/mcps
/triagebot.toml
/clippy.toml
35 changes: 6 additions & 29 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,8 @@ jobs:

- name: Launch postgres
run: |
cp .env.sample .env
mkdir -p ${DOCSRS_PREFIX}/public-html
docker compose up -d db
# Give the database enough time to start up
sleep 5
# Make sure the database is actually working
psql "${DOCSRS_DATABASE_URL}"
touch .docker.env
docker compose up --wait --wait-timeout 30 db

- name: install SQLX CLI
run: cargo install sqlx-cli --no-default-features --features postgres
Expand All @@ -66,9 +61,6 @@ jobs:
--database-url $DOCSRS_DATABASE_URL \
--target-version 0

- name: Clean up the database
run: docker compose down --volumes

test:
env:
SQLX_OFFLINE: 1
Expand All @@ -91,13 +83,8 @@ jobs:

- name: Launch postgres and min.io
run: |
cp .env.sample .env
mkdir -p ${DOCSRS_PREFIX}/public-html
docker compose up -d db s3
# Give the database enough time to start up
sleep 5
# Make sure the database is actually working
psql "${DOCSRS_DATABASE_URL}"
touch .docker.env
docker compose up --wait --wait-timeout 30 db s3

- name: run workspace tests
run: |
Expand All @@ -109,9 +96,6 @@ jobs:
run: |
cargo test --locked -- --ignored --test-threads=1

- name: Clean up the database
run: docker compose down --volumes

GUI_test:
runs-on: ubuntu-latest
steps:
Expand All @@ -129,19 +113,12 @@ jobs:

- name: Launch postgres and min.io
run: |
cp .env.sample .env
mkdir -p ${DOCSRS_PREFIX}/public-html
docker compose up -d db s3
# Give the database enough time to start up
sleep 5
# Make sure the database is actually working
psql "${DOCSRS_DATABASE_URL}"
touch .docker.env
docker compose up --wait --wait-timeout 30 db s3

- name: Run GUI tests
run: ./dockerfiles/run-gui-tests.sh

- name: Clean up the database
run: docker compose down --volumes

fmt:
name: Rustfmt
Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,7 @@ jobs:
steps:
- uses: actions/checkout@v4

- name: Build the Docker image
run: docker build -t docs-rs -f dockerfiles/Dockerfile .
- run: docker build --target web-server -f dockerfiles/Dockerfile .
- run: docker build --target build-server -f dockerfiles/Dockerfile .
- run: docker build --target registry-watcher -f dockerfiles/Dockerfile .
- run: docker build --target cli -f dockerfiles/Dockerfile .
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/ignored
/.env
/.docker.env
/src/web/badge/Cargo.lock
target
*.css
Expand Down
27 changes: 27 additions & 0 deletions Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,30 @@ lint:

lint-js:
deno run -A npm:eslint@9 static templates gui-tests eslint.config.js

# Initialize the docker compose database
[group('compose')]
compose-migrate:
docker compose run --build --rm cli database migrate

# Update last seen reference to the current index head, to only build newly published crates
[group('compose')]
compose-queue-head:
docker compose run --build --rm cli queue set-last-seen-reference --head

# Launch base docker services, ensuring the database is migrated
[group('compose')]
compose-up:
just compose-migrate
docker compose up --build -d

# Launch base docker services and registry watcher, ensuring the database is migrated
[group('compose')]
compose-up-watch:
just compose-migrate
docker compose --profile watch up --build -d

# Shutdown docker services and cleanup all temporary volumes
[group('compose')]
compose-down:
docker compose --profile all down --volumes --remove-orphans
69 changes: 51 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,7 @@ mkdir -p ignored/cratesfyi-prefix/crates.io-index
# Builds the docs.rs binary
SQLX_OFFLINE=1 cargo build
# Start the external services.
# It may be `docker compose` in newer versions
docker-compose up -d db s3
docker compose up --wait db s3
# anything that doesn't run via docker-compose needs the settings defined in
# .env. Either via `. ./.env` as below, or via any dotenv shell integration.
. ./.env
Expand Down Expand Up @@ -117,23 +116,57 @@ can take a look at its [documentation](https://github.com/GuillaumeGomez/browser

### Pure docker-compose

If you have trouble with the above commands, consider using `docker-compose up --build`,
If you have trouble with the above commands, consider using `docker compose up --build`,
which uses docker-compose for the web server as well.
This will not cache dependencies - in particular, you'll have to rebuild all 400 whenever the lockfile changes -
but makes sure that you're in a known environment so you should have fewer problems getting started.

You can also use the `web` container to run builds on systems which don't support running builds directly (mostly on Mac OS or Windows):
You'll need to `touch .docker.env` first, this file can have any environment
variable overrides you want to use in docker containers. Then run the migrations
before launching the main services:

```sh
docker compose run --build --rm cli database migrate
docker compose up --build -d
```

You can also use the `builder-a` container to run builds on systems which don't support running builds directly (mostly on Mac OS or Windows):

```sh
# update the toolchain
docker compose run --rm builder-a build update-toolchain
# run a build for a single crate
docker-compose run web build crate regex 1.3.1
# or build essential files
docker-compose run web build add-essential-files
# rebuild the web container when you changed code.
docker-compose build web
docker compose run --rm builder-a build crate regex 1.3.1
# rebuild containers when you changed code.
docker compose up --wait --build
```

You can also run other non-build commands like the setup steps above, or queueing crates for the background builders from within the `cli` container:

```sh
docker compose run --rm cli database migrate
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not clear that you have to run this command to make the service work.

docker compose run --rm cli queue add regex 1.3.1
```

If you want to run the registry watcher, you'll need to first set the "last seen
reference" from the registry index, e.g. to set it to the current head so only
newly published crates are built:

```sh
docker compose run --rm cli queue set-last-seen-reference --head
```

Then enable the docker-compose profile that includes the watcher:

```sh
docker compose --profile watch up --build -d
```

Note that running tests is not supported when using pure docker-compose.

Some of the above commands are included in the `Justfile` for ease of use,
check the `[compose]` group in `just --list`.

Please file bugs for any trouble you have running docs.rs!

### Docker-Compose
Expand All @@ -143,22 +176,22 @@ Three services are defined:

| name | access | credentials | description |
|------|-------------------------------------------------|----------------------------|----------------------------------------|
| web | http://localhost:3000 | N/A | A container running the docs.rs binary |
| db | postgresql://cratesfyi:password@localhost:15432 | - | Postgres database used by web |
| s3 | http://localhost:9000 | `cratesfyi` - `secret_key` | MinIO (simulates AWS S3) used by web |
| web | http://0.0.0.0:3000 | N/A | A container running the docs.rs binary |
| db | postgresql://cratesfyi:password@127.0.0.1:15432 | - | Postgres database used by web |
| s3 | http://127.0.0.1:9000 | `cratesfyi` - `secret_key` | MinIO (simulates AWS S3) used by web |

[docker-compose.yml]: ./docker-compose.yml

#### Rebuilding Containers

To rebuild the site, run `docker-compose build`.
To rebuild the site, run `docker compose --profile all build`.
Note that docker-compose caches the build even if you change the source code,
so this will be necessary anytime you make changes.

If you want to completely clean up the database, don't forget to remove the volumes too:

```sh
$ docker-compose down --volumes
$ docker compose down --volumes
```

#### FAQ
Expand All @@ -172,7 +205,7 @@ This is probably because you have `git.autocrlf` set to true,

##### I see the error `/opt/rustwide/cargo-home/bin/cargo: cannot execute binary file: Exec format error` when running builds.

You are most likely not on a Linux platform. Running builds directly is only supported on `x86_64-unknown-linux-gnu`. On other platforms you can use the `docker-compose run web build [...]` workaround described above.
You are most likely not on a Linux platform. Running builds directly is only supported on `x86_64-unknown-linux-gnu`. On other platforms you can use the `docker compose run --rm builder-a build [...]` workaround described above.

See [rustwide#41](https://github.com/rust-lang/rustwide/issues/41) for more details about supporting more platforms directly.

Expand Down Expand Up @@ -200,11 +233,11 @@ cargo run -- start-web-server
```sh
# Builds <CRATE_NAME> <CRATE_VERSION> and adds it into database
# This is the main command to build and add a documentation into docs.rs.
# For example, `docker-compose run web build crate regex 1.1.6`
# For example, `docker compose run --rm builder-a build crate regex 1.1.6`
cargo run -- build crate <CRATE_NAME> <CRATE_VERSION>

# alternatively, via the web container
docker-compose run web build crate <CRATE_NAME> <CRATE_VERSION>
# alternatively, within docker-compose containers
docker compose run --rm builder-a build crate <CRATE_NAME> <CRATE_VERSION>

# Builds every crate on crates.io and adds them into database
# (beware: this may take months to finish)
Expand Down
Loading