Issues setting up the dev environment

[audiodude]

Hi @audiodude @rgaudin ,

From my understanding:

• The web server is running on Flask
• The database is MySQL
• Available selection methods: Book, PetScan, SPARQL, WikiProject, and Simple selection

Currently, we can only use one selection method at a time. However, I would like to build a selection combination system that allows users to union, intersect, or subtract multiple selections. These operations may also be recursive.

So my plan is to build:

• The mathematical logic to combine multiple selection operations
• A GUI interface to allow users to create these combinations easily
  Am I understanding the idea correctly? Please let me know if I missed anything.

all development has been on Linux, use other platforms at your own risk.

Because of that, I first tried installing it on WSL with Ubuntu 22.04. However, I struggled to properly migrate the database in MySQL. After dealing with issues related to Redis and Mono, I switched to Docker Desktop instead.

Now, all the workers are running correctly:

Except for the zimfarm-worker-manager .

[2026-03-07 23:17:00,070: INFO] starting zimfarm worker-manager.
[2026-03-07 23:17:00,071: INFO] configuration:
	username=test_worker
	webapi_uris=['http://zimfarm-api:80/v2']
	workdir=/data
	worker_name=test_worker
	OFFLINERS=['mwoffliner', 'youtube', 'phet', 'gutenberg', 'sotoki', 'nautilus', 'ted', 'openedx', 'zimit', 'kolibri', 'wikihow', 'ifixit', 'freecodecamp', 'devdocs', 'mindtouch', 'maps']
	PLATFORMS_TASKS={}
	poll_interval=10
	sleep_interval=5
	selfish=False
[2026-03-07 23:17:00,071: WARNING] Declared memory 21474836480 appears to be greater than actual memory 9288482816
[2026-03-07 23:17:00,071: INFO] testing workdir at /data…
[2026-03-07 23:17:00,071: INFO] 	workdir is available and writable
[2026-03-07 23:17:00,071: INFO] testing private key at /etc/ssh/keys/zimfarm…
[2026-03-07 23:17:00,071: CRITICAL] 	private key is not a readable path

Although I have linked the SSH folder to a local folder, it still shows that the private key is not readable.

Could you please guide me on how to resolve this? I'm eager to learn anything required to work on this.

Originally posted by @sajjadhossanshimanto in #728

Let's discuss your issues with setting up the development environment here.

[sajjadhossanshimanto]

How I might set up a zimfarm-worker-manager on a Windows host using Docker. Despite mounting the SSH private key, the worker fails during the startup sequence with a critical error regarding the private key path.

Environment:

• os : Windows
• Docker Image: ghcr.io/openzim/zimfarm-worker-manager:latest
• Storage: Bind mount from host D: drive

Log:

[2026-03-07 23:17:00,070: INFO] starting zimfarm worker-manager.
[2026-03-07 23:17:00,071: INFO] configuration:
	username=test_worker
	webapi_uris=['http://zimfarm-api:80/v2']
	workdir=/data
	worker_name=test_worker
	OFFLINERS=['mwoffliner', 'youtube', 'phet', 'gutenberg', 'sotoki', 'nautilus', 'ted', 'openedx', 'zimit', 'kolibri', 'wikihow', 'ifixit', 'freecodecamp', 'devdocs', 'mindtouch', 'maps']
	PLATFORMS_TASKS={}
	poll_interval=10
	sleep_interval=5
	selfish=False
[2026-03-07 23:17:00,071: WARNING] Declared memory 21474836480 appears to be greater than actual memory 9288482816
[2026-03-07 23:17:00,071: INFO] testing workdir at /data…
[2026-03-07 23:17:00,071: INFO] 	workdir is available and writable
[2026-03-07 23:17:00,071: INFO] testing private key at /etc/ssh/keys/zimfarm…
[2026-03-07 23:17:00,071: CRITICAL] 	private key is not a readable path

configuration: (docker-compose.yml)

  zimfarm-worker-manager:
    image: ghcr.io/openzim/zimfarm-worker-manager:latest
    pull_policy: always
    container_name: zimfarm-worker-manager
    depends_on:
      zimfarm-api:
        condition: service_healthy
    command: worker-manager --webapi-uri 'http://zimfarm-api:80/v2' --username test_worker --name test_worker
    environment:
      - DEBUG=true
      - TASK_WORKER_IMAGE=ghcr.io/openzim/zimfarm-task-worker:latest
      - ENVIRONMENT=development
      - ZIMFARM_DISK=20Gb
      - ZIMFARM_MEMORY=20Gb
      - ZIMFARM_CPU=3
      - POLL_INTERVAL=10
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./docker/zimfarm/id_ed25519:/etc/ssh/keys/zimfarm
    networks:
      - wp1bot-dev
    profiles:
      - zimfarm-worker

What I've tried

• Generated a new Ed25519 key pair using ssh-keygen.

• Confirmed that the source path D:\coding\kiwi_wp1\docker\zimfarm\id_ed25519 is correctly recognized as a bind mount in Docker Inspect.

	"Mounts": [
		{
			"Type": "volume",
			"Name": "0971ad92156155c64a694d165cd2f8df331dd9c901063225510ec746d6771be9",
			"Source": "/var/lib/docker/volumes/0971ad92156155c64a694d165cd2f8df331dd9c901063225510ec746d6771be9/_data",
			"Destination": "/data",
			"Driver": "local",
			"Mode": "",
			"RW": true,
			"Propagation": ""
		},
		{
			"Type": "bind",
			"Source": "D:\\coding\\kiwi_wp1\\docker\\zimfarm\\id_ed25519",
			"Destination": "/etc/ssh/keys/zimfarm",
			"Mode": "rw",
			"RW": true,
			"Propagation": "rprivate"
		},
		{
			"Type": "bind",
			"Source": "/var/run/docker.sock",
			"Destination": "/var/run/docker.sock",
			"Mode": "rw",
			"RW": true,
			"Propagation": "rprivate"
		}
	],

Even with the key present at the destination /etc/ssh/keys/zimfarm, the application reports it is not a "readable path".

[benoit74]

wp1 and zimfarm are not meant to run on Windows.

I would recommend that you run these inside WSL2 or on a Linux VM (local or remote). I doubt we will spend time fixing issues on a platform we have no use of.