Self hosted docker Infisical exits with "Boot up migration failed"

[pseudokool]

Hi,

I am trying to get Infisical up and running on AWS EC2, using a t3a.medium instance with Ubuntu 24.

I have customized the samle .env, and supplying it to the docker run command. However...

ubuntu@ip-XXXXXX:~/apps/infisical$ docker run -p 80:8080 --env-file .env infisical/infisical:v0.115.0-postgres
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
(node:1) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
(Use `node --trace-warnings ...` to show where the warning was created)
{"level":50,"time":1741781823620,"pid":1,"hostname":"c4c95bd5c774","reqId":"UNKNOWN_REQUEST_ID","severity":"ERROR","err":{"type":"KnexTimeoutError","message":"Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?","stack":"KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?\n    at Client_PG.acquireConnection (/backend/node_modules/knex/lib/client.js:332:26)\n    at async Runner.ensureConnection (/backend/node_modules/knex/lib/execution/runner.js:305:28)\n    at async Runner.run (/backend/node_modules/knex/lib/execution/runner.js:30:19)","name":"KnexTimeoutError"},"msg":"Boot up migration failed"}

Same issue when running it via docker-compose

[akhilmhdh]

@pseudokool What's your postgres looking like? Most probably it's a really mini postgres.

[pseudokool]

Yes, both Infisical and postgres are on the same 2vCPU, 4GB ec2 instance.

Would it a matter or size, or it not able to connect to the db?

[erickne]

Same here. I'm using docker-compose.prod.yml

[Amahr]

Same here.
Docker-compose.prod.yml worked before, today I restarted the stack, now it won't come back up because of migration container failing. Running the stack without the migration container also won't work. The backend will then complain about table not lockable.

Running knex migrate:unlock as advised inside the backend container is not working. - command not found

Output of backend container log:

2025-03-17T19:53:21.490097471Z Can't take lock to run migrations: Migration table is already locked

2025-03-17T19:53:21.490143509Z If you are sure migrations are not running you can release the lock manually by running 'knex migrate:unlock'

2025-03-17T19:53:21.555767912Z 2025/03/17 08:53PM 50 | pid=1 hostname=8a95545082e1 reqId=UNKNOWN_REQUEST_ID severity=ERROR err={"type":"LockError","message":"Migration table is already locked","stack":"MigrationLocked: Migration table is already locked\n    at /backend/node_modules/knex/lib/migrations/migrate/Migrator.js:343:13\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Migrator._runBatch (/backend/node_modules/knex/lib/migrations/migrate/Migrator.js:359:7)","name":"MigrationLocked"} msg=Boot up migration failed ```

[pseudokool]

Same here. Docker-compose.prod.yml worked before, today I restarted the stack, now it won't come back up because of migration container failing. Running the stack without the migration container also won't work. The backend will then complain about table not lockable.

Running knex migrate:unlock as advised inside the backend container is not working. - command not found

Output of backend container log:

2025-03-17T19:53:21.490097471Z Can't take lock to run migrations: Migration table is already locked

2025-03-17T19:53:21.490143509Z If you are sure migrations are not running you can release the lock manually by running 'knex migrate:unlock'

2025-03-17T19:53:21.555767912Z 2025/03/17 08:53PM 50 | pid=1 hostname=8a95545082e1 reqId=UNKNOWN_REQUEST_ID severity=ERROR err={"type":"LockError","message":"Migration table is already locked","stack":"MigrationLocked: Migration table is already locked\n    at /backend/node_modules/knex/lib/migrations/migrate/Migrator.js:343:13\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Migrator._runBatch (/backend/node_modules/knex/lib/migrations/migrate/Migrator.js:359:7)","name":"MigrationLocked"} msg=Boot up migration failed ```

This is how I attempted to run it, which worked...

ubuntu@ip-X-X-X-X:~$ docker exec -it 69f25a1d9b41 sh
$ npm run auditlog-migration:unlock

> [email protected] auditlog-migration:unlock
> knex --knexfile ./dist/db/auditlog-knexfile.mjs migrate:unlock

Requiring external module /backend/node_modules/interpret/mjs-stub
Working directory changed to /backend/dist/db
Dedicated audit log database not found. No further migrations necessary
npm notice
npm notice New major version of npm available! 10.8.2 -> 11.2.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v11.2.0
npm notice To update run: npm install -g [email protected]
npm notice

...but now. This shows up next in my container logs. akhilmhdh is this what you meant by mini pg and it being a problem?

{"level":50,"time":1742275594409,"pid":1,"hostname":"69f25a1d9b41","reqId":"UNKNOWN_REQUEST_ID","severity":"ERROR","err":{"type":"KnexTimeoutError","message":"Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?","stack":"KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?\n    at Client_PG.acquireConnection (/backend/node_modules/knex/lib/client.js:332:26)\n    at async Runner.ensureConnection (/backend/node_modules/knex/lib/execution/runner.js:305:28)\n    at async Runner.run (/backend/node_modules/knex/lib/execution/runner.js:30:19)","name":"KnexTimeoutError"},"msg":"Boot up migration failed"}
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.