Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add dockerfile and docker-compose file for cdms databases #12

Merged
merged 4 commits into from
Sep 5, 2021
Merged

add dockerfile and docker-compose file for cdms databases #12

merged 4 commits into from
Sep 5, 2021

Conversation

faysal-ishtiaq
Copy link
Collaborator

@faysal-ishtiaq faysal-ishtiaq commented Aug 21, 2021

@isedwards

I have created a docker-compose file that starts required database engine for each CDMS.

Also created dockerfiles residing in containers directory. These dockerfiles usually fetches the required image and places ddl file into /docker-entrypoint-initdb.d/ directory so that they are executed when initializing a fresh instance.

There are few things to be discussed.

  • For CliDE there were a few extra users required in the database and so I created them in scripts/entrypoints/clide.sh and placed them in /docker-entrypoint-initdb.d/ to be executed before the ddl file is executed
  • Climsoft is working good
  • For MCH, I guess the docker image from dockerhub is built by you. I could not connect the db using the password I passed as MYSQL_ROOT_PASSWORD in docker-compose envirmonement. It would be great if there is initialization script support as there is in regular MySQL/PostgreSQL/Oracle docker images using scripts placed in /docker-entrypoint-initdb.d/
  • For MIDAS, I am not sure about it. But I think it's not working. Here is the logs
MIDAS Logs
Starting Oracle Net Listener.
Starting Oracle Database 11g Express Edition instance.


System altered.

/usr/sbin/startup.sh: running /docker-entrypoint-initdb.d/1.sql

SQL*Plus: Release 11.2.0.2.0 Production on Sat Aug 21 13:02:03 2021

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production


Table created.

SQL> Disconnected from Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production


Starting Oracle Net Listener.
Starting Oracle Database 11g Express Edition instance.


System altered.

/usr/sbin/startup.sh: running /docker-entrypoint-initdb.d/1.sql

SQL*Plus: Release 11.2.0.2.0 Production on Sat Aug 21 13:16:25 2021

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production

CREATE TABLE address
             *
ERROR at line 1:
ORA-00955: name is already used by an existing object


SQL> Disconnected from Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production

  • For WMDR, I guess it's working as expected. I used this DDL file schemas/wmdr/wmdr_ddl.sql I am attaching the logs for your reference.
WMDR logs
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
sh: locale: not found
2021-08-21 13:01:54.613 UTC [30] WARNING:  no usable system locales were found
performing post-bootstrap initialization ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok


Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....2021-08-21 13:01:55.096 UTC [35] LOG:  starting PostgreSQL 12.7 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, 64-bit
2021-08-21 13:01:55.097 UTC [35] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-21 13:01:55.109 UTC [36] LOG:  database system was shut down at 2021-08-21 13:01:54 UTC
2021-08-21 13:01:55.112 UTC [35] LOG:  database system is ready to accept connections
2021-08-21 13:01:55.112 UTC [42] LOG:  TimescaleDB background worker launcher connected to shared catalogs
 done
server started

/usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/000_install_timescaledb.sh
2021-08-21 13:01:55.382 UTC [51] WARNING:  
        WELCOME TO
         _____ _                               _     ____________  
        |_   _(_)                             | |    |  _  \ ___ \ 
          | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ / 
          | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \ 
          | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
          |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
                       Running version 2.3.0
        For more information on TimescaleDB, please visit the following links:
        
         1. Getting started: https://docs.timescale.com/getting-started
         2. API reference documentation: https://docs.timescale.com/api
         3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture
psql:/tmp/tmp.ghjBhH:1: WARNING:  
WELCOME TO
 _____ _                               _     ____________  
|_   _(_)                             | |    |  _  \ ___ \ 
  | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ / 
  | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \ 
  | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
  |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
               Running version 2.3.0
For more information on TimescaleDB, please visit the following links:

 1. Getting started: https://docs.timescale.com/getting-started
 2. API reference documentation: https://docs.timescale.com/api
 3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture

Note: TimescaleDB collects anonymous reports to better understand and assist our users.
For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.

        
        Note: TimescaleDB collects anonymous reports to better understand and assist our users.
        For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.
        
2021-08-21 13:01:55.382 UTC [51] CONTEXT:  PL/pgSQL function inline_code_block line 12 at RAISE
CREATE EXTENSION
2021-08-21 13:01:55.543 UTC [54] WARNING:  
        WELCOME TO
         _____ _                               _     ____________  
        |_   _(_)                             | |    |  _  \ ___ \ 
          | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ / 
          | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \ 
          | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
          |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
                       Running version 2.3.0
        For more information on TimescaleDB, please visit the following links:
        
         1. Getting started: https://docs.timescale.com/getting-started
         2. API reference documentation: https://docs.timescale.com/api
         3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture
        
        Note: TimescaleDB collects anonymous reports to better understand and assist our users.
        For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.
        
2021-08-21 13:01:55.543 UTC [54] CONTEXT:  PL/pgSQL function inline_code_block line 12 at RAISE
psql:/tmp/tmp.ghjBhH:1: WARNING:  
WELCOME TO
 _____ _                               _     ____________  
|_   _(_)                             | |    |  _  \ ___ \ 
  | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ / 
  | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \ 
  | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
  |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
               Running version 2.3.0
For more information on TimescaleDB, please visit the following links:

 1. Getting started: https://docs.timescale.com/getting-started
 2. API reference documentation: https://docs.timescale.com/api
 3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture

Note: TimescaleDB collects anonymous reports to better understand and assist our users.
For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.

2021-08-21 13:01:55.547 UTC [55] WARNING:  telemetry could not connect to "telemetry.timescale.com"
2021-08-21 13:01:55.547 UTC [55] DETAIL:  Interrupted system call
2021-08-21 13:01:55.547 UTC [56] ERROR:  TimescaleDB background worker connected to template database, exiting
CREATE EXTENSION

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/001_timescaledb_tune.sh
2021-08-21 13:01:55.548 UTC [35] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 56) exited with exit code 1
Using postgresql.conf at this path:
/var/lib/postgresql/data/postgresql.conf

shared_buffers = 3922MB
effective_cache_size = 11766MB
maintenance_work_mem = 1961MB
work_mem = 5020kB
Writing backup to:
/tmp/timescaledb_tune.backup202108211301

Recommendations based on 15.32 GB of available memory and 8 CPUs for PostgreSQL 12
timescaledb.max_background_workers = 8
max_worker_processes = 19
max_parallel_workers_per_gather = 4
max_parallel_workers = 8
wal_buffers = 16MB
min_wal_size = 512MB
default_statistics_target = 500
random_page_cost = 1.1
checkpoint_completion_target = 0.9
max_locks_per_transaction = 128
autovacuum_max_workers = 10
autovacuum_naptime = 10
effective_io_concurrency = 200
timescaledb.last_tuned = '2021-08-21T13:01:55Z'
timescaledb.last_tuned_version = '0.11.0'
Saving changes to: /var/lib/postgresql/data/postgresql.conf

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/1.sql
psql:/docker-entrypoint-initdb.d/1.sql:10: NOTICE:  table "Abstractenvironmentalmonitoringfacility" does not exist, skipping
DROP TABLE
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:13: NOTICE:  table "Attribution" does not exist, skipping
psql:/docker-entrypoint-initdb.d/1.sql:16: NOTICE:  table "Climatezone" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:19: NOTICE:  table "Controlcheckreport" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:22: NOTICE:  table "Datageneration" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:25: NOTICE:  table "Datapolicy" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:28: NOTICE:  table "Deployment" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:31: NOTICE:  table "Description" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:34: NOTICE:  table "Equipment" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:37: NOTICE:  table "EquipmentEquipment" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:40: NOTICE:  table "Equipmentlog" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:43: NOTICE:  table "Eventreport" does not exist, skipping
DROP TABLE
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:46: NOTICE:  table "Facilitylog" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:49: NOTICE:  table "Facilityset" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:52: NOTICE:  table "FacilitySetObservingFacility" does not exist, skipping
psql:/docker-entrypoint-initdb.d/1.sql:55: NOTICE:  table "Frequencies" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:58: NOTICE:  table "Geospatiallocation" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:61: NOTICE:  table "Header" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:64: NOTICE:  table "Instrumentoperatingstatus" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:67: NOTICE:  table "Log" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:70: NOTICE:  table "Logentry" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:73: NOTICE:  table "Maintenancereport" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:76: NOTICE:  table "Observingcapability" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:79: NOTICE:  table "Observingfacility" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:82: NOTICE:  table "Population" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:85: NOTICE:  table "Process" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:88: NOTICE:  table "Processing" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:91: NOTICE:  table "Programaffiliation" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:94: NOTICE:  table "Reporting" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:97: NOTICE:  table "Reportingstatus" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:100: NOTICE:  table "Responsibleparty" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:103: NOTICE:  table "Resultset" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:106: NOTICE:  table "Sampling" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:109: NOTICE:  table "Schedule" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:112: NOTICE:  table "Surfacecover" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:115: NOTICE:  table "Surfaceroughness" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:118: NOTICE:  table "Territory" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:121: NOTICE:  table "Timezone" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:124: NOTICE:  table "Topographybathymetry" does not exist, skipping
DROP TABLE
psql:/docker-entrypoint-initdb.d/1.sql:127: NOTICE:  table "Wigosmetadatarecord" does not exist, skipping
DROP TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT


waiting for server to shut down....2021-08-21 13:01:55.855 UTC [35] LOG:  received fast shutdown request
2021-08-21 13:01:55.856 UTC [35] LOG:  aborting any active transactions
2021-08-21 13:01:55.856 UTC [52] LOG:  terminating TimescaleDB job scheduler due to administrator command
2021-08-21 13:01:55.856 UTC [42] LOG:  terminating TimescaleDB background worker launcher due to administrator command
2021-08-21 13:01:55.856 UTC [42] FATAL:  terminating connection due to administrator command
2021-08-21 13:01:55.856 UTC [52] FATAL:  terminating connection due to administrator command
2021-08-21 13:01:55.856 UTC [52] LOG:  terminating TimescaleDB job scheduler due to administrator command
2021-08-21 13:01:55.856 UTC [35] LOG:  background worker "TimescaleDB Background Worker Launcher" (PID 42) exited with exit code 1
2021-08-21 13:01:55.856 UTC [35] LOG:  background worker "logical replication launcher" (PID 43) exited with exit code 1
2021-08-21 13:01:55.857 UTC [35] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 52) exited with exit code 1
2021-08-21 13:01:55.857 UTC [37] LOG:  shutting down
2021-08-21 13:01:55.918 UTC [35] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2021-08-21 13:01:55.960 UTC [1] LOG:  starting PostgreSQL 12.7 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, 64-bit
2021-08-21 13:01:55.960 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-08-21 13:01:55.960 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2021-08-21 13:01:55.962 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-21 13:01:56.005 UTC [77] LOG:  database system was shut down at 2021-08-21 13:01:55 UTC
2021-08-21 13:01:56.008 UTC [1] LOG:  database system is ready to accept connections
2021-08-21 13:01:56.009 UTC [83] LOG:  TimescaleDB background worker launcher connected to shared catalogs
2021-08-21 13:02:51.897 UTC [1] LOG:  received fast shutdown request
2021-08-21 13:02:51.898 UTC [1] LOG:  aborting any active transactions
2021-08-21 13:02:51.898 UTC [85] LOG:  terminating TimescaleDB job scheduler due to administrator command
2021-08-21 13:02:51.898 UTC [86] FATAL:  terminating connection due to administrator command
2021-08-21 13:02:51.898 UTC [83] LOG:  terminating TimescaleDB background worker launcher due to administrator command
2021-08-21 13:02:51.898 UTC [85] FATAL:  terminating connection due to administrator command
2021-08-21 13:02:51.898 UTC [83] FATAL:  terminating connection due to administrator command
2021-08-21 13:02:51.899 UTC [1] LOG:  background worker "TimescaleDB Background Worker Launcher" (PID 83) exited with exit code 1
2021-08-21 13:02:51.899 UTC [1] LOG:  background worker "logical replication launcher" (PID 84) exited with exit code 1
2021-08-21 13:02:51.899 UTC [85] LOG:  terminating TimescaleDB job scheduler due to administrator command
2021-08-21 13:02:51.900 UTC [1] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 85) exited with exit code 1
2021-08-21 13:02:51.901 UTC [78] LOG:  shutting down
2021-08-21 13:02:51.934 UTC [1] LOG:  database system is shut down

PostgreSQL Database directory appears to contain a database; Skipping initialization

2021-08-21 13:16:18.644 UTC [1] LOG:  starting PostgreSQL 12.7 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, 64-bit
2021-08-21 13:16:18.644 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-08-21 13:16:18.644 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2021-08-21 13:16:18.645 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-21 13:16:18.685 UTC [22] LOG:  database system was shut down at 2021-08-21 13:02:51 UTC
2021-08-21 13:16:18.687 UTC [1] LOG:  database system is ready to accept connections
2021-08-21 13:16:18.688 UTC [28] LOG:  TimescaleDB background worker launcher connected to shared catalogs

Let me know if I have to change my approach. Please advise on my next steps.

@cla-bot cla-bot bot added the cla-signed label Aug 21, 2021
@faysal-ishtiaq
Copy link
Collaborator Author

To recreate everything, please run docker-compose up -d --build from project root and check the logs

@isedwards
Copy link
Member

Thank you @faysal-ishtiaq - I'm really excited about this.

  • Mostly we will only run and test against one database image at a time. Could you create a separate docker-compose.yml for each system? (Although, if it's possible, it would be useful to also have a global docker-compose.yml that brings everything up together by including imports of the separate compose files?)
  • How did you know that CliDE requires the extra CREATE USER commands? Were errors reported when you tried to restore the clide DDL file?
  • MIDAS seems to be working. To get things started, I created a very simple DDL script for MIDAS consisting of a single table: midas_code_ddl.sql. The error seems to indicate that it is trying to run the script twice and create the same table twice. Is there a reason why this might be happening and could it also be happening in the Dockerfiles for the other CDMSs? (The other DDL files probably use CREATE IF NOT EXISTS table_name - so we wouldn't notice the error if the script was ran twice).
  • Not sure whether WMDR is working as expected - but let's not worry about it for now
  • The MySQL 5.1 docker image is created and documented in opencdms/mysql-5.1.73. Feel free to modify the Dockerfile there if necessary and we can push a new version to Docker Hub.

In case it's useful... I edited your comment above to collapse the logs using the <details> HTML tag, e.g.

<details>
  <summary>MIDAS Logs</summary>

    ```
    markdown code here (doesn't need to be indented)
    ```

</details>

This renders the logs section collapsed:

MIDAS Logs
Starting Oracle Net Listener.
Starting Oracle Database 11g Express Edition instance.


System altered.

/usr/sbin/startup.sh: running /docker-entrypoint-initdb.d/1.sql

SQL*Plus: Release 11.2.0.2.0 Production on Sat Aug 21 13:02:03 2021

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production


Table created.

SQL> Disconnected from Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production


Starting Oracle Net Listener.
Starting Oracle Database 11g Express Edition instance.


System altered.

/usr/sbin/startup.sh: running /docker-entrypoint-initdb.d/1.sql

SQL*Plus: Release 11.2.0.2.0 Production on Sat Aug 21 13:16:25 2021

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production

CREATE TABLE address
             *
ERROR at line 1:
ORA-00955: name is already used by an existing object


SQL> Disconnected from Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production

@faysal-ishtiaq
Copy link
Collaborator Author

faysal-ishtiaq commented Aug 21, 2021

@isedwards I saw your comment on the PR. I will sit with it tomorrow again and try to investigate on it. However,

  • we can run something like docker-compose up clide to start only clide service or docker-compose up midas to start midas only. Do we still need separate docker-compose file? I am happy to create one, I am just pointing out our options.
  • I actually saw some logs with error message while executing the sql file. So, I created them manually.
  • To my understanding, the initialization scripts are run only for fresh container. But it might be so that, if it's failed, it tries again on next run. I looked into the clide logs, the script is run only for the first time. For second time it logged the following line
    clide_db | PostgreSQL Database directory appears to contain a database; Skipping initialization . I am attaching all the logs for clide for your refrence. But I will look into it again tomorrow.
MIDAS Logs
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2021-08-21 13:01:55.106 UTC [51] LOG:  starting PostgreSQL 13.4 (Debian 13.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-08-21 13:01:55.107 UTC [51] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-21 13:01:55.110 UTC [52] LOG:  database system was shut down at 2021-08-21 13:01:54 UTC
2021-08-21 13:01:55.113 UTC [51] LOG:  database system is ready to accept connections
 done
server started

/usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/1.sh
Creating database role: clideadmin
CREATE ROLE
Creating database role: clidegui
CREATE ROLE
Creating database role: clide
CREATE ROLE
Creating database role: datacomp
CREATE ROLE
Creating database: clideDB
CREATE DATABASE

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/2.sql
SET
SET
SET
SET
SET
SET
2021-08-21 13:01:55.524 UTC [122] WARNING:  database "clideDB" does not exist
psql:/docker-entrypoint-initdb.d/2.sql:16: WARNING:  database "clideDB" does not exist
COMMENT
CREATE EXTENSION
ALTER LANGUAGE
SET
CREATE TYPE
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE TYPE
ALTER TYPE
COMMENT
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE DOMAIN
ALTER DOMAIN
CREATE TYPE
ALTER TYPE
CREATE TYPE
ALTER TYPE
CREATE TYPE
ALTER TYPE
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
COMMENT
CREATE FUNCTION
ALTER FUNCTION
COMMENT
CREATE FUNCTION
ALTER FUNCTION
COMMENT
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
COMMENT
CREATE FUNCTION
ALTER FUNCTION
COMMENT
CREATE FUNCTION
ALTER FUNCTION
COMMENT
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
COMMENT
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
COMMENT
CREATE FUNCTION
ALTER FUNCTION
COMMENT
CREATE FUNCTION
ALTER FUNCTION
COMMENT
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE FUNCTION
ALTER FUNCTION
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR
ALTER OPERATOR
CREATE OPERATOR CLASS
ALTER OPERATOR CLASS
CREATE OPERATOR CLASS
ALTER OPERATOR CLASS
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
SET
SET
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE VIEW
ALTER TABLE
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE VIEW
ALTER TABLE
CREATE VIEW
ALTER TABLE
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
CREATE VIEW
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
CREATE SEQUENCE
ALTER TABLE
COMMENT
CREATE TABLE
ALTER TABLE
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
COMMENT
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE TRIGGER
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
REVOKE
REVOKE
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT
REVOKE
REVOKE
GRANT
GRANT


waiting for server to shut down...2021-08-21 13:01:56.709 UTC [51] LOG:  received fast shutdown request
.2021-08-21 13:01:56.710 UTC [51] LOG:  aborting any active transactions
2021-08-21 13:01:56.711 UTC [51] LOG:  background worker "logical replication launcher" (PID 58) exited with exit code 1
2021-08-21 13:01:56.711 UTC [53] LOG:  shutting down
2021-08-21 13:01:56.748 UTC [51] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2021-08-21 13:01:56.837 UTC [1] LOG:  starting PostgreSQL 13.4 (Debian 13.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-08-21 13:01:56.837 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-08-21 13:01:56.837 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2021-08-21 13:01:56.839 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-21 13:01:56.842 UTC [125] LOG:  database system was shut down at 2021-08-21 13:01:56 UTC
2021-08-21 13:01:56.845 UTC [1] LOG:  database system is ready to accept connections
2021-08-21 13:02:51.913 UTC [1] LOG:  received fast shutdown request
2021-08-21 13:02:51.914 UTC [1] LOG:  aborting any active transactions
2021-08-21 13:02:51.915 UTC [1] LOG:  background worker "logical replication launcher" (PID 131) exited with exit code 1
2021-08-21 13:02:51.917 UTC [126] LOG:  shutting down
2021-08-21 13:02:51.930 UTC [1] LOG:  database system is shut down

PostgreSQL Database directory appears to contain a database; Skipping initialization

2021-08-21 13:16:18.332 UTC [1] LOG:  starting PostgreSQL 13.4 (Debian 13.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-08-21 13:16:18.332 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-08-21 13:16:18.332 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2021-08-21 13:16:18.334 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-21 13:16:18.339 UTC [27] LOG:  database system was shut down at 2021-08-21 13:02:51 UTC
2021-08-21 13:16:18.343 UTC [1] LOG:  database system is ready to accept connections
2021-08-21 17:28:52.273 UTC [1] LOG:  received fast shutdown request
2021-08-21 17:28:52.275 UTC [1] LOG:  aborting any active transactions
2021-08-21 17:28:52.276 UTC [1] LOG:  background worker "logical replication launcher" (PID 33) exited with exit code 1
2021-08-21 17:28:52.276 UTC [28] LOG:  shutting down
2021-08-21 17:28:52.283 UTC [1] LOG:  database system is shut down

PostgreSQL Database directory appears to contain a database; Skipping initialization

2021-08-21 17:29:07.598 UTC [1] LOG:  starting PostgreSQL 13.4 (Debian 13.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-08-21 17:29:07.598 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-08-21 17:29:07.598 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2021-08-21 17:29:07.600 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-21 17:29:07.603 UTC [26] LOG:  database system was shut down at 2021-08-21 17:28:52 UTC
2021-08-21 17:29:07.607 UTC [1] LOG:  database system is ready to accept connections
  • I will look into the custom mysql image again and make changes if required.

@faysal-ishtiaq
Copy link
Collaborator Author

@isedwards I just looked into the issues again. And here are my observations.

mysql -P 3307 --protocol tcp -u root -p
Enter password: 
ERROR 1043 (08S01): Bad handshake

Please let me know if I should split the docker-compose file into multiple files. However, I could not find any reference on importing a docker-compose file into another. But we can do something like this, docker-compsoe -f file1.yml -f file2.yml up . If you want my opinion, it's much cleaner and manageable to maintain a single docker-compose file and fire up the containers separately if needed like docker-compose up <service_name>

@isedwards
Copy link
Member

For the error below, the most likely cause of the problem is that you're using a more recent version of the MySQL CLI which is attempting to connect using a different method of authentication.

mysql -P 3307 --protocol tcp -u root -p
Enter password: 
ERROR 1043 (08S01): Bad handshake

You can see the version you have installed locally using mysql -V. I imagine there is probably a way to set the authentication method so that it will work with older instances of MySQL. We may need to work out how to do this from Python / SQLAlchemy but, for now, I'm happy for the MySQL 5.1 image to be considered "experimental" (we may not need to use it).

@isedwards
Copy link
Member

@faysal-ishtiaq would you be happy to make the following changes:

  1. Could you rename the /containers/ directory to /docker/?
  2. I've pushed the oracleinanutshell/oracle-xe-11g image to opencdms/oracle-xe-11g in case oracleinanutshell ever disappears (since I have trouble building the image myself). Could you change to using the opencdms docker hub link instead?
  3. Let's keep a single docker compose file with the modifications to the services shown below (if hyphens aren't permitted in names then could you substitute underscores where necessary?). Could you also add the suggested comments in the docker-compose.yml file above each service name if YAML will allow the comments?

Default

The default service, called opencdms-db, is currently just an empty TimescaleDB instance...

opencdms-db

(this service uses the default Dockerfile in /docker/Dockerfile)

# An empty TimescaleDB / PostGIS instance for deploying schemas from multiple supported CDMSs (currently empty)
services:
    opencdms-db:
        dockerfile: ./docker/Dockerfile
        container_name: opencdms-db
FROM timescale/timescaledb-postgis:latest-pg12

Groups

Database groups contain schemas for several CDMSs that use a comparable database technology.

postgresql

# A PostgreSQL instance containing schemas from CDMSs that use PostgreSQL (currently clide and wmdr)
services:
    postgresql:
        dockerfile: ./docker/groups/Dockerfile.postgresql
        container_name: postgresql
FROM postgres:13
COPY ./scripts/entrypoints/clide.sh /docker-entrypoint-initdb.d/1.sh
COPY ./schemas/clide/clidedb_schema.sql /docker-entrypoint-initdb.d/2.sql
COPY ./schemas/wmdr/wmdr_ddl.sql /docker-entrypoint-initdb.d/1.sql /docker-entrypoint-initdb.d/3.sql

mariadb

(for MCH to run on MariaDB 10.1 instead of MySQL 5.1 you'll need to use your extra sql_mode commands here)

# A MariaDB instance containing schemas from CDMSs that use MySQL/MariaDB (currently climsoft-4.1.1 and mch-english)
services:
    mariadb:
        dockerfile: ./docker/groups/Dockerfile.mariadb
        container_name: mariadb
FROM mariadb:10.1
COPY ./schemas/climsoft/v4/v4.1.1/climsoft_v4_all.sql /docker-entrypoint-initdb.d/1.sql
COPY ./schemas/mch/Create_MCH_English_basic_tables.sql /docker-entrypoint-initdb.d/2.sql

oracle-xe

# An Oracle Express Edition instance containing schemas from CDMSs that use Oracle (currently midas-core)
services:
    oracle-xe:
        dockerfile: ./docker/groups/Dockerfile.oracle-xe
        container_name: oracle-xe
FROM opencdms/oracle-xe-11g
COPY ./schemas/midas/midas_core_ddl.sql /docker-entrypoint-initdb.d/1.sql

Production

Production images provide a single CDMS's schema running on the database type and version that it runs on in production (midas is not available here since it runs on Oracle Enterprise or Oracle Standard in production rather than the free version of Oracle)

clide

# A PostgreSQL 13 instance containing the clide schema (comparible to how CliDE is used in production)
services:
    clide:
        dockerfile: ./docker/groups/Dockerfile.clide
        container_name: clide
FROM postgres:13
COPY ./scripts/entrypoints/clide.sh /docker-entrypoint-initdb.d/1.sh
COPY ./schemas/clide/clidedb_schema.sql /docker-entrypoint-initdb.d/2.sql

climsoft-4.1.1

# A MariaDB 10.1 instance containing the climsoft-4.1.1 schema (comparable to how Climsoft is used in production)
services:
    climsoft-4.1.1:
        dockerfile: ./docker/groups/Dockerfile..climsoft-4.1.1
        container_name: climsoft-4.1.1
FROM mariadb:10.1
COPY ./schemas/climsoft/v4/v4.1.1/climsoft_v4_all.sql /docker-entrypoint-initdb.d/1.sql

mch-english

MySQL 5.1.73 image is currently experimental

# MySQL 5.1.17 instance containing the mch-english schema (comparible to how MCH is used in production)
services:
    mch-english:
        dockerfile: ./docker/groups/Dockerfile.mch-english
        container_name: mch-english
FROM opencdms/mysql:5.1.73
COPY ./schemas/mch/Create_MCH_English_basic_tables.sql /docker-entrypoint-initdb.d/1.sql

wmdr

# A PostgreSQL 13 instance containing the wmdr schema
services:
    wmdr:
        dockerfile: ./docker/groups/Dockerfile.wmdr
        container_name: wmdr
FROM postgres:13
COPY ./schemas/wmdr/wmdr_ddl.sql /docker-entrypoint-initdb.d/1.sql

@isedwards isedwards marked this pull request as draft August 28, 2021 14:56
@isedwards
Copy link
Member

isedwards commented Aug 31, 2021

@faysal-ishtiaq I'm intending that these containers will be used for testing, therefore it's possible that developers will already have local database(s) running on the common port numbers. To avoid issues, let's prefix our port numbers with 1,2 or 3 depening on which category the image belongs to below.

Also:

  • Let's rename mariadb as mysql and rename oracle-xe as oracle to reflect the overall group rather than a specific implementation
  • Let's add midas to production using oracle-xe even though this isn't the same version that is used in production (it is the closest match and in the future we may want to run a container with just midas rather than having to run the oracle group with multiple schemas that all use oracle).
  • I've used port 3308 for Climsoft since this is often used in production (and prefixed it with as category 3)

Test images categories

1. Default

  • opencdms-db 15432

2. Groups

  • postgresql 25432
  • mysql 23306
  • oracle 21521

3. Production

  • clide 35432
  • climsoft-4.1.1 33308
  • mch-english 33306
  • midas 31521
  • wmdr 35433

@faysal-ishtiaq
Copy link
Collaborator Author

@isedwards

I have updated the PR with the port mappings.

Also there was an issue with mysql service. Climsoft ddl was being executed successfully. But during the execution of mch-english ddl, it was throwing No database selected error. So, I took following approach.

  • created a file in scripts/entrypoints/mch-english.sql with following content
CREATE DATABASE IF NOT EXISTS mch_english;
USE mch_english;
SET GLOBAL sql_mode = 'ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION';
  • Then in docker/groups/Dockerfile.mysql I have copied this new file and this file schemas/mch/Create_MCH_English_basic_tables.sql and then saved the output of cat for those files into /docker-entrypoint-initdb.d/2.sql so that we don't have to change this file ./schemas/mch/Create_MCH_English_basic_tables.sql. Here is the dockerfile for your reference:
FROM mariadb:10.1
COPY ./schemas/climsoft/v4/v4.1.1/climsoft_v4_all.sql /docker-entrypoint-initdb.d/1.sql
COPY ./scripts/entrypoints/mch-english.sql /tmp/settings.sql
COPY ./schemas/mch/Create_MCH_English_basic_tables.sql /tmp/ddl.sql
RUN cat /tmp/settings.sql /tmp/ddl.sql > /docker-entrypoint-initdb.d/2.sql

Please let me know if it's okay or there is a better way to do this.

@isedwards isedwards marked this pull request as ready for review September 5, 2021 08:17
Copy link
Member

@isedwards isedwards left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@faysal-ishtiaq - I like the solution of automating the cat command within the dockerfile.

The only alternative that springs to mind is MySQL's --init-command command line argument which would allow the three statements to be executed prior to restoring the SQL. But it's probably not relevant here when automating with containers.

@isedwards isedwards merged commit 6bf30e0 into opencdms:main Sep 5, 2021
isedwards pushed a commit that referenced this pull request Sep 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants