MILESTONE: 3.0.0 rc

[dylanjacob]

This issue is meant to be a "master" issue to discuss the remaining items to be resolved in order to bring us up to speed with bearded avenger release candidate.

I believe, at this point we are able to achieve a successful Ansible build (or will be able to once csirtgadgets.cif 0.0.9 is available from Ansible Galaxy), however, there are some glaring issues that make the resulting container unusable.

The main issue seems to revolve around the .cif.yml file and the token contained within. When running any command, such as a query or ping, an error is thrown saying "Unauthorized".

Lets start here and see where we end up, shall we?

[ventz]

@dylanjacob It looks like Wes added 0.0.9 already:

https://github.com/csirtgadgets/bearded-avenger-deploymentkit/blob/de201793f7e009426c1556308205d9dcfde88154/ansible.sh#L14

I tried building a new version with that, and it's running into the same issue - about "Unauthorized". It looked like the cif.yml was missing, and the router config yml was missing too.

I wonder if somewhere along the lines there is an over-reaching "DOCKER_BUILD" flag which is skipping something that it should not be.

[dylanjacob]

@ventz I'm not seeing 0.0.9 available from Ansible Galaxy yet, it's still at 0.0.8. I've temporarily added a line in Dockerfile to manually checkout csirtgadgets/cif-ansible-role master in the roles directory (which is detected and skips installation from galaxy) in order to bypass this.

Regardless, it sill gives me that Unauthorized error. I wonder the same regarding the extraneous flag...but I'm not sure as to where it would be.

One clue that I'll share here is that I believe there should be an environment variable that contains the token, but there is not. According to the deployment kit wiki (https://github.com/csirtgadgets/bearded-avenger-deploymentkit/wiki/FAQ#unauthorized) doing env | grep CIF should yield something with the token in it. I have yet to do a non-Docker build in order to compare this, but my bet is that it's related to a skipped Ansible build step.

[dylanjacob]

so, the environment variable thing may not be much of an issue after all. I spun up an ubuntu vm and did a non docker build for comparison and I realized that a) I was not becoming the cif user properly, i.e. sudo su cif rather than sudo su - cif which means that the environment variables are not set appropriately. Doing the latter in the Docker build yields the same results as the non docker build, which brings us to b) the CIF_TOKEN variable is empty in both instances. Setting this to the token does not seem to make a difference in either case.

Moving on, I did find that csirtg-smrt is logging to /var/log/cif/csirtg-smrt.log and there are some issues there that may lead us to the root issue. See below:

Process Process-25:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/smrt.py", line 309, in _run_smrt
    archiver = Archiver(dbfile=args.remember_path)
  File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/archiver.py", line 86, in __init__
    Base.metadata.create_all(self.engine)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 3817, in create_all
    tables=tables)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1855, in _run_visitor
    with self._optional_conn_ctx_manager(connection) as conn:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1848, in _optional_conn_ctx_manager
    with self.contextual_connect() as conn:
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
    e, dialect, self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1151, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
    self.connection = self.__connect()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
    connection = self.__pool._invoke_creator(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
    return self.dbapi.connect(*cargs, **cparams)
OperationalError: (sqlite3.OperationalError) unable to open database file

First thought is permissions, but the /var/lib/cif/cif.db file is owned by cif:cif and is chmod 660 which is the same as cif.db on my non docker build. From the log, it looks like more clues might be in csirtg_smrt/smrt.py and csirtg_smrt/archiver.py

[ventz]

@dylanjacob Thanks for investigating! Let me ping Wes and link him here too.