Error connecting clues to mysql

[tkald]

Can't start clues with mysql database. Seems like string parsing is broken in cpyutils or wrong sample format is provided in configuration file.

Connection string in clues2.cfg file:
DB_CONNECTION_STRING=mysql://clues:clpAss@cluesdb/clues

In log I get errors:
...

WARNING: Couldn't write lextab module 'lextab'. [Errno 20] Not a directory: '/usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/lextab.py'
WARNING: Couldn't create 'parsetab'. [Errno 20] Not a directory: '/usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/parsetab.py'
...

Traceback (most recent call last):
File "/usr/bin/cluesserver", line 569, in
main_loop()
File "/usr/bin/cluesserver", line 560, in main_loop
CLUES_DAEMON = CluesDaemon(PLATFORM, active_schedulers)
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 185, in init
self._db_system = DBSystem.create_from_connection_string()
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 57, in create_from_connection_string
return DBSystem(connection_string)
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 61, in init
self._db = cpyutils.db.DB.create_from_string(connection_string)
File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 80, in create_from_string
NameError: global name 'db' is not defined

If I use ip address for host in connection string I get last 2 lines of same error:

File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 95, in create_from_string
cpyutils.db.MalformedConnectionString: Connection string format not recognised

Latest master branches of clues and cpyutils on Ubuntu 16.04

[micafer]

Hi @tkald,
It is a bug of the cpyutils package (grycap/cpyutils#3).

[micafer]

Hi @tkald,
We have commited a fix to the cpyutils error.
Coul you try to download the last version and test it again?

[tkald]

Something's still wrong.

If I use connection string with ip address, I get

File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 96, in create_from_string
cpyutils.db.MalformedConnectionString: ('Connection string format not recognised')

connection_parameters returns None
Regex issue?

If I use DB_CONNECTION_STRING=mysql://clues:clpAss@cluesdb/clues I get:

Traceback (most recent call last):
File "/usr/bin/cluesserver", line 569, in
main_loop()
File "/usr/bin/cluesserver", line 560, in main_loop
CLUES_DAEMON = CluesDaemon(PLATFORM, active_schedulers)
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 185, in init
self._db_system = DBSystem.create_from_connection_string()
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 57, in create_from_connection_string
return DBSystem(connection_string)
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 62, in init
self._create_db()
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 78, in _create_db
result1, _, _ = self._db.sql_query("CREATE TABLE IF NOT EXISTS hostdata(name varchar(128) PRIMARY KEY, enabled bool)", True)
File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 190, in sql_query
AttributeError: DB_mysql instance has no attribute '_conn_string'

[micafer]

Hi @tkald,

As you said the regex was not prepaired for IP strings.
Also there was a bug in the DB connection.
I have uploaded a fix in cpyutils.

[tkald]

File "/usr/bin/cluesserver", line 569, in
main_loop()
File "/usr/bin/cluesserver", line 560, in main_loop
CLUES_DAEMON = CluesDaemon(PLATFORM, active_schedulers)
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 185, in init
self._db_system = DBSystem.create_from_connection_string()
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 57, in create_from_connection_string
return DBSystem(connection_string)
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 62, in init
self._create_db()
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 78, in _create_db
result1, _, _ = self._db.sql_query("CREATE TABLE IF NOT EXISTS hostdata(name varchar(128) PRIMARY KEY, enabled bool)", True)
File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 190, in sql_query
AttributeError: 'NoneType' object has no attribute 'cursor'

[micafer]

Hi @tkald,

Sorry for this failure it was may fault in one line.
Now it should work.

[tkald]

Yap, now I have 2 tables in mysql db. No data though in those tables. Nothing in log either.
Maybe misconfiguration in my conf.

/usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/db.py:178: Warning: Table 'hostdata' already exists
/usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/db.py:178: Warning: Table 'host_monitoring' already exists
[SERVER];DEBUG;2016-10-31 13:17:27,110;1477912647.111;CLUES server starting
[CLUES];DEBUG;2016-10-31 13:17:27,111;1477912647.111;monitoring jobs and nodes at the same time
[ONE];DEBUG;2016-10-31 13:17:27,119;OpenNebula version: 5.2.0
[CLUES];DEBUG;2016-10-31 13:17:27,142;1477912647.142;
First monitorization of LRMS:
List of nodes:
[NODE "r410-1"] state: off (since 1477912647), 3600/3600 (slots) - ID: 001 @1477912647
[NODE "vml-vm1"] state: used (since 1477912647), 0/3600 (slots) - ID: 002 @1477912647
[NODE "vml-vm2"] state: off (since 1477912647), 3600/3600 (slots) - ID: 003 @1477912647
[CLUES];DEBUG;2016-10-31 13:17:27,915;1477912647.916;first monitorisation of jobs in LRMS

[micafer]

Hi @tkald,

No information is stored in the DB untill there is any change in the information: A node state changes, a node is disabled, etc.
If the tables has been created correctly the DB access is correctly configured and working.
Try to disable a node and you will see as some information is stored in de DB.

[tkald]

Yes, got some data after changing host status.
Data type mismatches though:
/usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/db.py:178: Warning: Field 'x' doesn't have a default value
/usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/db.py:178: Warning: Incorrect integer value: 'True' for column 'enabled' at row 1

[tkald]

More related errors:

[CLUES];DEBUG;2016-10-31 15:20:27,135;1477920027.136;node vml-vm2 changed its state
Traceback (most recent call last):
File "/usr/bin/cluesserver", line 569, in
main_loop()
File "/usr/bin/cluesserver", line 566, in main_loop
CLUES_DAEMON.loop(options.RT_MODE)
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 557, in loop
cpyutils.eventloop.get_eventloop().loop()
File "build/bdist.linux-x86_64/egg/cpyutils/eventloop.py", line 299, in loop
File "build/bdist.linux-x86_64/egg/cpyutils/eventloop.py", line 138, in call
File "build/bdist.linux-x86_64/egg/cpyutils/eventloop.py", line 101, in call
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 351, in _monitor_lrms_nodes_and_jobs
self._monitor_lrms_nodes()
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 299, in _monitor_lrms_nodes
self._db_system.store_node_info(self._lrms_nodelist[n_id])
File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 106, in store_node_info
, True)
File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 190, in sql_query
_mysql_exceptions.IntegrityError: (1062, "Duplicate entry '0' for key 'PRIMARY'")
cluesd.service: Main process exited, code=exited, status=1/FAILURE

[tkald]

@micafer

No information is stored in the DB untill there is any change in the information: A node state changes, a node is disabled, etc.

I suppose monitoring host's free slots should change data.
Booted some extra vm-s, no change in mysql table.
I think this is unrelated to mysql though. Shall we open new issue?