You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
NOTE: THIS IS A PREVIEW TECHNOLOGY, FEATURES AND CONFIGURATION SETTINGS CAN CHANGE IN FUTURE RELEASES.
17
16
18
-
This repository only holds plugin artefacts. Source code is available at this [link](https://github.com/nextflow-io/nextflow/tree/master/plugins/nf-sqldb).
17
+
## Getting started
19
18
20
-
## Get started
21
-
22
-
Make sure to have Nextflow `22.08.1-edge` or later. Add the following snippet to your `nextflow.config` file.
19
+
This plugin requires Nextflow `22.08.1-edge` or later. You can enable the plugin by adding the following snippet to your `nextflow.config` file:
23
20
24
-
```
21
+
```groovy
25
22
plugins {
26
-
id 'nf-sqldb@0.5.0'
23
+
id 'nf-sqldb'
27
24
}
28
25
```
29
26
30
-
The above declaration allows the use of the SQL plugin functionalities in your Nextflow pipelines.
31
-
See the section below to configure the connection properties with a database instance.
32
-
33
27
34
28
## Configuration
35
29
36
-
The target database connection coordinates are specified in the `nextflow.config` file using the
37
-
`sql.db` scope. The following are available
30
+
You can configure any number of databases under the `sql.db` configuration scope. For example:
38
31
39
-
| Config option | Description |
40
-
|--- |--- |
41
-
| `sql.db.'<DB-NAME>'.url` | The database connection URL based on Java [JDBC standard](https://docs.oracle.com/javase/tutorial/jdbc/basics/connecting.html#db_connection_url).
42
-
| `sql.db.'<DB-NAME>'.driver` | The database driver class name (optional).
43
-
| `sql.db.'<DB-NAME>'.user` | The database connection user name.
44
-
| `sql.db.'<DB-NAME>'.password` | The database connection password.
45
-
46
-
For example:
47
-
48
-
```
32
+
```groovy
49
33
sql {
50
34
db {
51
35
foo {
52
-
url = 'jdbc:mysql://localhost:3306/demo'
53
-
user = 'my-user'
54
-
password = 'my-password'
55
-
}
36
+
url = 'jdbc:mysql://localhost:3306/demo'
37
+
user = 'my-user'
38
+
password = 'my-password'
39
+
}
56
40
}
57
41
}
58
-
59
42
```
60
43
61
-
The above snippet defines SQL DB named *foo* that connects to a MySQL server running locally at port 3306 and
62
-
using `demo` schema, with `my-name` and `my-password` as credentials.
44
+
The above example defines a database named `foo` that connects to a MySQL server running locally at port 3306 and
45
+
using the `demo` schema, with `my-name` and `my-password` as credentials.
63
46
64
-
## Available operations
47
+
The following options are available:
65
48
66
-
This plugin adds to the Nextflow DSL the following extensions that allows performing of queries and populating database tables.
49
+
`sql.db.'<DB-NAME>'.url`
50
+
: The database connection URL based on the [JDBC standard](https://docs.oracle.com/javase/tutorial/jdbc/basics/connecting.html#db_connection_url).
51
+
52
+
`sql.db.'<DB-NAME>'.driver`
53
+
: The database driver class name (optional).
54
+
55
+
`sql.db.'<DB-NAME>'.user`
56
+
: The database connection user name.
57
+
58
+
`sql.db.'<DB-NAME>'.password`
59
+
: The database connection password.
60
+
61
+
## Dataflow Operators
62
+
63
+
This plugin provides the following dataflow operators for querying from and inserting into database tables.
67
64
68
65
### fromQuery
69
66
70
-
The `fromQuery` factory method allows for performing a query against a SQL database and creating a Nextflow channel emitting
71
-
a tuple for each row in the corresponding result set. For example:
67
+
The `fromQuery` factory method queries a SQL database and creates a channel that emits a tuple for each row in the corresponding result set. For example:
72
68
73
-
```
69
+
```nextflow
74
70
include { fromQuery } from 'plugin/nf-sqldb'
75
71
76
-
ch = channel.fromQuery('select alpha, delta, omega from SAMPLE', db: 'foo')
72
+
channel.fromQuery('select alpha, delta, omega from SAMPLE', db: 'foo').view()
77
73
```
78
74
79
75
The following options are available:
80
76
81
-
| Operator option | Description |
82
-
|--- |--- |
83
-
| `db` | The database handle. It must must a `sql.db` name defined in the `nextflow.config` file.
84
-
| `batchSize` | Performs the query in batches of the specified size. This is useful to avoid loading the complete resultset in memory for query returning a large number of entries. NOTE: this feature requires that the underlying SQL database to support `LIMIT` and `OFFSET` capability.
85
-
| `emitColumns` | When `true` the column names in the select statement are emitted as first tuple in the resulting channel.
77
+
`db`
78
+
: The database handle. It must be defined under `sql.db` in the Nextflow configuration.
79
+
80
+
`batchSize`
81
+
: Query the data in batches of the given size. This option is recommended for queries that may return large a large result set, so that the entire result set is not loaded into memory at once.
82
+
: *NOTE:* this feature requires that the underlying SQL database supports `LIMIT` and `OFFSET`.
83
+
84
+
`emitColumns`
85
+
: When `true`, the column names in the `SELECT` statement are emitted as the first tuple in the resulting channel.
86
86
87
87
### sqlInsert
88
88
89
-
The `sqlInsert` operator provided by this plugin allows populating a database table with the data emitted
90
-
by a Nextflow channels and therefore produced as result by a pipeline process or an upstream operator. For example:
89
+
The `sqlInsert` operator collects the items in a source channel and inserts them into a SQL database. For example:
The above example creates and performs the following two SQL statements into the database with name `foo` as defined
103
-
in the `nextflow.config` file.
100
+
The above example executes the following SQL statements into the database `foo` (as defined in the Nextflow configuration).
104
101
105
-
```
102
+
```sql
106
103
INSERT INTO SAMPLE (NAME, LEN) VALUES ('HELLO', 5);
107
104
INSERT INTO SAMPLE (NAME, LEN) VALUES ('WORLD!', 6);
108
105
```
109
106
110
-
NOTE: the target table (e.g. `SAMPLE` in the above example) must be created ahead.
107
+
*NOTE:* the target table (e.g. `SAMPLE` in the above example) must be created beforehand.
111
108
112
109
The following options are available:
113
110
114
-
| Operator option | Description |
115
-
|-------------------|--- |
116
-
| `db` | The database handle. It must must a `sql.db` name defined in the `nextflow.config` file.
117
-
| `into` | The database table name into with the data needs to be stored.
118
-
| `columns` | The database table column names to be filled with the channel data. The column names order and cardinality must match the tuple values emitted by the channel. The columns can be specified as a `List` object or a comma-separated value string.
119
-
| `statement` | The SQL `insert` statement to be performed to insert values in the database using `?` as placeholder for the actual values, for example: `insert into SAMPLE(X,Y) values (?,?)`. When provided the `into` and `columsn` parameters are ignored.
120
-
| `batchSize` | The number of insert statements that are grouped together before performing the SQL operations (default: `10`).
121
-
| `setup` | A SQL statement that's executed before the first insert operation. This is useful to create the target DB table. NOTE: the underlying DB should support the *create table if not exist* idiom (i.e. the plugin will execute this statement every time the script is run).
111
+
`db`
112
+
: The database handle. It must be defined under `sql.db` in the Nextflow configuration.
122
113
123
-
## Query CSV files
114
+
`into`
115
+
: The target table for inserting the data.
124
116
125
-
The SQL plugin includes the [H2](https://www.h2database.com/html/main.html) database engine that allows the query of CSV files
126
-
as DB tables using SQL statements.
117
+
`columns`
118
+
: The database table column names to be filled with the channel data. The column names order and cardinality must match the tuple values emitted by the channel. The columns can be specified as a list or as a string of comma-separated values.
127
119
128
-
For example, create CSV file using the snippet below:
120
+
`statement`
121
+
: The SQL `INSERT` statement to execute, using `?` as a placeholder for the actual values, for example: `insert into SAMPLE(X,Y) values (?,?)`. The `into` and `columns` options are ignored when this option is provided.
129
122
130
-
```
123
+
`batchSize`
124
+
: Insert the data in batches of the given size (default: `10`).
125
+
126
+
`setup`
127
+
: A SQL statement that is executed before inserting the data, e.g. to create the target table.
128
+
: *NOTE:* the underlying database should support the *create table if not exist* idiom, as the plugin will execute this statement every time the script is run.
129
+
130
+
## Querying CSV files
131
+
132
+
This plugin supports the [H2](https://www.h2database.com/html/main.html) database engine, which can query CSV files like database tables using SQL statements.
133
+
134
+
For example, create a CSV file using the snippet below:
135
+
136
+
```bash
131
137
cat <<EOF > test.csv
132
138
foo,bar
133
139
1,hello
@@ -137,27 +143,31 @@ foo,bar
137
143
EOF
138
144
```
139
145
140
-
To query this file in a Nextflow script use the following snippet:
146
+
Then query it in a Nextflow script:
141
147
142
148
```nextflow
143
-
channel
144
-
.sql
145
-
.fromQuery("SELECT * FROM CSVREAD('test.csv') where foo>=2;")
146
-
.view()
149
+
include { fromQuery } from 'plugin/nf-sqldb'
150
+
151
+
channel
152
+
.fromQuery("SELECT * FROM CSVREAD('test.csv') where foo>=2;")
153
+
.view()
147
154
```
148
155
156
+
The `CSVREAD` function provided by the H2 database engine allows you to query any CSV file in your filesystem. As shown in the example, you can use standard SQL clauses like `SELECT` and `WHERE` to define your query.
157
+
158
+
## Caveats
149
159
150
-
The `CSVREAD` function provided by the H2 database engine allows the access of a CSV file in your computer file system,
151
-
you can replace `test.csv` with a CSV file path of your choice. The `foo>=2` condition shows how to define a filtering
152
-
clause using the conventional SQL WHERE constrains.
160
+
Like all dataflow operators in Nextflow, the operators provided by this plugin are executed asynchronously.
153
161
154
-
## Important
162
+
In particular, data inserted using the `sqlInsert` operator is *not* guaranteed to be available to any subsequent queries using the `fromQuery` operator, as it is not possible to make a channel factory operation dependent on some upstream operation.
155
163
156
-
This plugin is not expected to be used to store and access a pipeline status in a synchronous manner during the pipeline
157
-
execution.
158
164
159
-
This means that if your script has a `sqlInsert` operation followed by a successive `fromQuery` operation, the query
160
-
may *not* contain previously inserted data due to the asynchronous nature of Nextflow operators.
165
+
## Developtment
161
166
162
-
The SQL support provided by this plugin is meant to be used to fetch DB data from a previous run or to populate DB tables
0 commit comments