Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: numtel/mysql-live-select
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.0.14
Choose a base ref
...
head repository: numtel/mysql-live-select
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref

Commits on Feb 17, 2015

  1. Readme binlog path updated

    numtel committed Feb 17, 2015
    Copy the full SHA
    ca3f33c View commit details

Commits on Feb 25, 2015

  1. Error handler fixed

    numtel committed Feb 25, 2015
    Copy the full SHA
    ccb8272 View commit details

Commits on Feb 27, 2015

  1. Zongji updated to 0.3.1

    * Fixes stop() method so connections actually close
    numtel committed Feb 27, 2015
    Copy the full SHA
    debc639 View commit details

Commits on Mar 5, 2015

  1. Update Zongji to 0.3.2

    * Fixes temporal types in MySQL >= 5.6.4
    numtel committed Mar 5, 2015
    Copy the full SHA
    0ba457b View commit details

Commits on Apr 12, 2015

  1. Copy the full SHA
    3b70124 View commit details

Commits on Apr 22, 2015

  1. Travis CI badge on master

    numtel committed Apr 22, 2015
    Copy the full SHA
    6c9a2ff View commit details

Commits on May 6, 2015

  1. Copy the full SHA
    12ec0b4 View commit details

Commits on May 7, 2015

  1. Copy the full SHA
    fd387bf View commit details
  2. 0.0.21 Emit added/changed/removed event before updating data

    * Regression started in 0.0.19
    numtel committed May 7, 2015
    Copy the full SHA
    40829c4 View commit details

Commits on Jul 17, 2015

  1. Copy the full SHA
    ff50a83 View commit details
  2. Copy the full SHA
    254c759 View commit details
  3. Readme test details fix

    numtel committed Jul 17, 2015
    Copy the full SHA
    ad4382d View commit details

Commits on Aug 18, 2015

  1. isDeleted argument added to trigger condition functions

    * Test case updated to check condition function arguments
    numtel committed Aug 18, 2015
    Copy the full SHA
    6e20eaa View commit details
  2. Copy the full SHA
    e80dff3 View commit details
  3. Version bump 0.0.23

    numtel committed Aug 18, 2015
    Copy the full SHA
    e3f9d0a View commit details

Commits on Aug 19, 2015

  1. Better caching that fixes two major bugs

    Bugs: LiveMysql._resultsBuffer[query] never gets invalidated when no
    LiveMysqlSelect objects for that query are active; Even slightly large
    values of minInterval drastically slow down binlog event processing when
    there are many LiveMysqlSelect objects active
    wj32 authored and numtel committed Aug 19, 2015
    Copy the full SHA
    548a699 View commit details
  2. Copy the full SHA
    cfae27a View commit details
  3. Version bump 0.0.24

    * Update dependency package 'mysql' to 2.8.0
      Fixes #15
    numtel committed Aug 19, 2015
    Copy the full SHA
    a0ace73 View commit details

Commits on Aug 22, 2015

  1. Copy the full SHA
    8969116 View commit details
  2. 1.0.0: Result set differences now updated to use less data

    * BREAKING CHANGE: added, changed, removed, and diff events no longer
      exist. update event now includes a diff argument along with the data
      argument.
    * Differ adapted from pg-live-select
    * QueryCache revised
    * skipDiff setting now longer exists
    numtel committed Aug 22, 2015
    Copy the full SHA
    a44c86b View commit details
  3. Copy the full SHA
    8a9d0f6 View commit details

Commits on Aug 23, 2015

  1. 1.0.1: Fix added rows in diff calculation

    * multipleQueries test created
    numtel committed Aug 23, 2015
    Copy the full SHA
    b9c7feb View commit details
  2. Copy the full SHA
    346a98d View commit details

Commits on Aug 24, 2015

  1. Copy the full SHA
    e71953b View commit details

Commits on Oct 16, 2015

  1. Copy the full SHA
    3f13283 View commit details
  2. Copy the full SHA
    4710c9e View commit details

Commits on Mar 11, 2016

  1. 1.0.4: Update ZongJi to 0.4.0

    * Also update other dependencies
    * LiveMysql constructor callback now DEPRECATED, use 'error' and 'ready'
      events instead.
    numtel committed Mar 11, 2016
    Copy the full SHA
    060f709 View commit details
  2. Use Docker for Travis CI Build

    * Also, update my.cnf readme instructions
    numtel committed Mar 11, 2016
    Copy the full SHA
    978df05 View commit details

Commits on Mar 28, 2016

  1. 1.0.5: Update ZongJi to 0.4.1

    numtel committed Mar 28, 2016
    Copy the full SHA
    0670e88 View commit details

Commits on Apr 28, 2016

  1. 1.0.6: Update ZongJi to 0.4.2

    numtel committed Apr 28, 2016
    Copy the full SHA
    211512d View commit details
Showing with 873 additions and 431 deletions.
  1. +1 −0 .gitignore
  2. +13 −7 .travis.yml
  3. +63 −22 README.md
  4. +10 −2 example.js
  5. +120 −38 lib/LiveMysql.js
  6. +34 −95 lib/LiveMysqlSelect.js
  7. +149 −0 lib/QueryCache.js
  8. +113 −0 lib/differ.js
  9. +6 −4 package.json
  10. +75 −0 test/fixtures/multipleQueries.js
  11. +278 −75 test/index.js
  12. +0 −2 test/settings/mysql.js
  13. +0 −72 test/travis/install.sh
  14. +0 −36 test/travis/my.5.1.73.cnf
  15. +0 −36 test/travis/my.5.5.41.cnf
  16. +0 −36 test/travis/my.5.6.22.cnf
  17. +11 −6 test/travis/runner.sh
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
*.*~
*.swp
*.swo
*.swn
.npm
node_modules
20 changes: 13 additions & 7 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
language: node_js
sudo: required
dist: trusty
node_js:
- "0.10"
before_script:
- sudo apt-get install libaio1 libaio-dev
- ./test/travis/install.sh
- "4.1"
services:
- mysql
- docker
before_install:
- docker pull mysql
- docker pull mtirsel/mysql-5.1
- docker run -p 3351:3306 --name mysql-51 -e MYSQL_ROOT_PASSWORD=numtel -d mtirsel/mysql-5.1 mysqld --datadir=/var/lib/mysql --user=mysql --server-id=1 --log-bin=/var/lib/mysql/mysql-bin.log --binlog-format=row
- docker run -p 3355:3306 --name mysql-55 -e MYSQL_ROOT_PASSWORD=numtel -d mysql:5.5 --server-id=1 --log-bin=/var/lib/mysql/mysql-bin.log --binlog-format=row
- docker run -p 3356:3306 --name mysql-56 -e MYSQL_ROOT_PASSWORD=numtel -d mysql:5.6 --server-id=1 --log-bin=/var/lib/mysql/mysql-bin.log --binlog-format=row
- docker run -p 3357:3306 --name mysql-57 -e MYSQL_ROOT_PASSWORD=numtel -d mysql:5.7 --server-id=1 --log-bin=/var/lib/mysql/mysql-bin.log --binlog-format=row
script:
- ./test/travis/runner.sh
after_script:
- kill $(cat mysql-5.1.73-linux-x86_64-glibc23/mysql.pid)
- kill $(cat mysql-5.5.41-linux2.6-x86_64/mysql.pid)
- kill $(cat mysql-5.6.22-linux-glibc2.5-x86_64/mysql.pid)
85 changes: 63 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
# mysql-live-select [![Build Status](https://travis-ci.org/numtel/mysql-live-select.svg)](https://travis-ci.org/numtel/mysql-live-select)
# mysql-live-select [![Build Status](https://travis-ci.org/numtel/mysql-live-select.svg?branch=master)](https://travis-ci.org/numtel/mysql-live-select)

NPM Package to provide events when a MySQL select statement result set changes.

Built using the [`zongji` Binlog Tailer](https://github.com/nevill/zongji) and [`node-mysql`](https://github.com/felixge/node-mysql) projects.

* [Example Application using Express, SockJS and React](https://github.com/numtel/reactive-mysql-example)
* [Meteor package for reactive MySQL](https://github.com/numtel/meteor-mysql)
* [NPM Package for Sails.js connection adapter integration](https://github.com/numtel/sails-mysql-live-select)
* [Analogous package for PostgreSQL, `pg-live-select`](https://github.com/numtel/pg-live-select)

This package has been tested to work in MySQL 5.5.40 and 5.6.19. Expected support is all MySQL server version >= 5.1.15.
This package has been tested to work in MySQL 5.1, 5.5, 5.6, and 5.7. Expected support is all MySQL server version >= 5.1.15.

## Installation

@@ -19,18 +21,22 @@ This package has been tested to work in MySQL 5.5.40 and 5.6.19. Expected suppor
* Enable MySQL binlog in `my.cnf`, restart MySQL server after making the changes.

```
# binlog config
# Must be unique integer from 1-2^32
server-id = 1
# Row format required for ZongJi
binlog_format = row
log_bin = /usr/local/var/log/mysql/mysql-bin.log
binlog_do_db = employees # optional
expire_logs_days = 10 # optional
max_binlog_size = 100M # optional
# Directory must exist. This path works for Linux. Other OS may require
# different path.
log_bin = /var/log/mysql/mysql-bin.log
binlog_do_db = employees # Optional, limit which databases to log
expire_logs_days = 10 # Optional, purge old logs
max_binlog_size = 100M # Optional, limit log size
```
* Create an account with replication privileges:
* Create an account, then grant replication privileges:

```sql
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user'@'localhost'
GRANT REPLICATION SLAVE, REPLICATION CLIENT, SELECT ON *.* TO 'user'@'localhost'
```

## LiveMysql Constructor
@@ -41,17 +47,31 @@ The `LiveMysql` constructor makes 3 connections to your MySQL database:
* Replication slave connection
* `information_schema` connection for column information

#### Arguments

Argument | Type | Description
---------|------|---------------------------
`settings` | `object` | An object defining the settings. In addition to the [`node-mysql` connection settings](https://github.com/felixge/node-mysql#connection-options), the additional settings below are available.
`callback` | `function` | Optional callback on connection success/failure. Accepts one argument, `error`.
`callback` | `function` | **Deprecated:** callback on connection success/failure. Accepts one argument, `error`. See information below about events emitted.

#### Additional Settings

Setting | Type | Description
--------|------|------------------------------
`serverId` | `integer` | [Unique number (1 - 2<sup>32</sup>)](http://dev.mysql.com/doc/refman/5.0/en/replication-options.html#option_mysqld_server-id) to identify this replication slave instance. Must be specified if running more than one instance.<br>**Default:** `1`
`minInterval` | `integer` | Pass a number of milliseconds to use as the minimum between result set updates. Omit to refresh results on every update. May be changed at runtime.
`skipDiff` | `boolean` | If `true`, the `added`, `changed`, and `removed` events will not be emitted. May be changed at runtime.<br>**Default:** `false`
`checkConditionWhenQueued` | `boolean` | Set to `true` to call the condition function of a query on every binlog row change event. By default (when undefined or `false`), the condition function will not be called again when a query is already queued to be refreshed. Enabling this can be useful if external caching of row changes.

#### Events Emitted

Use `.on(...)` to handle the following event types.

Event Name | Arguments | Description
-----------|-----------|---------------
`error` | `Error` | An error has occurred.
`ready` | *None* | The database connection is ready.

#### Quick Start

```javascript
// Example:
@@ -66,8 +86,16 @@ liveConnection.select(function(esc, escId){
);
}, [ {
table: table,
condition: function(row, newRow){ return row.id === id; }
} ]).on('update', function(data){
condition: function(row, newRow){
// Only refresh the results when the row matching the specified id is
// changed.
return row.id === id
// On UPDATE queries, newRow must be checked as well
|| (newRow && newRow.id === id);
}
} ]).on('update', function(diff, data){
// diff contains an object describing the difference since the previous update
// data contains an array of rows of the new result set
console.log(data);
});
```
@@ -100,15 +128,32 @@ Name | Type | Description

#### Condition Function

A condition function accepts one or two arguments:
A condition function accepts up to three arguments:

Argument Name | Description
--------------|-----------------------------
`row` | Table row data
`newRow` | New row data (only available on `UPDATE` queries)
`newRow` | New row data (only available on `UPDATE` queries, `null` for others)
`rowDeleted` | Extra argument for aid in external caching: `true` on `DELETE` queries, `false` on `INSERT` queries, `null` on `UPDATE` queries.

Return `true` when the row data meets the condition to update the result set.

### LiveMysql.prototype.pause()

Temporarily skip processing of updates from the binary log.

### LiveMysql.prototype.resume()

Begin processing updates after `pause()`. All active live select instances will be refreshed upon resume.

### LiveMysql.prototype.end()

Close connections and stop checking for updates.

### LiveMysql.applyDiff(data, diff)

Exposed statically on the LiveMysql object is a function for applying a `diff` given in an `update` event to an array of rows given in the `data` argument.

## LiveMysqlSelect object

Each call to the `select()` method on a LiveMysql object, returns a `LiveMysqlSelect` object with the following methods:
@@ -126,18 +171,14 @@ As well as all of the other methods available on [`EventEmitter`](http://nodejs.

Event Name | Arguments | Description
-----------|-----------|---------------------------
`update` | `rows` | Single argument contains complete result set array. Called before `added`, `changed`, and `removed` events.
`added` | `row`, `index` | Row added to result set at index
`changed` | `row`, `newRow`, `index` | Row contents mutated at index
`removed` | `row`, `index` | Row removed at index
`diff` | `diff` | Aggregation of `added`, `changed`, `removed` events for current event into a single array for easier handling of multiple changes
`update` | `diff`, `data` | First argument contains an object describing the difference since the previous `update` event with `added`, `removed`, `moved`, and `copied` rows. Second argument contains complete result set array.
`error` | `error` | Unhandled errors will be thrown

## Running Tests

Tests must be run with a properly configured MySQL server. Configure test settings in `test/settings.mysql.js`.
Tests must be run with a properly configured MySQL server. Configure test settings in `test/settings/mysql.js`.

Execute `nodeunit` using the `npm test` command.
Execute [Nodeunit](https://github.com/caolan/nodeunit) using the `npm test` command.

## License

12 changes: 10 additions & 2 deletions example.js
Original file line number Diff line number Diff line change
@@ -22,8 +22,16 @@ liveConnection.select(function(esc, escId){
);
}, [ {
table: table,
condition: function(row, newRow){ return row.id === id; }
} ]).on('update', function(data){
condition: function(row, newRow){
// Only refresh the results when the row matching the specified id is
// changed.
return row.id === id
// On UPDATE queries, newRow must be checked as well
|| (newRow && newRow.id === id);
}
} ]).on('update', function(diff, data){
// diff contains an object describing the difference since the previous update
// data contains an array of rows of the new result set
console.log(data);
});

Loading