Skip to content

Commit de65dfd

Browse files
frknbasaranfrknbasaran
authored
Merge pull request #5 from Countly/master
sync master.
2 parents 63993aa + 367a990 commit de65dfd

File tree

376 files changed

+15649
-3844
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

376 files changed

+15649
-3844
lines changed

‎.travis.yml

+10-7
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,29 @@
11
language: nodejs
22
node_js: "8"
3-
services:
4-
- mongodb
53
env:
64
global:
5+
- MONGODB=3.6.6
76
- secure: ZXdUMP+8Nk9Kd5nX31D/DIB1t2xlJd+T4W7umWjwpy+KPA9d1W1OhUG6V2Gpv3PsJf+UtMosgZsywcLFVG4l/WcjAhs3c9pegW5ZRYAEfVjkk7aHhWQGvj5DID8iAAxs91Mm8LwJ43o68x+XN6746zArPqcVzQltDunJ0G3gfh8=
87
branches:
98
except:
109
- gh-pages
1110
before_install:
11+
- sudo apt-get remove -y mongodb-org mongodb-org-mongos mongodb-org-server mongodb-org-shell mongodb-org-tools
12+
- wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-${MONGODB}.tgz -O /tmp/mongodb.tgz
13+
- tar -xvf /tmp/mongodb.tgz
14+
- mkdir /tmp/data
15+
- ${PWD}/mongodb-linux-x86_64-${MONGODB}/bin/mongod --dbpath /tmp/data --bind_ip 127.0.0.1 --logpath ${PWD}/mongod.log &> /dev/null &
16+
- until nc -z localhost 27017; do echo Waiting for MongoDB; sleep 1; done
1217
install: sudo bash ./bin/countly.install_travis.sh
1318
before_script:
1419
- nodejs --version
15-
- mongod --version
16-
- cat /var/log/mongodb/mongod.log
17-
- until nc -z localhost 27017; do echo Waiting for MongoDB; sleep 1; done
18-
- sleep 10
20+
- ${PWD}/mongodb-linux-x86_64-${MONGODB}/bin/mongod --version
21+
- cat ${PWD}/mongod.log
1922
script: sudo npm test
2023
after_script:
2124
- cat log/countly-api.log
2225
- cat log/countly-dashboard.log
23-
- cat /var/log/mongodb/mongod.log
26+
- cat ${PWD}/mongod.log
2427
after_success:
2528
- cd $TRAVIS_BUILD_DIR
2629
- bash bin/scripts/deploy.sh

‎CHANGELOG.md

+160
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,163 @@
1+
## Version 19.02
2+
3+
**Fixes**
4+
* [alerts] change crash checking to once per hour
5+
* [api] limit loading meta data for events with high cardinality in segments
6+
* [api] make sure session duration increments are always numbers
7+
* [api] parse events only if needed
8+
* [api] properly delete app images on app delete
9+
* [api] record session frequency by sessions, not users
10+
* [compare] fixed bug with returning result
11+
* [compare] fixed missing icons
12+
* [compliance-hub] ui table trim longer device_ids
13+
* [config] fixed config extender not working with underscore or camelcase properties
14+
* [crashes] do not format app version
15+
* [crashes] do not return list if there are too many crashes
16+
* [crashes] fixed hidden comments
17+
* [crashes] properly refresh stacktrace with markup
18+
* [dashboard] fixed changing password
19+
* [dashboard] fixed sorting in some serverside tables
20+
* [dashboard] make age of unauthorized sessions much shorter
21+
* [dashboard] move app versions legend to the bottom
22+
* [data-migration] fixed click menus disappearing
23+
* [db] correctly replace database name
24+
* [db] fixed overriding poolsize in some cases
25+
* [frontend] fix when isoweek falls for the year in which it has thursday
26+
* [frontend] parse values as number when calculating percentage change
27+
* [frontend] server svg files with correct encoding
28+
* [monetization] crash bug when year is selected
29+
* [overview] fixed geo chart when date change
30+
* [push] respect restricted access rules
31+
* [reports] UI and table sorting fixes
32+
* [reports] allow manual data refresh for auto refresh tasks
33+
* [reports] allow rerunning tasks on http redirect settings
34+
* [reports] fix reports hour logic bug
35+
* [reports] removed the date picker from report manager
36+
* [screenshots] fixed ssl issue resolving
37+
* [security] add global $ handler in keys
38+
* [security] escape html in popups
39+
* [security] more proper error handling to prevent unwanted states
40+
* [server-stats] fix event count calculation
41+
* [slipping-away-users] use time of last api call instead of last session
42+
* [star-rating] fixed timestamp
43+
* [systemlogs] change comparison was recording too much
44+
* [systemlogs] exprot file name fix
45+
* [ui] correct flag sizes
46+
* [web] fixed error in pixel tracking passing user agent and added tests
47+
48+
**Enterprise fixes**
49+
* [attribution] optimizations for large click amount
50+
* [attribution] proper back behavior
51+
* [dashboards] fixed email subjects
52+
* [dashboards] remove current users from the edit view permission list
53+
* [dashboards] users list fix
54+
* [drill] Bug switching to hourly format on 7days
55+
* [drill] allow disabling meta recording
56+
* [drill] get correct user count in BY queries
57+
* [drill] handling paralel view duration updates
58+
* [flows] update for query performance
59+
* [funnels] fixed using correct minimal ts for each step
60+
* [groups] fixed output closed and null issues
61+
* [groups] fixed section restriction not showing in some cases
62+
* [heatmaps] are now not tied to view name (can use custom view names)
63+
* [push] Drill loading fix for no users case
64+
* [revenue] fixed loading data in app configuration
65+
66+
**New Features**
67+
* [alert] allow sharing with multiple email addresses
68+
* [api] Allow to delete multiple app users with force flag
69+
* [api] add region from geoip lite module
70+
* [api] added ability to upload app icon via API
71+
* [api] allow pausing receiving app data
72+
* [dashboard] bruteforce 0 attempts disabled bruteforce protection
73+
* [dashboard] change /logout api method from GET to POST
74+
* [dashboard] move to auth tokens
75+
* [dashboard] send link to authenticate user on different cases (time ban, password change)
76+
* [dashboard] use POST in most cases instead of GET
77+
* [dashboard] use argon2 for password hashing
78+
* [data-points] show data for larger periods
79+
* [dbviewer] add countly_fs and countly_out to list of databases too
80+
* [dbviewer] aggregation support
81+
* [dbviewer] reload data instead of whole view
82+
* [feedback] added systemlogs for widget create, edit and remove processes
83+
* [frontend] add rate limiting and configuration for it
84+
* [frontend] added more configuration options to version.info file
85+
* [frontend] allow controling language list from specific config file
86+
* [logger] log all canceled requests too
87+
* [loyalty] new loyalty view
88+
* [overview] faster top bar loading when it is high cardinality segment
89+
* [populator] add utm data generation
90+
* [push] ability to edit a automated push notification
91+
* [push] clear credentials on app reset
92+
* [push] close emoticon dialog on ESC key
93+
* [push] email notifications on push auto message fails
94+
* [push] new alert dialogs for errors
95+
* [push] proxy support
96+
* [remote-config] allow remote configuration for apps with SDK
97+
* [security] configurable session secret
98+
* [sources] add source character limitation configuration
99+
* [ui] added dropdown icons
100+
* [ui] first app creation dialog
101+
* [views] add column configuration
102+
* [views] added page scroll percentage numbers (avg)
103+
* [views] allow deleting views
104+
* [views] performance improvements for handling more views
105+
* [views] select which fields to show in table
106+
107+
108+
**New Enterprise Features**
109+
* [block] allow blocking internal events
110+
* [cohorts] allow segmenting user properties separately
111+
* [cohorts] allow to use location in user property query
112+
* [crashes] add crash name as drill segment
113+
* [dashboards] added systemlogs for dashboard and widget create, edit and remove
114+
* [dashboards] cohort widget
115+
* [drill] allow contains search on fields
116+
* [drill] drill data based estimation correct for users in overview and other sections (including previous period)
117+
* [drill] multiple by fields are allowed
118+
* [drill] new ui for querying
119+
* [drill][flows][funnels] do not cache if amount of data is low
120+
* [funnels] option for funnel start all users or entered users
121+
* [funnels] separate each step segmentation
122+
* [heatmaps] additional resolution options added
123+
* [retention_segments] Addd event based retention data
124+
* [users] column configuration for users table
125+
* [users] display notes about users on hover in the table
126+
127+
**Development related**
128+
* [api] add event for plugins to convert metric to collection data
129+
* [api] added api to return internal-events
130+
* [api] changed api key generation logic
131+
* [api] correct error states for APIcallback
132+
* [api] more multi purpose top fetching
133+
* [api] multiple metric refactoring and fixes
134+
* [api] respong only after app check
135+
* [cmd] healthcheck additions and fixes
136+
* [common] fixed countlyCommon.formatSecond() method
137+
* [db] adding country metric index for faster overview load
138+
* [db] ignore handled errors like duplicate index inserts
139+
* [db] implement basic collection cache
140+
* [db] improved logging
141+
* [db] properly log arguments
142+
* [db] provide a way to load database specific configs based on database name
143+
* [db] update compatability layer due to driver changes
144+
* [db] use count instead of aggregation for total users
145+
* [db] use estimated count when possible
146+
* [eslint] single config file and fixes
147+
* [frontend] add common.db reference for consistency with api process
148+
* [frontend] add constraints to unique fields like email
149+
* [frontend] add countlyCommon.getPeriodRange function
150+
* [frontend] fixed regex for page/refresh scripts
151+
* [groups] added group id for groups.
152+
* [nginx] increase worker file and connection limits
153+
* [nginx] provide https template
154+
* [package] remove bad dependencies
155+
* [package] remove kerberos support
156+
* [populator] add crash name property
157+
* [ui] allow cly dropdown to open up instead of down too
158+
* [upgrade] backup configs properly
159+
* [vagrant] modified Vagrantfile to provision development VM with 2048 MB memory
160+
1161
## Version 18.08.2
2162

3163
**Fixes**

‎Gruntfile.js

+2-1
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,8 @@ module.exports = function(grunt) {
9797
'frontend/express/public/javascripts/countly/countly.views.js',
9898
'frontend/express/public/javascripts/countly/countly.vue.components.js',
9999
'frontend/express/public/javascripts/countly/countly.token.manager.js',
100-
'frontend/express/public/javascripts/countly/countly.version.history.js'
100+
'frontend/express/public/javascripts/countly/countly.version.history.js',
101+
'frontend/express/public/javascripts/countly/countly.analytics.js'
101102
],
102103
dest: 'frontend/express/public/javascripts/min/countly.lib.concat.js'
103104
}

‎api/config.sample.js

-6
Original file line numberDiff line numberDiff line change
@@ -71,18 +71,12 @@ var countlyConfig = {
7171
* @property {number} [max_sockets=1024] - maximal amount of sockets to open simultaneously
7272
* @property {number} workers - amount of paralel countly processes to run, defaults to cpu/core amount
7373
* @property {number} [timeout=120000] - nodejs server request timeout, need to also increase nginx timeout too for longer requests
74-
* @property {object=} push_proxy - push proxy settings
7574
*/
7675
api: {
7776
port: 3001,
7877
host: "localhost",
7978
max_sockets: 1024,
8079
timeout: 120000
81-
/* GCM proxy server for push plugin
82-
push_proxy: {
83-
host: 'localhost',
84-
port: 8888
85-
} */
8680
},
8781
/**
8882
* Path to use for countly directory, empty path if installed at root of website

‎api/configs/config.db_out.sample.js

+47
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
var countlyConfig = {
2+
mongodb: {
3+
host: "localhost",
4+
db: "countly_out",
5+
port: 27017,
6+
max_pool_size: 500,
7+
//username: test,
8+
//password: test,
9+
/*
10+
dbOptions:{
11+
//db options
12+
native_parser: true
13+
},
14+
serverOptions:{
15+
//server options
16+
ssl:false
17+
}
18+
*/
19+
}
20+
/* or for a replica set
21+
mongodb: {
22+
replSetServers : [
23+
'192.168.3.1:27017',
24+
'192.168.3.2:27017'
25+
],
26+
replicaName: "test",
27+
db: "countly_out",
28+
username: test,
29+
password: test,
30+
max_pool_size: 1000,
31+
dbOptions:{
32+
//db options
33+
native_parser: true
34+
},
35+
serverOptions:{
36+
//server options
37+
ssl:false
38+
}
39+
},
40+
*/
41+
/* or define as a url
42+
//mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]
43+
mongodb: "mongodb://localhost:27017/countly_out",
44+
*/
45+
};
46+
47+
module.exports = countlyConfig;

‎api/jobs/ping.js

+1-1
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ class PingJob extends job.Job {
1414
* @param {done} done callback
1515
*/
1616
run(db, done) {
17-
request("http://localhost/configs", function(err, res, body) {
17+
request({strictSSL: false, uri: "http://localhost/configs"}, function(err, res, body) {
1818
log.d(err, body);
1919
});
2020
var countlyConfigOrig = JSON.parse(JSON.stringify(countlyConfig));

‎api/jobs/task.js

+9-4
Original file line numberDiff line numberDiff line change
@@ -27,10 +27,15 @@ class MonitorJob extends job.Job {
2727
log.d("job info:", self._json, tasks);
2828
tasks.forEach((task)=>{
2929
return Promise.coroutine(function *() { // eslint-disable-line require-yield
30-
taskmanager.rerunTask({
31-
db: common.db,
32-
id: task._id
33-
}, () => {});
30+
try {
31+
taskmanager.rerunTask({
32+
db: common.db,
33+
id: task._id
34+
}, () => {});
35+
}
36+
catch (e) {
37+
log.e(e, e.stack);
38+
}
3439
})();
3540
});
3641

‎api/lib/countly.common.js

+15-6
Original file line numberDiff line numberDiff line change
@@ -615,7 +615,7 @@ countlyCommon.extractTwoLevelData = function(db, rangeArray, clearFunction, data
615615
}
616616

617617
if (propertyNames.indexOf("u") !== -1 && Object.keys(tmpPropertyObj).length) {
618-
if (countlyCommon.periodObj.periodContainsToday && totalUserOverrideObj && totalUserOverrideObj[rangeArray[j]]) {
618+
if (totalUserOverrideObj && totalUserOverrideObj[rangeArray[j]]) {
619619

620620
tmpPropertyObj.u = totalUserOverrideObj[rangeArray[j]];
621621

@@ -1114,7 +1114,7 @@ countlyCommon.extractMetric = function(db, rangeArray, clearFunction, dataProper
11141114
}
11151115

11161116
if (propertyNames.indexOf("u") !== -1 && Object.keys(tmpPropertyObj).length) {
1117-
if (countlyCommon.periodObj.periodContainsToday && totalUserOverrideObj && totalUserOverrideObj[rangeArray[j]]) {
1117+
if (totalUserOverrideObj && totalUserOverrideObj[rangeArray[j]]) {
11181118

11191119
tmpPropertyObj.u = totalUserOverrideObj[rangeArray[j]];
11201120

@@ -1233,6 +1233,7 @@ countlyCommon.timeString = function(timespent) {
12331233
* @param {array} properties - array of all properties to extract
12341234
* @param {array} unique - array of all properties that are unique from properties array. We need to apply estimation to them
12351235
* @param {object} totalUserOverrideObj - using unique property as key and total_users estimation property as value for all unique metrics that we want to have total user estimation overridden
1236+
* @param {object} prevTotalUserOverrideObj - using unique property as key and total_users estimation property as value for all unique metrics that we want to have total user estimation overridden for previous period
12361237
* @returns {object} dashboard data object
12371238
* @example
12381239
* countlyCommon.getDashboardData(countlySession.getDb(), ["t", "n", "u", "d", "e", "p", "m"], ["u", "p", "m"], {u:"users"});
@@ -1247,7 +1248,7 @@ countlyCommon.timeString = function(timespent) {
12471248
* "m":{"total":86,"prev-total":0,"change":"NA","trend":"u","isEstimate":true}
12481249
* }
12491250
*/
1250-
countlyCommon.getDashboardData = function(data, properties, unique, totalUserOverrideObj) {
1251+
countlyCommon.getDashboardData = function(data, properties, unique, totalUserOverrideObj, prevTotalUserOverrideObj) {
12511252
/**
12521253
* Clear object, bu nulling out predefined properties, that does not exist
12531254
* @param {object} obj - object to clear
@@ -1372,17 +1373,25 @@ countlyCommon.getDashboardData = function(data, properties, unique, totalUserOve
13721373
}
13731374

13741375
//check if we can correct data using total users correction
1375-
if (_periodObj.periodContainsToday && totalUserOverrideObj) {
1376+
if (totalUserOverrideObj) {
13761377
for (let i = 0; i < unique.length; i++) {
13771378
if (current[unique[i]] && typeof totalUserOverrideObj[unique[i]] !== "undefined" && totalUserOverrideObj[unique[i]]) {
13781379
current[unique[i]] = totalUserOverrideObj[unique[i]];
13791380
}
13801381
}
13811382
}
13821383

1384+
if (prevTotalUserOverrideObj) {
1385+
for (let i = 0; i < unique.length; i++) {
1386+
if (previous[unique[i]] && typeof prevTotalUserOverrideObj[unique[i]] !== "undefined" && prevTotalUserOverrideObj[unique[i]]) {
1387+
previous[unique[i]] = prevTotalUserOverrideObj[unique[i]];
1388+
}
1389+
}
1390+
}
1391+
13831392
// Total users can't be less than new users
13841393
if (typeof current.u !== "undefined" && typeof current.n !== "undefined" && current.u < current.n) {
1385-
if (_periodObj.periodContainsToday && totalUserOverrideObj && typeof totalUserOverrideObj.u !== "undefined" && totalUserOverrideObj.u) {
1394+
if (totalUserOverrideObj && typeof totalUserOverrideObj.u !== "undefined" && totalUserOverrideObj.u) {
13861395
current.n = current.u;
13871396
}
13881397
else {
@@ -1409,7 +1418,7 @@ countlyCommon.getDashboardData = function(data, properties, unique, totalUserOve
14091418
}
14101419

14111420
//check if we can correct data using total users correction
1412-
if (_periodObj.periodContainsToday && totalUserOverrideObj) {
1421+
if (totalUserOverrideObj) {
14131422
for (let i = 0; i < unique.length; i++) {
14141423
if (dataArr[unique[i]] && typeof totalUserOverrideObj[unique[i]] !== "undefined" && totalUserOverrideObj[unique[i]]) {
14151424
dataArr[unique[i]].is_estimate = false;

0 commit comments

Comments
 (0)