Merge branch 'master' into next

Author: ar2rsawseen

README.md

@@ -1,7 +1,7 @@
 
-<h2 align="center"> Countly Analytics </h2>
+<h1 align="center"> Countly Analytics </h1>
 
-<p align="center">
+<p align="right">
 
 [![Build Status](https://api.travis-ci.org/Countly/countly-server.png?branch=master)](https://travis-ci.org/Countly/countly-server) [![Codacy Badge](https://api.codacy.com/project/badge/Grade/98c2726f2d734697a5f1ac0d453f0a06)](https://app.codacy.com/app/ar2rsawseen/countly-server?utm_source=github.com&utm_medium=referral&utm_content=Countly/countly-server&utm_campaign=Badge_Grade_Dashboard) [![Install Countly on DigitalOcean](https://do.count.ly/button.svg?v2)](http://do.count.ly)
 
@@ -46,7 +46,7 @@
 
 
 ## What is Countly?
-[Countly](http://count.ly) is an innovative, real-time, open source [mobile](https://count.ly/mobile-analytics) & [web analytics](http://count.ly/web-analytics), [rich push notifications](https://count.ly/plugins/rich-push-notifications) and [crash reporting](https://count.ly/plugins/crash-analytics) platform powering more than 2500 web sites, 14000 mobile applications and several tens of thousands of desktop applications as of 2017 Q3. It collects data from mobile, desktop, web apps including Apple Watch, TvOS and other internet-connected devices, and visualizes this information to analyze application usage and end-user behavior. 
+[Countly](http://count.ly) is an innovative, real-time, open source [mobile](https://count.ly/mobile-analytics) & [web analytics](http://count.ly/web-analytics), [rich push notifications](https://count.ly/plugins/rich-push-notifications) and [crash reporting](https://count.ly/plugins/crash-analytics) platform powering more than 2500 web sites, 16000 mobile applications and several desktop applications as of 2019. It collects data from mobile, desktop, web apps including Apple Watch, TvOS and other internet-connected devices, and visualizes this information to analyze application usage and end-user behavior. 
 
 With the help of [Javascript SDK](http://github.com/countly/countly-sdk-web), Countly is a web analytics platform with features on par with mobile SDKs. For more information about web analytics capabilities, [see this link](http://count.ly/web-analytics).
 
@@ -64,7 +64,7 @@ This repository includes server-side part of Countly, with following features:
 * Complete dashboard user interface
 * User, application and permission management
 * Read / write APIs
-* Plugin based system
+* Plugin based architecture
 * Analytics features for [mobile](https://count.ly/mobile-analytics), [web](https://count.ly/web-analytics) and [desktop](https://count.ly/desktop-analytics)
 * [Crash reporting](https://count.ly/plugins/crash-analytics) for iOS & Android and error reporting for Javascript
 * [Rich and interactive push notifications](https://count.ly/plugins/rich-push-notifications) for iOS & Android
@@ -79,7 +79,7 @@ This repository includes server-side part of Countly, with following features:
 ## Built with 
 
 * **MongoDB** One of the most popular NoSQL databases
-* **Node.js** An open-source, cross-platform JavaScript runtime environment for developing a diverse variety of tools and applications.
+* **Node.js** An open-source, cross-platform JavaScript runtime environment
 * **Express.js** Popular Node.js web application framework 
 * **Linux** What we love using ;-)
 
@@ -91,22 +91,24 @@ Security is very important to us. If you discover any issue regarding security,
 
 ## What makes Countly unique?
 
-Countly is a privacy-focused and 360-degree analytics approach with several, unique values:
+Countly is a privacy-focused and 360-degree analytics platform with several, unique values:
 
 * Real-time mobile analytics, web analytics and push notifications.
 * Your data, your rules - since you can install Countly on your own server, or let us do a private cloud deployment for you.
 * Configurable and extensible via open source [plugins](https://count.ly/plugins).
-* Modern and easy to use web dashboard with a focus on user experience, helping to get complex insights easily.
-* Tracking more than 2500 web sites and 14000 mobile applications.
+* Modern and easy to use web based dashboard with a focus on user experience, which makes getting complex insights a breeze.
+* Tracking more than 2500 web sites and 16000 mobile applications.
 * Collecting more than 60 billion datapoints worlwide.
 
 ## Differences between Community Edition & Enterprise Edition
 
-* **Audience:** Community Edition is for product managers, developers and analytics enthusiasts, whereas banks, operators, healthcare companies and businesses that run the world's leading websites choose Enterprise Edition.
-* **Deployment:** Community Edition deployments are for environments where the information stored is not critical since it does not include professional assistance. You must have a qualified technician. Enterprise Edition is for corporate environments that require availability and reliability where intellectual capital is critical.
-* **Technology:** Community Edition has a number of limitations, including no corporate support for sharding, replica sets or installation.
-* **Guarantee:** Community Edition is on the bleeding edge regarding version upgrades and with no bugfix guarantee. Enterprise Edition has bugfix guarantee, immediate resolution, verified builds, on-site and automatic version upgrades.
-* **Features:** Enterprise Edition has more features compared to Countly, with a focus on end-to-end analytics and marketing platform. Additional features include but not limited to automated push notifications, drilling on the raw data, user profiles, in-app purchase analytics, retention & engagement, user flows, cohorts and custom dashboards.
+* **Audience:** Community Edition is suitable for individual developers and small development houses whereas Enterprise Edition is a better fit for companies that require more advanced analytics and marketing capabilities together with ongoing support.
+* **Features:** Enterprise Edition has additional features compared to Community Edition including automated push notifications, advanced segmentation, user profiles, in-app purchase analytics, retention, user flows, behavioral cohorts and custom dashboards.
+* **Data granularity:** Community Edition stores data (only) in an aggregated format, which reduces the required storage and makes reporting incredibly fast. Enterprise Edition takes advantage of the same format but also stores individual occurrences of data points which enables more advanced capabilities such as segmentation, funnels, user profiles and behavioral cohorts to name a few.
+* **Working with raw data:** Granular data, exclusive to Enterprise Edition, enables customers to take advantage of external BI tools or work directly with MongoDB to process and report data as they need.
+* **Deployment:** Running and maintaining a Community Edition instance requires technical know-how of several technologies including Linux, Nginx, Node.js and MongoDB. Whereas an Enterprise Edition hosted or on-premise subscription includes hands on support.
+* **High availability and scalability:** Countly engineers provide active support to Enterprise Edition customers for deployment planning and realization of this plan including replica set, sharding and a load balanced deployment setup on-premises or on popular cloud environments such as Google Cloud, AWS and Azure.
+* **Service-level agreement:** Enterprise Edition subscriptions include an SLA with response and issue resolution guarantees. Community Edition users can take advantage of the community forum or GitHub to post issues.
 
 ## Installing and upgrading Countly server
 
@@ -151,7 +153,9 @@ Also, you are encouraged to read an extended contribution section on [how to con
 [![4](https://sourcerer.io/fame/ar2rsawseen/Countly/countly-server/images/4)](https://sourcerer.io/fame/ar2rsawseen/Countly/countly-server/links/4)
 [![5](https://sourcerer.io/fame/ar2rsawseen/Countly/countly-server/images/5)](https://sourcerer.io/fame/ar2rsawseen/Countly/countly-server/links/5)
 [![6](https://sourcerer.io/fame/ar2rsawseen/Countly/countly-server/images/6)](https://sourcerer.io/fame/ar2rsawseen/Countly/countly-server/links/6)
-[![7](https://sourcerer.io/fame/ar2rsawseen/Countly/countly-server/images/7)](https://sourcerer.io/fame/ar2rsawseen/Countly/countly-server/links/7)
+[![7](https://sourcerer.io/fame/ar2rsawseen/Countly/countly-server/images/7)](https://sourcerer.io/fame/ar2rsawseen/Countly/countly-server/links/7) 
+
+[![Greenkeeper badge](https://badges.greenkeeper.io/Countly/countly-server.svg)](https://greenkeeper.io/)
 
 ## Badges
 

api/parts/data/usage.js

@@ -81,6 +81,7 @@ function locFromGeoip(loc, ip_address) {
             if (data) {
                 loc.country = loc.country || (data && data.country);
                 loc.city = loc.city || (data && data.city);
+                loc.region = loc.region || (data && data.region);
                 loc.lat = loc.lat === undefined ? (data && data.ll && data.ll[0]) : loc.lat;
                 loc.lon = loc.lon === undefined ? (data && data.ll && data.ll[1]) : loc.lon;
                 resolve(loc);
@@ -112,10 +113,17 @@ function updateLoc(params, optout, loc) {
 
     loc.gps = loc.gps || false;
     if (optout) {
-        loc.country = loc.city = 'Unknown';
+        loc.country = loc.region = loc.city = 'Unknown';
         update.$unset = {loc: 1};
     }
     else {
+        if (loc.country && loc.region && loc.region.length) {
+            loc.region = loc.country + "-" + loc.region;
+        }
+        else {
+            loc.region = 'Unknown';
+        }
+
         if (!loc.country) {
             loc.country = loc.city = 'Unknown';
         }
@@ -125,6 +133,7 @@ function updateLoc(params, optout, loc) {
     }
 
     params.user.country = loc.country;
+    params.user.region = loc.region;
     params.user.city = loc.city;
 
     if (!params.app_user || params.app_user[common.dbUserMap.country_code] !== loc.country) {
@@ -136,6 +145,12 @@ function updateLoc(params, optout, loc) {
         }
         update.$set[common.dbUserMap.city] = loc.city;
     }
+    if (!params.app_user || params.app_user[common.dbUserMap.region] !== loc.region) {
+        if (!update.$set) {
+            update.$set = {};
+        }
+        update.$set[common.dbUserMap.region] = loc.region;
+    }
     if (!params.app_user || (loc.tz !== undefined && params.app_user.tz !== loc.tz)) {
         if (!update.$set) {
             update.$set = {};
@@ -382,11 +397,12 @@ usage.endUserSession = function(params, done) {
 **/
 usage.processSessionDuration = function(params, callback) {
     var updateUsers = {},
-        session_duration = parseInt(params.qstring.session_duration);
+        session_duration = parseInt(params.qstring.session_duration),
+        session_duration_limit = parseInt(plugins.getConfig("api", params.app && params.app.plugins, true).session_duration_limit);
 
     if (session_duration) {
-        if (plugins.getConfig("api", params.app && params.app.plugins, true).session_duration_limit && session_duration > plugins.getConfig("api", params.app && params.app.plugins, true).session_duration_limit) {
-            session_duration = plugins.getConfig("api", params.app && params.app.plugins, true).session_duration_limit;
+        if (session_duration_limit && session_duration > session_duration_limit) {
+            session_duration = session_duration_limit;
         }
 
         if (session_duration < 0) {

api/utils/common.js

@@ -185,6 +185,7 @@ common.dbUserMap = {
     'device': 'd',
     'carrier': 'c',
     'city': 'cty',
+    'region': 'rgn',
     'country_code': 'cc',
     'platform': 'p',
     'platform_version': 'pv',
@@ -984,7 +985,7 @@ common.returnRaw = function(params, returnCode, body, heads) {
                 params.res = {};
             }
             params.res.finished = true;
-            params.APICallback(returnCode === 200, body, heads, returnCode, params);
+            params.APICallback(returnCode !== 200, body, heads, returnCode, params);
         }
         return;
     }
@@ -1025,7 +1026,7 @@ common.returnMessage = function(params, returnCode, message, heads) {
                 params.res = {};
             }
             params.res.finished = true;
-            params.APICallback(returnCode === 200, JSON.stringify({result: message}), heads, returnCode, params);
+            params.APICallback(returnCode !== 200, JSON.stringify({result: message}), heads, returnCode, params);
         }
         return;
     }
@@ -1083,7 +1084,7 @@ common.returnOutput = function(params, output, noescape, heads) {
                 params.res = {};
             }
             params.res.finished = true;
-            params.APICallback(true, output, heads, 200, params);
+            params.APICallback(false, output, heads, 200, params);
         }
         return;
     }

api/utils/requestProcessor.js

@@ -373,7 +373,7 @@ const processRequest = (params) => {
                                 common.returnMessage(params, 400, 'No users matching criteria');
                                 return false;
                             }
-                            if (count > 1) {
+                            if (count > 1 && !params.qstring.force) {
                                 common.returnMessage(params, 400, 'This query would delete more than one user');
                                 return false;
                             }
@@ -616,8 +616,7 @@ const processRequest = (params) => {
 
                 break;
             }
-            case '/i/events':
-            {
+            case '/i/events': {
                 switch (paths[3]) {
                 case 'edit_map':
                 {
@@ -1111,7 +1110,7 @@ const processRequest = (params) => {
                         .digest('hex');
                 }
 
-                if (params.qstring.events) {
+                if (params.qstring.events && typeof params.qstring.events === "string") {
                     try {
                         params.qstring.events = JSON.parse(params.qstring.events);
                     }
@@ -1983,6 +1982,8 @@ const validateAppForWriteAPI = (params, done, try_times) => {
     if (params.qstring.device_id === "00000000-0000-0000-0000-000000000000") {
         common.returnMessage(params, 400, 'Ignoring device_id');
         common.log("request").i('Request ignored: Ignoring zero IDFA device_id', params.req.url, params.req.body);
+        params.cancelRequest = "Ignoring zero IDFA device_id";
+        plugins.dispatch("/sdk/cancel", {params: params});
         return done ? done() : false;
     }
     common.db.collection('apps').findOne({'key': params.qstring.app_key}, (err, app) => {
@@ -2010,6 +2011,8 @@ const validateAppForWriteAPI = (params, done, try_times) => {
                 }
                 if (payloads.indexOf((params.qstring.checksum + "").toUpperCase()) === -1) {
                     console.log("Checksum did not match", params.href, params.req.body, payloads);
+                    params.cancelRequest = 'Request does not match checksum sha1';
+                    plugins.dispatch("/sdk/cancel", {params: params});
                     if (plugins.getConfig("api", params.app && params.app.plugins, true).safe) {
                         common.returnMessage(params, 400, 'Request does not match checksum');
                     }
@@ -2023,6 +2026,8 @@ const validateAppForWriteAPI = (params, done, try_times) => {
                 }
                 if (payloads.indexOf((params.qstring.checksum256 + "").toUpperCase()) === -1) {
                     console.log("Checksum did not match", params.href, params.req.body, payloads);
+                    params.cancelRequest = 'Request does not match checksum sha256';
+                    plugins.dispatch("/sdk/cancel", {params: params});
                     if (plugins.getConfig("api", params.app && params.app.plugins, true).safe) {
                         common.returnMessage(params, 400, 'Request does not match checksum');
                     }
@@ -2031,6 +2036,8 @@ const validateAppForWriteAPI = (params, done, try_times) => {
             }
             else {
                 console.log("Request does not have checksum", params.href, params.req.body);
+                params.cancelRequest = "Request does not have checksum";
+                plugins.dispatch("/sdk/cancel", {params: params});
                 if (plugins.getConfig("api", params.app && params.app.plugins, true).safe) {
                     common.returnMessage(params, 400, 'Request does not have checksum');
                 }

bin/scripts/device_list/package.json

@@ -23,6 +23,6 @@
   },
   "homepage": "https://count.ly/",
   "dependencies": {
-    "csvtojson": "^1.1.9"
+    "csvtojson": "^2.0.8"
   }
 }

frontend/express/app.js

@@ -487,7 +487,7 @@ app.get(countlyConfig.path + "*/screenshots/*", function(req, res) {
 var oneYear = 31557600000;
 app.use(countlyConfig.path, express.static(__dirname + '/public', { maxAge: oneYear }));
 app.use(session({
-    secret: 'countlyss',
+    secret: countlyConfig.web.session_secret || 'countlyss',
     cookie: { httpOnly: true, maxAge: 1000 * 60 * 60 * 24 * 365, secure: countlyConfig.web.secure_cookies || false },
     store: new SkinStore(countlyDb),
     saveUninitialized: false,

frontend/express/config.sample.js

@@ -70,6 +70,7 @@ var countlyConfig = {
     * @property {boolean} use_intercom - true, to use intercom in dashboard for communication with Countly
     * @property {boolean} secure_cookies - true, to use secure cookies, enable only if you have https enabled
     * @property {string} track - allow Countly to collect stats about amount of apps and datapoints as well as feature usage. 
+    * @property {string} session_secret - secret used to sign the session ID cookie. 
     * Possible values are: 
     *    "all" - track all, 
     *    "GA" - track only Global admins, 
@@ -81,7 +82,8 @@ var countlyConfig = {
         host: "localhost",
         use_intercom: true,
         secure_cookies: false,
-        track: "all"
+        track: "all",
+        session_secret: "countlyss"
     },
     /**
     * Legacy value, not supported

frontend/express/public/javascripts/countly/countly.common.js

@@ -3113,6 +3113,7 @@
                 periodMin = 1;
                 dateString = "D MMM";
                 numberOfDays = moment(currentTimeStamp || undefined).format("D");
+                daysInPeriod = numberOfDays;
                 break;
             case "yesterday":
                 var yesterday = moment(currentTimeStamp || undefined).subtract(1, 'days'),
@@ -3785,6 +3786,45 @@
             return tmpAvgVal.toFixed(2);
         };
 
+        /**
+        * Get timestamp range in format as [startTime, endTime] with period and base time
+        * @param {object} period - period has two format: array or string
+        * @param {number} baseTimeStamp - base timestamp to calc the period range
+        * @returns {array} period range
+        */
+        countlyCommon.getPeriodRange = function(period, baseTimeStamp) {
+            var periodRange;
+            if (Object.prototype.toString.call(period) === '[object Array]' && period.length === 2) { //range
+                periodRange = period;
+                return periodRange;
+            }
+            var endTimeStamp = baseTimeStamp;
+            var start;
+            switch (period) {
+            case 'hour':
+                start = moment(baseTimeStamp).hour(0).minute(0).second(0);
+                break;
+            case 'yesterday':
+                start = moment(baseTimeStamp).subtract(1, 'day').hour(0).minute(0).second(0);
+                endTimeStamp = moment(baseTimeStamp).subtract(1, 'day').hour(23).minute(59).second(59).toDate().getTime();
+                break;
+            case 'day':
+                start = moment(baseTimeStamp).date(1).hour(0).minute(0).second(0);
+                break;
+            case 'month':
+                start = moment(baseTimeStamp).month(0).date(1).hour(0).minute(0).second(0);
+                break;
+            default:
+                if (/([0-9]+)days/.test(period)) {
+                    var match = /([0-9]+)days/.exec(period);
+                    if (match[1] && (parseInt(match[1]) > 1)) {
+                        start = moment(baseTimeStamp).subtract(parseInt(match[1]) - 1, 'day').hour(0).minute(0);
+                    }
+                }
+            }
+            periodRange = [start.toDate().getTime(), endTimeStamp];
+            return periodRange;
+        };
     };
 
     window.CommonConstructor = CommonConstructor;