WDI-Spurious.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spurious correlations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import libraries and functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import glob\n",
    "import os\n",
    "from zipfile import ZipFile\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "import functools as ft\n",
    "import ipywidgets as widgets\n",
    "from ipywidgets import Layout\n",
    "from ipywidgets import interact, interact_manual\n",
    "import plotly.express as px\n",
    "from scipy import stats\n",
    "from scipy.stats import shapiro\n",
    "from scipy.stats import pearsonr\n",
    "from scipy.stats import spearmanr\n",
    "from pandas.api.types import is_numeric_dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, there is an interesting phenomenon, in some cases there are correlations that have a high coefficient and also an adequate graphics, but they do not make sense in the analysis, these are called **spurious correlations**. Here are some examples:\n",
    "\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/devonfw-forge/python-data-driven-decisions/main-the-big-three/Logos/chart%20(2).jpeg\" width=\"800\" height=\"400\" />\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/devonfw-forge/python-data-driven-decisions/main-the-big-three/Logos/chart%20(1).jpeg\" width=\"800\" height=\"400\" />\n",
    "\n",
    "\n",
    "Therefore, we have to be carefull with our results because correlation does not imply causation, it may have happened by chance that both variables are really similar.\n",
    "So, after some thought and experimenting, we have developed a method that we think, it will allow us to find out if the correlation has happened by chance or if there is really a correlation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method, consists of the following:\n",
    "\n",
    "\n",
    "Firstly we have classified the indicators by a group, which can be one of the following: *A&D*, *Agriculture*, *Demography*, *Economy*, *Employment*, *Environment*, *Equality*, *Exports*, *Health*, *Mortality* or *Principal*. Moreover inside each group we have also assigned each varible a level, *primary* or *secondary*, depending on their level of relevance. For example we have consider more relevant the *Population in the largest city* over the *Rural population*, thus the first will be *primary* and the latter *secondary*, while both are part of the *Demography* group. \n",
    "\n",
    "With this set, we can expose our hypothesis:\n",
    "\n",
    "\"It is assumed that the correlation in the primary indicators can be caused by randomness, however if this correlation also appears in the secondary indicators for at least X% of the countries that appears in the primaries (Pareto's rule), we can suppose that there is no randomness affecting each group. Furthermore, the first assumption has to happen in Y% of the secondary indicators to avoid any fortuity.\" \n",
    "\n",
    "This hypothesis can be used in a global level, all the countries, or in the different regions. \n",
    "\n",
    "For example if, X and Y =80% a primary indicator is repeated 20 times the secondary indicators must have repeated 18 times. And if there are 10 secondary indicators, it has to happen for, at least, 8 indicators.\n",
    "\n",
    "Finally, we will finish with two possible errors of 20%, which combined (20%*20%), leaves us with a 4% of margin of error, which is lower than the wildly spread of 5%.\n",
    "\n",
    "**This will only work if the data has been collected by independent sources and uses different methods to collect it. Therefore, this step has only been developed for this data (WDI), which we have checked that comes from different sources and is gathered differently.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e154ff7a4c684bebb8a1f62906fac3fe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatSlider(value=0.8, continuous_update=False, description='% of primary indicators:', max=1.0, step=0.05)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1436ccab37804d3e82fbc5b3bd07c8f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatSlider(value=0.8, continuous_update=False, description='% of secondary indicators:', max=1.0, step=0.05)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "limita=widgets.FloatSlider( value=0.8, min=0, max=1.0, step=0.05, description='% of primary indicators:', disabled=False, continuous_update=False, orientation='horizontal', readout=True)\n",
    "limitb=widgets.FloatSlider( value=0.8, min=0, max=1.0, step=0.05, description='% of secondary indicators:', disabled=False, continuous_update=False, orientation='horizontal', readout=True)\n",
    "display(limita,limitb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The margin of error in this combination is: 3.9999999999999982\n"
     ]
    }
   ],
   "source": [
    "a=(1-limita.value)*(1-limitb.value)*100\n",
    "print('The margin of error in this combination is:',a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have selected both percentages and we agree with the margin of error, we can proceed to put into action our method. Firstly, we filter the primary indicators and get the minimun of times thatthe secondary have to be repeated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Min</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Group</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A&amp;D</th>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Agriculture</th>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Demography</th>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Economy</th>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Employment</th>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Environment</th>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Equality</th>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Exports</th>\n",
       "      <td>28.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Health</th>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Internet</th>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mortality</th>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Principal</th>\n",
       "      <td>23.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Min\n",
       "Group            \n",
       "A&D          13.0\n",
       "Agriculture  10.0\n",
       "Demography   24.0\n",
       "Economy      26.0\n",
       "Employment   11.0\n",
       "Environment  13.0\n",
       "Equality     14.0\n",
       "Exports      28.0\n",
       "Health        9.0\n",
       "Internet     27.0\n",
       "Mortality    14.0\n",
       "Principal    23.0"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "selected_p=categories.loc[categories['Level']=='primary']\n",
    "minprimary=selected_p.groupby('Group').min()\n",
    "minprimary['Min']=round(minprimary['Number of times repeated']*limita.value)\n",
    "minprimary.drop(columns=['Indicator','Number of times repeated','Level'], inplace=True)\n",
    "minprimary\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouplist=minprimary.index.to_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we test if the repetition are accomplished. \n",
    "\n",
    "- H_0 data has correlation buy has not happened by randomness.\n",
    "- H_1 data has correlation due to randomness \n",
    "\n",
    "**If Number of times repeated the secondary indicator < Minimun per group, then H_0 denied and H_1 accepted**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Does it have some global casuallity implied?</th>\n",
       "      <th>% of count (Global)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Agriculture</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Demography</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Economy</th>\n",
       "      <td>Yes</td>\n",
       "      <td>71.039604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Employment</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Environment</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Equality</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Exports</th>\n",
       "      <td>No</td>\n",
       "      <td>82.379863</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Health</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Internet</th>\n",
       "      <td>Yes</td>\n",
       "      <td>37.190083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mortality</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Principal</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Does it have some global casuallity implied?  % of count (Global)\n",
       "Group                                                                        \n",
       "Agriculture                                           No           100.000000\n",
       "Demography                                            No           100.000000\n",
       "Economy                                              Yes            71.039604\n",
       "Employment                                            No           100.000000\n",
       "Environment                                           No           100.000000\n",
       "Equality                                              No           100.000000\n",
       "Exports                                               No            82.379863\n",
       "Health                                                No           100.000000\n",
       "Internet                                             Yes            37.190083\n",
       "Mortality                                             No           100.000000\n",
       "Principal                                             No           100.000000"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "secondary=final.loc[final['Level']=='secondary']\n",
    "secondary=pd.merge(secondary,minprimary, left_on='Group',right_on='Group')\n",
    "secondaryp=secondary.loc[:,['Group','Min']]\n",
    "Global_Count=secondaryp.groupby('Group').count()\n",
    "Global_Count.rename(columns={'Min':'Global Count'},inplace=True)\n",
    "secondary['H_0']=np.where(secondary['Number of times repeated']-secondary['Min']>0,'Not Discarded', 'Denied')\n",
    "seco=secondary.groupby(['H_0','Group']).count()\n",
    "sec=seco.loc['Not Discarded']\n",
    "secondarycount=sec.drop(columns=['Indicator','R^2 Spearman','Behaviour','Country','Moved','Type','Continent','Number of times repeated','Level'])\n",
    "secondarycount.rename(columns={'Min':'Secondary Count'},inplace=True)\n",
    "continentlist=final['Continent'].unique()\n",
    "namescontinents=['European', 'North African', 'Asian', 'Pair', 'Persian', 'South African', 'Latino-American']\n",
    "finalcount=pd.merge(Global_Count,secondarycount, left_on='Group',right_on='Group')\n",
    "finalcount['Does it have some global casuallity implied?']=np.where(finalcount['Secondary Count']/finalcount['Global Count']>limitb.value,'No', 'Yes')\n",
    "finalcount['% of count (Global)']=finalcount['Secondary Count']/finalcount['Global Count']*100\n",
    "finalcount.drop(columns=['Global Count','Secondary Count'],inplace=True)\n",
    "finalcount"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, in a Global situation for the groups that have a **NO**, we do not need to worry about casualities, however for the rest of the groups correlation can still be a great indicator as a basis for decision making, if we carefully analyze the variables and found some sort of real relationship between them.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Does it have some global casuallity implied?</th>\n",
       "      <th>% of count (Global)</th>\n",
       "      <th>Does it have some European casuallity implied?</th>\n",
       "      <th>% of count (European)</th>\n",
       "      <th>Does it have some North African casuallity implied?</th>\n",
       "      <th>% of count (North African)</th>\n",
       "      <th>Does it have some Asian casuallity implied?</th>\n",
       "      <th>% of count (Asian)</th>\n",
       "      <th>Does it have some Pair casuallity implied?</th>\n",
       "      <th>% of count (Pair)</th>\n",
       "      <th>Does it have some Persian casuallity implied?</th>\n",
       "      <th>% of count (Persian)</th>\n",
       "      <th>Does it have some South African casuallity implied?</th>\n",
       "      <th>% of count (South African)</th>\n",
       "      <th>Does it have some Latino-American casuallity implied?</th>\n",
       "      <th>% of count (Latino-American)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Agriculture</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Demography</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Economy</th>\n",
       "      <td>Yes</td>\n",
       "      <td>71.039604</td>\n",
       "      <td>Yes</td>\n",
       "      <td>75.799087</td>\n",
       "      <td>Yes</td>\n",
       "      <td>71.812081</td>\n",
       "      <td>Yes</td>\n",
       "      <td>72.794118</td>\n",
       "      <td>Yes</td>\n",
       "      <td>70.348837</td>\n",
       "      <td>Yes</td>\n",
       "      <td>66.968326</td>\n",
       "      <td>Yes</td>\n",
       "      <td>70.542636</td>\n",
       "      <td>Yes</td>\n",
       "      <td>66.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Employment</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Environment</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Equality</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Exports</th>\n",
       "      <td>No</td>\n",
       "      <td>82.379863</td>\n",
       "      <td>Yes</td>\n",
       "      <td>77.142857</td>\n",
       "      <td>No</td>\n",
       "      <td>80.701754</td>\n",
       "      <td>No</td>\n",
       "      <td>86.842105</td>\n",
       "      <td>No</td>\n",
       "      <td>84.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>86.075949</td>\n",
       "      <td>No</td>\n",
       "      <td>83.333333</td>\n",
       "      <td>No</td>\n",
       "      <td>83.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Health</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Internet</th>\n",
       "      <td>Yes</td>\n",
       "      <td>37.190083</td>\n",
       "      <td>Yes</td>\n",
       "      <td>47.619048</td>\n",
       "      <td>Yes</td>\n",
       "      <td>35.294118</td>\n",
       "      <td>Yes</td>\n",
       "      <td>26.315789</td>\n",
       "      <td>Yes</td>\n",
       "      <td>42.857143</td>\n",
       "      <td>Yes</td>\n",
       "      <td>36.842105</td>\n",
       "      <td>Yes</td>\n",
       "      <td>37.500000</td>\n",
       "      <td>Yes</td>\n",
       "      <td>28.571429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mortality</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Principal</th>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>No</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Does it have some global casuallity implied?  % of count (Global)  \\\n",
       "Group                                                                           \n",
       "Agriculture                                           No           100.000000   \n",
       "Demography                                            No           100.000000   \n",
       "Economy                                              Yes            71.039604   \n",
       "Employment                                            No           100.000000   \n",
       "Environment                                           No           100.000000   \n",
       "Equality                                              No           100.000000   \n",
       "Exports                                               No            82.379863   \n",
       "Health                                                No           100.000000   \n",
       "Internet                                             Yes            37.190083   \n",
       "Mortality                                             No           100.000000   \n",
       "Principal                                             No           100.000000   \n",
       "\n",
       "            Does it have some European casuallity implied?  \\\n",
       "Group                                                        \n",
       "Agriculture                                             No   \n",
       "Demography                                              No   \n",
       "Economy                                                Yes   \n",
       "Employment                                              No   \n",
       "Environment                                             No   \n",
       "Equality                                                No   \n",
       "Exports                                                Yes   \n",
       "Health                                                  No   \n",
       "Internet                                               Yes   \n",
       "Mortality                                               No   \n",
       "Principal                                               No   \n",
       "\n",
       "             % of count (European)  \\\n",
       "Group                                \n",
       "Agriculture             100.000000   \n",
       "Demography              100.000000   \n",
       "Economy                  75.799087   \n",
       "Employment              100.000000   \n",
       "Environment             100.000000   \n",
       "Equality                100.000000   \n",
       "Exports                  77.142857   \n",
       "Health                  100.000000   \n",
       "Internet                 47.619048   \n",
       "Mortality               100.000000   \n",
       "Principal               100.000000   \n",
       "\n",
       "            Does it have some North African casuallity implied?  \\\n",
       "Group                                                             \n",
       "Agriculture                                                 No    \n",
       "Demography                                                  No    \n",
       "Economy                                                    Yes    \n",
       "Employment                                                  No    \n",
       "Environment                                                 No    \n",
       "Equality                                                    No    \n",
       "Exports                                                     No    \n",
       "Health                                                      No    \n",
       "Internet                                                   Yes    \n",
       "Mortality                                                   No    \n",
       "Principal                                                   No    \n",
       "\n",
       "             % of count (North African)  \\\n",
       "Group                                     \n",
       "Agriculture                  100.000000   \n",
       "Demography                   100.000000   \n",
       "Economy                       71.812081   \n",
       "Employment                   100.000000   \n",
       "Environment                  100.000000   \n",
       "Equality                     100.000000   \n",
       "Exports                       80.701754   \n",
       "Health                       100.000000   \n",
       "Internet                      35.294118   \n",
       "Mortality                    100.000000   \n",
       "Principal                    100.000000   \n",
       "\n",
       "            Does it have some Asian casuallity implied?  % of count (Asian)  \\\n",
       "Group                                                                         \n",
       "Agriculture                                          No          100.000000   \n",
       "Demography                                           No          100.000000   \n",
       "Economy                                             Yes           72.794118   \n",
       "Employment                                           No          100.000000   \n",
       "Environment                                          No          100.000000   \n",
       "Equality                                             No          100.000000   \n",
       "Exports                                              No           86.842105   \n",
       "Health                                               No          100.000000   \n",
       "Internet                                            Yes           26.315789   \n",
       "Mortality                                            No          100.000000   \n",
       "Principal                                            No          100.000000   \n",
       "\n",
       "            Does it have some Pair casuallity implied?  % of count (Pair)  \\\n",
       "Group                                                                       \n",
       "Agriculture                                         No         100.000000   \n",
       "Demography                                          No         100.000000   \n",
       "Economy                                            Yes          70.348837   \n",
       "Employment                                          No         100.000000   \n",
       "Environment                                         No         100.000000   \n",
       "Equality                                            No         100.000000   \n",
       "Exports                                             No          84.000000   \n",
       "Health                                              No         100.000000   \n",
       "Internet                                           Yes          42.857143   \n",
       "Mortality                                           No         100.000000   \n",
       "Principal                                           No         100.000000   \n",
       "\n",
       "            Does it have some Persian casuallity implied?  \\\n",
       "Group                                                       \n",
       "Agriculture                                            No   \n",
       "Demography                                             No   \n",
       "Economy                                               Yes   \n",
       "Employment                                             No   \n",
       "Environment                                            No   \n",
       "Equality                                               No   \n",
       "Exports                                                No   \n",
       "Health                                                 No   \n",
       "Internet                                              Yes   \n",
       "Mortality                                              No   \n",
       "Principal                                              No   \n",
       "\n",
       "             % of count (Persian)  \\\n",
       "Group                               \n",
       "Agriculture            100.000000   \n",
       "Demography             100.000000   \n",
       "Economy                 66.968326   \n",
       "Employment             100.000000   \n",
       "Environment            100.000000   \n",
       "Equality               100.000000   \n",
       "Exports                 86.075949   \n",
       "Health                 100.000000   \n",
       "Internet                36.842105   \n",
       "Mortality              100.000000   \n",
       "Principal              100.000000   \n",
       "\n",
       "            Does it have some South African casuallity implied?  \\\n",
       "Group                                                             \n",
       "Agriculture                                                 No    \n",
       "Demography                                                  No    \n",
       "Economy                                                    Yes    \n",
       "Employment                                                  No    \n",
       "Environment                                                 No    \n",
       "Equality                                                    No    \n",
       "Exports                                                     No    \n",
       "Health                                                      No    \n",
       "Internet                                                   Yes    \n",
       "Mortality                                                   No    \n",
       "Principal                                                   No    \n",
       "\n",
       "             % of count (South African)  \\\n",
       "Group                                     \n",
       "Agriculture                  100.000000   \n",
       "Demography                   100.000000   \n",
       "Economy                       70.542636   \n",
       "Employment                   100.000000   \n",
       "Environment                  100.000000   \n",
       "Equality                     100.000000   \n",
       "Exports                       83.333333   \n",
       "Health                       100.000000   \n",
       "Internet                      37.500000   \n",
       "Mortality                    100.000000   \n",
       "Principal                    100.000000   \n",
       "\n",
       "            Does it have some Latino-American casuallity implied?  \\\n",
       "Group                                                               \n",
       "Agriculture                                                 No      \n",
       "Demography                                                  No      \n",
       "Economy                                                    Yes      \n",
       "Employment                                                  No      \n",
       "Environment                                                 No      \n",
       "Equality                                                    No      \n",
       "Exports                                                     No      \n",
       "Health                                                      No      \n",
       "Internet                                                   Yes      \n",
       "Mortality                                                   No      \n",
       "Principal                                                   No      \n",
       "\n",
       "             % of count (Latino-American)  \n",
       "Group                                      \n",
       "Agriculture                    100.000000  \n",
       "Demography                     100.000000  \n",
       "Economy                         66.666667  \n",
       "Employment                     100.000000  \n",
       "Environment                    100.000000  \n",
       "Equality                       100.000000  \n",
       "Exports                         83.333333  \n",
       "Health                         100.000000  \n",
       "Internet                        28.571429  \n",
       "Mortality                      100.000000  \n",
       "Principal                      100.000000  "
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for i in range(0,len(continentlist)):\n",
    "    apfinal=final.loc[final['Continent']==continentlist[i]]\n",
    "    \n",
    "    selected_p=categories.loc[categories['Level']=='primary']\n",
    "    minprimary=selected_p.groupby('Group').min()\n",
    "    minprimary['Min']=round(minprimary['Number of times repeated']*limita.value)\n",
    "    minprimary.drop(columns=['Indicator','Number of times repeated','Level'], inplace=True)\n",
    "\n",
    "    grouplist=minprimary.index.to_list()\n",
    "\n",
    "    secondary=apfinal.loc[apfinal['Level']=='secondary']\n",
    "    secondary=pd.merge(secondary,minprimary, left_on='Group',right_on='Group')\n",
    "\n",
    "    secondaryp=secondary.loc[:,['Group','Min']]\n",
    "    Global_Count=secondaryp.groupby('Group').count()\n",
    "    Global_Count.rename(columns={'Min':'Global Count'},inplace=True)\n",
    "\n",
    "    secondary['H_0']=np.where(secondary['Number of times repeated']-secondary['Min']>0,'Not Discarded', 'Denied')\n",
    "    seco=secondary.groupby(['H_0','Group']).count()\n",
    "    sec=seco.loc['Not Discarded']\n",
    "    secondarycount=sec.drop(columns=['Indicator','R^2 Spearman','Behaviour','Country','Moved','Type','Continent','Number of times repeated','Level'])\n",
    "    secondarycount.rename(columns={'Min':'Secondary Count'},inplace=True)\n",
    "\n",
    "    apfinalcount=pd.merge(Global_Count,secondarycount, left_on='Group',right_on='Group')\n",
    "    apfinalcount['Does it have some '+namescontinents[i]+' casuallity implied?']=np.where(apfinalcount['Secondary Count']/apfinalcount['Global Count']>limitb.value,'No', 'Yes')\n",
    "    apfinalcount['% of count ('+namescontinents[i]+')']=apfinalcount['Secondary Count']/apfinalcount['Global Count']*100\n",
    "    apfinalcount.drop(columns=['Global Count','Secondary Count'],inplace=True)\n",
    "    finalcount=pd.merge(finalcount,apfinalcount, left_on='Group',right_on='Group')\n",
    "\n",
    "finalcount"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can observe into much more detail the differnt regions that we defined before, and for groups that have a **NO**, we do not need to worry about casualities. Meanwhile, for the rest of the groups correlation can still be a great indicator as a basis for decision making, if we carefully analyze the variables and found some sort of real relationship between them.   "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.4 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "e21cf16a31979fe9d2d6e9786ebc932e404e707404f15260e38504afc6c53159"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}