Chapter 2 - Bayes' Theorem.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#-Concept-of-Bayes'-Theorem\" data-toc-modified-id=\"-Concept-of-Bayes'-Theorem-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Concept of Bayes' Theorem</font></a></span><ul class=\"toc-item\"><li><span><a href=\"#-Example-1\" data-toc-modified-id=\"-Example-1-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 1</font></a></span></li><li><span><a href=\"#-Example-2\" data-toc-modified-id=\"-Example-2-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 2</font></a></span></li><li><span><a href=\"#-Example-3\" data-toc-modified-id=\"-Example-3-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 3</font></a></span></li><li><span><a href=\"#-Example-4\" data-toc-modified-id=\"-Example-4-1.4\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 4</font></a></span></li><li><span><a href=\"#-Example-5\" data-toc-modified-id=\"-Example-5-1.5\"><span class=\"toc-item-num\">1.5&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 5</font></a></span></li><li><span><a href=\"#-Example-6\" data-toc-modified-id=\"-Example-6-1.6\"><span class=\"toc-item-num\">1.6&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 6</font></a></span></li></ul></li></ul></div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font face=\"gotham\" color=\"purple\"> Concept of Bayes' Theorem</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **Bayes' Theorem** is occasionally called **inverse probability theorem**. As the name suggests, once we have known $p(A|B)$, the Bayes' Theorem can tell us something about $P(B|A)$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see how Bayes' Theorem is derived, start from conditional probability\n",
    "\n",
    "$$\n",
    "P(A|B)P(B)=P(A\\cap B)\\\\\n",
    "P(B|A)P(A)=P(A\\cap B)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Equate them and arrange, we obtain the famous Bayes' Theorem:\n",
    "\n",
    "$$\n",
    "P(B|A) = \\frac{P(A|B)P(B)}{P(A)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there are $k$ partitions of $B$, $P(A)$ can bee explicitly written in total probability form\n",
    "\n",
    "$$\n",
    "P(B_i|A)=\\frac{P(A|B_i)P(B_i)}{\\sum_{i=0}^k P(A|B_i)P(B_i)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since Bayes' Theorem can be somewhat counter-intuitive, it might take some time to grasp the tricky logic. We will go through some examples, strongly advise to study them thoroughly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 1</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A certain disease affects about 1 out of 10,000 people. There is a test to check whether the person has the disease. The test is quite accurate. In particular, we know that\n",
    "\n",
    "1. the probability that the test result is positive (suggesting the person has the disease), given that the person does not have the disease, is only 2 percent;\n",
    "\n",
    "2. the probability that the test result is negative (suggesting the person does not have the disease), given that the person has the disease, is only 1 percent.\n",
    "\n",
    "A random person gets tested for the disease and the result comes back positive. What is the probability that the person has the disease?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "$\\text{Event A}$: $\\text{the test result is positive}$<br>\n",
    "$\\text{Event B}$: $\\text{the person has the disease}$\n",
    "\n",
    "\n",
    "\n",
    "$$P(B|A) = \\frac{P(A|B)P(B)}{P(A|B)P(B)+P(A|B^c)P(B^c)}= \\frac{(1- .01)\\times.0001}{(1- .01)\\times.0001+.02\\times(1-.0001)}$$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability that a person have positive results and having the disease is 0.493%.\n"
     ]
    }
   ],
   "source": [
    "pB_A =( .99*.0001)/(.99*.0001+.02*.999)\n",
    "print('The probability that a person have positive results and having the disease is %.3f%%.'%(pB_A*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "The result comes with a surprise. But if you dwell on it a while, you will notice an obvious fact that there are much more 'healthy people'(people who do not have this type of disease) i.e.  $9999:1$. Even if you got to guess, you should consider this fact, which we call **prior**. It means even if a person has a positive result, yet he is still likely to be from healthy group.<br><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 2</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose a patient has an unidentified lesion in the liver, that internist recommend abdominal MRI for firmer evidences to exclude critical diseases. The sensitivity (true positive) rate of MRI is $92\\%$, the prevalence of critical liver disease is $.06\\%$. \n",
    "\n",
    "The question is if the result is positive, how likely this patient has critical liver disease?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "$\\text{Event A}$: $\\text{MRI positive results}$<br>\n",
    "$\\text{Event B}$: $\\text{the person has critical disease}$\n",
    "\n",
    "$$p(B|A) = \\frac{P(A|B)P(B)}{P(A|B)P(B)+P(A|B^C)P(B^C)} = \\frac{.92\\times .0006}{.92\\times .0006+.08\\times .9994}$$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability that a person who has symptoms truly have critical liver disease is 0.686%.\n"
     ]
    }
   ],
   "source": [
    "pB_A = (0.92*.0006)/(.92*.0006+.08*.9994)\n",
    "print('The probability that a person who has symptoms truly have critical liver disease is %.3f%%.'%(pB_A*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 3</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A factory produces an item using three different brands of machines—$\\text{A}$, $\\text{B}$, and $\\text{C}$—which account for $20\\%$, $30\\%$, and $50\\%$ of output respectively. Of the items produced by machine $\\text{A}$, $5\\%$ are defective; similarly, $3\\%$ of machine $\\text{B}$'s items and $1\\%$ of machine $\\text{C}$'s are defective. If a randomly selected item is defective, what is the probability it was produced by machine $\\text{C}$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "$\\text{Event A}$: $\\text{Produced by machine A}$<br>\n",
    "$\\text{Event B}$: $\\text{Produced by machine B}$<br>\n",
    "$\\text{Event C}$: $\\text{Produced by machine C}$<br>  \n",
    "$\\text{Event D}$: $\\text{Defective product}$  \n",
    "$$\n",
    "P(C|D) = \\frac{P(D|C)P(C)}{P(D|A)P(A)+P(D|B)P(B)+P(D|C)P(C)}\n",
    "$$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability that a defective product made by machine C is 20.833%.\n"
     ]
    }
   ],
   "source": [
    "pB_A = .5*.01/(.2*.05+.3*.03+.5*.01)\n",
    "print('The probability that a defective product made by machine C is %.3f%%.'%(pB_A*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 4</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bag I contains 4 white and 6 black balls while Bag II contains 4 white and 3 black balls. One ball is drawn randomly from one of the bags and it is found to be black. Find the probability that it was drawn from Bag I."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "$\\text{Event A}$: $\\text{black ball}$<br>\n",
    "$\\text{Event B}$: $\\text{from bag I}$<br>\n",
    "\n",
    "$$p(B|A) = \\frac{p(A|B)p(B)}{p(A)}= \\frac{\\frac{6}{10}\\frac{1}{2}}{\\frac{6}{10}\\frac{1}{2}+\\frac{3}{7}\\frac{1}{2}}$$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability that the ball is from Bag I given it is black is 58.33333%.\n"
     ]
    }
   ],
   "source": [
    "pB_A = 6/10*.5/(6/10*.5+3/7*.5)\n",
    "print('The probability that the ball is from Bag I given it is black is %.5f%%.'%(pB_A*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 5</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A man is known to speak truth 2 out of 3 times. He throws a die and reports that number obtained is a four. Find the probability that the number obtained is actually a four."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "$\\text{Event A}$: $\\text{reports number 4}$<br>\n",
    "$\\text{Event B}$: $\\text{speak truth}$<br>\n",
    "$$   \n",
    "P(A|B)=\\frac{P(B|A)P(A)}{P(B|A)P(A)+P(B^C|A)P(A)}=\\frac{\\frac{2}{3}\\frac{1}{6}}{\\frac{2}{3}\\frac{1}{6}+\\frac{1}{3}\\frac{5}{6}}\n",
    "$$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability that he truly obtains 4 given he speaks the truth is 28.571%.\n"
     ]
    }
   ],
   "source": [
    "pB_A = (1/6*2/3)/(1/6*2/3+5/6*1/3)\n",
    "print('The probability that he truly obtains 4 given he speaks the truth is %.3f%%.'%(pB_A*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font face=\"gotham\" color=\"purple\"> Bayesian Inference </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bayes' Theorem won't be much helpful in scientific researches if it is only a tool for calculate the inverse probability. Without changing the theory, just reframe the definition, we will have a completely revolutionary view. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\text{Event A}$: $\\text{Newly collected data}$<br>\n",
    "$\\text{Event B}$: $\\text{Hypothesis}$<br>\n",
    "$$\n",
    "P(B|A) = \\frac{P(A|B)P(B)}{P(A)} = \\frac{P(A|B)P(B)}{P(A|B)P(B)+P(A|B^c)P(B^c)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hypothesis $\\text{B}$ has a probability $P(B)$ which is called **prior** in **Bayesian Inference**, $P(A|B)$ is the **likelihood** (exactly the same meaning as in maximum likelihood), $P(B|A)$ is the **posterior** meaning which is also your new prior before next data collection.\n",
    "\n",
    "The reason why $P(A|B)$ is called likelihood is actually quite intuitive, it measures how likely data $A$ will present given $B$ hypothesis is true. $P(A)$ is calculated by law of total probability based on $B$ and $B^C$ hypothesis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If $B$ is called a hypothesis, then it expresses a subjective belief. Broadly speaking, you start with a scientific question and set forth two or more alternative hypotheses. You then assign a prior probability that each alternative hypothesis is true. Next, you collect data. Finally, you use Bayes’ Theorem to update the probability for each hypothesis considered."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we change the prior gradually in the first example above, from $1$ to $1000$ people out of $10000$, and rest parameters are the same, let's see how the posteriors change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtMAAAGHCAYAAAByA95NAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA+5ElEQVR4nO3deXxU1eH+8edMAoQQEpIZkkASFgERwQWIoCAgkOJuca17kS9Vi1ttbd2tXbDYuv1UKLYgUu2iVutW1wiKEheURXYIArIkZGdJSEhyz++PC5E0LCHL3Ezm83698kpm5k7muRy1Ty/nnmOstVYAAAAAjprP6wAAAABAqKJMAwAAAA1EmQYAAAAaiDINAAAANBBlGgAAAGggyjQAAADQQJRpAMBhPfjgg+rdu7fXMQCgRaJMA0AzmjBhgowxdb5iYmK8jhZ0f/nLXzR27FglJCTIGKNPP/3U60gA0GiUaQBoZiNGjFBOTk6tr2+//dbrWEFXVlamMWPG6E9/+pPXUQCgyVCmAaCZtW3bVsnJybW+EhMTJUmFhYVKS0vTbbfdVnN8Xl6eunTpojvvvFOSZK3VT37yE/Xq1Uvt27fXMccco3vuuUcVFRU179k/FeOll15Snz59FB0drfHjx2vnzp169dVX1bdvX3Xs2FGXXHKJduzYUfO+CRMmKCMjQ4899phSUlIUHR2tiy++WAUFBYc9pw8++EDDhw9X+/btlZKSouuuu06FhYWHfc/PfvYz3XvvvRo7duxR/xkCQEtFmQYAD/n9fv3973/X9OnT9eabb8paq2uuuUbdu3fX73//e0lumU5KStI//vEPrVq1Sk888YRmz56thx56qNbvysnJ0Zw5c/TKK6/onXfe0YIFC3TJJZdo5syZeumll/T222/rk08+qfO+L7/8Uh999JHeffddvf322/rmm280ceLEQ2aeO3eufvjDH+ryyy/XN998o9dee00bN27UhRdeKGtt0/8hAUALFul1AABo7T766KM6c6RHjx6tN998U5I0cuRI3Xfffbruuuv04x//WF988YUWL16sNm3aSJJ8Pl9NsZakHj16aP369Zo+fbp+85vf1DxfUVGhOXPmKBAISJIuu+wyzZgxQ7m5uercubMk6fLLL9eHH35YK4vjOHr++ecVFxcnSZo2bZrOPPNMrVu3Tn369KlzPr/97W9166236pZbbql5bs6cOerevbuWLl2qk08+uaF/VAAQcijTANDMhg4dqjlz5tR6Ljo6utbj+++/X++9954ee+wx/etf/1LPnj1rvf7Xv/5VM2fO1MaNG1VaWqqqqio5jlPrmJSUlJoiLalmSsn+Ir3/uby8vFrvO/7442uKtCQNHz5ckrRq1aqDlumFCxfq888/19NPP13ntXXr1lGmAYQVyjQANLP27dsfcWm5nJwcrV27VhEREVq7dm2t115++WXddNNNmjp1qkaNGqXY2Fi9/PLLuvfee2sdt/9K9n7GmIM+978l/Gg5jqM777xT11xzTZ3XkpOTG/W7ASDUUKYBwGOO4+jqq69W//79dfvtt+vSSy/V6NGjdfrpp0uS5s+fr4EDB+rnP/95zXs2btzYZJ+/atUq7dy5U7GxsZKkrKwsSVK/fv0Oenx6erpWrFjB2tMAIMo0ADS7vXv3Kjc3t87zSUlJMsZoypQpWrZsmZYsWaLU1FTdeOONuuqqq7RkyRLFx8erb9++mjVrll5//XUNGDBAb731ll599dUmy2eM0bXXXqvf//73Kioq0k033aRzzz33oFM8JHfO9Lhx43T77bfrxz/+sTp27Kh169bp5Zdf1tNPP6327dsf9H25ubnKzc3Vtm3bJEnZ2dmKiYmpmY4CAKGI1TwAoJl98skn6tKlS52vwsJCZWVl6be//a2effZZpaamSpIeeeQRderUSZMmTZIk3XDDDbrmmmt03XXXaeDAgfriiy/04IMPNlm+IUOG6PTTT9cPfvADnXnmmerfv79mz559yONHjx6tuXPnatmyZRoxYoROPPFE3X777erYsWOdaSUHmjFjhgYOHKhzzz1XkmrOZ8aMGU12LgAQbMayjhEAhK0JEyZoy5YtyszM9DoKAIQkrkwDAAAADUSZBgAAABqIaR4AAABAA3FlGgAAAGggyjQAAADQQJRpAAAAoIFCftOW/Yv/B1MgEFBBQUHQPxfBxTiHB8Y5PDDOrR9jHB68HOeuXbse9HmuTAMAAAANRJkGAAAAGogyDQAAADQQZRoAAABoIMo0AAAA0ECUaQAAAKCBKNMAAABAA1GmAQAAgAaiTAMAAAANRJkGAAAAGogyDQAAADRQpNcBAAAAgIOx1koV5dKuHdKuHarYsFo2rZdMZBuvo9WgTAMAACBobFWltGuntKtE2rlDdteOmp+1a9/jnSVugd69Q9q7t+a9JZJ8U2dJ/s7ehD8IyjQAAAAaxZbvcQvwvi+7vwzX/Pz9Y5WVHvyXRLaRYuOkjp2kjrEyXbtJHeP2PRcn0zFOndK6q6RDp+CcVD1RpgEAAFCHrSiXdhTvK8jFbineccDPB37trTj4L+nQcV8h7iST0kPq18ktx7GdZDp2qnlNsXFSu/Yyxhw2U5tAQKagoKlOsUlQpgEAAMKEra52rxLvKJZ2FMvuL8s7imV3FteUZe0okSr21P0FxrgFOS7eLcS9jttXht0vs//n/VeXW9Dc5uZCmQYAAAhxdm+FW5BLiqQdRbI7SqQdRfsKc1FNedbunZK1dX9BdAcpNl6Ki5fp3ntfWY6X4vYX5Ph9JTlOJiIiyGfXslGmAQAAWqgDS7ItKZJ2FLqFuebxvpK85yDzkCMiagqy/Ikyx/StKclm35VldUpwryi3aRv0c2stKNMAAABBZqur9xXhIqm4ULbkgJK8o6jmZ5XtrvvmyDZuKe6UIHXtJtPvJPfnuAS3JHeKl+ISpA4dZXxsKdLcKNMAAABNyJaXScVFUkmhbHGBVOwWZVtc8H1J3lkiWaf2GyMi3JIclyAldZXpO8D9uZNfplOCW5g7JUjRMUe8UQ/BQ5kGAACoB2utu6xbcb57NXl/US4ukC0ukooLpJJCaU9Z3Td36OgW4Xi/TFrPfcXYL9PJv+/5BCkmjivJIYgyDQAAIMnuKZOK8qWiAtli93tNaS4qcMvy/y4BZ3xSXCcpPiB1SZU5/mQp3v99UY7fV5rbtvPilBAElGkAANDq2apK9ypyYZ7svsKsovzaRfl/rygbnzvtIiEgpXaXOSHdvbKcEHDLc7zfvZkvkjoVzhh9AAAQ0qy1cnbtlP3uW6koT7awQCrKkwrzvy/OO4vrLgnXMU5K6CwldpU57kS3NCd0lonfV5Y7JbAMHI6IMg0AAFo0a617w17BdrccF+a5V5gL9/+cr/z/3WCkTVu3KPs7y5ww2P05ISCT0NktygkBpl6gSQStTC9ZskSzZ8+W4zgaO3asxo8fX+v1srIyPfnkkyosLFR1dbXOP/98jR49OljxAACAR2qV5YLtNQXZFn7/syr31n5TdIzk33dVud9J6tCtp8raRbvPJXR2NxdhxQsEQVDKtOM4mjVrlu677z75/X7dfffdSk9PV2pqas0x7777rlJTU3XXXXdp586duu222zRixAhFMg8JAICQZ0t3SwXbvy/MB3xXYV7dshwTK/kTpZTuMicOkQKJMgmJUiDRvdocFV3r8A6BgPYUFATxjABXUJpqdna2kpOTlZSUJEkaNmyYFi5cWKtMG2NUXl4ua63Ky8sVExMjH8vDAAAQEmxVpXsFOT93X0nOlc13vyt/e90d+qJjpECSu+nIienuDn2BJMmftK8st/fmRICjFJQyXVRUJL/fX/PY7/dr3bp1tY4566yz9Mc//lE33HCD9uzZo9tvv50yDQBAC2LLdrtlOS9Xys9xry7n5Uj5ue5KGQduQhLZxi3LgSSZXse53wPJ3z8X3cG7EwGaUFDKtP3fu2elOvOYli5dqu7du+uBBx7Q9u3b9bvf/U7HHXecoqNr/zVOZmamMjMzJUlTp05VIBBovuCHEBkZ6cnnIrgY5/DAOIcHxrl+rLVydhSrOmeL+5W7VdW5W1SVu1XVOVtkd++sdbwvLl6RySmKGDBQEUkpikhOUURyV0UkpcgX7w/qBiSMcXhoieMclDLt9/tVWFhY87iwsFDx8fG1jpk3b57Gjx8vY4ySk5OVmJiobdu2qXfv3rWOy8jIUEZGRs3jAg/mRwUCAU8+F8HFOIcHxjk8MM7fq7nZb/s22bxtUl6OlJcjm+9+V/kBq2IYn3tDX+dkmcHDZDp3kemcLCUmS4Fkmaj2ciQ5kiprfYikoqJgnhZjHCa8HOeuXbse9PmglOlevXopJydHeXl5SkhIUFZWlm699dZaxwQCAS1btkz9+vVTSUmJtm3bpsTExGDEAwCg1bG7d7qFefs2afs2aftWtzBvz5EOXEYuIkIKJEuJXWT69Jc6d5FJ7CJ1TnZv+ots491JACEgKGU6IiJCEydO1JQpU+Q4jkaPHq20tDS9//77kqRx48bp4osv1vTp0/WLX/xCknTVVVcpNjY2GPEAAAhJdm+FlOeWZZu7Vcrd6l5t3r5NKt31/YE+nztXObHr94U5qYuU2NW98Y+NSYAGM/ZgE5pDyLZt24L+mfxVUnhgnMMD4xweQnmcrbVSSZGUu2VfYd73fftWqSi/9q5+8QH3CnNyiluck1KkpK7uDX+tfKnZUB5j1F/YTvMAAACHZysr3TnLuVtkczYfUJ631p6W0a69lJwi07uflJTh/pzU1S3PLCcHBB1lGgCAILLlZVLOvsJ8wHcV5ErOAUvLJXSWklNlTt9fmFOkLqlSXAI7+wEtCGUaAIBmYMt2u2V523fSts3u95zNUvEBf0UdEelOw0jrITNkhFuek1Pd8twuyrvwAOqNMg0AQCPYPWXStu/2lebvv6vkgKXh2raVktNk+g6QuqTJdElzrzJ37sLNf0CIo0wDAFAPtnKve6V56yZp6yb3+7ZNUtEBV5r3l+bjTnK3ye7aTeqa5q6Ywa6+QKtEmQYA4ADWcaTCPGnLRtmtG6Ut+4pz3rbv5zRHREpdUmV6H++W5pRuUtfu7rrMPq40A+GEMg0ACFu2rNQtzVs27Pu+0Z2iUVHuHmCMuz5zSg+ZwcPc7ynd3JUzWvlScwDqh/8SAABaPes4snk50pYNspvdL23Z6F6B3q9DRym1h8zwDPd7Snf3qjPLzQE4DMo0AKBVsZWV0rZNst99K+0rzvlbN7o3CkqS8bmrZRzTVxp5pkxaTymlhxTvZ8k5AEeNMg0ACFl2T5m0+Vu3OH/3rezmb93l56qr3QOi2kupPRR1xtkq79xFJrWnlNJNpm07b4MDaDUo0wCAkGBLd7mF+bv10qb1spvWuzcF7hcXL6UdI3PiKTLdjpHSekqBZBmfT7GBgPay1TSAZkCZBgC0OLZ0t/TdetmN2dKmbNlN2VLB9u8P8CdK3XvJDBvjFuduvWTi4r0LDCBsUaYBAJ6y5Xv2Fed10sZs93t+7vcHBJJkuveWRp4l072X1O0YmZhY7wIDwAEo0wCAoLFVle6GJ9+ulTauc4tzzhbJ7lu/OaGz1KO3zIhxbnHu3lumQ0dvQwPAYVCmAQDNwlor5efIblgnbVgru2Gt9N23UlWle0DHOKlHH5nBw2R69HF/ju3kaWYAOFqUaQBAk7Blpe7V5m/XyH67RtqwRtq9y32xbTt3jvOYc6Uex8r07ONusc1SdABCHGUaAHDUrONI27fKrl8trV/tlueczZK17gFd0mROGiL17Ouu59y1m0wE22wDaH0o0wCAI7IV5dK3a2TXr3YL9LdrpLLd7ovRMdIxfWVOOd0tzj36yETHeBsYAIKEMg0AqMMWF8pmr5LWr3K/b/5WchzJGPeq8+BhUq/jZI45TkrqKuPzeR0ZADxBmQaAMGetlXI2y2avlNatlF23UirMc19s29adqnH2JTK9+7lXoLnqDAA1KNMAEGZsdbW7k+C65W5xzl75/Y2CHeOkPv1lMs6X6XW8lNZTJpL/qQCAQ+G/kADQytmqSneVjbUrZNcul7JXSxV73BcTu7g3CvY+XqZPf/cxK2wAQL1RpgGglakpz2uWy65ZJq1fJe3d677YtZvMaWdIxw6Q6dNfplOCp1kBINRRpgEgxNnqarc8r/6mbnlO7Skz4kyZY/u70zc6xnkbFgBaGco0AIQY6zjSlo1ueV79jbRuhVS+b9pGao995XmAdGx/mZhYb8MCQCtHmQaAEGDzc2VXLZFWLpVd8833NwwmpcgMHSXT7yTp2BNkOlKeASCYKNMA0ALZ3Tul1d/Irloqu2qplJ/rvtDJL3NCutTvZJm+J8gkBLwNCgBhjjINAC2ArapydxhcuVh25RJp4zp3a+720VLfE2QyLpDpd7KUnMJqGwDQglCmAcAjNj9XdsUi2eWLpDXL3HnPPp/U81iZ8y6XOf5k9+eICK+jAgAOgTINAEFi91ZIa1fILv9adsUiKXer+4I/UWbIKJn+A6XjTmCHQQAIIZRpAGhGtmC77LKvZJd9La35xl2yrk1bqe8AmVFnywwYLCV1ZeoGAIQoyjQANCFbVSWtXyX7zVeyy76Scja7L3ROdpesGzDI3TClbTtvgwIAmgRlGgAayZbucovz0oXu9I09ZVJEpLvO88hxMgPSZZJTvI4JAGgGlGkAaACbu1V26Zey33wprVslWUeK7SQzeLjMiadI/U6UiYr2OiYAoJlRpgGgHqzjSBvWyi75QnbJF1LuFveF1B4yZ18ic9IpUo8+Mj6ft0EBAEFFmQaAQ7CVle7GKUs+l136pbSj2F26ru8JMmecI3PyEBl/otcxAQAeokwDwAFsRbm0fJHsos9kly105z+3a+/eOHjyUJkT0mU6sHQdAMBFmQYQ9mxZqew3X8ou+kxaschdvi6mo8ygYTKDTpP6nSTTpq3XMQEALRBlGkBYsmW7ZZd8qeJlC+Us+UKqqpI6JcgM/4FboPv0Z+dBAMARUaYBhA23QH8h+9UCaeUSqbpKVYEkmTPOlUkf7m7dzQ2EAICjQJkG0KrZinJ3CbuFn0jLv3avQCd0lhl7nsygYQqcMkyFRUVexwQAhCjKNIBWx1bulZZ9LbvwE3cd6L173SkcZ5wrc8rp7hXofdt3cyUaANAYlGkArYJ1qqU1y2W/+Ni9kXBPqRQTKzNsrMwpI6Tex1OcAQBNjjINIGRZa6XN37oF+sv5UkmRu4zdoFNlhoxyV+HgJkIAQDOiTAMIOba4UPaLj2Q/mydt+06KiJQGDJK5bJTMiUNk2rXzOiIAIExQpgGEBFtRLrv4M7dAr1oqWSv1Ok7mqp/KpA+XiYn1OiIAIAxRpgG0WNZaad0K2QUfyn69QKool/yJMuf+SOa0M2QSu3odEQAQ5ijTAFocW5Qv+9k82QWZUn6uFNVe5pQRMqeNkXr340ZCAECLQZkG0CLYykp3Q5UFH7gbqlgr9T1B5vwrZAadJtMuyuuIAADUQZkG4Cmbs0X2k/fcudC7d7obqpx7mbukXedkr+MBAHBYlGkAQWf3Vsh+tUD2k/el7JVSRIR00lD5RoyTjj9JxsdydgCA0ECZBhA0NmeL7MfvyH42VyorlRK7yFz8Y5lhY2Ri472OBwDAUaNMA2hWtqpKWvK5nI/ekdYskyIi3TnQo86Sjh1Qs603AAChiDINoFnYogLZ+e/KfvqBtKPYXdLuwmtkTs/gKjQAoNWgTANoMvvXhXbmviUt/txdkeOEdPnOOFvqP5C50ACAVocyDaDRbEWFu733vP9KWzZK0TEyP/ihzBnnyASSvI4HAECzoUwDaDBbVCA777+y89+TynZLqT1krr1ZZsgomXbtvI4HAECzo0wDOGp2w1rZD153t/i2kgadKt+Y86U+x3NDIQAgrFCmAdSLdaqlxV/IyXxdyl4ltY+WGXu+zJjzmMoBAAhblGkAh2UrKmSzPpT94DUpP1cKJMn8aJK7KkdUtNfxAADwFGUawEHZXTvd+dDz/utu831MX/kumSCdPJRVOQAA2IcyDaAWm58r+8Frsgsypb17pZOGyHfmRVLvfsyHBgDgfwStTC9ZskSzZ8+W4zgaO3asxo8fX+eYFStW6LnnnlN1dbU6duyo3/zmN8GKB4Q9u/U72Xf/LfvlfMn4ZE49Q2bceJmu3byOBgBAixWUMu04jmbNmqX77rtPfr9fd999t9LT05WamlpzTGlpqWbOnKl7771XgUBAO3bsCEY0IOzZDevkvP2ytORzqW0796bCH4yXifd7HQ0AgBYvKGU6OztbycnJSkpy7/gfNmyYFi5cWKtMf/rppxo6dKgCgYAkKS4uLhjRgLBl1yyX8/ZL0solUnQHmfMulxl7nkxMrNfRAAAIGUEp00VFRfL7v7/K5ff7tW7dulrH5OTkqKqqSg8++KD27Nmjc845R6NGjarzuzIzM5WZmSlJmjp1ak35DqbIyEhPPhfB1RrH2VqryuWLtPvFZ1W5YrF8nRIUfe1Nan/WePnad/A6nida4zijLsa59WOMw0NLHOeglGlrbZ3n/vdGpurqam3YsEH333+/9u7dq/vuu099+vRR165dax2XkZGhjIyMmscFBQXNE/owAoGAJ5+L4GpN42ytlVZ/I+fNf0rrVkpxCTKX/0QaMU572rbTntI9Uuker2N6ojWNMw6NcW79GOPw4OU4/28n3S8oZdrv96uwsLDmcWFhoeLj4+sc07FjR0VFRSkqKkr9+vXTpk2bDhkcwJHVlOg3/illr5Q6Jchccb3MiHEybdp6HQ8AgJDnC8aH9OrVSzk5OcrLy1NVVZWysrKUnp5e65j09HStXr1a1dXVqqioUHZ2tlJSUoIRD2iVbPZKOY/cK+ex+6WC7TJX3iDfQ3+Rb8x5FGkAAJpIUK5MR0REaOLEiZoyZYocx9Ho0aOVlpam999/X5I0btw4paam6uSTT9Ydd9whn8+nMWPGqFs3luQCjpbdtF7Oay9Iy7+WYjvJXP4TmZFnUqABAGgGxh5sQnMI2bZtW9A/k3lZ4SHUxtlu+07O6/+QFmVJ0TEyZ10sM+ZcmXZRXkdr0UJtnNEwjHPrxxiHh7CdMw2g+diifNk3/iGbNU9q107m/MtlMn4oEx2eq3MAABBMlGkgRNnS3bLvvCz74VuSrMwPLpA56xKZjqwTDQBAsFCmgRBjK/fKzn1L9u2XpT1l7rbfP7xKxp/odTQAAMIOZRoIEdZxZL/8WPY/z0tFBdKAwfJdfK1Mak+vowEAELYo00AIsOtWynlxprQpW+reW77rfiZz3IlexwIAIOxRpoEWzObnynnlOenrLKmTX2bi7TJDR8n4grJEPAAAOALKNNAC2bJS2bdfkv3wTckXIXPBlTLjxrPMHQAALQxlGmhBrOPIfjZP9tU50q4dMqeNkRl/tUy83+toAADgICjTQAthN66T88+/SN+ukXoeK98t98v06ON1LAAAcBiUacBjdtcO2f88L/vpB1LHOJnrbpM5dTTzogEACAGUacAj1qmW/fg92deelyrKZTIukDnvcnYuBAAghFCmAQ/YTevlvDBd2rhO6neSfFdcL9MlzetYAADgKFGmgSCye8pkX/+77Nz/Sh1jZSb9QmbISBljvI4GAAAagDINBIG1Vvp6gZx/zZR2Fsuccba7Skd0jNfRAABAI1CmgWZmC/Pl/P3P0rKvpG695LvpXpmerNIBAEBrQJkGmol1HNmP3pZ99XlJVuZHk2TGnCvji/A6GgAAaCKUaaAZ2JzNcuY8Ja1fLfUfKN81N8n4E72OBQAAmhhlGmhCtqpS9t1XZf/7otQ2Sua6n8mcNpobDAEAaKUo00ATsZs3yHn2cWnLRpn002Wu+IlMbLzXsQAAQDOiTAONZKurZd/5t+xbL0odYuSbfI/MwFO9jgUAAIKAMg00gt32nZxnn5A2ZcucMkLmyhtkYmK9jgUAAIKEMg00gHWqZT94Xfa1v0tR7eW74Vcy6ad7HQsAAAQZZRo4SjY/150bnb1KGniqfFf/lLnRAACEKco0UE/WWtnPP5L9xwzJGJn/u11m6Bms1AEAQBijTAP1YMt2y77wZ9mFn0h9jpfv/37OutEAAIAyDRyJXbtczqzHpR1FMuOvljn7YnYxBAAAkijTwCHZqio5/3le9p1/S52T5bvzYZmex3odCwAAtCCUaeAgbGG+ih+9V3b1MpnhGTKX/0Qmqr3XsQAAQAtDmQb+h136pZxnn5C1jsxP7pBvyEivIwEAgBaKMg3sY6sqZV/9m+wHr0vdjlHCnX9QSVuuRgMAgEOjTAOSbMF2OX/5k7Rhrczoc2QunajILl2lggKvowEAgBaMMo2w507reFyyVr4b75QZPNzrSAAAIETUq0xba5WXl6fOnTvL5/M1dyYgKKxTLfvGP2X/+5LUrZdbpDsnex0LAACEkHo1Y2OM7rjjjubOAgSNLd0l56nfyf73JZnhY+W7cypFGgAAHLV6T/Po0aOHcnJylJKS0px5gGZnv1sv589TpeJCmasny4w8ky3BAQBAg9S7TPfv318PPfSQRo0apUAgUOu1MWPGNHkwoDk4WXNlX5gudego36/+IHNMX68jAQCAEFbvMr1mzRolJiZq1apVdV6jTKOls9XVsi8/K/vhm1LfE+S7/pcysZ28jgUAAEJcvcv0r3/96+bMATQbW7pbzl/+KK1cIpNxgcwl18lERHgdCwAAtAJHtTTe7t279fXXX6uoqEgJCQkaPHiwYmJimisb0Gg2d4ucp34vFebJ/PgW+U7/gdeRAABAK1Lvde7Wrl2rW265RR988IE2bdqkzMxM3XLLLVq7dm1z5gMazC5fJOehX0p7SuX7xe8p0gAAoMnV+8r0c889p0mTJmn48O83tMjKytLs2bP1hz/8oVnCAQ1hrZX98E3Zl56VUrrJd/N9Mv5Er2MBAIBWqN5XpnNycnTaaafVeu7UU09Vbm5uk4cCGspWV8u+MF32xZnSSUPku/NhijQAAGg29S7TycnJysrKqvXcZ599pqSkpCYPBTSELS+T8/TvZOe/J3P2JfL99C6ZqPZexwIAAK1Yvad5TJgwQVOnTtU777yjQCCg/Px85eTk6K677mrOfEC92JJCOU/+Vtq6Seaam+QbeabXkQAAQBiod5nu27evnnrqKS1atEjFxcUaPHiwBg0axGoe8JzdstEt0mWl8t18v8wJg72OBAAAwsRRLY0XExOjkSNHNlcW4KjZVUvl/PkPUrsod0fDbsd4HQkAAISRw5bpKVOm6N5775UkPfDAAzLGHPS43/zmN02fDDgCJ2uu7N+ekpJT5bv1AZmEzl5HAgAAYeawZXrUqFE1P7NlOFoS5/3/yL48W+p3knw33iUT3cHrSAAAIAwdtkyffvrpkiTHcbR9+3ZddNFFatOmTVCCAQdjrZX9z99k33lFZvBwmf/7uQz/TAIAAI/Ua2k8n8+n9957TxEREc2dBzgk61TLPj/NLdIjz5K5/g6KNAAA8FS915keNWqUPvjgg+bMAhySrdwr55k/yn7yvsy5l8lc/VMZH//nDgAAeKveq3lkZ2fr3Xff1RtvvCG/31/rZkRuQERzsuVlcqY9JK3+RuZH/ydfxg+9jgQAACDpKMr02LFjNXbs2ObMAtRhS3fJeeJB6bv1MhNvl++00V5HAgAAqFHvMn3GGWc0YwygLrtrp5zH75dyNss3+R6Zk4Z4HQkAAKCWepdpa60+/PBDLViwQLt27dIjjzyilStXqqSkRMOGDWvOjAhDdmexnEfvl/Jz5bvpPpkBg7yOBAAAUEe9b0B88cUXNW/ePGVkZKigoECS5Pf79frrrzdbOIQnW1Io50/3SgXb5bvlfoo0AABosepdpj/++GPdeeedGj58eM3Nh4mJicrLy2u2cAg/tihfzp/ukYoL5bvtQZl+J3kdCQAA4JDqPc3DcRxFRUXVeq68vLzOc0BD2YLtch65VyrbLd/tv5HpdZzXkQAAAA6r3lemBw4cqL/97W+qrKyU5M6hfvHFFzV48OBmC4fwYQu2y/nT3dKeUvlu/x1FGgAAhIR6l+lrr71WRUVFmjBhgsrKynTttdcqPz9fV155ZXPmQxiwRflyHr1PKt8j3y9+L9Ozj9eRAAAA6qXe0zyio6P1q1/9Sjt27FB+fr4CgYA6derUjNEQDmxJkbtqx+6d8v389zLdenkdCQAAoN7qfWX6V7/6lSQpLi5OvXv3rinSd911V73ev2TJEt1222265ZZb9Nprrx3yuOzsbP3oRz/S559/Xt9oCFF21w45j90v7SiS77Zfc0UaAACEnHqX6dzc3DrPWWu1ffv2I77XcRzNmjVL99xzjx5//HEtWLBAW7ZsOehxf//733XyySfXNxZClC3dJeexB75f/q738V5HAgAAOGpHnObx9NNPS5Kqqqpqft4vPz9faWlpR/yQ7OxsJScnKykpSZI0bNgwLVy4UKmpqbWOe+eddzR06FCtX7++3ieA0GPLSuU8/mspd7N8N98v0/cEryMBAAA0yBGvTCclJdWU4P0/JyUlKTk5WaeffnrN9I/DKSoqkt/vr3ns9/tVVFRU55gvv/xS48aNO9pzQAix5XvkPPkbacsG+W68W6b/QK8jAQAANNgRr0xfeumlkqQ+ffo0ePqFtbbOc/s3ftnvueee01VXXSWf7/D9PjMzU5mZmZKkqVOnKhAINChTY0RGRnryuaHOVlaqZPoU7d2wVnF3/E5Rp432OtJhMc7hgXEOD4xz68cYh4eWOM71Xs0jMjJSeXl5SkxMVElJiV544QX5fD5deeWVR1zVw+/3q7CwsOZxYWGh4uPjax2zfv16/b//9/8kSTt37tTixYvl8/k0ZMiQWsdlZGQoIyOj5vH+rc2DKRAIePK5ocw6juysx2QXfyFz7c3a3ecE7W7hf4aMc3hgnMMD49z6Mcbhwctx7tq160Gfr/cNiLNmzaq5ajxnzhxVV1fLGKNnnnnmiO/t1auXcnJylJeXp6qqKmVlZSk9Pb3WMdOmTav5OvXUUzVp0qQ6RRqhyVor+9Is2S/ny1x0rXwjmMoDAABah3pfmS4qKlIgEFB1dbWWLl2q6dOnKzIyUjfccMMR3xsREaGJEydqypQpchxHo0ePVlpamt5//31JYp50K2fffln2wzdlMn4oc9bFXscBAABoMvUu0+3bt1dJSYk2b96s1NRURUVFqaqqSlVVVfV6/6BBgzRo0KBazx2qRN900031jYUWzpn/nuxrL8iceobMpdfVmSsPAAAQyupdps866yzdfffdqqqq0oQJEyRJq1evVkpKSnNlQ4iziz6TfeHP0oDBMj++VeYIN5cCAACEmnqX6fHjx2vIkCHy+XxKTk6WJCUkJOjGG29stnAIXXbtCjl/fUTq2Ue+G++Uiaz3P2oAAAAh46gaTlJSktasWaPs7GwlJCSob9++ioiIaK5sCFF2+zY50x+SAonu7obtoryOBAAA0CzqXaa3bt2qhx9+WHv37q1Z6q5Nmza688476+xkiPBld+90N2UxRr5bHpCJifU6EgAAQLOpd5meOXOmMjIydP7559fcRPbGG29o1qxZ+vWvf91sARE6bGWlnGkPSUUF8v3i9zKJXbyOBAAA0KzqfUfYxo0bdd5559VajeHcc8/Vxo0bmyMXQoy1VnbOk1L2SpmJP5Pp3c/rSAAAAM2u3mU6ISFBK1eurPXcqlWr6uxkiPBk3/yn7Bcfy4y/Wr5TRngdBwAAICjqPc3jiiuu0MMPP6zBgwfXbOW4aNEi3XLLLc2ZDyHA+Wye7Jv/khk+VuacS72OAwAAEDRHLNMVFRV65ZVXtHnzZp1yyilKSUlRcXGx0tLSdNlllx1yn3KEB7t2heycp6TjTpS5ejKbsgAAgLByxDI9a9YsrV+/XgMHDtTixYvVoUMHTZo0KRjZ0MLZwnw5M6ZKnZPku/Eumcg2XkcCAAAIqiPOmV6yZInuu+8+XX311br77ru1aNGiYORCC2f3VrhrSVdVynfTvTIdYryOBAAAEHRHLNMVFRU1NxkGAgGVlZU1eyi0bNZa2eenSZu/lW/SL2SSWWccAACEpyNO86iurtby5ctrHjuOU+uxJA0YMKDpk6HFsh+8Lvv5RzLjr5Y58RSv4wAAAHjmiGU6Li5Of/7zn2sex8TE1HpsjNHTTz/dPOnQ4tiVS2T//Zw0aBgrdwAAgLB3xDI9bdq0YORACLD5uXL+8iepa5p8193Gyh0AACDs1XvTFoQ3W1EuZ9oUyTryTb5HJqq915EAAAA8R5nGEblbhT8lbdss309+KZPYxetIAAAALQJlGkdkP35HduEnMj+8UmbAIK/jAAAAtBiUaRyW3bRe9sWZ0oDBMmdf4nUcAACAFoUyjUOyZaVynnlY6thJvom3y/j4xwUAAOBAtCMclLVWzpynpMI8+a6/Q6ZjrNeRAAAAWhzKNA7Kzv2vtChL5qJrZXof73UcAACAFokyjTrshnWyLz8rnXiKzA/Gex0HAACgxaJMoxZbutudJx0XL9/EnzFPGgAA4DBoSqhhrZXzt6ekkkL5rv+lTIeOXkcCAABo0SjTqGE//UBa9JnMhdfK9DrO6zgAAAAtHmUakiSbt81dT/q4E2V+8EOv4wAAAIQEyjRkq6rkzHxMioiU7zrmSQMAANQXrQmyb78kbVgr3zWTZRICXscBAAAIGZTpMGfXr5Z96yWZ00bLpJ/udRwAAICQQpkOY7a8TM6sx6SEgMwVN3gdBwAAIORQpsOY/ddfpYI8+Sb9XKZ9tNdxAAAAQg5lOkzZr7NkF3woc84lbBcOAADQQJTpMGR3Fst5YZrUo4/MeZd7HQcAACBkUabDkPOPZ6Tycne78MhIr+MAAACELMp0mLFfL5C+zpK54AqZLmlexwEAAAhplOkwYnftlPP3GVL33jLjLvQ6DgAAQMijTIcR++JfpbLd8k24RSYiwus4AAAAIY8yHSbs0i9lv/hY5pxLZVJ7eh0HAACgVaBMhwFbtlvOC9OllO4y51zqdRwAAIBWgzIdBuxLz0o7S+SbcKtMZBuv4wAAALQalOlWzq5YLLsgU+bMC2V69PE6DgAAQKtCmW7FbPkeOX97WkpOkTn/Cq/jAAAAtDqU6VbMvvUvqShfvmtvkWnT1us4AAAArQ5lupWyWzfJZr4hMzxDps/xXscBAABolSjTrZC1Vs4Lf5aiomUunuB1HAAAgFaLMt0K2c/mStkrZS7+sUzHWK/jAAAAtFqU6VbGlu6SfXm21Os4meEZXscBAABo1SjTrYx99Xl3y/CrfirjY3gBAACaE22rFbHfrpH95D2ZMefLpLFlOAAAQHOjTLcStrpazt//LMXFy/yQNaUBAACCgTLdStiP3pa++1a+H02SiYr2Og4AAEBYoEy3AnZHsexrL0jHD5QGD/c6DgAAQNigTLcC9j/PS5WV8l15g4wxXscBAAAIG5TpEGe/Wy+b9aHMmHNlkrp6HQcAACCsUKZDmLVWzkvPSh1iZM77kddxAAAAwg5lOpQt+UJas0zmgqtkomO8TgMAABB2KNMhylZVyvn3bKlLmszIM72OAwAAEJYo0yHKzv2vlJcj32UTZSIivI4DAAAQlijTIcju2in71ovSgEEyAwZ7HQcAACBsUaZDkH3zH1LFHvkuneh1FAAAgLBGmQ4xdtt3sh+/KzPyLJmu3byOAwAAENYig/VBS5Ys0ezZs+U4jsaOHavx48fXev2TTz7R66+/LkmKiorSpEmT1KNHj2DFCxnOy7Oldu1lLrjS6ygAAABhLyhXph3H0axZs3TPPffo8ccf14IFC7Rly5ZaxyQmJurBBx/UI488oosvvlh/+ctfghEtpNgVi6XlX8ucd5lMx1iv4wAAAIS9oJTp7OxsJScnKykpSZGRkRo2bJgWLlxY65i+ffsqJsZdK7lPnz4qLCwMRrSQYa2V8+rfJH+izOjzvI4DAAAABWmaR1FRkfx+f81jv9+vdevWHfL4uXPnauDAgQd9LTMzU5mZmZKkqVOnKhAING3YeoiMjAz655Z/Nk87vluv2FvvV/suXYL62eHKi3FG8DHO4YFxbv0Y4/DQEsc5KGXaWlvnOWPMQY9dvny55s2bp9/+9rcHfT0jI0MZGRk1jwsKCpom5FEIBAJB/VzrVMv525+lLmna3X+QSj0453AU7HGGNxjn8MA4t36McXjwcpy7du160OeDMs3D7/fXmrZRWFio+Pj4Osdt2rRJzzzzjH75y1+qY8eOwYgWEuznH0m5W+Qbf5WMjw1aAAAAWoqglOlevXopJydHeXl5qqqqUlZWltLT02sdU1BQoEceeUQ333zzIZt/OLKVlbJv/FPq3lsaeJrXcQAAAHCAoEzziIiI0MSJEzVlyhQ5jqPRo0crLS1N77//viRp3Lhx+ve//63du3dr5syZNe+ZOnVqMOK1aPbT96XCPPmuuemQU2MAAADgDWMPNqE5hGzbti3onxms+Tq2olzOPddLyany3TGFMh1kzL8LD4xzeGCcWz/GODyE7ZxpNIyd+5a0s0S+C6+mSAMAALRAlOkWypbtln33FemEdJnex3sdBwAAAAdBmW6h7HuvSWWl8l14jddRAAAAcAiU6RbI7iyW/fANmVNGyKT19DoOAAAADoEy3QLZd1+VKvfKXHCl11EAAABwGJTpFsbu2iH78bsyQ0fJJKd4HQcAAACHQZluYWzmG+5V6bMv9ToKAAAAjoAy3YLYst2y8/4rM2iYTJdUr+MAAADgCCjTLYid+19pT5nMuZd5HQUAAAD1QJluIWz5HneKx0lDWMEDAAAgRFCmWwj78btS6S75zmGuNAAAQKigTLcAdm+F7AevSf1Okjmmr9dxAAAAUE+U6RbALsiUdhTLd+6PvI4CAACAo0CZ9pitqnQ3aendTzq2v9dxAAAAcBQo0x6zn38kFeXLd+5lMsZ4HQcAAABHgTLtIVtdLfvOv6XuvaX+g7yOAwAAgKNEmfaQ/epTKS9HvnMu5ao0AABACKJMe8Ra686V7pImnTzU6zgAAABoAMq0V1Z/I23ZIDNuvIyPYQAAAAhFtDiPOJlvSB3jZIaO8joKAAAAGogy7QGbu0X6ZqHMGefItGnrdRwAAAA0EGXaAzbzDSmyjcwZZ3sdBQAAAI1AmQ4yu3un7GdzZU49Qya2k9dxAAAA0AiU6SCzH78r7d0rk3GB11EAAADQSJTpILJVlbLz3paOHyiT0t3rOAAAAGgkynQQ2YWfSjuK5PsBV6UBAABaA8p0kFhrZT94zd2kha3DAQAAWgXKdLCsXS5t3iCTcQFbhwMAALQSlOkgcT54XYqJlTn1DK+jAAAAoIlQpoPA5m7dt0nL2TJt23kdBwAAAE2EMh0E9sM3pYgImTPO8ToKAAAAmhBlupnZPWXuJi2njJCJi/c6DgAAAJoQZbqZ2S8+lirKxVVpAACA1ocy3Yyste6Oh2k9pZ7Heh0HAAAATYwy3Zy+XSNt2SAz6myWwwMAAGiFKNPNyH78rtSuvczQkV5HAQAAQDOgTDcTW7pL9qtPZU4dJRMV7XUcAAAANAPKdDOxn82VKvfKjDrb6ygAAABoJpTpZlBz4+ExfWXSenodBwAAAM2EMt0c1q6QcrfKjDrL6yQAAABoRpTpZmA/fkeK7iCTfrrXUQAAANCMKNNNzO4skV30mcywsTJt23kdBwAAAM2IMt3E7IIPpeoqmZFM8QAAAGjtKNNNyDqO7Px3pWMHyHRJ9ToOAAAAmhlluimtXCIVbOfGQwAAgDBBmW5CzsfvSh3jZAae5nUUAAAABAFluonYncXSN1+6Nx62aeN1HAAAAAQBZbqJ2C/mS44jM3ys11EAAAAQJJTpJmKz5ko9+sh0SfM6CgAAAIKEMt0E7OYN0pYNMsPGeB0FAAAAQUSZbgI2a64UESlzygivowAAACCIKNONZKurZb/4SDoxXSYm1us4AAAACCLKdGOtWCTt2iEfUzwAAADCDmW6kWzWXCkmVhow2OsoAAAACDLKdCPY0t2yS7+UGTpKJpK1pQEAAMINZboR7FefSlWVMqeN9joKAAAAPECZbgT72VypazepWy+vowAAAMADlOkGstu3SetXywwbI2OM13EAAADgAcp0A9nP5krGJzN0lNdRAAAA4BHKdANYx5H9bJ50/EkynfxexwEAAIBHKNMNULlisVSUL3Maa0sDAACEs8hgfdCSJUs0e/ZsOY6jsWPHavz48bVet9Zq9uzZWrx4sdq1a6fJkyfrmGOOCVa8o7Lno3ekqPYyJ5/qdRQAAAB4KChXph3H0axZs3TPPffo8ccf14IFC7Rly5ZaxyxevFi5ubl68skndf3112vmzJnBiHbUbEW5KrI+kkk/XaZdO6/jAAAAwENBKdPZ2dlKTk5WUlKSIiMjNWzYMC1cuLDWMV999ZVGjhwpY4yOPfZYlZaWqri4OBjxjs7OErU5boDMsLFeJwEAAIDHglKmi4qK5Pd/f6Oe3+9XUVFRnWMCgcBhj2kJTOdkxf/6CZk+x3sdBQAAAB4Lypxpa22d5/53beb6HCNJmZmZyszMlCRNnTq1VgEPlsjISE8+F8HFOIcHxjk8MM6tH2McHlriOAelTPv9fhUWFtY8LiwsVHx8fJ1jCgoKDnuMJGVkZCgjI6Pm8YHvCZZAIODJ5yK4GOfwwDiHB8a59WOMw4OX49y1a9eDPh+UaR69evVSTk6O8vLyVFVVpaysLKWnp9c6Jj09XfPnz5e1VmvXrlV0dPRByzQAAADQUgTlynRERIQmTpyoKVOmyHEcjR49WmlpaXr//fclSePGjdPAgQO1aNEi3XrrrWrbtq0mT54cjGgAAABAgwVtnelBgwZp0KBBtZ4bN25czc/GGE2aNClYcQAAAIBGYwdEAAAAoIEo0wAAAEADUaYBAACABqJMAwAAAA1EmQYAAAAaiDINAAAANBBlGgAAAGggyjQAAADQQMZaa70OAQAAAIQirkw3wF133eV1BAQB4xweGOfwwDi3foxxeGiJ40yZBgAAABqIMg0AAAA0EGW6ATIyMryOgCBgnMMD4xweGOfWjzEODy1xnLkBEQAAAGggrkwDAAAADRTpdYCWZMmSJZo9e7Ycx9HYsWM1fvz4Wq9bazV79mwtXrxY7dq10+TJk3XMMcfU671oORo6zgUFBZo2bZpKSkpkjFFGRobOOeccb04CR9SYf58lyXEc3XXXXUpISGiRd4/D1ZhxLi0t1YwZM7R582YZY/TTn/5Uxx57rAdngSNpzDi/9dZbmjt3rowxSktL0+TJk9W2bVsPzgJHcqRx3rp1q6ZPn64NGzbo8ssv1wUXXFDv9zYrC2uttdXV1fbmm2+2ubm5trKy0t5xxx128+bNtY75+uuv7ZQpU6zjOHbNmjX27rvvrvd70TI0ZpyLiors+vXrrbXWlpWV2VtvvZVxbqEaM877vfnmm/aJJ56wf/jDH4IZHUehseP81FNP2czMTGuttZWVlXb37t1BzY/6acw4FxYW2smTJ9uKigprrbWPPvqonTdvXrBPAfVQn3EuKSmx69ats//4xz/s66+/flTvbU5M89gnOztbycnJSkpKUmRkpIYNG6aFCxfWOuarr77SyJEjZYzRscceq9LSUhUXF9frvWgZGjPO8fHxNVc62rdvr5SUFBUVFXlxGjiCxoyzJBUWFmrRokUaO3asF/FRT40Z57KyMq1atUpjxoyRJEVGRqpDhw5enAaOoLH/PjuOo71796q6ulp79+5VfHy8F6eBI6jPOMfFxal3796KiIg46vc2J8r0PkVFRfL7/TWP/X5/naJUVFSkQCBQ55j6vBctQ2PG+UB5eXnasGGDevfu3byB0SCNHefnnntOV199tYwxwQmMBmnMOOfl5Sk2NlbTp0/Xr371K82YMUPl5eVBy476a8w4JyQk6Pzzz9dPf/pTXX/99YqOjtZJJ50UtOyov8Z0Ka97GGV6H3uQRU3+939ID3VMfd6LlqEx47xfeXm5Hn30UU2YMEHR0dFNHxKN1phx/vrrrxUXF1dr/jRapsaMc3V1tTZs2KBx48bpj3/8o9q1a6fXXnutuaKiERozzrt379bChQs1bdo0PfPMMyovL9f8+fObLSsarjFdyuseRpnex+/3q7CwsOZxYWFhnb8K8vv9KigoqHNMfd6LlqEx4yxJVVVVevTRRzVixAgNHTo0OKFx1BozzmvWrNFXX32lm266SU888YSWL1+uJ598MmjZUX+N/e+23+9Xnz59JEmnnnqqNmzYEJzgOCqNGedly5YpMTFRsbGxioyM1NChQ7V27dqgZUf9NaZLed3DKNP79OrVSzk5OcrLy1NVVZWysrKUnp5e65j09HTNnz9f1lqtXbtW0dHRio+Pr9d70TI0ZpyttZoxY4ZSUlJ03nnneXQGqI/GjPOVV16pGTNmaNq0afrZz36mAQMG6NZbb/XoTHA4jRnnTp06ye/3a9u2bZKkZcuWKTU11YvTwBE0ZpwDgYDWrVuniooKWWu1bNkypaSkeHQmOJzGdCmvexibthxg0aJFmjNnjhzH0ejRo3XRRRfp/ffflySNGzdO1lrNmjVLS5cuVdu2bTV58mT16tXrkO9Fy9TQcV69erUeeOABdevWreavj6644goNGjTIy9PBITTm3+f9VqxYoTfffJOl8Vqwxozzxo0bNWPGDFVVVSkxMVGTJ09WTEyMl6eDQ2jMOL/00kvKyspSRESEevTooRtvvFFt2rTx8nRwCEca55KSEt11113as2ePjDGKiorSY489pujoaE97GGUaAAAAaCCmeQAAAAANRJkGAAAAGogyDQAAADQQZRoAAABoIMo0AAAA0ECUaQAIE9dcc422b9/udQwAaFVYGg8AQthNN92kkpIS+Xw+RUVFaeDAgZo4caKioqK8jgYAYYEr0wAQ4u688049//zzevjhh7V+/Xq98sortV6vrq5u1O9v7PsBoDWL9DoAAKBpJCQk6OSTT9bmzZt12WWXaeLEiXr77bdVXV2tadOm6bLLLtOTTz6p5ORklZWV6dlnn9XixYvVrl07jR07VhdeeKF8Pp8++ugjffjhh+rVq5c+/vhjnXnmmbr88su9Pj0AaJEo0wDQShQUFGjx4sUaMmSIFi1apIULF+qhhx5S27Zt6xz77LPPqqysTE8//bR27dqlKVOmKD4+XmPGjJEkrVu3TsOGDdPMmTO5Mg0Ah0GZBoAQ96c//UkRERGKjo7WoEGDdNFFF+nVV1/VhRdeqJiYmDrHO46jrKws/fGPf1T79u3Vvn17nXfeeZo/f35NmY6Pj9fZZ58tSYqIiAjq+QBAKKFMA0CI++Uvf6kTTzyxzvN+v/+gx+/cuVNVVVUKBAI1z3Xu3FlFRUU1jw98DQBwaNyACACtlDHmoM/HxsYqIiJCBQUFNc8VFBQoISEhWNEAoNWgTANAmPH5fDrttNP0z3/+U3v27FF+fr7eeustjRgxwutoABBymOYBAGFo4sSJevbZZ3XzzTerbdu2Gjt2rEaPHu11LAAIOWzaAgAAADQQ0zwAAACABqJMAwAAAA1EmQYAAAAaiDINAAAANBBlGgAAAGggyjQAAADQQJRpAAAAoIEo0wAAAEADUaYBAACABvr/GbLcPy84RgAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "p_B = np.linspace(.0001, .1, 100)\n",
    "pB_A =( .99*p_B)/(.99*p_B+.02*(1-p_B))\n",
    "fig, ax = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "ax.plot(p_B, pB_A)\n",
    "ax.set_xlabel('Prior')\n",
    "ax.set_ylabel('Posterior')\n",
    "ax.set_title('Example 1')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Probability Density Form of Bayesian Inference</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To handle complicated issues, the integral version of Bayes' Theorem is our guideline, every Bayesian computation requires solving an integral numerically. \n",
    "\n",
    "$$\n",
    "p(\\theta \\mid D)=\\frac{ p(D \\mid \\theta)p(\\theta)}{p(D)}=\\frac{p(D \\mid \\theta)p(\\theta) }{\\int p(D \\mid \\theta) p(\\theta) d \\theta}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $\\theta$ is the parameter that we hypothesized which characterizes the distribution $p(\\theta)$, note that we change the notation from $P$ to $p$, the former represents probability mass function and latter represents probability density distribution, which will be illustrated in details in next chapter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some important differences has to be pointed out, density form has an _infinite number_ of hypotheses though bounds could be specified. And also the integral in the denominator won't always have a closed-form solution, which is why **Markov Chain Monte Carlo** is usually used for numerical computation of the integral.\n",
    "\n",
    "In some special cases you can derive posterior in closed-form, usually both prior and posterior share the same distribution but different parameters, such prior is called **conjugate prior**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}