examples/dnnr_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0ac57d55",
   "metadata": {},
   "source": [
    "# Tutorial of DNNR with a comparison to KNN and Catboost\n",
    "\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/younader/dnnr/blob/main/examples/dnnr_tutorial.ipynb)\n",
    "\n",
    "DNNR works by estimating the gradient of the target function at each neighbor, and then instead of averaging the labels of the neighbors it averages the n-th order taylor approximations of the target function. remember that the taylor approximation around a point is given by $ \\eta_{\\text{DNNR}}(x) = \\frac{1}{k} \n",
    "    \\sum_{\n",
    "        X_m \\in B_{x, \\#k}\n",
    "    }\n",
    "        \\left(\n",
    "            Y_m + \\hat \\gamma_m (x - X_m) \n",
    "        \\right).$ the taylor approximation also includes the averaging of targets that KNN does but accounts for the function changes with the estimated derivative. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "203f404e",
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m pip install dnnr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01337708-f296-4ae1-a23f-a52cf3a1f022",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing,make_friedman1\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "\n",
    "from dnnr import DNNR"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fae45235-47fa-4bec-92fb-6ce0b9509443",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Data fetching and preprocessing\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df2cb3ec-7713-459f-b982-fdb34d0498f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "\n",
    "# Friedman1 is a simple synthetic dataset. See: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_friedman1.html\n",
    "dataset = 'friedman1'\n",
    "# Uncomment the following line to use the California housing dataset (https://www.dcc.fc.up.pt/~ltorgo/Regression/cal_housing.html).\n",
    "# dataset = 'california_housing'\n",
    "\n",
    "if dataset == 'friedman1':\n",
    "    X, y = make_friedman1(n_samples=20000)\n",
    "\n",
    "elif dataset == 'california_housing':\n",
    "    cali = sklearn.datasets.fetch_california_housing()\n",
    "    y = cali.target\n",
    "    X = cali.data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d01257c1",
   "metadata": {},
   "source": [
    "It is critical to scale the data for good performance. Here, we use the `sklearn.preprocessing.StandardScaler` to scale the data to have mean 0 and standard deviation 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6b970bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = StandardScaler().fit_transform(X)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=2022\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "383021a8-7848-4e64-8ebe-54c6db989af3",
   "metadata": {},
   "source": [
    "## DNNR: fitting and evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d530fc88-fe19-4c7e-8328-3f70cb9caef6",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = DNNR(n_derivative_neighbors=32)\n",
    "model.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9485cc91-593a-4f5b-b8f0-a55b823aa6a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Evaluating DNNR Model\")\n",
    "mse_error = mean_squared_error(y_test, model.predict(X_test))\n",
    "print(\"MSE={error}\".format(error=mse_error))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5bd0993-71f5-473b-a76c-a8aa6a353ea4",
   "metadata": {},
   "source": [
    "## Comparing to KNN model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e561f3d6-8a61-4ea8-ad84-1476b52ab03a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "\n",
    "print(\"Evaluating KNN Regression\")\n",
    "knn_model = KNeighborsRegressor(n_neighbors=5)\n",
    "knn_model.fit(X_train, y_train)\n",
    "knn_mse_error = mean_squared_error(y_test, knn_model.predict(X_test))\n",
    "print(\"MSE={error}\".format(error=knn_mse_error))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02dcbd75",
   "metadata": {},
   "source": [
    "## Comparing to [CatBoost](https://catboost.ai/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "022204c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import catboost\n",
    "except ImportError:\n",
    "    ! pip install catboost\n",
    "\n",
    "from catboost import CatBoostRegressor\n",
    "\n",
    "print(\"Evaluating Catboost\")\n",
    "\n",
    "catboost = CatBoostRegressor(verbose=False)\n",
    "catboost.fit(X_train, y_train)\n",
    "catboost_mse_error = mean_squared_error(y_test, catboost.predict(X_test))\n",
    "print(\"MSE={error}\".format(error=catboost_mse_error))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fdaabca",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Method    |  MSE\")\n",
    "print(\"DNNR      |  {error}\".format(error=mse_error))\n",
    "print(\"KNN Reg   |  {error}\".format(error=knn_mse_error))\n",
    "print(\"CatBoost  |  {error}\".format(error=catboost_mse_error))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2e21af2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.13 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "21d31839b08a1115e47633d1eed7a4eac2b86e082da6e385b446d5de6db36cb8"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}