notebooks/mathematical-statistics/matrix_calculus_short.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4d88997e-48b3-4ebe-9d89-c1f51158c412",
   "metadata": {},
   "source": [
    "### Gradient\n",
    "\n",
    "- **Definition**: The gradient of a scalar-valued function $f(\\textbf{x})$ is a column vector of partial derivatives with respect to each element of a column vector $\\textbf{x}$.\n",
    "  \n",
    "- **Mathematical Expression**: If $f(\\textbf{x})$ is a scalar function of $\\textbf{x} = \\begin{bmatrix} x_1 & x_2 & \\dots & x_n \\end{bmatrix}^T$, then the gradient is:\n",
    "  $$\n",
    "  \\nabla f(\\textbf{x}) = \\begin{bmatrix} \n",
    "  \\frac{\\partial f}{\\partial x_1} & \\frac{\\partial f}{\\partial x_2} & \\dots & \\frac{\\partial f}{\\partial x_n}\n",
    "  \\end{bmatrix}^T\n",
    "  $$\n",
    "\n",
    "**Example**: If $f(x, y) = x^2 + 3xy$, then \n",
    "$$\n",
    "\\nabla f(x, y) = \\begin{bmatrix} \n",
    "\\frac{\\partial f}{\\partial x} & \\frac{\\partial f}{\\partial y}\n",
    "\\end{bmatrix}^T\n",
    "= \\begin{bmatrix} \n",
    "2x + 3y & 3x\n",
    "\\end{bmatrix}^T\n",
    "$$\n",
    "This is the gradient of the scalar function $f(x, y)$, representing the rate of change with respect to each variable and pointing in the direction of the steepest ascent at $(x, y)$.\n",
    "\n",
    "\n",
    "### Jacobian\n",
    "\n",
    "- **Definition**: The Jacobian is a matrix of all first-order partial derivatives of a vector-valued function.\n",
    "\n",
    "- **Mathematical Expression**: For a vector-valued function $\\mathbf{y}(\\textbf{x}) = \\begin{bmatrix} y_1 & y_2 \\end{bmatrix}^T$, where $\\mathbf{x} = \\begin{bmatrix} x_1 & x_2 \\end{bmatrix}^T$, the Jacobian matrix is:\n",
    "  $$\n",
    "  J = \\begin{bmatrix} \n",
    "  \\frac{\\partial y_1}{\\partial x_1} & \\frac{\\partial y_1}{\\partial x_2} \\\\\n",
    "  \\frac{\\partial y_2}{\\partial x_1} & \\frac{\\partial y_2}{\\partial x_2}\n",
    "  \\end{bmatrix}\n",
    "  $$\n",
    "\n",
    "**Example**: If $\\mathbf{y}(x, y) = \\begin{bmatrix} x^2 + 3xy & 2x + y \\end{bmatrix}^T$, the Jacobian matrix is:\n",
    "$$\n",
    "J = \\begin{bmatrix} \n",
    "\\frac{\\partial (x^2 + 3xy)}{\\partial x} & \\frac{\\partial (x^2 + 3xy)}{\\partial y} \\\\\n",
    "\\frac{\\partial (2x + y)}{\\partial x} & \\frac{\\partial (2x + y)}{\\partial y}\n",
    "\\end{bmatrix}\n",
    "\\hspace{1cm}\n",
    "J = \\begin{bmatrix} \n",
    "2x + 3y & 3x \\\\\n",
    "2 & 1\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "This **2 x 2 matrix** represents the rate of change of each component of the output vector $\\mathbf{y}$ with respect to each input variable.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9f0fe83d-7b7b-42e1-b2e2-e670cd6e810b",
   "metadata": {},
   "source": [
    "### Top-5 Matrix Calculus Rules ###\n",
    "\n",
    "### Rule-1 ###\n",
    "\n",
    "Given a function $f(x) = a^T x$, where:\n",
    "- $a$ is a $n \\times 1$ vector,\n",
    "- $x$ is a $n \\times 1$ vector,\n",
    "\n",
    "the gradient of $f(x)$ with respect to $x$ is:\n",
    "\n",
    "$$\n",
    "\\nabla_x f = a\n",
    "$$\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "eebf8e94-2ae6-4720-aa0d-ed1332065ff3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculated Gradient:\n",
      "[[-1.4624308347702026], [0.7523223161697388]]\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "torch.manual_seed(47)\n",
    "\n",
    "a = torch.randn(2, 1)\n",
    "x = torch.randn(2, 1, requires_grad=True)\n",
    "\n",
    "def grad_f(x, a):\n",
    "    f = a.T @ x\n",
    "    f.backward()\n",
    "    return x.grad\n",
    "\n",
    "expected_gradient = a\n",
    "calculated_gradient = grad_f(x, a)\n",
    "\n",
    "assert torch.allclose(expected_gradient, calculated_gradient)\n",
    "print(calculated_gradient.tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3ef6b19-09b9-40b1-b3a4-0e31debdb7e3",
   "metadata": {},
   "source": [
    "### Rule-2 ###\n",
    "\n",
    "Given a function $ f(x) = A x $, where:\n",
    "- $ A $ is an $ m \\times n $ matrix,\n",
    "- $ x $ is an $ n \\times 1 $ vector,\n",
    "\n",
    "the Jacobian of $ f(x) $ with respect to $ x $ is:\n",
    "\n",
    "$$\n",
    "\\mathbf{J}_{f(x)} = A\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "83d82f43-e701-47a9-8cdf-6f64cc2feaff",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "torch.manual_seed(47)\n",
    "\n",
    "# Define A as a 2x3 matrix and x as a 3x1 vector\n",
    "A = torch.randn(2, 3)\n",
    "x = torch.randn(3, 1, requires_grad=True)\n",
    "\n",
    "# f is a vector-valued function (in and out: vector)\n",
    "def f(x):\n",
    "    return A @ x\n",
    "\n",
    "jacobian = (\n",
    "    torch.autograd\n",
    "    .functional\n",
    "    .jacobian(f, x)\n",
    "    .reshape(2, -1)\n",
    ")\n",
    "\n",
    "expected_jacobian = A\n",
    "assert torch.allclose(jacobian, expected_jacobian)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f52686f2-e772-47aa-a77a-22ed0ac9e5c6",
   "metadata": {},
   "source": [
    "### Rule-3\n",
    "\n",
    "Given a function $f(x) = x^T A x$, where:\n",
    "- $A$ is a $n \\times n$ matrix,\n",
    "- $x$ is a $n \\times 1$ vector,\n",
    "\n",
    "the gradient of $f(x)$ with respect to $x$ is:\n",
    "\n",
    "$$\n",
    "\\nabla_x f = A x + A^T x\n",
    "$$\n",
    "\n",
    "#### Condition on $A$:\n",
    "- If $A$ is **symmetric** ($A = A^T$), the gradient simplifies to, $\\nabla_x f = 2 A x$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d054dd81-d9de-4066-86c9-05b87bf96653",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-2.916311502456665], [0.7888590097427368]]\n"
     ]
    }
   ],
   "source": [
    "import torch; torch.manual_seed(47)\n",
    "\n",
    "A = torch.randn(2, 2); \n",
    "x = torch.randn(2, 1, requires_grad=True)\n",
    "\n",
    "def grad_f(A, x):\n",
    "    f = x.T @ A @ x\n",
    "    f.backward()\n",
    "    return x.grad\n",
    "\n",
    "expected_gradient = A @ x + A.T @ x\n",
    "calculated_gradient = grad_f(A, x)\n",
    "\n",
    "assert torch.allclose(expected_gradient, calculated_gradient)\n",
    "print(calculated_gradient.tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be9b0247-af3f-4265-a2e6-de7711ce5f39",
   "metadata": {},
   "source": [
    "### Rule-4 ###\n",
    "\n",
    "Given a function $f(x, y) = x^T A y$, where:\n",
    "- $A$ is a $n \\times n$ matrix,\n",
    "- $x$ is a $n \\times 1$ vector,\n",
    "- $y$ is a $n \\times 1$ vector,\n",
    "\n",
    "the gradients of $f(x, y)$ with respect to $x$ and $y$ are:\n",
    "\n",
    "$$\n",
    "\\nabla_x f = A y\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\nabla_y f = A^T x\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5ecc47b9-6bec-4919-9d5c-d60046702c00",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculated Gradient with respect to y:\n",
      "tensor([[-2.9293],\n",
      "        [ 1.1403]])\n"
     ]
    }
   ],
   "source": [
    "import torch; torch.manual_seed(47)\n",
    "\n",
    "A = torch.randn(2, 2)\n",
    "x = torch.randn(2, 1, requires_grad=True)\n",
    "y = torch.randn(2, 1, requires_grad=True)\n",
    "\n",
    "def grad_f(A, x, y):\n",
    "    f = x.T @ A @ y\n",
    "    f.backward()\n",
    "    return x.grad, y.grad\n",
    "\n",
    "expected_grad_x = A @ y\n",
    "expected_grad_y = A.T @ x\n",
    "calculated_grad_x, calculated_grad_y = grad_f(A, x, y)\n",
    "\n",
    "assert torch.allclose(expected_grad_x, calculated_grad_x)\n",
    "assert torch.allclose(expected_grad_y, calculated_grad_y)\n",
    "\n",
    "\n",
    "print(\"Calculated Gradient with respect to y:\")\n",
    "print(calculated_grad_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29d459c8-8ace-4dfc-9d35-6479d8c7ac8c",
   "metadata": {},
   "source": [
    "### Rule-5\n",
    "\n",
    "Given a function $ f(X) = a^T X b $, where:\n",
    "- $ a $ is a $n \\times 1 $ column vector,\n",
    "- $ X $ is a $n \\times m $ matrix,\n",
    "- $ b $ is a $m \\times 1 $ column vector,\n",
    "\n",
    "the gradient of $ f(X) $ with respect to $X $ is:\n",
    "\n",
    "$$\n",
    "\\nabla_X (a^T X b) = a b^T\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "433ddff6-4da4-48d3-b521-569d1d5b9f02",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.8419, -0.8833],\n",
      "        [ 0.4331,  0.4544],\n",
      "        [-0.9886, -1.0373]])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "torch.manual_seed(47)\n",
    "\n",
    "a = torch.randn(3, 1)  \n",
    "b = torch.randn(2, 1)  \n",
    "zX = torch.randn(3, 2, requires_grad=True) \n",
    "\n",
    "def grad_f(X, a, b):\n",
    "    f = a.T @ X @ b \n",
    "    f.backward()  \n",
    "    return X.grad\n",
    "\n",
    "calculated_grad_X = grad_f(X, a, b)\n",
    "expected_grad_X = a @ b.T\n",
    "assert torch.allclose(expected_grad_X, calculated_grad_X)\n",
    "print(calculated_grad_X)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "threads",
   "language": "python",
   "name": "threads"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}