Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pin and compile requirements #7

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Pin and compile requirements #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4839681
pinned and compiled
May 9, 2022
e704b48
Name appropraitely based on pip-tools
May 9, 2022
8275513
add installation of requirements
May 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions examples/macos-py3.10-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
# pip-compile requirements.in
#
cloudpickle==2.0.0
# via
# -r requirements.in
# shap
cycler==0.11.0
# via matplotlib
findspark==2.0.1
# via -r requirements.in
fonttools==4.33.3
# via matplotlib
joblib==1.1.0
# via scikit-learn
kiwisolver==1.4.2
# via matplotlib
llvmlite==0.38.0
# via numba
matplotlib==3.5.2
# via seaborn
networkx==2.8
# via -r requirements.in
numba==0.55.1
# via shap
numpy==1.21.6
# via
# matplotlib
# numba
# pandas
# scikit-learn
# scipy
# seaborn
# shap
packaging==21.3
# via
# matplotlib
# shap
pandas==1.4.2
# via
# seaborn
# shap
pillow==9.1.0
# via matplotlib
py4j==0.10.9.3
# via pyspark
pydot==1.4.2
# via -r requirements.in
pyparsing==3.0.8
# via
# matplotlib
# packaging
# pydot
pyspark==3.2.1
# via -r requirements.in
python-dateutil==2.8.2
# via
# matplotlib
# pandas
pytz==2022.1
# via pandas
scikit-learn==1.0.2
# via
# shap
# sklearn
scipy==1.8.0
# via
# scikit-learn
# seaborn
# shap
seaborn==0.11.2
# via -r requirements.in
shap==0.40.0
# via -r requirements.in
six==1.16.0
# via python-dateutil
sklearn==0.0
# via -r requirements.in
slicer==0.0.7
# via shap
threadpoolctl==3.1.0
# via scikit-learn
tqdm==4.64.0
# via shap

# The following packages are considered to be unsafe in a requirements file:
# setuptools
9 changes: 9 additions & 0 deletions examples/requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
cloudpickle==2.0.0
findspark==2.0.1
networkx==2.8
pydot==1.4.2
pyspark==3.2.1
seaborn==0.11.2
shap==0.40.0
sklearn==0.0

8 changes: 0 additions & 8 deletions examples/requirements.txt
View file
Open in desktop

This file was deleted.

69 changes: 59 additions & 10 deletions examples/simple.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,58 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Install Requirements"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: cloudpickle==2.0.0 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from -r requirements.in (line 1)) (2.0.0)\n",
"Requirement already satisfied: findspark==2.0.1 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from -r requirements.in (line 2)) (2.0.1)\n",
"Requirement already satisfied: networkx==2.8 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from -r requirements.in (line 3)) (2.8)\n",
"Requirement already satisfied: pydot==1.4.2 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from -r requirements.in (line 4)) (1.4.2)\n",
"Requirement already satisfied: pyspark==3.2.1 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from -r requirements.in (line 5)) (3.2.1)\n",
"Requirement already satisfied: seaborn==0.11.2 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from -r requirements.in (line 6)) (0.11.2)\n",
"Requirement already satisfied: shap==0.40.0 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from -r requirements.in (line 7)) (0.40.0)\n",
"Requirement already satisfied: sklearn==0.0 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from -r requirements.in (line 8)) (0.0)\n",
"Requirement already satisfied: pyparsing>=2.1.4 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from pydot==1.4.2->-r requirements.in (line 4)) (3.0.8)\n",
"Requirement already satisfied: py4j==0.10.9.3 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from pyspark==3.2.1->-r requirements.in (line 5)) (0.10.9.3)\n",
"Requirement already satisfied: pandas>=0.23 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from seaborn==0.11.2->-r requirements.in (line 6)) (1.4.0)\n",
"Requirement already satisfied: scipy>=1.0 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from seaborn==0.11.2->-r requirements.in (line 6)) (1.8.0)\n",
"Requirement already satisfied: numpy>=1.15 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from seaborn==0.11.2->-r requirements.in (line 6)) (1.21.6)\n",
"Requirement already satisfied: matplotlib>=2.2 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from seaborn==0.11.2->-r requirements.in (line 6)) (3.5.1)\n",
"Requirement already satisfied: numba in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from shap==0.40.0->-r requirements.in (line 7)) (0.56.0.dev0+418.g637a75023)\n",
"Requirement already satisfied: tqdm>4.25.0 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from shap==0.40.0->-r requirements.in (line 7)) (4.64.0)\n",
"Requirement already satisfied: scikit-learn in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from shap==0.40.0->-r requirements.in (line 7)) (1.0.2)\n",
"Requirement already satisfied: packaging>20.9 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from shap==0.40.0->-r requirements.in (line 7)) (21.3)\n",
"Requirement already satisfied: slicer==0.0.7 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from shap==0.40.0->-r requirements.in (line 7)) (0.0.7)\n",
"Requirement already satisfied: fonttools>=4.22.0 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from matplotlib>=2.2->seaborn==0.11.2->-r requirements.in (line 6)) (4.29.1)\n",
"Requirement already satisfied: pillow>=6.2.0 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from matplotlib>=2.2->seaborn==0.11.2->-r requirements.in (line 6)) (9.1.0)\n",
"Requirement already satisfied: python-dateutil>=2.7 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from matplotlib>=2.2->seaborn==0.11.2->-r requirements.in (line 6)) (2.8.2)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from matplotlib>=2.2->seaborn==0.11.2->-r requirements.in (line 6)) (1.3.2)\n",
"Requirement already satisfied: cycler>=0.10 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from matplotlib>=2.2->seaborn==0.11.2->-r requirements.in (line 6)) (0.11.0)\n",
"Requirement already satisfied: pytz>=2020.1 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from pandas>=0.23->seaborn==0.11.2->-r requirements.in (line 6)) (2022.1)\n",
"Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from numba->shap==0.40.0->-r requirements.in (line 7)) (0.39.0.dev0+61.gf27ac6f)\n",
"Requirement already satisfied: setuptools in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from numba->shap==0.40.0->-r requirements.in (line 7)) (60.6.0)\n",
"Requirement already satisfied: joblib>=0.11 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from scikit-learn->shap==0.40.0->-r requirements.in (line 7)) (1.1.0)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from scikit-learn->shap==0.40.0->-r requirements.in (line 7)) (3.1.0)\n",
"Requirement already satisfied: six>=1.5 in /Users/isaacjoseph/miniforge3/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib>=2.2->seaborn==0.11.2->-r requirements.in (line 6)) (1.16.0)\n"
]
}
],
"source": [
"!pip install -r requirements.in"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand All @@ -14,7 +67,6 @@
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"slideshow": {
"slide_type": "skip"
}
Expand Down Expand Up @@ -140,7 +192,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home\r\n"
"/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home\n"
]
}
],
Expand All @@ -163,7 +215,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"/usr/local/opt/apache-spark/libexec\r\n"
"/usr/local/opt/apache-spark/libexec\n"
]
}
],
Expand Down Expand Up @@ -220,7 +272,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"5863 org.apache.spark.deploy.SparkSubmit pyspark-shell -Xmx1g\r\n"
"5863 org.apache.spark.deploy.SparkSubmit pyspark-shell -Xmx1g\n"
]
}
],
Expand Down Expand Up @@ -336,7 +388,6 @@
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"slideshow": {
"slide_type": "skip"
}
Expand Down Expand Up @@ -684,7 +735,6 @@
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"slideshow": {
"slide_type": "skip"
}
Expand Down Expand Up @@ -1282,7 +1332,6 @@
"end_time": "2020-11-02T04:03:16.424298Z",
"start_time": "2020-11-02T04:02:55.183451Z"
},
"scrolled": false,
"slideshow": {
"slide_type": "slide"
}
Expand Down Expand Up @@ -1793,9 +1842,9 @@
],
"metadata": {
"kernelspec": {
"display_name": "3.7",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "3.7"
"name": "python3"
},
"language_info": {
"codemirror_mode": {
Expand All @@ -1807,7 +1856,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
"version": "3.10.2"
},
"toc": {
"base_numbering": 1,
Expand Down