Data Loader Can No Longer Fiind Correct Python Interpreter

[WillHWThompson]

foo.csv.py. At the beginning of this data loader, I import several packages. Here is a snippet of the code:

#!/Users/my_user/miniconda3/envs/my_env/bin/python
import numpy as np
import pandas as pd

a = np.random.rand(10, 10)
df = pd.DataFrame(a)
df_results_dict = {'my_df': df}
# MORE CODE TO SAVE DF HERE 

I also added the following line to my observablehq.config.ts:

interpreters: {
  ".py": ["/Users/my_user/miniconda3/envs/my_env/bin/python"],
},

When I run the dashboard with npm run dev, with the my_env environment active, I get the following error:

ModuleNotFoundError: No module named 'pandas'

This error occurs for any package that is not part of the standard library. It seems like Observable is not using the correct interpreter despite the shebang and the interpreter specification in the config.ts file.

When I try to determine what interpreter is being used by adding a line like:

raise SystemError(f"python env: {sys.executable}")

I get:

SystemError: python env: /opt/homebrew/opt/[email protected]/bin/python3.12

The Python interpreter was installed with Homebrew.

I have tried to create a fresh Observable project and a fresh conda environment, and the problem persists. I am not sure exactly what changed, but I did upgrade to the latest version of Observable.

How can I get Observable to use the correct Python interpreter?

[mbostock]

It seems like Observable is not using the correct interpreter despite the shebang and the interpreter specification in the config.ts file.

The shebang has no effect unless you are using an executable (.exe) data loader. When you use .py data loader, by default you’ll get python3 as the interpreter, though you should be able to override the interpreter by using the interpreters config option.

Have you tried running your Python data loader manually? Assuming you are in your project root, and foo.csv.py lives in your source root src, it should be like this:

/Users/my_user/miniconda3/envs/my_env/bin/python src/foo.csv.py

(If you’re not on the latest version 1.7.0, then your source root is probably docs instead of src, so you’ll need to change the above command accordingly.)

It should also be possible for you to set your PATH environment variable so that you don’t need to specify the full path to your python executable.

Lastly have you considered using a virtual environment? I’m not a Python expert but I believe it’s common practice. We use it in our penguin-classification example, and it is setup like this:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
yarn
yarn dev

Then you can exit the virtual environment like so:

deactivate

[WillHWThompson]

Thanks for the reply!

I have tried running my data loader manually and it works. The only problem occurs when observable calls the dataloader. This is what makes it so strange - it is a problem with how observable calls Python, not with the Python interpreter or the environment itself.

I am using a virtual environment. The path to the interpreter I am using is specifically for the Python interpreter associated with my Python virtual environment. This interpreter is in my PATH. I first tried to specify the relative path to the interpreter and when this did not work, I tried the absolute path to ensure the issue wasn't with my PATH.

Do you have any suggestions on how I could start to debug the issue? I don't know what else I can try? How can I begin to determine how Observable determines what interpreter to use for a dataloader?

[mbostock]

Did you start the preview server (npm run dev) inside your virtual environment too?

The only thing Framework is doing is invoking the interpreter as defined in the config, which defaults to python3 for .py data loaders.