Skip to content

Coerce dtype __props__ to string due to invalid hash of np.dtype() objects #1436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 2, 2025

Conversation

ricardoV94
Copy link
Member

@ricardoV94 ricardoV94 commented May 31, 2025

There were some odd errors cropping up in pymc-marketing and a private project.

The issue happened when you created an Arange with float64 and another with np.dtype("float64") (or any combo of dtype and string), which evaluate to the same thing, but hash differently (see below why). The PyTensor C-cache has a sanity-check mechanism to make sure the eq and hash of our Ops is correctly implemented, as the validity of the cache depends on this.

PyTensor uses the properties of Op to define equality and hash automatically, and for Arange that is it's dtype argument. Surprisingly though, np.dtype("float64") == "float64" and hash(np.dtype("float64")) != hash("float64"), which seems to be a fundamental bug in Numpy: numpy/numpy#17864

This only became a problem once we added a C-implementation to Arange in #1392 as this is where the automatic checks for hash/eq are performed.

I tried to look for other Ops parametrized by dtype of some form or another, and make sure those are always converted to the equivalent strings.

CC @lucianopaz and @juanitorduz


📚 Documentation preview 📚: https://pytensor--1436.org.readthedocs.build/en/1436/

@ricardoV94 ricardoV94 added bug Something isn't working C-backend labels May 31, 2025
@ricardoV94 ricardoV94 changed the title Coerce dtype __props__ to string due to invalid hash of np.dtype() objects Coerce dtype __props__ to string due to invalid hash of np.dtype() objects May 31, 2025
@ricardoV94 ricardoV94 force-pushed the numpy_dtype_hash_bug branch 2 times, most recently from eacddf1 to a118b03 Compare June 2, 2025 10:38
@ricardoV94 ricardoV94 force-pushed the numpy_dtype_hash_bug branch from a118b03 to fd371c3 Compare June 2, 2025 10:40
Copy link

codecov bot commented Jun 2, 2025

Codecov Report

Attention: Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 82.12%. Comparing base (92eef5e) to head (fd371c3).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
pytensor/tensor/random/op.py 0.00% 1 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (80.00%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1436   +/-   ##
=======================================
  Coverage   82.12%   82.12%           
=======================================
  Files         211      211           
  Lines       49751    49757    +6     
  Branches     8817     8819    +2     
=======================================
+ Hits        40857    40862    +5     
- Misses       6714     6715    +1     
  Partials     2180     2180           
Files with missing lines Coverage Δ
pytensor/sparse/sandbox/sp2.py 82.79% <100.00%> (ø)
pytensor/tensor/basic.py 91.69% <100.00%> (+0.01%) ⬆️
pytensor/tensor/elemwise.py 90.01% <100.00%> (ø)
pytensor/tensor/io.py 77.41% <100.00%> (ø)
pytensor/tensor/random/op.py 92.51% <0.00%> (-1.01%) ⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ricardoV94 ricardoV94 merged commit ff98ab8 into pymc-devs:main Jun 2, 2025
72 of 73 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working C-backend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants