Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rule Execution Benchmark Suite #1199

Open
dhanur-sharma opened this issue Jan 14, 2025 · 2 comments · May be fixed by #1227
Open

Rule Execution Benchmark Suite #1199

dhanur-sharma opened this issue Jan 14, 2025 · 2 comments · May be fixed by #1227
Assignees
@bishwaspraveen

Comments

@dhanur-sharma
Copy link
Contributor

Description

The execution speed increase needs a benchmarking suite to quantify gains.

Implementation Considerations

Deliverable

  • Rule execution benchmarking

Dependencies

depends on
@dhanur-sharma dhanur-sharma mentioned this issue Jan 14, 2025
Open
@bishwaspraveen
Copy link
Contributor 

Running benchmark for Title Pattern...

================================================================================
Detailed Profile for Title Pattern
================================================================================

Dataset Size: 10

APPLY OPERATION
Function Name                                      | Calls    | Total Time   | Cumulative  
--------------------------------------------------------------------------------------
/app/sde_collections/models/delta_patterns.py:477(apply) | 1        | 0.0000       | 0.5870      
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:383(__iter__) | 4        | 0.0000       | 0.5800      
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:1879(_fetch_all) | 5        | 0.0000       | 0.5800      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py:1532(execute_sql) | 4        | 0.0000       | 0.5790      
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:85(__iter__) | 3        | 0.0000       | 0.5790      
/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py:66(execute) | 4        | 0.0000       | 0.5720      
/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py:76(_execute_with_wrappers) | 4        | 0.0000       | 0.5720      
/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py:82(_execute) | 4        | 0.0000       | 0.5720      
{method 'execute' of 'psycopg2.extensions.cursor' objects} | 4        | 0.5720       | 0.5720      
/app/sde_collections/models/delta_patterns.py:109(update_affected_delta_urls_list) | 1        | 0.0000       | 0.0080      
/usr/local/lib/python3.10/site-packages/django/db/models/manager.py:85(manager_method) | 15/5     | 0.0000       | 0.0070      
/usr/local/lib/python3.10/site-packages/django/db/models/fields/related_descriptors.py:1206(set) | 1        | 0.0000       | 0.0060      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py:725(as_sql) | 17/4     | 0.0000       | 0.0060      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py:78(pre_sql_setup) | 17/5     | 0.0000       | 0.0060      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py:541(compile) | 176/58   | 0.0000       | 0.0040      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py:70(setup_query) | 17/5     | 0.0000       | 0.0040      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py:229(get_select) | 17/5     | 0.0000       | 0.0040      
/app/sde_collections/models/delta_url.py:60(get_queryset) | 3        | 0.0000       | 0.0030      
/app/sde_collections/models/delta_url.py:13(with_exclusion_status) | 3        | 0.0000       | 0.0030      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/query.py:1557(_add_q) | 21       | 0.0000       | 0.0030      

UNAPPLY OPERATION
Function Name                                      | Calls    | Total Time   | Cumulative  
--------------------------------------------------------------------------------------
/app/sde_collections/models/delta_patterns.py:555(unapply) | 1        | 0.0000       | 0.0130      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py:1532(execute_sql) | 6        | 0.0000       | 0.0060      
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:383(__iter__) | 2        | 0.0000       | 0.0050      
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:1879(_fetch_all) | 2        | 0.0000       | 0.0050      
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:85(__iter__) | 2        | 0.0000       | 0.0050      
/usr/local/lib/python3.10/site-packages/django/db/models/fields/related_descriptors.py:1173(clear) | 2        | 0.0000       | 0.0040      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py:725(as_sql) | 10/2     | 0.0000       | 0.0030      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py:78(pre_sql_setup) | 10/2     | 0.0000       | 0.0030      
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:1124(delete) | 4        | 0.0000       | 0.0020      
/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py:66(execute) | 6        | 0.0000       | 0.0020      
/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py:76(_execute_with_wrappers) | 6        | 0.0000       | 0.0020      
/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py:82(_execute) | 6        | 0.0000       | 0.0020      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/query.py:1557(_add_q) | 22/20    | 0.0000       | 0.0020      
{method 'execute' of 'psycopg2.extensions.cursor' objects} | 6        | 0.0020       | 0.0020      
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:1457(_filter_or_exclude_inplace) | 16       | 0.0000       | 0.0020      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/query.py:1532(add_q) | 16       | 0.0000       | 0.0020      
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:1584(annotate) | 4        | 0.0000       | 0.0020      
/usr/local/lib/python3.10/site-packages/django/db/models/sql/query.py:1130(add_annotation) | 20       | 0.0000       | 0.0020      
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:1446(_filter_or_exclude) | 16       | 0.0000       | 0.0020      
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:1599(_annotate) | 4        | 0.0000       | 0.0020

@bishwaspraveen
Copy link
Contributor 

================================================================================
COMPARATIVE SUMMARY
================================================================================

Dataset Size: 10
Pattern Type         | Operation | Total Functions | Total Time   | DB Queries
--------------------------------------------------------------------------------
DeltaExcludePattern  | Apply    | 486             | 0.5960       | 0         
                     | Unapply  | 486             | 0.0020       | 0         
DeltaIncludePattern  | Apply    | 486             | 0.5810       | 0         
                     | Unapply  | 481             | 0.0010       | 0         
DeltaDivisionPattern | Apply    | 488             | 0.5890       | 0         
                     | Unapply  | 497             | 0.0020       | 0         
DeltaDocumentTypePattern | Apply    | 488             | 0.5720       | 0         
                     | Unapply  | 497             | 0.0020       | 0         
DeltaTitlePattern    | Apply    | 486             | 0.5720       | 0         
                     | Unapply  | 504             | 0.0020       | 0         

Dataset Size: 100
Pattern Type         | Operation | Total Functions | Total Time   | DB Queries
--------------------------------------------------------------------------------
DeltaExcludePattern  | Apply    | 486             | 0.5780       | 0         
                     | Unapply  | 473             | 0.0020       | 0         
DeltaIncludePattern  | Apply    | 486             | 0.5900       | 0         
                     | Unapply  | 473             | 0.0010       | 0         
DeltaDivisionPattern | Apply    | 488             | 0.5840       | 0         
                     | Unapply  | 483             | 0.0020       | 0         
DeltaDocumentTypePattern | Apply    | 488             | 0.5710       | 0         
                     | Unapply  | 483             | 0.0030       | 0         
DeltaTitlePattern    | Apply    | 486             | 0.5660       | 0         
                     | Unapply  | 482             | 0.0020       | 0         

Dataset Size: 1000
Pattern Type         | Operation | Total Functions | Total Time   | DB Queries
--------------------------------------------------------------------------------
DeltaExcludePattern  | Apply    | 486             | 0.5720       | 0         
                     | Unapply  | 473             | 0.0020       | 0         
DeltaIncludePattern  | Apply    | 486             | 0.5670       | 0         
                     | Unapply  | 473             | 0.0020       | 0         
DeltaDivisionPattern | Apply    | 488             | 0.6130       | 0         
                     | Unapply  | 483             | 0.0020       | 0         
DeltaDocumentTypePattern | Apply    | 488             | 0.5710       | 0         
                     | Unapply  | 483             | 0.0010       | 0         
DeltaTitlePattern    | Apply    | 486             | 0.5760       | 0         
                     | Unapply  | 482             | 0.0020       | 0         

Dataset Size: 10000
Pattern Type         | Operation | Total Functions | Total Time   | DB Queries
--------------------------------------------------------------------------------
DeltaExcludePattern  | Apply    | 486             | 0.5810       | 0         
                     | Unapply  | 473             | 0.0010       | 0         
DeltaIncludePattern  | Apply    | 486             | 0.5900       | 0         
                     | Unapply  | 473             | 0.0010       | 0         
DeltaDivisionPattern | Apply    | 488             | 0.5730       | 0         
                     | Unapply  | 483             | 0.0010       | 0         
DeltaDocumentTypePattern | Apply    | 488             | 0.5650       | 0         
                     | Unapply  | 483             | 0.0020       | 0         
DeltaTitlePattern    | Apply    | 486             | 0.5740       | 0         
                     | Unapply  | 482             | 0.0020       | 0         

Dataset Size: 100000
Pattern Type         | Operation | Total Functions | Total Time   | DB Queries
--------------------------------------------------------------------------------
DeltaExcludePattern  | Apply    | 486             | 0.5850       | 0         
                     | Unapply  | 473             | 0.0020       | 0         
DeltaIncludePattern  | Apply    | 486             | 0.5820       | 0         
                     | Unapply  | 473             | 0.0020       | 0         
DeltaDivisionPattern | Apply    | 488             | 0.5850       | 0         
                     | Unapply  | 483             | 0.0010       | 0         
DeltaDocumentTypePattern | Apply    | 488             | 0.5770       | 0         
                     | Unapply  | 483             | 0.0020       | 0         
DeltaTitlePattern    | Apply    | 486             | 0.5770       | 0         
                     | Unapply  | 482             | 0.0020       | 0         

Dataset Size: 200000
Pattern Type         | Operation | Total Functions | Total Time   | DB Queries
--------------------------------------------------------------------------------
DeltaExcludePattern  | Apply    | 486             | 0.5890       | 0         
                     | Unapply  | 473             | 0.0020       | 0         
DeltaIncludePattern  | Apply    | 486             | 0.5760       | 0         
                     | Unapply  | 473             | 0.0010       | 0         
DeltaDivisionPattern | Apply    | 488             | 0.5900       | 0         
                     | Unapply  | 483             | 0.0020       | 0         
DeltaDocumentTypePattern | Apply    | 488             | 0.5740       | 0         
                     | Unapply  | 483             | 0.0020       | 0         
DeltaTitlePattern    | Apply    | 486             | 0.5920       | 0         
                     | Unapply  | 482             | 0.0020       | 0         
.
----------------------------------------------------------------------
Ran 1 test in 263.684s

@bishwaspraveen bishwaspraveen linked a pull request Feb 20, 2025 that will close this issue
added a script for benchmarking #1227
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bishwaspraveen bishwaspraveen

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

added a script for benchmarking NASA-IMPACT/COSMOS
2 participants
@dhanur-sharma @bishwaspraveen