Here's the top entries of time spent in the BSHD attention test:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   28.997   28.997 main.py:276(wrap_session)
        1    0.000    0.000   28.507   28.507 main.py:339(_main)
    16/12    0.000    0.000   28.466    2.372 runner.py:323(from_call)
        1    0.000    0.000   27.737   27.737 main.py:356(pytest_runtestloop)
        4    0.000    0.000   27.737    6.934 runner.py:114(pytest_runtest_protocol)
        4    0.000    0.000   27.736    6.934 runner.py:122(runtestprotocol)
       10    0.000    0.000   27.736    2.774 runner.py:230(call_and_report)
       10    0.000    0.000   27.735    2.773 runner.py:246(<lambda>)
        2    0.000    0.000   27.730   13.865 runner.py:167(pytest_runtest_call)
        2    0.000    0.000   27.730   13.865 python.py:1669(runtest)
        2    0.000    0.000   27.730   13.865 python.py:150(pytest_pyfunc_call)
        2    0.001    0.000   27.730   13.865 vanilla_attention_test.py:1153(testAttentionBSHD_Prefetch_MultiBuffer)
        2    0.012    0.006   24.627   12.313 compile.py:899(wave_compile)
        2    0.000    0.000   23.359   11.679 compile.py:655(_trace_launchable_and_get_kernel_signature)
37018/36089    1.034    0.000   16.628    0.000 basic.py:965(subs)
        2    0.001    0.001   14.784    7.392 compile.py:567(compile_launchable_to_mlir)
        2    0.000    0.000   14.661    7.330 emitter.py:229(emit)
      8/2    0.001    0.000   14.636    7.318 emitter.py:455(_emit_graph)
2670/2022    0.003    0.000   14.636    0.007 emitter.py:463(_emit_function_call_node)
      600    0.006    0.000   12.689    0.021 read_write.py:668(handle_read)
    10226    0.079    0.000    8.503    0.001 emitter.py:592(gen_sympy_index)
   839782    1.601    0.000    7.258    0.000 basic.py:2153(is_same)
   100/90    0.000    0.000    7.088    0.079 print_utils.py:763(try_apply_pass)
      632    0.012    0.000    6.419    0.010 read_write.py:150(_build_start_indices)
      936    0.009    0.000    6.052    0.006 mapping_utils.py:323(transform_index_on_mapping)
  1841872    0.614    0.000    4.859    0.000 numbers.py:1965(__eq__)
       20    0.000    0.000    4.638    0.232 _jit_internal.py:1027(_overload)
3192308/3192092    2.434    0.000    4.500    0.000 sympify.py:124(sympify)
  1817353    0.522    0.000    4.226    0.000 numbers.py:1622(__eq__)
1976395/1820915    0.656    0.000    4.183    0.000 cache.py:69(wrapper)
  1853295    0.345    0.000    3.721    0.000 sympify.py:514(_sympify)
        2    0.000    0.000    3.333    1.667 handlers.py:1477(handle_iterate)
      271    0.003    0.000    3.275    0.012 __init__.py:1(<module>)
92578/41997    0.055    0.000    3.144    0.000 basic.py:1176(_subs)
      114    0.006    0.000    3.072    0.027 tracing.py:193(walk)
908269/462645    0.622    0.000    2.874    0.000 sorting.py:202(ordered)
      632    0.009    0.000    2.786    0.004 read_write.py:457(_create_vec_read_write)
        2    0.003    0.002    2.721    1.360 partition_strided_operators.py:395(partition_gather_like_ops)
59157/36336    0.076    0.000    2.674    0.000 basic.py:1248(fallback)
     4680    0.004    0.000    2.624    0.001 mapping_utils.py:337(<genexpr>)
     2770    0.001    0.000    2.506    0.001 partition_strided_operators.py:414(has_gather_mapping)
      194    0.004    0.000    2.453    0.013 mapping_utils.py:242(check_is_mapping_contiguous)
372265/350006    0.155    0.000    2.114    0.000 wave_ops.py:584(get_custom)
        2    0.000    0.000    1.991    0.996 compile.py:218(__call__)
        2    0.000    0.000    1.991    0.996 compile.py:221(invoke)
        2    0.000    0.000    1.991    0.995 launch.py:198(__call__)
        2    0.000    0.000    1.975    0.988 device.py:657(get_device_from_torch)
        2    0.000    0.000    1.975    0.988 device.py:625(lookup_device_from_torch)

(obtained by cprofiler)

monstrous amounts of time are spent in symbolic manipulation, at a point where these symbols don't matter much since decisions have been taken (local storage, expansion) based on their specific values.

Bring the CI time to <10 minutes #612

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions