Guidelines on how to contribute

[jplehr]

Hi everyone. We would like to start contributing to HeCBench with fixes and potentially more, if there is an appetite for it.

We have a few questions on how to contribute best:

• Should we file issues and then PRs to address those issues?
• Should we just open PRs with fixes, or should we open issues and then PRs?
• Is there a policy around AI-assisted contributions (human-reviewed before PRs)?

Thanks for your answers.

[zjin-lcf]

Hello @jplehr,

I appreciate your descriptions of an issue as I don't fully understand the implementation of every program. However, a PR with fixes may include descriptions of an issue as well. If AI creates a PR or a few PRs, they may address issues for one benchmark. There are issues that may exist in many programs. For example, "sm_60" in Makefiles is not supported by a recent CUDA compiler, then a PR (sm_60 -> sm_75) for all benchmarks is better and simpler. Thank you for your questions and contributions!

[keitaTN]

This is good suggestion. We will discuss this with the leads (me, Aaron Young and Jeff Vetter) team of this project.
There are action items.

1. I will reach out Olga Pearce at LLNL who is leading Benchmark Software Ecosystem across DOE. She is working across multiple benchmark software projects.
2. I am leaning toward opening issues/bugs, and then PR for update/fix indicating which issues being addressed.
3. We need to address AI contibutions as RIKEN is preparing for this. Again, I will talk to Olga for recommendations.

[keitaTN]

Internally, we are leaning toward to establish criteria for adding new benchmark code, and some repository structure changes.

[Geekdude]

Hi all, here are my thoughts:

1. We need to establish a contributing policy, either in the README.md or in a CONTRIBUTING.md file. This statement should also have statements about our expectation around AI use.
2. As far as contribution procedure, if it's a new benchmark or feature, I'm good with a PR only without an issue as there is no issue it is addressing. If instead the PR is addressing a bug or issue with an existing benchmark, then creating the issue first, then addressing it with the PR is my preference.
3. Regarding AI, this is a fast-moving area in the community, but my current position is that the quality of the code must be ensured by a human, I do not want fully agentic contributions without a human review, and a human must take ownership/responsibility of the code. If AI is used, I would like the committer to include the Assisted-by tag in the commit with information about the module used. For example Assisted-by: Claude Opus 4.5, these kind of tags are easy to add if the Agent is creating the commits. Just tell them to use Assisted-by instead of Co-authored-by which they seem to default to. See Assisted-by: How open source projects are drawing the line on AI contributions for a recent article I read on this.

[keitaTN]

We had a monthly meeting today. We are going to post a draft for the rule of PR and commit. Issues and PRs can be submitted by anybody. Any PR must be corresponding to a fork/branch. Each PR needs approvals from 2 reviewers before merging. We will implement this very soon once the formal draft is ready (will be posted soon).

@zjin-lcf I want you to be aware of this change. Please let me know if you have any questions.

[jplehr]

Thank you for all the remarks and thoughts.
We are sorting out a few things on our end as well. Once done, I'll check again for a contribution guide.

I agree with the human-ownership of the code that is submitted, as it is ultimately taking reviewer bandwidth.