Description
When a test fails, it can be helpful to add a comment to the PR to specify exactly what failed, with appropriate links.
When using the Develocity integration, we could possibly do it:
- On the first failed build, we create a comment on the PR (see below for content) -- and include a hidden HTML marker (HTML comment) containing the commit hash this PR comment refers to.
- On subsequent failed builds, we look for comments from the bot on the PR, and minimize those whose HTML marker refers a different (older) commit hash. If we find a PR comment containing the HTML marker for the current commit, we update that PR comment; if we don't find one, we create a new PR comment.
The content of the PR comment could take inspiration from what Guillaume did on Quarkus. But:
- We'd want a single comment for all "workflows/jobs" (GitHub Actions/Jenkins)
- The code from Quarkus is based on Maven Surefire output and cannot (easily) be reused for Gradle-based builds.
- So I'd suggest to base all our data on Develocity metadata: I think the Develocity APIs allow us to list failing tests, so that would be a great start?
Something like the example below would be a start. Note the idea with the tests is to present, for each failed test, links to test failures as well as test successes, labelling both failures and tests with relevant Develocity Build Scan labels.
3 builds failed
Job/Workflow | OS | Java | DB | Goals/Tasks | Results |
---|---|---|---|---|---|
Hibernate ORM build | Linux |
17 |
sybase |
check |
🔴 ❌ 📃 |
Hibernate ORM build | Linux |
17 |
mssql |
check |
🔴 ❌ 📃 |
Hibernate ORM build | Linux |
17 |
mysql |
check |
🔴 ❌ 📃 |
12 builds succeeded
Job/Workflow | OS | Java | DB | Goals/Tasks | Results |
---|---|---|---|---|---|
Hibernate ORM build | Linux |
17 |
hsqldb |
check |
🟢 ✅ 📃 |
Hibernate ORM build | Linux |
17 |
h2 |
check |
🟢 ✅ 📃 |
Hibernate ORM build | Linux |
17 |
edb |
check |
🟢 ✅ 📃 |
Hibernate ORM build | Linux |
17 |
pgsql |
check |
🟢 ✅ 📃 |
Hibernate ORM build | Linux |
17 |
mariadb |
check |
🟢 ✅ 📃 |
Hibernate ORM build | Linux |
17 |
oracle |
check |
🟢 ✅ 📃 |
Hibernate ORM build | Linux |
17 |
db2 |
check |
🟢 ✅ 📃 |
Hibernate ORM build-Atlas | Linux |
21 |
oracle_cloud_autonomous |
check |
🟢 ✅ 📃 |
Hibernate ORM build-Atlas | Linux |
21 |
oracle_cloud_db19c |
check |
🟢 ✅ 📃 |
Hibernate ORM build-Atlas | Linux |
21 |
oracle_cloud_db23c |
check |
🟢 ✅ 📃 |
Hibernate ORM build-Atlas | Linux |
21 |
oracle_cloud_db21c |
check |
🟢 ✅ 📃 |
hibernate-orm-pipeline/PR-8932 | Linux |
17 |
tidb |
check |
🟢 ✅ 📃 |
14 tests failed
:hibernate-core:test
Test | Failures ❌ | Successes ✅ | |
---|---|---|---|
|
|||
↳ |
|
|
|
↳ |
|
|
|
↳ |
|