Native Test Report Integration Proposal

[timhuynh94]

In this proposal, there 2 approaches being proposed for native test-report step.

1. Dedicate docker image
2. Leveraging container output enhancement

[wass3rw3rk]

just first round of feedback. i might do a separate one for Details section which looks pretty good, but want to review more closely. Will probably just respond as comments instead of reviewing PR.

[wass3rw3rk]

maybe as part of the description it would be appropriate to list out similar capabilities from other CICD systems, here's a couple of examples:

• https://docs.semaphoreci.com/using-semaphore/tests/test-reports
• https://circleci.com/docs/collect-test-data/
• https://docs.travis-ci.com/user/uploading-artifacts/
• https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/storing-and-sharing-data-from-a-workflow#uploading-build-and-test-artifacts
• https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html#how-to-set-it-up
• https://buildkite.com/docs/test-engine/ci-environments
• https://codefresh.io/docs/docs/testing/test-reports/

alternatively, this could also be represented in the "Why is this required?" to support that section.

[wass3rw3rk]

Phase 1: Basic Test Reporting
    Implement the test-report step and backend support for storing test results.
    Simple UI to display test results.
Phase 2: Advanced Features
    Add code coverage, performance metrics, and code quality reporting.
    Integrate object storage for test artifacts.
    Enhance the UI to visualize test data and metrics.
Phase 3: Slack Integration
    Implement Slack notifications for test results and flaky tests.
    Allow users to configure Slack settings in .vela.yml.
Phase 4: Historical Data and Analytics
    Develop a dashboard to show historical test data and trends.
    Add analytics features to track test performance over time.

using your implementation phases as a guide, what about the following order:

1. Basic Test Reporting
   • Integrate Object Storage (ability to hook up Vela to a storage system)
     • includes backend, database, api work
   • Implement means for a user to store test results from a step to the storage system
   • UI/UX research
2. UI
   • UI to display test results/data, including code coverage, perf metrics, etc
3. Enhanced UI
   • Enhance UI with visualizations and dashboards/historical data (trends)

In terms of chunks of work and adding incremental value, those phases seem appropriate. Phase 1 is pretty heavy, so i pulled UI out and turned that into two phases for the whole effort and combined some points. There probably could be more detailed subsections within each phase but at a high level, what do you think?

You will notice that I dropped the Slack integration part. I think this could be a bigger play and separate effort to integrate native ways of triggering notifications, not limited to Slack or this test reporting feature. In the meantime, there are ways to leverage something like the Slack plugin today, I think.

[JordanSussman]

As discussed within Slack, it would be nice if the storage pattern developed as part of this approach could be re-usable to enable future use-cases similar to GitHub Actions artifacts.