How to show the result of offline CI for SYCL backend

[NeoZhangJianyu]

Background:
Current CI of SYCL backend is only building code, without running unit test.
Run the unit test need Intel GPU, which is not provided by github runner.
It depends on developer running local unit test before create PR.
For many reasons, the unit test pass rate is easy to be broken from 100%.
That leads to the result of OPs have obvious error, performance drop.

That impacts the whole SYCL backend quality and user experience.

Solution:
Offline CI for SYCL backend:
1. github doesn't recommend to setup self-hosted runner for public repo, due to security issue.
2. I run the unit test/performance for SYCL backend in my personal computer with Intel iGPU.
I choose the existed commits which change SYCL backend.
I could know the quality and performance of every commit.

Question:
How to show the result of offline CI for SYCL backend？
I hope to publish the result of every key commit.
It will help developer to improve the quality and user to choose the release with best quality and performance.

Alternative:
a. Update the summary of unit test result of SYCL backend in README-sycl.md
b. Insert the unit test pass rate in the title of merged PR.
c. Update the summary of test result in another public repo MD file and insert the link in README-sycl.md. So user is easy to read it to check the latest SYCL backend test result.

I choose c.
How about your idea?

@ggerganov @slaren @airMeng

[phymbert]

We successfully deployed an approach with self-hosted github runner for CUDA, maybe the same can be done for SYCL ?

• JIT GitHub docker runner ci#2
• server: continuous performance monitoring and PR comment #6283
• server: bench: continuous performance testing #6233

[tonym97]

Perfect.

[tonym97]

How many systems would you need @ggerganov with Intel GPUs? What's your typical CI/CD requirements?

[ggerganov]

One system should be enough so let's start with that. In the future if you have a variety of Intel GPU hardware that you would like to be exercised we can add more nodes

The CI requires at least a 128 GB disk since it downloads a few models that can be tens of GBs. It runs Ubuntu 22.04 and requires the following packages (+ any SYCL related stuff) to be installed:

sudo apt install vim curl cmake g++ python3-pip python3-venv unzip ccache

No other requirements are necessary except for an SSH access so I can login and setup ggml-ci to run

[tonym97]

Got it. Will get back to you in short order. Thanks!

[tonym97]

@ggerganov I have some buy in, I wanted to chat with you to align hardware requirements since you already provided software above. I'm going to drop you an email if that's ok.