Interest in Project 14 – My Previous OpenVINO Contributions (PR 5544, #6676, #4953)

[yunji-yunji]

Hello mentors, @alexsu52 @AlexanderDokuchaev

My name is Yunji Kim, and I recently completed a Master’s in Computer Science at the University of Waterloo, Canada. I am writing to express my strong interest in Project 14, "Accelerating Inference of NNCF-Compressed LLMs with Triton." After reviewing all the project ideas, I found that this one perfectly aligns with my previous experiences and passion for machine learning inference acceleration.

Relevant Work Experience

1. GPU Kernel Implementation (Internship, OpenVINO Team)

• Skills: GPU kernel programming (OpenCL), C++, OpenVINO, various ML frameworks (ONNX, PyTorch, TensorFlow, Caffe)
• My contributions:
  • PR [IE CLDNN] Fix DetectionOutput cpu implementation to be aligned with ngraph #5544
  • PR [GPU] Add new operation GatherElements to IE clDNN plugin #6676
  • PR [IE CLDNN] lower the tolerance of scatter elements update fusing test from 1 to 1e-5 #4953

2. Porting Transformer model to C Porting for migration on chip (SK Hynix)

• Skills: C, Python, PyTorch
• Contribution: I ported SwinIR Vision Transformer from PyTorch to C, implementing all necessary functions (e.g., matrix transpose, convolution, multiplication, activation functions) without external libraries.

Questions about the project

1. Dynamic INT8 group quantization: Are there any implementation differences or anything that should be considered when comparing dynamic quantization in frameworks such as TensorFlow or PyTorch to dynamic quantization in the inference engine like OpenVINO?
2. NF4 Implementation: Should my own NF4 be implemented by following the concepts in the QLoRA paper, or are there additional code or documentations that I can refer to?
3. When conducting performance profiling, is there a specific target hardware where performance improvement is particularly desired? Or is the goal to achieve general performance improvements across multiple hardware platforms?

Through all these experiences, I have gained a solid understanding of OpenVINO’s execution flow and backend integration while deepening my expertise in GPU kernel programming. Additionally, I am familiar with implementing various matrix operations for ML.

While I have not yet written a Triton kernel, I have already begun studying Triton and NNCF’s algorithms to prepare for this project. I am highly motivated to contribute to OpenVINO again. If you believe additional experience in Triton or CUDA would be beneficial before applying, I would greatly appreciate your advice.

Thank you for your time and guidance. I look forward to your response.

Best regards,
Yunji Kim

[alexsu52]

Hello @yunji-yunji, Thanks for your interest to the project and for the submitted proposal.

Answers to your questions:

1. The implementation of dynamic group quantization of int8 in OpenVINO is similar to the implementation in the PyTorch framework.
2. NNCF supports NF4 for OpenVINO backend. Details in the code: https://github.com/openvinotoolkit/nncf/blob/develop/nncf/quantization/algorithms/weight_compression/weight_lowering.py#L31
3. We want to exploit the Triton advantage to gain performance improvements across multiple hardware platforms.

[yunji-yunji]

Hello @alexsu52 ,

Thank you so much for your clear answers to my questions.
I'll definitely dive into the NNCF code you shared regarding NF4 implementation and focus my stuies on these points and the information you've provided.

Thanks again for your guidance.