Dear developers,
on your latest release you report significant speed up on prefill times using NPU (~9x) while at the same time decode seems to be much faster on CPU (~1.5x).
Is there a reason why inference is not split such that prefill is done by NPU and decode by CPU, essentially combining the best of both worlds?
Dear developers,
on your latest release you report significant speed up on prefill times using NPU (~9x) while at the same time decode seems to be much faster on CPU (~1.5x).
Is there a reason why inference is not split such that prefill is done by NPU and decode by CPU, essentially combining the best of both worlds?