[Feature][TPU host offload] Revise KV Cache slicing, gathering, ... operations

[juncgu-google]

Description

1. small tweaks in host-hbm dma functions, and re-define the interface of the jitted kv_cache_swap_fn for both pallas and jax mode.
2. use the jitted gather / slice kv_cache functions from KVCacheManager to replaces the corresponding operations in the save / load workflow.
3. Due to a core crash issue, currently only allow to use jax.device_put for both swap-in / out. For the swap-in operations, the host kv data will apply SingleDeviceSharding(CpuDevice) instead of NamedSharding(memory_kind="pinned_host").

Tests

pytest -sv tests/distributed/host_offloading_accuracy_test.py
pytest -sv tests/distributed/cpu_offloading_kv_roundtrip_test.py

Checklist

Before submitting this PR, please make sure:

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have made or will make corresponding changes to any relevant documentation.

[github-actions]

Description

Start with a short description of what the PR does and how this is a change from
the past.

The rest of the description includes relevant details and context, examples:

• why is this change being made,
• the problem being solved and any relevant context,
• why this is a good solution,
• some information about the specific implementation,
• shortcomings of the solution and possible future improvements.

If the change fixes a bug or a Github issue, please include a link, e.g.,:
FIXES: b/123456
FIXES: #123456

Tests

Please describe how you tested this change, and include any instructions and/or
commands to reproduce.

Checklist

Before submitting this PR, please make sure:

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have made or will make corresponding changes to any relevant documentation.