Skip to content

nvcr container & backend hosting question #6454

@here4dadata

Description

@here4dadata

The nvcr.io/nvidia/tensorrt-llm/release:0.20.0 container does not include the triton_backend/ folder, is this the norm going forward for future containers? Should these templates just be copied out of the github project sep. before creating a repos folder for further triton backend hosting?

Is the guidance to also use llm_api/tensorrt_llm going forward, or is there a future for the inflight_batcher_llm and dissaggregated serving c++ methods?

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    Triton BackendRelated to NVIDIA Triton Inference Server backend

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions