feat(file_processors): route docling VLM processing through stack inference API#6225
feat(file_processors): route docling VLM processing through stack inference API#6225alinaryan wants to merge 3 commits into
Conversation
alinaryan
requested review from
bbrowning,
cdoern,
franciscojavierarceo,
leseb,
mattf,
raghotham and
skamenan7
as code owners
…erence API Add VLM-based document processing to the inline docling provider by creating a custom VLM engine that calls the stack's inference API in-process via dependency injection, avoiding external HTTP round-trips. When vlm_model is configured and an inference provider is available, the provider initializes docling's VlmPipeline and replaces the default API engine with OgxInferenceVlmEngine, which bridges docling's synchronous pipeline to the async inference API using asyncio.run_coroutine_threadsafe. When no VLM model is configured or no inference API is available, the provider gracefully degrades to the standard non-VLM pipeline. Signed-off-by: Alina Ryan <aliryan@redhat.com>
alinaryan
force-pushed
the
feat/docling-vlm-inference
branch
from
0672897 to
de4c4d4
Compare
mattf
left a comment
mattf
left a comment
There was a problem hiding this comment.
some inline comments.
which llm did you use for this?
| log.warning( | ||
| "VLM model configured but no inference API available, falling back to standard pipeline", | ||
| vlm_model=self.config.vlm_model, | ||
| ) |
There was a problem hiding this comment.
this could be a startup error. the admin configured use of a vlm model and it's not being honored.
| from docling.datamodel.pipeline_options import VlmConvertOptions | ||
| from docling.datamodel.vlm_engine_options import ApiVlmEngineOptions, VlmEngineType | ||
| from docling.pipeline.vlm_pipeline import VlmPipeline | ||
|
|
||
| from .vlm_engine import OgxInferenceVlmEngine |
| self.config.vlm_preset, | ||
| engine_options=ApiVlmEngineOptions(engine_type=VlmEngineType.API_OPENAI), | ||
| ) | ||
| except KeyError: |
There was a problem hiding this comment.
please me more specific. check for the expected values and error if they aren't present.
| stream=False, | ||
| ) | ||
| future = asyncio.run_coroutine_threadsafe( | ||
| self.inference_api.openai_chat_completion(request), |
There was a problem hiding this comment.
will this handle provider data credentials?
There was a problem hiding this comment.
Yes the engine calls the stack's inference API in-process via the Python object (inference_api.openai_chat_completion()), not over HTTP. Auth credentials (API keys, tokens, etc.) are configured on the inference provider itself and are already resolved by the time the call reaches the provider implementation. This is the same dependency injection pattern used by the responses and file_search providers.
|
This pull request has merge conflicts that must be resolved before it can be merged. @alinaryan please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork |
Raise a startup error when vlm_model is configured but no inference provider is available, instead of silently falling back. Validate vlm_preset against docling's registered presets with a clear error listing available values. Move VLM-related imports to module level. Signed-off-by: Alina Ryan <aliryan@redhat.com>
Keep both the new supported_mime_types() method from upstream and the VLM pipeline methods from this branch. Signed-off-by: Alina Ryan <aliryan@redhat.com>
@mattf I used Claude Opus 4.6 |
3 participants
What does this PR do?
Adds VLM-based document processing to the inline docling file processor provider. When vlm_model is configured and an inference provider is available, the provider uses docling's VlmPipeline for richer document understanding (layout analysis, OCR via vision models). Inference is routed through the stack's model-serving infrastructure in-process — no separate GPU resources or external HTTP calls are needed for document processing.
A custom BaseVlmEngine subclass (OgxInferenceVlmEngine) bridges docling's synchronous pipeline to the stack's async inference API using asyncio.run_coroutine_threadsafe(). The engine is injected into the docling pipeline at initialization time, replacing the default ApiVlmEngine.
When no VLM model is configured or no inference API is available, the provider gracefully degrades to the existing non-VLM pipeline with no behavior change.
file_processors:
- provider_id: docling
provider_type: inline::docling
config:
vlm_model: granite-docling-258M
vlm_preset: granite_docling
Test Plan