Problem
Intermittent embedding failures surface as:
Failed to generate batch embeddings: TEI embedding request failed: [Errno 9] Bad file descriptor
Observed ~once daily in a production deployment, typically after idle periods or infrastructure disturbances (e.g. shortly after a database failover shuffled worker activity) — consistent with the HTTP client reusing a connection whose socket died underneath it, rather than any TEI-server fault (TEI pods healthy, zero restarts across occurrences).
Task-level retries absorb it today, but each hit fails a consolidation/retain attempt and re-runs the whole operation.
Proposed fix
- Enable/verify connection health-checking on the TEI HTTP client pool (e.g. httpx/aiohttp keepalive expiry shorter than infra idle timeouts, or retry-once-on-
OSError at the request level before failing the batch).
- A single request-level retry on stale-socket
OSErrors is cheap and converts a full task retry into a transparent reconnect.
Problem
Intermittent embedding failures surface as:
Observed ~once daily in a production deployment, typically after idle periods or infrastructure disturbances (e.g. shortly after a database failover shuffled worker activity) — consistent with the HTTP client reusing a connection whose socket died underneath it, rather than any TEI-server fault (TEI pods healthy, zero restarts across occurrences).
Task-level retries absorb it today, but each hit fails a consolidation/retain attempt and re-runs the whole operation.
Proposed fix
OSErrorat the request level before failing the batch).OSErrors is cheap and converts a full task retry into a transparent reconnect.