[BLOCKED] [Search] Caching for VA Search API

[jilladams]

Description

This is a clone of the same ticket for the Forms API, except in this case, for Search: #15359
We only need to SPIKE once, so if we prefer to focus on Search first or Forms first, doesn't matter. But the solution would need to be implemented separately for each API.

Background

The VA Search API gets heavy traffic spikes at times, which has caused some issues with the API returning 5xx responses, causing Veterans the inability to access their necessary Search.

[todo: link to issues, slack threads, etc].

User story

AS A Veteran or other user of Search available via VA.gov
I WANT to have access to the Search I need without encountering any errors
SO THAT I can complete them without frustration and get the services/aid/etc. that I need

Engineering notes / background

With no caching in place, each and every request to the VA Search API is routed through Lighthouse and ultimately to the postgres database.

In order to understand what our caching options are, we need to explore the path a request takes from browser to Lighthouse. We may find that implementing a Redis cache at the /v0/search endpoint could greatly impact the Veteran performance. Or, we may find that the path to the /v0/search endpoint takes won't allow for an in-memory cache. In that case, perhaps there is an HTTP proxy cache we could leverage.

Analytics considerations

Quality / testing notes

Acceptance criteria

• TBD whether SPIKE has already occurred. If we SPIKE here, convert this to a spike ticket. If we already spiked in [Forms] SPIKE: Caching for VA Forms API #15359, this ticket is only for implementation for Search API

[FranECross]

@dsasser Regarding a Spike to investigate caching, would it make a difference in ease of investigation whether you "spiked" search or forms? If not, then I'll leave this one a Spike and you can look at it first, then use the info to implement for Search. cc @jilladams

[FranECross]

Removed from [this epic](Monitoring enhancements for API-driven Public Websites products#15262), and added to the Misc API Q3 2024 epic.