Cosmetology - investigate 503 errors from OpenSearch in test environment

[landonshumway-ia]

While testing the staff user search capability in the test environment, OpenSearch returned several 503 status codes after the very first license records were uploaded into the new system, as shown in these logs:

"stack_trace": {
        "type": "TransportError",
        "value": "TransportError(503, 'search_phase_execution_exception')",
        "module": "opensearchpy.exceptions",
        "frames": [
            {
                "file": "/opt/python/cc_common/utils.py",
                "line": 136,
                "function": "caught_handler",
                "statement": "'body': json.dumps(fn(event, context), cls=ResponseEncoder),"
            },
            {
                "file": "/opt/python/aws_lambda_powertools/logging/logger.py",
                "line": 554,
                "function": "decorate",
                "statement": "return lambda_handler(event, context, *args, **kwargs)"
            },
            {
                "file": "/opt/python/cc_common/utils.py",
                "line": 271,
                "function": "authorized",
                "statement": "return fn(event, context)"
            },
            {
                "file": "/var/task/handlers/search.py",
                "line": 42,
                "function": "search_api_handler",
                "statement": "return _search_providers(event, context)"
            },
            {
                "file": "/var/task/handlers/search.py",
                "line": 78,
                "function": "_search_providers",
                "statement": "response = opensearch_client.search(index_name=index_name, body=search_body)"
            },
            {
                "file": "/var/task/opensearch_client.py",
                "line": 156,
                "function": "search",
                "statement": "return self._client.search(index=index_name, body=body)"
            },
            {
                "file": "/var/task/opensearchpy/client/utils.py",
                "line": 176,
                "function": "_wrapped",
                "statement": "return func(*args, params=params, headers=headers, **kwargs)"
            },
            {
                "file": "/var/task/opensearchpy/client/__init__.py",
                "line": 2445,
                "function": "search",
                "statement": "return self.transport.perform_request("
            },
            {
                "file": "/var/task/opensearchpy/transport.py",
                "line": 455,
                "function": "perform_request",
                "statement": "raise e"
            },
            {
                "file": "/var/task/opensearchpy/transport.py",
                "line": 418,
                "function": "perform_request",
                "statement": "status, headers_response, data = connection.perform_request("
            },
            {
                "file": "/var/task/opensearchpy/connection/http_requests.py",
                "line": 242,
                "function": "perform_request",
                "statement": "self._raise_error("
            },
            {
                "file": "/var/task/opensearchpy/connection/base.py",
                "line": 315,
                "function": "_raise_error",
                "statement": "raise HTTP_EXCEPTIONS.get(status_code, TransportError)("
            }
        ]
    }

The search functionality later worked as expected. This error could potentially be the result of the OpenSearch Domain being empty when we attempted to search against it, but more investigation is required.

Notes

Questions

Assumptions

Estimate

Tasks

• TODO
• Automated tests
• API Docs
• Postman collection
• PR opened with labels / reviewers / assignee / linked-issue

Implementation Notes