Errors are being swallowed (I think)

[gadomski]

I believe that exceptions aren't being allowed to propagate up correctly, leading to a server error (instead of a more informative message). The error occurs when attempting to decode the body of a error response. Apologies that I don't have an actual reproducer, I'll try to create one and will post back here if I can.

One guess on the source is that in ProcessLinksMiddleware, should_transform_response doesn't have a guard for status, so it's trying to transform any response with JSON content type?

Here's the traceback from an innocuous /search call to a STAC API w/ stac-auth-proxy in front of it. Notice how we're in the error handling code of starlette when we throw our own error b/c we can't decode the JSON.

Traceback (most recent call last):
  File "/var/task/mangum/protocols/http.py", line 66, in run
    await app(self.scope, self.receive, self.send)
  File "/var/task/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/var/task/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/var/task/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/var/task/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/var/task/starlette_cramjam/middleware.py", line 124, in __call__
    await self.app(scope, receive, send)
  File "/var/task/stac_auth_proxy/utils/middleware.py", line 101, in __call__
    return await self.app(scope, receive, transform_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/stac_auth_proxy/middleware/EnforceAuthMiddleware.py", line 113, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/starlette/middleware/base.py", line 182, in __call__
    with recv_stream, send_stream, collapse_excgroups():
                                   ~~~~~~~~~~~~~~~~~~^^
  File "/var/lang/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/var/task/starlette/_utils.py", line 83, in collapse_excgroups
    raise exc
  File "/var/task/starlette/middleware/base.py", line 184, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/stac_auth_proxy/middleware/AddProcessTimeHeaderMiddleware.py", line 15, in dispatch
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/starlette/middleware/base.py", line 159, in call_next
    raise app_exc
  File "/var/task/starlette/middleware/base.py", line 144, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/var/task/stac_auth_proxy/utils/middleware.py", line 101, in __call__
    return await self.app(scope, receive, transform_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/stac_auth_proxy/utils/middleware.py", line 101, in __call__
    return await self.app(scope, receive, transform_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/var/task/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/var/task/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/var/task/starlette/routing.py", line 716, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/var/task/starlette/routing.py", line 736, in app
    await route.handle(scope, receive, send)
  File "/var/task/starlette/routing.py", line 290, in handle
    await self.app(scope, receive, send)
  File "/var/task/starlette/routing.py", line 78, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/var/task/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/var/task/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/var/task/starlette/routing.py", line 76, in app
    await response(scope, receive, send)
  File "/var/task/starlette/responses.py", line 165, in __call__
    await send({"type": prefix + "http.response.body", "body": self.body})
  File "/var/task/starlette/_exception_handler.py", line 39, in sender
    await send(message)
  File "/var/task/starlette/_exception_handler.py", line 39, in sender
    await send(message)
  File "/var/task/stac_auth_proxy/utils/middleware.py", line 81, in transform_response
    data = json.loads(body)
  File "/var/lang/lib/python3.13/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/var/lang/lib/python3.13/json/decoder.py", line 345, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lang/lib/python3.13/json/decoder.py", line 363, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[alukach]

One guess on the source is that in ProcessLinksMiddleware, should_transform_response doesn't have a guard for status, so it's trying to transform any response with JSON content type?

I think this is a correct interpretation, however I am not sure what harm is being done by attempting to augment the links of any JSON response. Our get_links function is pretty gentle:

stac-auth-proxy/src/stac_auth_proxy/utils/stac.py

Lines 6 to 18 in 34ba6b7

	def get_links(data: dict) -> chain[dict]:
	"""Get all links from a STAC response."""
	return chain(
	# Item/Collection
	data.get("links", []),
	# Collections/Items/Search
	(
	link
	for prop in ["features", "collections"]
	for object_with_links in data.get(prop, [])
	for link in object_with_links.get("links", [])
	),
	)

As you said above, the error itself is due to a failure to decode the JSON response. To this specific instance of the error, it seems as though either 1) the upstream STAC API is not returning JSON, or 2) we somehow mutated the response body to be invalid JSON.

In a larger view, we could also ask what we should do with a response if it's not JSON. Perhaps just skip any specific middleware step and return it?

[gadomski]

Perhaps just skip any specific middleware step and return it?

I think this would be the most graceful option, in an error case for sure.

[alukach]

I think this would be the most graceful option, in an error case for sure.

My concern is that it's possible that the middleware is enforcing some auth function. We wouldn't want bad data to circumvent security. For now, I think perhaps just sending an error to the user (see #73) may be the most security-conscious solution. Perhaps in the future, we could add a flag so each middleware could decide its fallback mode in the event of decoding errors.