ValueError: metadata was invalid occurs when key in request metadata contains uppercase charactors

[parthea]

See the following test which fails when an uppercase is used in the metadata key

gapic-generator-python/tests/system/test_request_metadata.py

Lines 20 to 28 in c72411d

	def test_metadata_string(echo):
	echo.echo(
	showcase.EchoRequest(
	content="The hail in Wales falls mainly on the snails.",
	request_id="some_value",
	other_request_id="",
	),
	metadata=[('some-key', 'some_value')]
	)

def test_metadata_string(echo):
    echo.echo(
        showcase.EchoRequest(
            content="The hail in Wales falls mainly on the snails.",
            request_id="some_value",
            other_request_id="",
        ),
        metadata=[('Some-Key', 'some_value')] <----- `Some-key` will fail but `some-key` works
    )

======================================================================================================================================== FAILURES =========================================================================================================================================
_______________________________________________________________________________________________________________________________ test_metadata_string[grpc] ________________________________________________________________________________________________________________________________

echo = <google.showcase_v1beta1.services.echo.client.EchoClient object at 0x7f35d5f12460>

    def test_metadata_string(echo):
>       echo.echo(
            showcase.EchoRequest(
                content="The hail in Wales falls mainly on the snails.",
                request_id="some_value",
                other_request_id="",
            ),
            metadata=[('Some-Key', 'some_value')]
        )

tests/system/test_request_metadata.py:21: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/tmp/tmptwfw_f5e/google/showcase_v1beta1/services/echo/client.py:684: in echo
    response = rpc(
.nox/showcase-3-9/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py:131: in __call__
    return wrapped_func(*args, **kwargs)
.nox/showcase-3-9/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py:293: in retry_wrapped_func
    return retry_target(
.nox/showcase-3-9/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py:153: in retry_target
    _retry_error_helper(
.nox/showcase-3-9/lib/python3.9/site-packages/google/api_core/retry/retry_base.py:212: in _retry_error_helper
    raise final_exc from source_exc
.nox/showcase-3-9/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py:144: in retry_target
    result = target()
.nox/showcase-3-9/lib/python3.9/site-packages/google/api_core/timeout.py:120: in func_with_timeout
    return func(*args, **kwargs)
.nox/showcase-3-9/lib/python3.9/site-packages/google/api_core/grpc_helpers.py:76: in error_remapped_callable
    return callable_(*args, **kwargs)
.nox/showcase-3-9/lib/python3.9/site-packages/grpc/_channel.py:1178: in __call__
    ) = self._blocking(
.nox/showcase-3-9/lib/python3.9/site-packages/grpc/_channel.py:1146: in _blocking
    call = self._channel.segregated_call(
src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi:547: in grpc._cython.cygrpc.Channel.segregated_call
    ???
src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi:416: in grpc._cython.cygrpc._segregated_call
    ???
src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi:410: in grpc._cython.cygrpc._segregated_call
    ???
src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi:262: in grpc._cython.cygrpc._call
    ???
src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi:305: in grpc._cython.cygrpc._call
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   ValueError: metadata was invalid: [('Some-Key', 'some_value'), ('x-goog-api-version', 'v1_20240408'), ('x-goog-api-client', 'gl-python/3.9.16 grpc/1.68.0 gax/2.23.0 gapic/0.0.0')]

src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi:62: ValueError
---------------------------------------------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------------------------------------------
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1732628608.393651 1738620 filter_stack_call.cc:399] validate_metadata: INTERNAL:Illegal header key

[vchudnov-g]

Is this the error being returned by Showcase?

[parthea]

The exception ValueError: metadata was invalid is raised by gRPC.

From https://github.com/grpc/grpc/blob/ca0d85cea5242d6740ba6cf1dc5c0b9647856fca/src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi#L22-L23,

cdef str _call_error_metadata(metadata):
  return 'metadata was invalid: %s' % metadata

From https://github.com/grpc/grpc/blob/ca0d85cea5242d6740ba6cf1dc5c0b9647856fca/src/core/lib/surface/validate_metadata.cc#L33-L42,

Uppercase characters are not allowed in gRPC as per the following code

class LegalHeaderKeyBits : public BitSet<256> {
 public:
  constexpr LegalHeaderKeyBits() {
    for (int i = 'a'; i <= 'z'; i++) set(i);
    for (int i = '0'; i <= '9'; i++) set(i);
    set('-');
    set('_');
    set('.');
  }
};

Also see grpc/grpc#9863 and b/367263573