How to reuse a pact.v3.Pact in consumer testing

[lotruheawea]

Hi,

I am having some difficulties with implementing version 3 of pact-python. Specifically, on how the Consumer Testing can be done. In the end, I wonder why I cannot set the scope of the pytest fixture to session in the example examples/tests/v3/test_00_consumer.py like so:

@pytest.fixture(scope="session")
def pact() -> Generator[Pact, None, None]:
    """
    Set up the Pact fixture.

    This fixture configures the Pact instance for the consumer test. It defines
    where the pact file will be written and the consumer and provider names.
    This fixture also sets the Pact specification to `V4` (the latest version).

    The use of `yield` allows this function to return the Pact instance to be
    used in the test cases, and then for this function to continue running after
    the test cases have completed. This is useful for writing the pact file
    after the test cases have run.

    Yields:
        The Pact instance for the consumer tests.
    """
    pact_dir = Path(Path(__file__).parent.parent.parent / "pacts")
    pact = Pact("v3_http_consumer", "v3_http_provider")
    yield pact.with_specification("V4")
    pact.write_file(pact_dir)

This returns a RuntimeError: The provider state could not be specified. when running the second test and all following. I assume that the lib.pactffi_given call in the second and following calls somehow still assumes that the flag mock_server_started is True (in Rust code). Is that correct? Why can I not set the scope to session?

My use case is that I would like to attach the Pact to an object yielded by a session-scoped fixture.

Your help is much appreciated. Thank you.

[JP-Ellis]

For the sake of completeness, I'll copy my answer from the issue here in case anyone stumbles on this (and apologies for not answering you earlier).

---

When the underlying server is started for the verification process, the underlying server expects all of the configured interactions to be replayed and will result in an error if any of the interactions are not replayed. For example, if I configure the Pact to have a GET /user/{id} and PUT /user/{id}, then both interactions must happen or the verification fails.

As a result, it is not exactly possible to have a session-scoped fixture like you have here assuming that each test has the combination of:

1. Defining the expected interaction, and then
2. Interacting with the mock server to test the interaction.

Having said all that though, Pact by default merges interactions within the output file if it already exists. So simply changing your fixture to be function-scoped (the default) should work fine.

Hopefully this helps! And apologies for net answering you earlier.

[lotruheawea]

Thank you for your reply. Your explanation supports my assumption that this might be a design problem. A pact file contains multiple interactions that can and, in practical terms, are probably usually generated by multiple tests. But the pact-python framework does not support that scheme that multiple tests feed interactions to the same Pact object (while they end up in the same file and Pact on the broker). I cannot really see the actual causal connection between the underlying server expects all of the configured interactions to be replayed and it is not exactly possible to have a session-scoped fixture. When I call pact.serve() in the context manager, the server is shut down again and all interactions need to have been replayed. I initially assumed the Pact object to behave like a stack of interactions that is emptied during the pact.serve call and needs to be empty after that but can be added to in subsequent tests.

In our test suite we reuse multiple interactions with the same description in multiple tests. But in order to avoid overwriting of interactions with the same name (description) when saving to file, we came up with additional code to handle the interaction descriptions and its properties. It effectively creates a new description name if a given set of interaction properties (request_body, request_query_params, return_code, response_body) is not defined in an interaction already used in this pact.

I personally would like to see this as part of pact-python and that could be achieved by making the pact object session-scoped and allow to add new interactions and replays after a server was shut down.

[lotruheawea]

Any update here? We are thinking about integrating a generalized version of the code above in our own library which we would not if you could picture that as part of pact-python.

[JP-Ellis]

Is there any strong reason for using the scope="session" option for the Pytest fixture?

The underlying Rust library does make the assumption that the workflow is:

1. Define interactions to be verified
2. Launch server, and start verification
3. Check each request (and respond) against the defined interactions. This must verify all interactions defined in (1).
   • The reason being that it might verify a sequence of steps as part of one larger logical operation. E.g., perhaps ensuring that a DELETE request has a GET request first to validate the resource being delete.
4. Output data to JSON, merging interactions if the JSON exists already

Using your example, but modifying it by:

1. Removing scope="session" (i.e. @pytest.fixture(scope="session") changed to @pytest.fixture)
2. Adding an identical post_foo test, but with my_identifier=2 instead and foo_id=2.

The output JSON is:

{
  "consumer": {
    "name": "MyApp"
  },
  "interactions": [
    {
      "description": "A valid post /v1/foo/1 request (0)",
      "pending": false,
      "providerStates": [
        {
          "name": "some kind of foo with id 1 already exists."
        }
      ],
      "request": {
        "method": "POST",
        "path": "/v1/foo/1",
        "query": {
          "bar": [
            "baz"
          ]
        }
      },
      "response": {
        "status": 201
      },
      "type": "Synchronous/HTTP"
    },
    {
      "description": "A valid post /v1/foo/2 request (0)",
      "pending": false,
      "providerStates": [
        {
          "name": "some kind of foo with id 2 already exists."
        }
      ],
      "request": {
        "method": "POST",
        "path": "/v1/foo/2",
        "query": {
          "bar": [
            "baz"
          ]
        }
      },
      "response": {
        "status": 201
      },
      "type": "Synchronous/HTTP"
    }
  ],
  "metadata": {
    "pactRust": {
      "ffi": "0.4.22",
      "models": "1.2.3"
    },
    "pactSpecification": {
      "version": "4.0"
    }
  },
  "provider": {
    "name": "OtherAPI"
  }
}

[lotruheawea]

Is there any strong reason for using the scope="session" option for the Pytest fixture?

In the end, the strong reason is to prevent overwriting of interactions with the same description string but with differing properties. What is the proposed (by pact) naming convention of interaction descriptions in order to avoid that? I assume there is none, so the user must handle that.

As I have written before: "A pact file contains multiple interactions that can and, in practical terms, are probably usually generated by multiple tests. But the pact-python framework does not support that scheme that multiple tests feed interactions to the same Pact object (while they end up in the same file and Pact on the broker)." It is left to the user to ensure that descriptions of interactions are unique for differing interaction properties. I think, that should be handled by pact-python and not the user. My code above would be one possible avenue for this.

I see that your modification of the proposal works. It still leaves the user with the task to prevent creating duplicate interactions. if the my_identifier is not part of interaction_description (via path) anymore, the JSON file would contain only one interaction.

interaction_description = (
            f"{description_valid_str} {method} {path} request ({n_existing_keys})"
        )

[mefellows]

What is the proposed (by pact) naming convention of interaction descriptions in order to avoid that? I assume there is none, so the user must handle that.
...
It is left to the user to ensure that descriptions of interactions are unique for differing interaction properties

The description of the interaction is an essential aspect of Pact. It's key to the BDD and "specification by example" elements of the test. Pact is not a general purpose API testing automation solution, it's a targeted contract testing solution - the descriptions (along with the states or "pre-conditions") are the basis for explaining the scenario being tested.

That is to say that the description is for human comprehension of the contract. It's not there as some incidental detail of the test itself.

[lotruheawea]

Okay, I translate this as the user indeed needs to ensure that the description is comprehensible and unique. Then we very well might do this, I guess.

[tu-pm]

I don't think users have any problems with pact requiring interaction description to be unique. After all, it forces us to write a small and tidy test suite, which is essential to avoid burdening the provider side with a lot of duplicated test cases (and provider states) to handle.

What we are asking for is a way to identify test duplication before writing interactions to the contract file. The current behavior is to override the existing interaction with the new one with the same description, causing missing test cases from the contract file.

Or may be if we can have a mechanism to define all the interactions in a single dictionary with the keys are the interactions description, and then inside each PactServer context we can choose which interactions to replay.

I think asking users who might have thousands of test cases scattered all around the code base to keep all the description strings unique is a bit unrealistic.

[JP-Ellis]

It certainly would be possible to introduce a mode whereby a warning is emitted if an interaction is going to be overwritten within a JSON file.

As you correctly point it, having the description be a unique descriptor of the interaction

forces us to write a small and tidy test suite, which is essential to avoid burdening the provider side with a lot of duplicated test cases

and goes some way to avoid having thousands of test cases. Pact tests are not intended to replace unit tests; they are instead intended to verify contracts with other services and thus should test only what is actually used (at least, from the consumer's perspective).

Having said that, your point about choosing which interactions to replay is quite valid and we just recently had a pre-RFC over at pact-foundation/roadmap#117