Mpc api

[matthewjohnpayne]

Introduces an mpc_api package that is intended to be a pip-installable package on PyPi.
mpc_api provides a single MPCClient class with methods for all 10 MPC public APIs:

• identify,
• get_orbit,
• get_observations,
• get_observatory,
• get_mpecs,
• check_near_duplicates,
• get_submission_status,
• submit_xml/psv,
• request_action_code, and
• get_neocp_observations.

Includes optional pandas DataFrame output, custom exception hierarchy, and 60 mocked tests.

This PR also

• updates all api tutorial notebooks with mpc_api usage examples
• adds a Getting Started notebook for the mpc_api package

I have verified that the code works and that the main demo notebook, mpc_tutorial_mpcapi.ipynb, also works and provides simple-but-cogent explanations of the individual apis, as well as linking out to the tutorials & documentation for the APIs.

NB:

• As per my comment in PR31 , It's probably reasonably obvious that I created this using claude-code.
• It's unclear to me how we should be reviewing such code going forward: It works, but it was all produced in a few hours, and reviewers will have a hard time digesting such a large volume of code.

[matthewjohnpayne]

@federicaspoto : I've asked @fiftymillionfeltcapfanscantbewrong to review this PR, but I'm just flagging you on it so that you are aware of it (e.g. for newsletter/similar).

[federicaspoto]

I haven't tested this, but in principle this should not work unless the tracklet is currently on the NEOCP. This doesn't seem to be the expected behavior though.

[matthewjohnpayne]

I'm not quite sure what you mean here, but I suspect it is lack of clarify in the tests regarding what @responses.activate does (it mocks API responses).

As such, I have added more tests, and then split the tests into 3 main sections:

1. tests that actually, for real, call the API (but are allowed to fail/skip if the API is unavailable)

• N.B. I'm allowing these to pass if the API is unavailable because it is not strictly the job of mpc_api to be the api, and also, I don't want the CI to get screwed-up due to (e.g.) downtime in mpcweb6

2. tests that mock the API response, so that we are purely testing this mpc_api.MPCClient
3. tests that are independent of the API (such as error handling within mpc_api)

I note that I have done the same for all of the various test files in tests/test_*.py, and added a bunch of comments to make clear what is going on.

[matthewjohnpayne]

OK, @federicaspoto , @fiftymillionfeltcapfanscantbewrong , I have addressed the previous comments regarding mpc_api.

I have also verified that the version currently released to TestPyPI works when downloaded.

I have clarified the README

I have also added various .github/workflows/ to ensure that

1. the code will be released to TestPyPI on every push to a PR involving mpc_api, and
2. the code will be released to PyPI on every merge to main of a PR involving mpc_api.

[fiftymillionfeltcapfanscantbewrong]

Hey Matt, I think this project is really cool, but I have concerns about the maintainability of it. This PR introduces validation steps that duplicate the validation that we perform in the APIs, and each endpoint requires a special mixin class. Yes, an LLM could produce these quickly, but we need to be able to read it. Perhaps a simpler solution would be to provide a generic, pip-installable utility? Calls could be structured like:

from mpc_api import call_mpc_api

endpoint = 'mpecs'
payload = {
  ...
}
method = 'GET'
# 'POST' should be the default method; we need to move to consistently using that, anyway.
call_mpc_api(endpoint, payload, method=method) 


def call_mpc_api(endpoint: str, payload: dict, method: Literal['GET', 'POST'], retries: int = 5) -> dict:
  # call the API, attempting retries
  # If the Pydantic validation fails, which some endpoints report, it's already formatted in JSON
  # If successful, just return what the API does.

Irrespective of this suggestion, this package should attempt retries intelligently; currently it seems like it only attempts requests once?

Perhaps it's also a philosophical question: If we provide APIs for the public to use, with attendant documentation, should we obfuscate them behind a package such as this? Users will ultimately be subject to rate-limiting imposed by the API, which may give them the wrong impression of this package.

[pkgw]

OK, a few early comments.

First: we should keep in mind that distributing and maintaining a real (not halfassed) API client library is really valuable, but also a lot of work. Documentation, testing, examples, keeping up-to-date with internal changes, support, avoiding compatibility breaks, etc. If we're going to provide a package like this and not work ourselves into an annoyingly fragmented ecosystem, it needs to become the thing that we use every single time anyone asks for or needs example code to use MPC APIs. I'm all for that, but we'd need really embrace "eating our own dogfood" as much as possible. Doing a decent job of this would demand time and attention from multiple MPC staff.

Nomenclature: these kinds of libraries are usually called "clients", à la the MPCClient type. I would prefer that this package be named mpc_client or mpc_api_client rather than mpc_api, which to me makes it sound like something that would implement the server side of the API. It would be a hassle to change this, but we'll be stuck with whatever name we adopt for a very long time.

Documentation: the package needs documentation. Example notebooks are a component of this but there is a lot else that is needed. Fortunately I just spent a year developing a scientific software documentation checklist. Docs should probably be built with Sphinx (or mkdocs) and wired up to publish to readthedocs.org. The mixin pattern might be a bit of a hassle for autogenerated API docs.

Typing: in terms of the actual implementation, this package fails to do what I think is the most useful thing for a language-specific client library, which is to (consistently) map the JSON HTTP API responses into data structures useful in the target language. In my experience the most useful pattern for a library like this is to rigorously define Python dataclasses representing all of the request and response types, using something like Pydantic to map to/from JSON. Among other things, this enables you to write docstrings on the classes and their fields that turn into published documentation that actually rigorously describe their semantics, and get Intellisense in a modern editor. Low-level API invocation methods pretty much only transmute between the Python types and the JSON API request, and then you can build smarter and higher-level workflows based on those low-level primitives.

Throughout, function signatures should have thorough type annotations. There should probably be linting automation to enforce this in CI.

Scope: Thinking about the above leads to a question of library scope. If we have APIs that return data in obs80 format, we want to provide something low-level that does just that, but it would also likely be helpful to provide something a bit more "opinionated" that can parse the obs80 out of its bespoke textual format. So then we have to start thinking about providing public obs80 parsing code, or depending on a blessed external obs80 library, etc. This is probably especially relevant here since we have a lot of bespoke data formats flying around.

Examples of docs describing similar client libraries that I've built:

• daschlab
• wwt_api_client

[matthewjohnpayne]

@pkgw : Thanks for your various thoughts and suggestions.

I've made numerous updates, much of which I describe below.

OK, a few early comments.

First: we should keep in mind that distributing and maintaining a real (not halfassed) API client library is really valuable, but also a lot of work. Documentation, testing, examples, keeping up-to-date with internal changes, support, avoiding compatibility breaks, etc. If we're going to provide a package like this and not work ourselves into an annoyingly fragmented ecosystem, it needs to become the thing that we use every single time anyone asks for or needs example code to use MPC APIs. I'm all for that, but we'd need really embrace "eating our own dogfood" as much as possible. Doing a decent job of this would demand time and attention from multiple MPC staff.

• I agree!
• I'd like this package to become the standard place that we point users too, so I'm all in favor of putting in the required effort to build, document & maintain it.

Nomenclature: these kinds of libraries are usually called "clients", à la the MPCClient type. I would prefer that this package be named mpc_client or mpc_api_client rather than mpc_api, which to me makes it sound like something that would implement the server side of the API. It would be a hassle to change this, but we'll be stuck with whatever name we adopt for a very long time.

• I've changed it to mpc_client.

Documentation: the package needs documentation. Example notebooks are a component of this but there is a lot else that is needed. Fortunately I just spent a year developing a scientific software documentation checklist. Docs should probably be built with Sphinx (or mkdocs) and wired up to publish to readthedocs.org. The mixin pattern might be a bit of a hassle for autogenerated API docs.

• I believe that I have followed most/all of the suggestions in https://onegoodtutorial.org/ , and implemented them in mpc-client.
• I note that I have set up readthedocs-integration with mpc-public, and this now produces documentation for the latest/main branch as well as a build for the PR, https://mpc-public--32.org.readthedocs.build/en/32/

Typing: in terms of the actual implementation, this package fails to do what I think is the most useful thing for a language-specific client library, which is to (consistently) map the JSON HTTP API responses into data structures useful in the target language. In my experience the most useful pattern for a library like this is to rigorously define Python dataclasses representing all of the request and response types, using something like Pydantic to map to/from JSON. Among other things, this enables you to write docstrings on the classes and their fields that turn into published documentation that actually rigorously describe their semantics, and get Intellisense in a modern editor. Low-level API invocation methods pretty much only transmute between the Python types and the JSON API request, and then you can build smarter and higher-level workflows based on those low-level primitives.

• I believe that I have now implemented pydantic validation on the request side, as well as consistent class+attribute structures for the results side.
• I note that I have opted to put all code (both request & result classes) for a given API (e.g. cnd) into a single mpc-client script (e.g. _cnd.py). I could have opted to (e.g.) put all request structures for all apis into a single _requests.py file, but I felt like the _cnd.py pattern would be easier for folks to follow when adding additional future APIs into the package.

Throughout, function signatures should have thorough type annotations. There should probably be linting automation to enforce this in CI.

• I have added .github/workflows/mpc_client_lint.yml to the repo (and fixed various problems).
• I am not at all experienced in using/deploying this stuff, so I'd appreciate any suggestions for improving it.

Scope: Thinking about the above leads to a question of library scope. If we have APIs that return data in obs80 format, we want to provide something low-level that does just that, but it would also likely be helpful to provide something a bit more "opinionated" that can parse the obs80 out of its bespoke textual format. So then we have to start thinking about providing public obs80 parsing code, or depending on a blessed external obs80 library, etc. This is probably especially relevant here since we have a lot of bespoke data formats flying around.

• I agree that we should probably do this. In many respects I'm resistant to it, as I ultimately want to stop users submitting in this format, but logically, just because I might prevent submission in a format doesn't mean that format doesn't exist!
• We have a python package (named obs80) in mpc-software that will parse obs80 strings into python classes/namedtuples. It is not perfect, but it would provide a solid base for a piece of public obs80 parsing code.
• Alternatively, one could use the ADES package to convert to XML and thence on to python dicts/classes.