ADES header for orbit computers

[federicaspoto]

Hi @stevechesley ,

this came from Marco Fenucci (NEOCC) as a Jira ticket.
I'll send this link to Marco in the ticket and we can continue the discussion here if needed.

Thanks!
Federica

Dear MPC,

I am writing because we have a request for ADES. As per ADES format, observers are allowed to put more specific information and data in the header of their submission, i.e. the obsContext block.

On the contrary, a similar header/block to be filled by orbit computers is currently not part of ADES (provided I understood correctly the current ADES standard). However, such header would be really useful for us orbit computers to put more information about the computation of their residuals. For example: 1) we could put a line with the weighting scheme adopted in the computations; 2) the total WRMS error; 3) information about the software used, etc. These is only a non-exhaustive list of examples, and I would like to start a discussion about this topic. Would it be possible to design a header also for orbit computers?

In addition, fields that already belong to the residuals group would fit nicely in such header. I am referring specifically to orbProd and orbID fields. I do not clearly understand why these two fields should be repeated in every single entry of the residuals group, when they are typically common for the whole orbit. Is there a specific reason why ADES has been designed this way?

Please, feel free to add JPL colleagues if you think they are needed for this discussion.

Best

Marco, on behalf of NEOCC

[stevechesley]

@Fenu24, I wonder if what you are requesting can already be handled with the comment elements of obsContext. But given that obsContext was originally intended as context for observation submission, I'm not sure it makes sense to add new (optional) elements specifically for OD information, but I would not be opposed to it.

I suppose that a new context element, e.g., orbContext as an alternative to obsContext could serve the need. So you could have, e.g.,

<ades>
    <obsBlock>
        <orbContext>
           ...
        </orbContext>
        <obsData>
           ...
        </obsData>
    </obsBlock>
</ades>

This structure, of course, would not be allowed in MPC submissions.

I do appreciate the problem with orbProd and orbID elements. When this was developed more than 10 years ago there was not a lot of clarity on who would be using ADES to communicate residuals, but we wanted to get some framework in place. At JPL, we have an internal variant of ADES that has some additional elements and does put orbProd and orbID in the preamble.

If this seems like a valuable change then I encourage you propose a new framework for discussion, which would eventually lead to a PR.

[Bill-Gray]

I'm glad this has come up; I've long wondered about the redundant use of orbProd and orbID for every entry. To which I'd add photProd and photMod; all these are attributes of the overall orbit fit, and should go into an orbContext header.

At the risk of going afield here with something perhaps best raised as a separate issue : if you're going to give astrometric and/or photometric residuals, I'd think you should give the orbital elements or state vector and the H, G, etc. photometric parameters that were used in the fit. Or at least be able to do so. All of this, again, would go into the orbContext preamble.

I don't, at present, make use of any of the ADES residuals features. I've been thinking about doing so because people frequently will load up data for an object in Find_Orb, manually toggle some observations off or on, and possibly change some of the initial astrometric/photometric/timing uncertainties for some observations. It'd be pretty nifty if they could save the results for future use, or to send to another Find_Orb user or another orbit determination program. The ADES residuals groups would store just about everything needed to do that.

[Fenu24]

First, thanks @federicaspoto for moving this issue here.

@stevechesley at the moment we are in fact using the comment tag of the obsContext block, however when we check our output xml against the ADES xsd schema, the check fails with errors. Automating the check against the schema adds a layer of robustness for the products of the software, and since there is a standard for residuals as well, it would be ideal that also the output products comply with the standard.

I understand it is easy to slightly modify the standard used locally by our corresponding software to overcome this issue. However, in the moment the products are made public, I think they should definitely comply with the standard, for many reasons (some of them were already nicely reported by @Bill-Gray). From our side, we plan to make the NEOCC residuals in ADES standard available to users once we transition to ADES in operations.

At NEOCC, we like the idea of introducing a new block such as orbContext, and I think for the sake of keeping the standard as easy as possible, we can think about defining some useful common tags along with a comment tag. This would solve also the issue of redundant tags, which can be easily moved here and do not overlap with a block dedicated to submissions.

Another possibility is changing slightly the obsContext block and/or the xsd schema, so that an obsContext containing only the comment tag would pass the checks. But this is potentially dangerous, since the schema is used as a first filter to prevent incomplete submissions to the MPC. But @federicaspoto or someone else from MPC could better comment on this.

@Bill-Gray thanks for commenting on this, I agree that also the photProd and photMod tags could be moved in a orbContext block as well. As for the photometric parameters and state vector/orbital elements, at NEOCC we give them already in the orbit files which are also available to users, so that reconstructing the fit for the magnitude is possible (this is actually possible already with the data we provide in the .orb and .rwo files). However, in ADES we introduced some custom tags in the localUse field for distances from Earth, Sun, and phase angle computed from the best fit orbit, so that reconstructing the fit of the magnitude would be even easier (i.e. no need to compute ephemerides). These are also not part of the standard, but I would not be opposed for them to be included in the residuals group as well.

[Bill-Gray]

@Fenu24 - "at NEOCC we give [orbital and photometric parameters] already in the orbit files..."

True. But I could see some management issues, where you have a file of orbital/photometric parameters that is from a previous or later fit. It'd be nice to be able to say, "here are the residuals I got, and here's what you'd need to replicate that result with somebody else's software." (And if you don't get that result, it'd be a good idea to figure out why that is.)

Also, you have such data in your files, MPC has it in theirs, and JPL and NEOfixer and I each have it in our own files, none with the same format. Interoperability is a Good Thing. (Which may lead me onto my soapbox about the desirability of being able to exchange orbital elements in a common format... but that's another rant!)

[Fenu24]

Dear @stevechesley ,

We would like to proceed with the proposal for modifying ADES by introducing a new orbContext header. How do you suggest we should proceed with this? Should we start a discussion group somewhere else?

Thank you,

Marco on behalf of NEOCC

[stevechesley]

This github issue seems a suitable place to discuss your proposal, but we could carry on by email if you prefer and then update here at appropriate intervals.

An orbContext element could carry a range of information about the orbit on which the residuals are based. This would likely include the orbit itself, nongravs and nongrav models, N-body model assumptions, reference frame orientation (ecliptic vs. equatorial) and origin (e.g., barycentric, heliocentric, planetocentric). Also ancillary documentation like orbID and orbProd, plus realted photometric info.

My preference would be to limit, at least as a first pass, the number of alternate representations for an orbit. There are a great number of variations of orbital elements the could be supported, but we might do well by starting with only heliocentric cartesian and comet elements, in the equatorial and ecliptic frames respectively, which is what I think most users will want.

Suggest looking into the old PS DES (Sec. 4.3) from Milani to see if there is any wisdom to be gleaned there.

[federicaspoto]

@stevechesley @Fenu24 before moving forward in the discussion, I'd like to remind everyone that the MPC is in the process of developing the mpc_orb json format, that contains exactly all these informations (cc @matthewjohnpayne).
For reference: https://minorplanetcenter.net/mpcops/documentation/mpc-orb-json/

I personally don't have any problems about moving it to ADES, I don't know if Matt feels differently, but I believe that we (MPC) already went through a couple of iterations with the community and the JSON might contain lots of the information that we need here, so that we don't need to start from the beginning.

Also, I don't think it would be then worth replicating the information in two different formats.

[Fenu24]

Hi @stevechesley @federicaspoto,

Thanks for the comments, this is a very interesting discussion. We second what Federica said, we should definitely avoid to replicate information between ADES and orbit formats, and take advantage of the work the MPC has already been doing for the orbit format.

We also agree with @Bill-Gray that residuals should go along with full information about the orbit, but probably unifying everything in ADES would be too confusing or complicated. In the end even the name of ADES refers to astrometry specifically. We think should find a way to make the two dataset correspond to each other, and try to keep the header of ADES residuals as minimalistic as possible. For example, only information/assumptions on the statistical treatment of data weights could be kept, along with an ID to identify the corresponding orbit.

One way to go could be something like this:
- Move the field orbID to a orbContext header, mandatory for producers of ADES residuals. The same field should be mandatory in the mpc_orb format as well, so that orbits and residuals correspond through this field (similar to a key field of a database). This way, it is clear (in theory) what orbit data is used to obtain the residuals.
- orbProd/photProd: these could be moved to orbContext header, so that it is clear who computed those data to a user who has access only to the residuals file. We can also consider to remove photProd because it seems like a duplicate of orbProd. Is there any chance this assumption is not valid?
- photMod: to my understanding, this is present also in the mpc_orb format. We propose to remove this from ADES and keep it only in the orbit file. This is an information about the model used to fit the data, in a similar fashion to N-body model and ephemerides for the orbit. It would be consistent to remove it, as we don't have info on the dynamical model in ADES.
- normalized RMS/not normalized RMS (of both astrometry and photometry): The RMS for astrometry is also in the mpc_orb format, I didn't see a corresponding field for photometry RMS. We think it would be more consistent to report these values in the ADES orbContext header, since the orbit is not needed to obtain these values once the residuals are known. However, it is not a strong opinion.
- Possible other fields that could go to the orbContext header:
○ Debiasing model used: we think is this information could be useful to report
○ Time uncertainty flag: a flag to report whether time uncertainties were used in the fit or not
○ Default uncTime/rmsTime: fields to report the default time uncertainties in the case they were not reported.
○ A comment field to report non-standard information

We think this could be a good start for a definition of such header, but we would like to know other opinions of course.

As a final remark, we think it could be useful to have the orbit standard also in XML/PSV formats, so that users would have the option to use the same data type for both orbits and observations/residuals, without juggling between different data types.

[Fenu24]

Hello, @stevechesley @federicaspoto @matthewjohnpayne any new thoughts on this matter?