Move to TOML for manifest and lock

[sdboyer]

DECISION IS MADE - we'll be using TOML.

---

At the time dep went public, the committee was generally in agreement that we would prefer to use TOML or YAML over JSON for the manifest and lock files. There are a few reasons for this, but the primary one is that those formats, unlike JSON, allow users to include arbitrary comments. Because the manifest is the locus of dependency decisionmaking for teams, such comments can add a lot of value:

# we pinned this to a rev because of <link to bug>. remember to unpin when that's fixed!

However, in looking through the existing TOML and YAML implementations in Go, we found that none of them actually support preserving arbitrary comments when regenerating the files. dep is constantly rewriting the manifest and lock files; without support for comment preservation, each rewrite would blow away any user comments. So, without this feature, TOML and YAML lose much of their advantage over JSON.

There are other arguments for using TOML or YAML, and it is possible that the we may still elect to use them even without support for comment preservation. However, we are putting a call out - if anyone feels particularly itchy at the thought of using JSON for these files, then please, contribute comment preservation support to an existing implementation, and we'll make the switch! The deadline for this change is the relevant milestone, after which the format and structure of the manifest and lock must be stable.

Alternatively, if we've missed an implementation of TOML or YAML that does support comment preservation, or there's some other crafty way around this problem, please tell us here!

---

UPDATE: we've decided to generally not have the tool update the manifest, which relieves us of the need for a comment-preserving implementation.

[kardianos]

In what instances would users be expected to manually modify any of the files, manifest or lock?

[freeformz]

FWIW: I don't think it should generally be necessary with the exception of adding comments or the possibility that there are some seldom used features that are (at least to start) implemented through directly modifying the manfiest file. That's my 2c anyway.

[jessfraz]

agreed, don't think there should be much reason other than comments to directly edit the manifest, but if there is we should think about if that is the user experience we desire

[adg]

I vote for sticking with JSON.

[sdboyer]

In what instances would users be expected to manually modify any of the files, manifest or lock?

echo what @freeformz and @jessfraz said. it's also possible that, when we start looking at direct integration into the go toolchain, we have to trim the command set down.

[kardianos]

when we start looking at direct integration into the go toolchain, we have to trim the command set down.

@sdboyer Is there a design document for a go tool integration? Are there docs from internal discussions? Are you just hypothesizing?

[jessfraz]

somewhere around here: #25 (comment)

[sdboyer]

Is there a design document for a go tool integration?

Nope, there are no more design docs hiding anywhere.

Are there docs from internal discussions? Are you just hypothesizing?

I'm basing this on some email chains with @rsc. He set out some desired constraints for how he wants the overall go toolchain to work, and the first thought I had was, "it would be easier to meet these if we weren't trying to provide a command for ever possible manifest change." I shared that thought in an email about two weeks ago, and there's been no further discussion since. That's where we're at.

[kevinburke]

Funny, I wrote about this recently - this is out of left field, but the only Go parsers I found that preserved comments were for XML, HCL and Go source files themselves, and they are about equally difficult to work with. https://kev.inburke.com/kevin/more-comment-preserving-configuration-parsers/

Here's a parser that finds a version number in a Go source file and bumps it: https://github.com/shyp/bump_version. I've also written code to manipulate HCL files and it's a little finicky but the resulting file is recognizable. https://github.com/hashicorp/hcl

[nathany]

There is also JSON5, which does allow comments, though I don't think it's anywhere near a standard.

Though I prefer the readability of TOML > YAML > JSON, another consideration is the licensing and CLA if dep is later incorporated into the go toolchain. As far as I understand, the TOML or YAML libraries would need to be contributed under the Google CLA or rewritten (with comment preservation as well). /cc @willnorris