Refactor Punycode (IDNA/ACE) handling

[tlimoncelli]

IDNA is something that needs to be fixed in DNSControl. Currently it takes the string from dnsconfig.js and convert it by calling dc.Punycode(). At that point we lose the original.

A better way would be: The DomainConfig struct should store the original string from dnsconfig.js and the string converted to punycode. (Maybe call the punycode version .Name and the Unicode version .NameUnicode?) It should then use the original for displaying to the users and the punycode for everything else.

Or, if we're lazy we can just output "Domain {.NameUnicode} being converted to {.Name}" once and refer to the punycode for all other output.

[tlimoncelli]

The IDNA part of dnscontrol is kind of a mess.

First, I must confess that I'm not an expert here. I had to look up what ACE-encoding it just now. Please correct me if I use any terminology incorrectly.

The way things work now: Users can put Unicode in dnsconfig.js. The providers call dc.Punycode() which converts everything (zone names, labels, targets) to ACE-encoded (i.e. xn--bcher-kva) strings. After that all code assumes everything is ASCII. The "preview/push" output is all ASCII (I think).

The problem I see are:

• some providers call dc.Punycode() and others do not.
• preview/push output doesn't try to convert ACE-encoded to Unicode. As a result, a user could input a domain in Unicode but then be confused that the preview/push output is ACE-encoded.

I'm not 100% sure what the best way forward is. Some thoughts:

• It would be less confusing to users if preview/push messages output ascii when the user input was ascii, and Unicode when the user input was Unicode. (This would be a simple matter of recording (bool) if the input was Unicode, and using that to convert back during preview/push output.)
• I think it would be better if all providers called dc.Punycode(). It would be better if the conversion was done by the system, so that each provide didn't have a choice.

[tlimoncelli]

dkim1970 writes:

I would say this RFC covers the "opinions" about how IDN should be handled in applications. https://datatracker.ietf.org/doc/rfc3490/

More about IDN can be found by this author: https://datatracker.ietf.org/person/paf@paftech.se

[tlimoncelli]

Thanks, @dkim1970, for those URLs. I read rfc3490 and now I feel much more confident about what we should do:

My proposal:

• Users should be permitted to use unicode or ascii in D("") statements.
• Internally DNSControl should processed everything as ascii (i.e. run ToASCII on all labels as early as possible after "runninig" dnsconfig.js to create the DomainConfig structure). This means that dc.Punycode() should be called by dnscontrol and not be optional for providers.
• Any output to stdout should pass all labels through ToUNICODE for display. Optionally it should display the ACE-formatted version too. Normally apps are discouraged from exposing the ACE-formatted strings except for debugging purposes (RFC3490 Section 6.1) but I would say that dnscontrol users probably always in a debugging mindset. I propose a CLI flag with options for ACE-only, Unicode-only, or both.
• When creating zonefiles, output should be ACE, with a comment showing the Unicode.
• Most provider APIs require everything to be ACE-formatted. If a provider sends or received Unicode, it is up to the provider code to properly translate. This goes for ListZones() and GetZoneRecords() too.

[dkim1970]

@tlimoncelli no worries happy to be of help.
I agree with all your point on how a IDN domain should be handle by DNSControl from a users point of view and how the providers should conform to being force to accept that dc.Punycode() is run by DNSControl.

[flokli]

For visibility: #2874 is a draft PR doing this.

[tlimoncelli]

Maybe I'm jumping to the end too fast but I want to write this down before I forget it.

The fundamental problem is that the .Name field is used for multiple things, and that changes based on whether dc.Punycode() has been run yet.

The most simple thing might be deprecate the .Name field and add fields for "as input by the user", "plain ascii", and "IDN". This way there are fields that are unambiguous in their use. We can leave the .Name field around unchanged, but slowly remove it over time.

I'm hearing that some people want preview/push to display "as input by the user" strings, but others want "IDN". Maybe we add a CLI flag to choose how they should be displayed, and we add a field that stores the appropriate version.