Native Syntax: allow non-decimal integer literals

[herrgahr]

Hello!

I'm considering HCL as a configuration language for an embedded software project. For this, it would be useful to have support for hexadecimal literals, maybe even binary and octal.

For simplicity reasons, I'm suggesting only a subset of Go's integer literals - some of the variations allowed there are not great for readability, especially octal literals like 07 or 0O7.

Since go-cty supports unsigned 64bit integers, it would make sense to limit the numeric range to that (uint64_t).

For existing HCL files, there should be no compatibility issues as far as I can tell. numeric literals as specified today will still parse unchanged, the only additions suggested are number literals of the form:

• 0x12345678123ABCDE
• 0b00001111000
• 0o0666

Suggested Change in spec.md

NumericLit = BinaryLit | OctalLit | HexLit | DecimalLit;

BinaryLit = "0b" binary+;
binary = ('0' | '1');

OctalLit = "0o" octal+;
octal = '0' .. '7';

HexLit = "0x" hex+;
hex = (decimal | 'a' .. 'f' | 'A' .. 'F');

DecimalLit = decimal+ ("." decimal+)? (expmark decimal+)?;
decimal    = '0' .. '9';
expmark    = ('e' | 'E') ("+" | "-")?;

Notes Regarding Readability and (in-)Consistency in the Spec

For consistency reasons, one might be tempted to allow "0x" | "0X", "0b" | "0B" and "0o" | "0O", given that today's langauge spec allows for upper case E and lower case e in numbers (1.0e5 == 1.0E5). I am not suggesting this here. As mentioned above, the octal prefix "0O" is too easy to misinterpret IMHO.

[apparentlymart]

Hi! I'm not an HCL maintainer so this comment is not an answer on behalf of the HCL project. However, I am the maintainer of the cty library that HCL uses as the foundation of its type system and in this comment I'm intending to draw attention to some hazards that the HCL maintainers might wish to consider when considering this feature request.

HCL currently makes lots of use -- both direct and indirect -- of cty's convert.Convert function, and other related functionality in that package.

Relevant to this feature request, HCL relies on cty for automatic conversions between string and number values:

• "1" can convert to number as 1
• 1 can convert to string as "1"

These conversions intentionally use decimal representation exclusively. I don't intend to add upstream support for automatically converting "0xf" to 15, for example, and in the reverse direction the type system does not have any way to represent that a number was "originally written in binary", so a hypothetical HCL 0b10 would evaluate to decimal 2 and then convert to string as "2".

None of this means that HCL's own parser could not allow alternative representations of numbers by implementing its own parser for producing a cty.Number from a numeric literal token, of course. Currently it delegates directly to cty's parser which only allows decimal values¹:

hcl/hclsyntax/parser.go

Lines 1132 to 1151 in ba07594

	func (p *parser) numberLitValue(tok Token) (cty.Value, hcl.Diagnostics) {
	// The cty.ParseNumberVal is always the same behavior as converting a
	// string to a number, ensuring we always interpret decimal numbers in
	// the same way.
	numVal, err := cty.ParseNumberVal(string(tok.Bytes))
	if err != nil {
	ret := cty.UnknownVal(cty.Number)
	return ret, hcl.Diagnostics{
	{
	Severity: hcl.DiagError,
	Summary: "Invalid number literal",
	// FIXME: not a very good error message, but convert only
	// gives us "a number is required", so not much help either.
	Detail: "Failed to recognize the value of this number literal.",
	Subject: &tok.Range,
	},
	}
	}
	return numVal, nil
	}

The main thing I'm trying to draw attention to is that it might potentially be confusing to allow numeric literals in other bases without also allowing string representations of those same literals to convert automatically to number values, and so HCL might also wish to take responsibility for number-to-string conversions too for consistency. However, that's ultimately a design tradeoff for the HCL maintainers to make; I have no particular opinion about it.

Footnotes

1. This function can only be called on a token that matched the Ragel production for NumberLit during the tokenization pass, so in practice these two need to be aligned and would need to change together. ↩