Code is completely de-indented when no line-wrapping solution is found

[zaneduffield]

In the rare cases where no line-wrapping solution is found for a logical line, currently all the code is de-indented all the way to the margin.

A contrived example that triggers this behaviour is

type 
  A = (
    a a a a a a a a a a a a a a a a a a a a a a a a a a a a a 
    a a a a a a a a a a a a a a a a a a a a a a a a a a a a a 
    a a a a a a a a a a a a a a a a a a a a a a a a a a a a a 
    a a a a a a a a a a a a a a a a a a a a a a a a a a a a a 
  );

which is formatted like

type
A = (
a a a a a a a a a a a a a a a a a a a a a a a a a a a a a
a a a a a a a a a a a a a a a a a a a a a a a a a a a a a
a a a a a a a a a a a a a a a a a a a a a a a a a a a a a
a a a a a a a a a a a a a a a a a a a a a a a a a a a a a
);

The formatter does this because it has already messed with the spacing between tokens, even if they span different lines. This means that if the optimising line formatter did nothing on no solution found, then most of the wrapped lines would be put within one space of the margin anyway, because it's the thing that's supposed to set the indentation level; the most sensible 'default' indentation if the process has failed is zero.

I think in cases like this it would be better to disable formatting for the entire line, as if it were wrapped in {pasfmt off/on} comments. Yes this will make it inconsistent with the surrounding code, but it's an unlikely error case and it would always be worse to de-indent to the margin.

[zaneduffield]

On second thought, it wouldn't be possible with the current exposed API to mark tokens as ignored in execution of a 'formatter'.

What would work (and might be closer to what we actually want), would be to 'reset' the whitespace data for the sequence of tokens.

To 'reset' the whitespace:

fn foo(tokens: &mut FormattedTokens, index: usize) {
    if let Some((tok, data)) = tokens.get_token_mut(index) {
        *data = FormattingData::from(tok.get_leading_whitespace());
    }
}

but this could easily be a method on FormattedTokens if we wanted.