statemachine: Error messages should release (optionally)?

[schmichael]

Problem

There are a number of circumstances where a task cannot be executed on a node that has claimed it:

• A new task type is being added via a rolling deploy and a node without the necessary code has claimed the a new task type.
• A task cannot complete due to a node-specific condition: eg a node has insufficient diskspace for a task to use as scratch.

Possible Solution 1: Errors always release

Seems like the worst case scenario to Error messages always causing releases is extra broker load. Probably worth it.

Possible Solution 2: Make Release metadata on a Message

Allow any message to cause a release via a Release bool field on the statemachine.Message struct.

This might actually simplify other code... not sure.