Major new language feature - defer

Added defer generator. This generator will defer outputting until a
scope is exited. This adds some burden on generator writers to be
aware of the need to document their scope enter/exit and explicit
exits.

It was implemented by putting the defer output into a splice, then
having the Writer keep track of scopes and output the splices in the
appropriate locations.

* Speculatively fix double std::move on newStringOutput which seems to
be completely invalid, but didn't seem to break anything.
* Add documentation for path and field

Author: makuto

doc/Roadmap.org

@@ -83,10 +83,6 @@ It needs to be possible to mix strict C and C++ modules, because it's unlikely t
 Currently, defining which headers to include from compile-time functions is not possible. This is important to support complex compile-time code bases spread across multiple functions and headers. It's also necessary to be able to include Cakelisp headers optionally, so compile-time functions can help macros/generators/etc.
 
 This shouldn't be a hard add, but it is tricky to decide where import list should go: within each function, at the module level, both of those, etc.
-** ~defer~ support
-This one is going to be a bit tricky, because scopes are going to need to be tracked. I think it's a really nice to have feature, but one of the harder, more error-prone ones.
-
-I don't want to just use C++ destructors because I want a pure-C option.
 ** ~for~ loop
 In GameLib, almost all loops would be fine with a number range, e.g. here are some ideas:
 #+BEGIN_SRC lisp
@@ -214,6 +210,11 @@ Mostly listing this because it's what I use (LSP is a bit too heavyweight for my
 The following are things that were on the Roadmap that are now complete.
 
 These are sorted with most recently completed appearing first.
+
+** ~defer~ support
+This one is going to be a bit tricky, because scopes are going to need to be tracked. I think it's a really nice to have feature, but one of the harder, more error-prone ones.
+
+I don't want to just use C++ destructors because I want a pure-C option.
 ** Conditionals based on build configuration
 It is useful to be able to define blocks of code which are e.g. operating-system specific, e.g. something like:
 #+BEGIN_SRC lisp

runtime/CHelpers.cake

@@ -247,9 +247,9 @@
      (array Keyword ";" -1)
      (array Expression null update)
      (array CloseParen null -1)
-     (array OpenBlock null -1)
+     (array OpenContinueBreakableScope null -1)
      (array Body null body)
-     (array CloseBlock null -1)))
+     (array CloseContinueBreakableScope null -1)))
   (return (c-statement-out statement)))
 
 ;; This only works for arrays where the size is known at compile-time

src/Evaluator.hpp

@@ -337,8 +337,9 @@ struct EvaluatorEnvironment
 	ArtifactCrcTable comptimeNewCommandCrcs;
 
 	// When a definition is replaced (e.g. by ReplaceAndEvaluateDefinition()), the original
-	// definition's output is still used, but no longer has a definition to keep track of it. We'll
-	// make sure the orphans get destroyed
+	// definition's output is still used, but no longer has a definition to keep track of it. This
+	// is also used for splices that don't have an owning object. We'll make sure the orphans get
+	// destroyed so as to not leak memory.
 	std::vector<GeneratorOutput*> orphanedOutputs;
 
 	// Create a named splice point for later output splicing. Used so the user can insert things

src/EvaluatorEnums.hpp

@@ -17,6 +17,8 @@ enum StringOutputModifierFlags
 
 	// Curly braces ({}) will be added for open and close, respectively
 	// These are also clues to the output formatter to indent all within, etc.
+	// If you expect statements to run within, use StringOutMod_OpenScopeBlock and
+	// StringOutMod_CloseScopeBlock instead.
 	StringOutMod_OpenBlock = 1 << 7,
 	StringOutMod_CloseBlock = 1 << 8,
 
@@ -35,6 +37,22 @@ enum StringOutputModifierFlags
 
 	// Signals the Writer that it needs to splice in another output list
 	StringOutMod_Splice = 1 << 15,
+
+	// The Writer should paste the spliceOutput of this operation before exiting the current scope
+	StringOutMod_SpliceOnScopeExit = 1 << 16,
+
+	StringOutMod_ScopeEnter = 1 << 17,
+	StringOutMod_ScopeExit = 1 << 18,
+	// for/while/switch all can have continue or break, which "exits" the scope or maybe just
+	// restarts it. We need to handle these specially because they can exist within another scope,
+	// so the writer needs to peel back all scopes
+	StringOutMod_ScopeContinueBreakableEnter = 1 << 19,
+	StringOutMod_ScopeContinueBreakableExit = 1 << 20,
+	StringOutMod_ScopeContinueOrBreak = 1 << 21,
+	StringOutMod_ScopeExitAll = 1 << 22,
+
+	StringOutMod_OpenScopeBlock = StringOutMod_OpenBlock | StringOutMod_ScopeEnter,
+	StringOutMod_CloseScopeBlock = StringOutMod_CloseBlock | StringOutMod_ScopeExit,
 };
 
 enum ImportLanguage

src/GeneratorHelpers.cpp

@@ -451,7 +451,24 @@ void addSpliceOutput(GeneratorOutput& output, GeneratorOutput* spliceOutput,
 
 	// Splice marker must be pushed to both source and header to preserve ordering in case
 	// spliceOutput has both source and header outputs
-	output.source.push_back(std::move(newStringOutput));
+	output.source.push_back(newStringOutput);
+	// Now we can have the header just take ownership
+	output.header.push_back(std::move(newStringOutput));
+}
+
+void addSpliceOutputWithModifiers(GeneratorOutput& output, GeneratorOutput* spliceOutput,
+                                  const Token* startToken, StringOutputModifierFlags modifiers)
+{
+	StringOutput newStringOutput = {};
+	newStringOutput.modifiers = (StringOutputModifierFlags)((int)StringOutMod_Splice | (int)modifiers);
+	newStringOutput.startToken = startToken;
+
+	newStringOutput.spliceOutput = spliceOutput;
+
+	// Splice marker must be pushed to both source and header to preserve ordering in case
+	// spliceOutput has both source and header outputs
+	output.source.push_back(newStringOutput);
+	// Now we can have the header just take ownership
 	output.header.push_back(std::move(newStringOutput));
 }
 
@@ -1044,6 +1061,38 @@ bool CStatementOutput(EvaluatorEnvironment& environment, const EvaluatorContext&
 			case CloseParen:
 				addLangTokenOutput(output.source, StringOutMod_CloseParen, &nameToken);
 				break;
+			case OpenScope:
+				addLangTokenOutput(output.source,
+				                   (StringOutputModifierFlags)((int)StringOutMod_OpenBlock |
+				                                               (int)StringOutMod_ScopeEnter),
+				                   &nameToken);
+				break;
+			case CloseScope:
+				addLangTokenOutput(output.source,
+				                   (StringOutputModifierFlags)((int)StringOutMod_CloseBlock |
+				                                               (int)StringOutMod_ScopeExit),
+				                   &nameToken);
+				break;
+			case OpenContinueBreakableScope:
+				addLangTokenOutput(
+				    output.source,
+				    (StringOutputModifierFlags)((int)StringOutMod_OpenBlock |
+				                                (int)StringOutMod_ScopeContinueBreakableEnter),
+				    &nameToken);
+				break;
+			case CloseContinueBreakableScope:
+				addLangTokenOutput(
+				    output.source,
+				    (StringOutputModifierFlags)((int)StringOutMod_CloseBlock |
+				                                (int)StringOutMod_ScopeContinueBreakableExit),
+				    &nameToken);
+				break;
+			case ContinueOrBreakInScope:
+				addLangTokenOutput(output.source, StringOutMod_ScopeContinueOrBreak, &nameToken);
+				break;
+			case ExitAllScopes:
+				addLangTokenOutput(output.source, StringOutMod_ScopeExitAll, &nameToken);
+				break;
 			case OpenBlock:
 				addLangTokenOutput(output.source, StringOutMod_OpenBlock, &nameToken);
 				break;

src/GeneratorHelpers.hpp

@@ -101,6 +101,9 @@ CAKELISP_API void addLangTokenOutput(std::vector<StringOutput>& output,
 void addSpliceOutput(GeneratorOutput& output, GeneratorOutput* spliceOutput,
                      const Token* startToken);
 
+void addSpliceOutputWithModifiers(GeneratorOutput& output, GeneratorOutput* spliceOutput,
+                                  const Token* startToken, StringOutputModifierFlags modifiers);
+
 struct FunctionArgumentTokens
 {
 	int startTypeIndex;

src/GeneratorHelpersEnums.hpp

@@ -8,11 +8,25 @@ enum CStatementOperationType
 
 	OpenParen,
 	CloseParen,
+
+	// Open a scope. Always use when regular statements can be executed within
+	OpenScope,
+	CloseScope,
+	// Anything which responds to continue or break should open its scope with these
+	OpenContinueBreakableScope,
+	CloseContinueBreakableScope,
+	// Do not open a scope. Use only for declarations
 	OpenBlock,
 	CloseBlock,
+	// Used only for e.g. array initializers
 	OpenList,
 	CloseList,
 
+	// "Unnatural" flow control, i.e., cases where the normal CloseScope will not be hit
+	ContinueOrBreakInScope,
+	// return keyword
+	ExitAllScopes,
+
 	Keyword,
 	KeywordNoSpace,
 	// End the statement if it isn't an expression