Explanation / Documentation Request: use_llm_feedback, cascade_evaluation, and use_template_stochasticity

[theahura]

I am fairly uncertain what all of these config options actually do. The readme (https://github.com/codelion/openevolve/blob/78c43bb0104036eb575b49e0a9189bacefc88039/README.md?plain=1#L470) is not super descriptive on any of these. Is there any documentation or even paper pointers that explain what these do in more detail?

[codelion]

Great question! Here's a comprehensive explanation of these config options with code pointers:

---

1. use_llm_feedback (Evaluator Config)

What it does: Enables AI-powered code quality assessment using an LLM to evaluate code readability, maintainability, and efficiency alongside traditional metrics.

Code locations:

• Defined: openevolve/config.py:308
• Used: openevolve/evaluator.py:189

How it works:

# evaluator.py:189-205
if self.config.use_llm_feedback and self.llm_ensemble:
    llm_result = await self._llm_evaluate(program_code, program_id=program_id)
    # Adds metrics like llm_readability, llm_maintainability, llm_efficiency
    # Each weighted by llm_feedback_weight (default 0.1)

Related config:

• llm_feedback_weight - Weighting factor for LLM metrics (used in evaluator.py:196)

Example output: Adds metrics like llm_readability: 0.092, llm_maintainability: 0.082 to evaluation results.

---

2. cascade_evaluation (Evaluator Config)

What it does: Implements multi-stage evaluation where programs must pass thresholds at each stage before advancing to more expensive tests. Saves computation by filtering out poor candidates early.

Code locations:

• Defined: openevolve/config.py:300
• Used: openevolve/evaluator.py:163

How it works:

# evaluator.py:163-168
if self.config.cascade_evaluation:
    result = await self._cascade_evaluate(temp_file_path)
else:
    result = await self._direct_evaluate(temp_file_path)

The cascade evaluator (evaluator.py:131-319) runs:

1. Stage 1 (evaluate_stage1 in your evaluator file) - Quick validation
2. Stage 2 (evaluate_stage2) - Basic performance (only if stage1 passes threshold)
3. Stage 3 (evaluate_stage3) - Comprehensive tests (only if stage2 passes threshold)

Related config:

• cascade_thresholds - List of thresholds for each stage (used in evaluator.py:195, 256)

Important: Your evaluator file must implement evaluate_stage1(), evaluate_stage2(), and/or evaluate_stage3() functions. Otherwise, it falls back to direct evaluation with a warning (evaluator.py:115-120).

---

3. use_template_stochasticity (Prompt Config)

What it does: Randomizes prompt templates to introduce variation in LLM requests, helping prevent repetitive responses and encouraging diverse code generations.

Code locations:

• Defined: openevolve/config.py:201
• Used: openevolve/prompt/sampler.py:129

How it works:

# sampler.py:129-130
if self.config.use_template_stochasticity:
    user_template = self._apply_template_variations(user_template)

# sampler.py:309-319
def _apply_template_variations(self, template: str) -> str:
    # Replaces placeholders with random variations from config
    for key, variations in self.config.template_variations.items():
        if variations and f"{{{key}}}" in result:
            chosen_variation = random.choice(variations)
            result = result.replace(f"{{{key}}}", chosen_variation)

Related config:

• template_variations - Dict mapping placeholder names to lists of random variations (used in sampler.py:558)

Example usage:

prompt:
  use_template_stochasticity: true
  template_variations:
    greeting:
      - "Let's enhance this code:"
      - "Time to optimize:"
      - "Improving the algorithm:"

Hope this helps! Let me know if you need clarification on any of these features.

[theahura]

Thanks for the response! Is there a way to adjust the use_llm_feedback measure? For example, say I do not care at all about readability, but I do care about providing the llm some way to understand or provide an explanation of what the most recent iteration did that was different from the previous one?

[theahura]

@codelion wanted to check back in here.

Is there a way for the evaluator to pass back more specific information to the next iteration of program generation besides just the score?

[theahura]

o nevermind I think the artifacts sidechannel is what I need here