Agentic AIPythonverifiedVerified

Evaluator-Optimizer Agent Pattern in Python

An iterative refinement loop where an 'Evaluator' provides granular feedback on an 'Optimizer’s' output until quality thresholds are met.

How to Implement the Evaluator-Optimizer Agent Pattern in Python

1Step 1: Define the Feedback, Optimizer, and Evaluator types

from dataclasses import dataclass


@dataclass
class Feedback:
    is_pass: bool
    critique: str
    score: float


class Optimizer:
    async def generate(self, task: str) -> str:
        """Generate initial output for the task."""
        ...

    async def refine(self, current: str, feedback: str) -> str:
        """Refine output based on evaluator feedback."""
        ...


class Evaluator:
    async def check(self, output: str) -> Feedback:
        """Evaluate output against quality rubric."""
        ...

2Step 2: Implement the iterative refinement loop

MAX_ITERATIONS = 5


async def refinement_loop(
    task: str,
    optimizer: Optimizer,
    evaluator: Evaluator,
) -> str:
    current_output = await optimizer.generate(task)

    for _ in range(MAX_ITERATIONS):
        feedback = await evaluator.check(current_output)

        if feedback.is_pass:
            return current_output

        current_output = await optimizer.refine(
            current_output, feedback.critique
        )

    return current_output

Evaluator-Optimizer Agent Pattern Architecture

hourglass_empty

Rendering diagram...

lightbulb

Evaluator-Optimizer Agent Pattern in the Real World

“Think of a student writing an essay (Optimizer) and a teacher grading it with detailed feedback (Evaluator). The student revises based on the red-ink comments and resubmits. This cycle repeats until the essay meets the teacher’s standards—or the deadline (max iterations) is reached.”