Building the Iterative Pipeline

A naive approach to machine translation is a single API call: send English text, receive translated text. This fails at scale for several reasons:

1. No nuance. Single-shot translations are often “correct enough” but rarely optimal for brand voice or regional idioms.
2. No quality signal. Without evaluation, there’s no way to know if a translation is acceptable before shipping it to users.
3. No recovery path. When a translation fails validation, the system has no mechanism to try again with corrective feedback.

We needed a system that could generate options, critique them, and iterate, a workflow that mirrors how human translators actually work. This led us to an architecture where multiple specialized processes translate and evaluate through structured handoffs.

The “Drafter” — Translation Generation

The job of the Drafter is primarily creative as it aims to produce diverse, high-quality translation candidates. We configure it to generate three distinct candidates for every source string.

Why three? We find a single translation often converges on the most likely phrasing, which may not be optimal for Lyft’s brand voice or the specific UI context. Multiple candidates increase the probability that at least one captures the right tone, handles edge cases correctly, and uses terminology naturally.

Model selection: For the Drafter, we use a fast, non-reasoning model here as translation is primarily a generative task where standard models already perform very well. Additionally, a faster model comes with lower cost and allows us to iterate.

Sample Prompt

DRAFTER_PROMPT = """
You are a professional translator for Lyft.
Translate into {language} for {country}.
Give {num_translations} translations of the following text.

GLOSSARY: {glossary}
PLACEHOLDERS (preserve exactly): {placeholders}

Text: {source_text}
"""

Sample Input/Output

Note: The LLM interactions return structured data via Pydantic schemas rather than free-form text. This ensures type safety, reliable parsing, and clear contracts between Drafter and Evaluator.

# Input 
source_text = "Your {vehicle_type} is arriving in {eta_minutes} minutes"
language = "French"
country = "Canada"

# Output (parsed)
DrafterOutput(
    candidates=[
        TranslationCandidate(text="Votre {vehicle_type} arrive dans {eta_minutes} minutes"),
        TranslationCandidate(text="Votre {vehicle_type} sera là dans {eta_minutes} minutes"),
        TranslationCandidate(text="Votre {vehicle_type} arrivera d'ici {eta_minutes} minutes"),
    ]
)

The “Evaluator” — Translation Evaluation

The Evaluator acts as a strict quality gate. It receives all candidates from the Drafter and scores each against a rubric, ultimately selecting the best one or rejecting them all.

Model selection: We use a reasoning-focused model for evaluation. Unlike generation, evaluation requires analytical comparison: checking source versus target for semantic drift, verifying terminology compliance, catching subtle tone mismatches. The deliberate reasoning process helps surface errors that a faster model might miss.

The Evaluator grades each candidate on four dimensions:

1. Accuracy & Clarity: Does the translation preserve the full meaning of the source? Is it unambiguous?
2. Fluency & Adaptation: Does it read naturally to a native speaker? Is it culturally appropriate for the target region?
3. Brand Alignment: Does it use official Lyft terminology? Are proper nouns, airport codes, and brand names preserved in English?
4. Technical Correctness: Is it free of spelling and grammar errors? Are all Lyft terms/phrases applied correctly?

Each candidate receives a grade: pass or revise. If any candidate passes, we ask the Evaluator to select the best one. If all fail, the Evaluator provides a detailed critique explaining why each failed.

Sample Output

EvaluatorOutput(
    evaluations=[
        CandidateEvaluation(candidate_index=0, grade=Grade.PASS, 
                           explanation="Accurate, natural phrasing."),
        CandidateEvaluation(candidate_index=1, grade=Grade.PASS,
                           explanation="Natural and conversational."),
        CandidateEvaluation(candidate_index=2, grade=Grade.REVISE,
                           explanation="'d'ici' implies uncertainty, inappropriate for ETA."),
    ],
    best_candidate_index=0,
)

Why separate Drafter and Evaluator?

Now that we’ve seen both components, it’s worth explaining why we separate them. The critique-and-refine pattern has several benefits:

1. Easier Evaluation: Spotting errors is simpler than perfect generation, so the Evaluator doesn’t need to be a flawless translator.
2. Context Preservation: The original translator retains the reasoning for its choices when refining based on feedback.
3. Bias Avoidance: Separating roles prevents the self-approval bias of a single model translating and evaluating its own work.
4. Flexibility/Cost: Different models can be used for each role (e.g., a fast drafting model and a more capable evaluator).

Retry, Reflection, and Self-Correction

The feedback loop between Drafter and Evaluator is a continuous mechanism to ensure if all candidates fail evaluation, the system doesn’t give up. It learns from the failure and tries again, up to three times.

We find this iterative refinement yields the largest gains in the first 1–2 cycles, so the three-attempt limit balances quality improvement against latency and cost.