Evaluation is the top hiring filter right now. This project proves you take it seriously, if you can defend it.
On your resume
Built and published a Python library and CLI that generates single-hop, multi-hop, and distractor question-answer pairs from document corpora for reproducible RAG evaluation and regression testing.
How ready are you?
Cross-questions
3
A strong answer covers
Compare them with real query logs, have SMEs review a sample, and mix synthetic with real questions.
Admit the bias: synthetic sets skew toward easy, extractive questions.
Avoid
Assuming synthetic equals real.
A strong answer covers
Specific rubrics, binary or small scales, and few-shot examples.
Calibrate against human labels and report agreement, for example Cohen’s kappa.
Use a different model family from the generator, and randomize order to reduce position bias.
Avoid
Trusting judge scores without checking them.
A strong answer covers
Questions whose answers aren’t in the corpus. They test that the system says “not found” instead of inventing an answer.
Avoid
Not being able to explain your own feature.
Your numbers
Interviewers probe numbers first. Write exact values so you never have to improvise.