Evaluation Engineer

Overview

In this role you build and run tests to assess AI applications quality and reliability within legal and business tasks. You maintain evaluation datasets, investigate failures, and document how AI systems perform against defined criteria. You coordinate with legal specialists and QA teams to ensure secure, compliant data usage and robust test coverage. You will help scale evaluation workflows, support CI/CD integration, and turn validated failure cases into new tests. This is a hands-on role at the intersection of AI and law with a focus on measurable, evidence-based improvements.

Responsibilities

• Implement automated evaluation cases and supporting scripts using established frameworks
• Prepare and maintain test datasets, source documents and reference examples with legal specialists
• Follow confidentiality, permitted data use and version control requirements
• Run evaluations of answer accuracy, citations, retrieval relevance, and instruction following; escalate when needed
• Execute functional and integration tests for AI workflows (document ingestion, tool calls, outputs, error handling)
• Review outputs and traces to identify failures; reproduce issues and document evidence
• Compare results across model, prompt and application versions; report changes with uncertainty and limitations
• Support calibration of automated graders by comparing scores with human reviews; investigate inconsistencies
• Maintain regression suites and contribute to CI/CD integration; create new test cases from approved production failure examples

Key requirements

• Minimum five years’ software development experience (Python, Java or C#)
• Foundational understanding of large language models and their limitations
• Ability to use coding agents for development, testing and analysis; knowledge of prompting best practices, context management, validation, security, and human review
• Working knowledge of APIs, Git, automated testing and CI/CD tools (Jira, Jenkins)
• Strong analytical skills with statistics basics (sampling, precision, recall) and clear issue documentation
• Practical experience evaluating AI applications is a plus, including academic or project work
• Curiosity and ability to learn in a rapidly evolving AI landscape
• Attention to detail
• Ability to document issues clearly and seek guidance when needed
• Python, Java or C# programming
• APIs, Git
• Automated testing

Report this job