Forward Deployed Engineer - AVP

Overview

As Forward Deployed Engineer, you will build automated evaluation harnesses, curate datasets, and optimize prompts to ensure reliable, accurate, and safe production AI agents. You’ll work within the Applied AI and Agents Factory to deliver production-ready AI solutions that integrate with enterprise systems.

Expect to tackle complex engineering challenges, drive automated testing and CI/CD pipelines, and collaborate with cross-functional teams to reduce risk and improve agent performance. This role offers hands-on development with a focus on impact, scale, and end-to-end delivery of AI capabilities for Citi.

You’ll contribute to reusable tooling and best practices that accelerate AI adoption

Pay / Benefits

• 27 days annual leave (plus bank holidays)
• discretionary annual bonus
• Private Medical Care
• Life Insurance
• Employee Assistance Program
• Pension Plan

Responsibilities

• Construct and run automated evaluation test suites using Agentflow Studio; measure task completion, tool call correctness, latency, and hallucination metrics.
• Curate and maintain domain-specific golden datasets, synthetic benchmarks, and regression testbeds with domain experts and senior engineers.
• Optimize prompt templates, system instructions, few-shot examples, and JSON output parsers to ensure deterministic agent behavior.
• Author and maintain automated test execution and deployment pipelines within Lightspeed CI/CD, enforcing evaluation gates before staging and production.
• Analyze complex agent errors and performance using Python applications development to recommend advanced coding and prompting solutions.
• Implement security measures and Lakera Guardrail policies in post-implementation analysis to ensure safe functionality.
• Monitor evaluation cycles, deliver performance metrics, and provide prompt diagnostic reporting to stakeholders and squad leads.
• Operate with limited supervision as an evaluation SME, supporting pod team members and maintaining compliance with laws and Citi policies.

Key requirements

• Solid Python programming proficiency with experience integrating REST/gRPC APIs, Docker, and Git workflows.
• Experience with AI evaluation frameworks, benchmarking (BLEU, ROUGE, semantic similarity), and automated testing suites.
• Experience building automated test pipelines in CI/CD environments (Lightspeed, Tekton, Jenkins).
• Ability to work under pressure with tight sprint deadlines and changing model outputs or requirements.
• Strong written and verbal communication with a methodical, detail-oriented approach to software quality and data integrity.
• Relevant software engineering or systems analysis experience, with a focus on Python and Generative AI applications.
• strong communication
• methodical and detail-oriented
• ability to work under pressure
• Python
• REST/gRPC APIs
• Docker

Report this job