QA Engineer - AI Systems III

in execution and impact We're looking for an AI Engineer who is passionate about both building AI-powered applications and ensuring they work reliably in production. This role combines AI engineering with quality engineering, focusing on developing LLM-based solutions while creating automated testing and evaluation frameworks to measure quality, accuracy, safety, and performance. You will work closely with engineering, product, and business teams to deliver AI solutions that are not only innovative, but also measurable, stable, and production-ready.

Responsibilities:

• AI Engineering: Build and maintain AI-powered applications using LLMs, RAG pipelines, and agent-based workflows.
• Develop integrations with foundation models and AI services using Python and modern AI frameworks.
• Design prompt workflows, structured outputs, tool calling, and retrieval-based systems.
• Build APIs, backend services, and internal tools using FastAPI or similar frameworks.
• Collaborate with stakeholders to translate business requirements into AI solutions.
• AI Quality & Automation Design and execute automated test suites for AI applications and agentic workflows.
• Create and maintain evaluation datasets, benchmark scenarios, and regression test cases.
• Build pipelines to measure response quality, hallucinations, accuracy, latency, and cost.
• Implement LLM-as-a-judge evaluations and automated scoring frameworks using tools such as Dee pEval, RAGAS, or PromptFoo.
• Develop AI-powered test generation for functional, edge-case, and negative testing scenarios.
• Perform adversarial testing including prompt injection, jailbreak attempts, and safety validation.
• Monitor production AI systems through logging, tracing, and quality dashboards.
• Work with engineering teams to establish quality gates before AI features are released.

• 2- 4 years of software engineering, QA automation, AI engineering, or related experience.
• Strong Python and Golang programming skills and understanding of software engineering fundamentals.
• Familiarity with LLM applications, prompt engineering, RAG, or agent-based AI systems.
• Experience in test automation using frameworks such as PyTest or equivalent tools.
• Understanding of API testing, backend services, and REST architectures.
• Knowledge of AI evaluation concepts such as hallucination detection, response quality measurement, benchmark testing, and regression testing.
• Experience with SQL and basic cloud platforms (AWS, Azure, or GCP).
• Strong analytical and problem-solving skills.

Why Proofpoint?

At Proofpoint, we believe that an exceptional career experience includes a comprehensive compensation and benefits package. Here are just a few reasons you'll love working with us:

• Competitive compensation

• Comprehensive benefits

• Career success on your terms

• Flexible work environment

• Annual wellness and community outreach days

• Always on recognition for your contributions

• Global collaboration and networking opportunities

Our Culture:

Our culture is rooted in values that inspire belonging, empower purpose and drive success-every day, for everyone. We encourage applications from individuals of all backgrounds, experiences, and perspectives. If you need accommodation during the application or interview process, please reach out to accessibility@proofpoint.com .