Director, Clinical AI Research & Evaluation

Research Leadership

• Define and own the scientific methodology for evaluating clinical AI systems: evaluation harnesses, benchmark construction, failure mode taxonomy, red-team protocols, and monitoring strategies across the system lifecycle. Build a structured catalog of ways the system can fail clinically and run structured testing before any deployment.
• Advance methods for evaluation in clinically complex conditions such as noisy or delayed ground truth, expert disagreement, multi-step agentic workflows, human in-the-loop performance, and automation bias or where simple metrics fail.
• Develop human factors and workflow-integration methods beyond usability testing: how AI output shapes human judgment, what oversight mechanisms are effective, how clinical professionals calibrate trust in AI-assisted work, and what human review effectively ensure patient safety.
• Publish and communicate findings externally where appropriate to build AstraZeneca's scientific credibility and attract research talent.

Applied Delivery

• Design and implement evaluation frameworks that determine when a system is ready to move from prototype to controlled pilot, from pilot to deployment, and from non-GxP to regulated use.
• Participating in defining human review requirements for user facing AI outputs: what requires sign-off before action, what can be surfaced directly, and how oversight should be structured in the validation plan.
• Build evaluation assets such as test sets, simulation environments, scoring rubrics, regression suites, and analysis pipelines, some or all of which are reusable across
• Lead hands-on failure analysis: fabrications/falsifications, unsupported claims, data leakage, subgroup performance, and unsafe workflow behavior.
• Be part of a team of scientists and engineers focused on evaluation, human factors, and responsible deployment.

Cross-Functional Governance

• Work with Quality, Regulatory, Privacy, Security, and R&D IT to ensure evaluation of evidence meets GxP, data integrity, and inspection-readiness requirements from the outset.
• Advise product owners and leadership on evidence-based go or no-go decisions; communicate residual risk clearly.
• Create reusable evaluation standards and clinical AI governance practices that can be adopted across AISI's Clinical Development portfolio and contribute to broader industry standards.
• Represent the team's scientific position in external forums, standards bodies, and regulatory engagements where relevant.

• PhD in Machine Learning, Computer Science, Human-Computer Interaction, Statistics, Biomedical Informatics, or a related quantitative discipline; or an MD or equivalent experience with a research record that demonstrates comparable rigor.
• 8 or more years of post-PhD (or equivalent) experience in AI/ML research, evaluation science, or human factors in healthcare, life sciences, clinical research, or another safety-critical domain.
• Deep expertise in evaluation methodology for AI/ML systems: experimental design, statistical inference, uncertainty quantification, measurement reliability, and evaluation of LLM-based or multi-step agentic workflows.
• Demonstrated research leadership: ability to set a scientific agenda, direct a team, produce research outputs, and translate principled methodology into practical, shipping systems.
• Current, hands-on coding ability in Python and SQL, including evaluation pipelines, data analysis, statistical testing, and version-controlled collaborative development.
• Strong grounding in human-computer interaction or human factors and its application to AI-assisted decision-making in high-stakes environments.
• Excellent communication skills across technical, clinical, quality, regulatory, and executive audiences; ability to represent uncertainty and residual risk without overstating or obscuring.
• Demonstrated ability to lead cross-functional work through influence, including go or no-go recommendations under ambiguity.
• Experience with LLM evaluation platforms, LLMOps/MLOps & observability for deployed AI systems.

• Direct experience with clinical trial conduct, site monitoring, risk-based monitoring, medical monitoring, or clinical data management in a pharmaceutical, biotech, or CRO environment.
• Experience producing audit-ready evidence under GxP requirements, including model risk management, audit trails, traceability, and inspection readiness for regulated AI systems.
• Experience designing or conducting prospective validations, implementation studies and/or human-computer interaction studies in live clinical or healthcare workflows.
• Peer-reviewed publications, standards body contributions, open source benchmarks, or regulatory and industry consortium engagement in AI evaluation, human factors for AI, or clinical AI governance.
• Familiarity with causal inference, calibration, subgroup analysis, weak supervision, active learning, or evaluation methods for learning with imperfect labels.
• Experience working with Quality or regulatory functions on computerized system validation, validation plans, or submission-relevant AI evidence.

• The team establishes an evaluation standard for clinical AI that is recognized internally as the bar for production readiness and externally as a principled contribution to the field.
• Clinical experts, Quality partners, and regulatory functions trust that evaluation evidence reflects real clinical risk rather than optimistic benchmarks.
• Every agentic release by the team is traceable from intended use through test evidence to deployment decisions, with human oversight requirements defined, enforced, and captured in the validation plan.
• Demonstrated scientific leadership via publications, standards of contributions, or methodological advances.
• Evaluation of assets and human factors methods are reused across the Clinical Development AI portfolio without reinvention.

Office working requirements When we put unexpected teams in the same room, we unleash bold thinking with the power to inspire life-changing medicines. In-person working gives us the platform we need to connect, work at pace, and challenge perceptions. That's why we work, on average, a minimum of three days per week from the office. We balance this expectation with individual flexibility. Join us in our unique and ambitious world.