AI Engineer (Evaluation)

• Develop and maintain evaluation frameworks and pipelines to measure the capabilities of Large Language Models (LLMs).
• Keep up to date and experiment with the latest research in multilingual, multicultural, and multimodal LLM evaluations such as the LLM-as-a-Judge paradigm.
• Work with partners to collect, translate and verify evaluation datasets.
• Perform the necessary data preparation and analysis, AI modelling, coding, testing, validation and deployment to ensure reliable and scalable AI solutions.
• Collaborate with cross-functional teams within AI Products to design and resolve issues.
• Maintain code repository and documentation standards.
• Contribute to community engagement activities such as sharings via technical session meet-ups and article write-ups, and participating in discussion forums.

• Graduate from a degree program in computer science, AI, data science or related fields (or equivalent practical experience).
• Deep understanding of LLM evaluation experimental design and the advantages/limitations of various LLM evaluation methods.
• Experience in LLM inference frameworks such as vLLM and AI/Deep Learning frameworks such as PyTorch.
• Writing production level code in Python and using version control systems such as Git.
• Strong written and verbal communication skills.
• Independent learner that is capable of reading and understanding research papers.
• Fluent in English and one other Southeast Asian language for the purposes of understanding how to build quality evaluations from a multilingual and multicultural perspective.

We regret that only shortlisted candidates will be notified.

Hiring Institution: NTU