Data Quality & AI Operations Manager

Job Summary

This role is responsible for managing content and data operations for the Extend platform, focusing on improving profiling accuracy and automating annotation processes using AI. The position includes leading the content operations team and overseeing the evaluation and data feedback loop.

Responsibilities

• Develop and maintain golden sets and evaluation suites, setting acceptance criteria for AI Admin iterations.
• Manage data aspects of AI Admin iterations, including curating training and evaluation data, running benchmarks, analyzing failures, and collaborating with Shanghai engineering on fine tuning and deployment.
• Design the human-in-the-loop system by defining the limits between human review and automatic approval, and integrate human corrections into prompts, evaluation sets, and training data.
• Create and update Arabic and English labeling guidelines and resolve complex cases requiring cultural and linguistic judgment.
• Lead the content operations team with full responsibility for annotation operations, including rater training, workflows, throughput, and quality service level agreements.
• Set up and monitor a quality metrics framework covering accuracy, automation rate, inter-annotator agreement, cost, and turnaround time per profile.

Must haves

• Fluency in Arabic with strong understanding of the Arab world’s media, brands, publishers, and cultural context.
• Practical experience with AI agent or LLM systems, including evaluation design, golden sets, and model feedback loops; familiarity with fine tuning concepts such as SFT, RLHF, and prompt iteration without needing to train models.
• At least 2 years of experience in AI data operations or applied AI quality from a data labeling company, in-house annotation team, or AI product team.
• Ability to collaborate effectively across Riyadh and Shanghai time zones.

Nice to haves

• Experience with LLM-assisted labeling automation, taxonomy design, or social/media data.

We refresh listings regularly, but some roles close early on the source platform.