Project Overview:

Join a growing community of professionals advancing the next wave of AI. As an AI Trainer, you’ll play a hands-on role by analyzing and providing feedback on data to improve LLM performance, helping ensure that the next generation of AI technology is accurate and trustworthy.

We are seeking a skilled Trust & Safety / AI Evaluation professional to work as a project consultant in our AI Labor Marketplace. This is not a full-time employment position — you will be engaged as an expert project consultant on a contract basis.

Location: U.S.-based experts only

Engagement: Part-time, project-based expert evaluation work

Work Type: Remote

Project Summary:

Contributors will design realistic prompts and multi-turn conversational scenarios to evaluate behavioral risks in large language models, along with structured annotations. Work focuses on nuanced, policy-relevant edge cases across high-risk categories such as social profiling, crisis handling, and human-AI interaction risks.

Consultant Engagement Terms:

This is a project-based consultant role. Consultants will be paid on a per-project basis; hourly rates are estimates based on anticipated completion time. Consultants control their own schedule, provide their own tools, and may simultaneously provide services to other vendors/employers (subject to those vendors’ allowances).

Responsibilities:

• Create high-quality single-turn prompts and multi-turn scenarios for AI evaluation
• Design realistic, policy-relevant edge cases reflecting real-world user behavior
• Apply structured annotations accurately and consistently
• Interpret and follow evolving safety and evaluation guidelines
• Review peer submissions for quality and policy alignment (for qualified contributors)
• Provide concise feedback on flagged or low-quality tasks

Expected Outcomes:

• Consistent production of high-quality, realistic prompts and scenarios
• Accurate and reliable annotations aligned with project taxonomy
• Identification and surfacing of meaningful edge cases in AI behavior
• Maintained throughput and quality standards over time

Qualifications:

• 1–3+ years of experience in trust & safety, AI evaluation, content moderation, prompt engineering, or related fields
• Strong writing and scenario design skills
• Demonstrated ability to apply judgment in ambiguous or policy-sensitive contexts
• Experience with annotation or structured evaluation workflows preferred
• Familiarity with AI systems, LLMs, or human-AI interaction patterns is a plus