AI Response Labeler, Annotator

• Perform side-by-side comparisons of AI-generated responses and select the stronger response using established evaluation criteria
• Assess responses for factual accuracy, relevance, completeness, reasoning, instruction following, clarity, safety, tone, and usefulness
• Evaluate questions and answers, web-search results, file-based and image-based responses, content generation, and single-turn or multi-turn conversations
• Identify unsupported claims, missed instructions, weak reasoning, incomplete answers, and other meaningful differences
• Apply detailed, scenario-specific guidelines and make sound decisions when examples are ambiguous
• Write concise, evidence-based explanations for decisions when required
• Meet productivity expectations while maintaining accuracy and consistent judgment
• Participate in training, guided practice, calibration, qualification reviews, and ongoing quality reviews
• Incorporate feedback as evaluation guidelines and quality standards evolve
• Complete approximately 25 evaluation tasks per day, with most tasks taking about 15 minutes
• Work with Blueprint Technologies, a technology solutions firm delivering AI, cloud, data, product development, and emerging technology solutions

• Excellent written English comprehension and communication skills
• Ability to read complex prompts and guidelines and explain evaluation decisions clearly
• Strong critical-thinking skills
• Ability to assess content across a wide range of topics
• Sound judgment when evaluating factuality, reasoning, user intent, and subtle differences in response quality
• Excellent attention to detail
• Ability to apply structured criteria consistently
• Comfort with repetitive, focused work and a high volume of evaluations
• Ability to work independently, respond to feedback, and stay aligned with shared quality standards
• Must successfully complete structured onboarding and qualification program before production work
• Must demonstrate the ability to evaluate broader response quality, follow detailed annotation guidelines, explain decisions, and complete work within expected timeframes
• During approximately 30-day training and qualification period, must work from 9:00 a.m. to 5:00 p.m. Pacific Time
• Preferred: experience evaluating, ranking, or comparing AI-generated responses
• Preferred: experience with data annotation, content quality assessment, search relevance evaluation, or model-quality review
• Preferred: experience with detailed rubrics, annotation guidelines, or quality benchmarks
• Preferred: experience writing clear rationales supporting evaluation decisions

• Medical, dental, and vision coverage
• Flexible Spending Account (FSA)
• 401(k) retirement plan
• Competitive paid time off
• Parental leave
• Professional growth and development opportunities
• Employment through a local PEO or EOR partner, as applicable
• Local statutory benefits according to the hiring country and employment terms
• Remote work arrangement