Benchmark & Evaluation Engineer

• Design and stand up multi-system enterprise environments, including warehouses, lakes, APIs, and databases, at scale
• Build environments across domains such as healthcare, finance, and product analytics
• Generate calibrated, large-scale benchmark datasets with realistic noise, seasonality, drift, and synthetic PII
• Test query scalability and reasoning using benchmark datasets
• Write and verify benchmark tasks, golden SQL, and rubric-scored reasoning keys
• Eliminate data leakage and ambiguity from benchmark tasks
• Extend grading systems using deterministic checks and LLM judges
• Maintain regression-tracked leaderboards
• Ensure evaluation scoring integrity

• Expert proficiency in production-grade SQL, including CTEs, window functions, query plans, and cardinality
• Expertise in data modeling, including star/snowflake schemas, SCDs, and referential integrity
• Strong Python skills with pandas, NumPy, and Polars
• Hands-on experience standing up and loading data into at least one major warehouse: Snowflake, BigQuery, Redshift, or Postgres
• Proven experience designing evaluation sets or benchmark tasks
• Experience with golden-answer/golden-SQL verification, rubric/LLM-judge calibration, avoiding data leakage, and tracking regressions
• Ability to construct realistic multi-system setups including warehouses, data lakes, operational databases, and APIs
• Experience with large-scale, semi-structured, or dirty synthetic data
• Domain knowledge in Healthcare/Health Insurance, Finance/FP&A, Product Analytics, or Supply Chain is nice to have
• Experience with LLM evaluation frameworks or streaming/data-lake stacks is nice to have

• Remote work in the Philippines
• US Shift schedule