Senior Data Infrastructure Engineer – Scraping, Scale

• Build and maintain the core ingestion engine powering the social media data lab
• Build and manage stealth scraping clusters using residential proxy networks, TLS fingerprinting, headful/headless browser farms, and session rotation
• Build fault-tolerant, scalable web-scraping pipelines for Instagram, TikTok, YouTube, X, and web sources
• Design distributed queues and workflow engines to manage millions of asynchronous scraping tasks daily
• Architect structured and unstructured storage environments for downstream AI modeling
• Implement automated alerts for platform UI/API changes, blocking patterns, and proxy failures
• Deliver millions of profile, post, and video records daily with near-zero downtime

• 4+ years of experience in high-volume web scraping, data engineering, or reverse engineering
• Deep experience defeating advanced anti-bot providers (Cloudflare, Akamai, PerimeterX) via TLS impersonation, browser automation, and proxy management
• Mastery of Python or Go
• Deep knowledge of Playwright, Puppeteer, Scrapy, or Selenium
• Experience with distributed systems and task queues (Temporal, Ray, Kafka, Redis)
• Strong SQL skills
• Experience with modern analytical data lakes / data warehouses
• Preferred: Direct experience extracting short-form video content and user metadata from TikTok, Instagram, and YouTube
• Preferred: Experience integrating scraping outputs directly into vector databases and real-time AI processing queues

• Remote work arrangement