Customer Reliability Engineer

• Own customer issues from reproduction and diagnosis through resolution or escalation and closure
• Debug Linux issues involving processes, networking, storage, kernel logs, resource contention, systemd, and journald
• Investigate Kubernetes problems involving pending or crash-looping pods, node conditions, scheduling failures, and resource limits
• Troubleshoot GPU failures including driver and device-plugin issues, XID errors, thermal throttling, cordoning or draining nodes, and multi-node job failures
• Escalate issues with evidence already gathered
• Participate in a 24/7 on-call rotation
• Respond first to alerts and customer-reported outages, assess impact, set severity, and involve appropriate personnel
• Keep customers updated during incidents
• Document incidents and convert findings into runbooks, alerts, or fixes
• Determine whether faults are provider-side, Andromeda-side, or customer-side
• Open and follow up on tickets with compute providers
• Track recurring provider failures and report patterns to sourcing decision-makers
• Write Python or Bash automation scripts
• Build and improve cluster and node health monitoring, GPU telemetry, dashboards, and alerts
• Maintain current runbooks and customer documentation

• Real Linux troubleshooting ability from the command line, including logs, processes, networking, and disk
• Working knowledge of Kubernetes, including pods, nodes, deployments, services, scheduling, and investigation of failures
• Ability to write Python or Bash scripts to automate repetitive tasks
• Strong technical writing and ability to explain problems clearly to frustrated customers
• Good judgment under pressure
• Willingness to join a 24/7 on-call rotation
• NVIDIA GPU production experience, drivers, CUDA, DCGM, and Kubernetes device plugin experience may be advantageous
• High-performance networking experience with InfiniBand, RoCE, or NCCL may be advantageous
• HPC or Slurm experience may be advantageous
• Previous customer-facing technical role may be advantageous
• Knowledge of Prometheus, Grafana, Datadog, or similar may be advantageous
• Infrastructure as code experience with Terraform, Ansible, or Helm may be advantageous
• Genuine interest in AI infrastructure may be advantageous

• Meaningful equity
• Healthcare coverage for you and your dependents
• Dental coverage for you and your dependents
• Vision coverage for you and your dependents
• 401(k)
• Unlimited PTO
• High-growth environment at a company at the center of the AI infrastructure boom