Senior Product Manager, Observability Tools

• Own scenario creation via the simulation agent, including tools for generating and curating realistic test scenarios.
• Own evaluation surfaces, mistake monitoring, and structured data evaluations.
• Define methods for detecting, surfacing, and categorizing GenerativeAgent mistakes at scale.
• Own the Previewer for assessing configuration changes in simulated conversations.
• Own Conversation Explorer for reviewing full production conversations, including agent thoughts and actions.
• Build the case for simulation-driven quality with Sales, Customer Success, and Research.
• Partner with Research and Engineering on agent architecture, reasoning traces, tool calls, and scenario generation.
• Manage the trust layer of the review experience by resolving desynchronization, incomplete traces, and unclear mistake categorization.
• Own the data layer, including models, dashboards, feeds, and APIs, that makes ASAPP’s platform trustworthy and actionable.

• 5+ years of product management experience, with meaningful time spent on testing, quality, or review/analytics products in enterprise B2B SaaS.
• Direct experience building products around LLMs, AI agents, or other applied AI systems.
• Ability to reason about agentic behavior, prompt and scenario design, or model output evaluation.
• Experience shipping tools that simulate, test, or preview system behavior before production.
• Experience building review or audit tooling for complex, unstructured content.
• Ability to turn ambiguous quality signals into clear, defensible categorizations.
• Comfort partnering with research and engineering teams on reasoning traces, tool-use logs, and evaluation methodology.
• Track record of running discovery with internal or external users and translating it into structured requirements.
• Excellent written and verbal communication.
• Experience in contact centers, conversational AI, or customer experience analytics is a plus.
• Experience with evaluation frameworks or methodologies for generative AI systems.
• Prior experience in a startup or fast-scaling environment.

• Competitive compensation
• Stock options
• Healthcare for the family group
• Mac equipment
• USD 150 per month in flexible benefits
• 3 weeks of vacation
• English lessons