Senior Software Engineer II - Agentic Intelligence

Honeycomb’s Agentic Intelligence team builds production-grade AI agents that investigate, reason, and act on live observability data inside Canvas. You’ll help shape an agentic workspace where engineers go to understand their systems, taking early prototypes through to code that can handle real production traffic.

This role spans agent engineering plus product and fullstack ownership. You’ll extend what agents can do in Canvas, improve the performance of Honeycomb’s agent loop, and help define what “good” means using measurable evaluations on real behavior.

What you’ll do

• Design and deliver production-grade agents that investigate, reason, and act on live observability data within Canvas, moving from early versions to something dependable under production load.
• Own the agent lifecycle: scope, build, ship, and maintain agent capabilities, including evals to verify agents improve or whether changes simply alter behavior.
• Build agent workflows only Honeycomb can support by leveraging a data store that returns high-cardinality queries in seconds, enabling agents to correlate across services, drill into a single trace, and compare before-and-after deploys.
• Expand the Canvas ecosystem by shipping new capability into Canvas, the Honeycomb MCP server, and Canvas Skills memory, including spatial awareness and a faster Bedrock loop, supported by working prototypes for what comes next.
• Set an engineering bar for agent quality: agents should be measurable against real evals, maintainable over time, and honest about their limitations.

What you’ll bring

• AI and agent engineering experience, with evidence you have shipped LLM-based systems people rely on in production, not demos. You understand agent failure modes and how to design around them, and you have worked beyond fine-tuned research contexts.
• End-to-end ownership, taking rough prototypes through to production-grade implementations without requiring someone to handle the durable engineering behind you.
• Current judgment informed by what has changed in agent design over the last six to twelve months, and how those shifts would affect how you build today.
• Agent architecture depth, including how a fast, high-cardinality data store changes what an agent can reason about and how to design for it.
• Product judgment: assess what the agent layer can do now and identify the next best step, backed by a prototype rather than a deck.

Helpful background

• Observability or developer-tools experience.
• Familiarity with eval frameworks, agent tooling, RAG, and prompt engineering.

Team context

Honeycomb’s Agentic Intelligence team has shipped Canvas, the Honeycomb MCP server, and the Canvas Agent and Canvas Skills surfaces. Some work starts as a prototype, with an expectation that it moves through the full arc from a rough first version to a production-ready implementation.

Example projects

• Multiple agents collaborating on a single shared Canvas investigation, each proposing hypotheses, publishing findings, and narrowing the search space to resolve faster.
• Auto-investigation triggered when an SLO burn alert fires: the agent forms hypotheses and prepares visualizations before a human looks, reducing mean-time-to-insight for on-call.
• Skills encoding a team’s domain expertise (for example, Kubernetes thresholds) so agents and colleagues can reuse that knowledge.

Compensation

Base salary: USD 183,340 - 206,000 per year (based on level of experience).

Benefits

• Generous equity with an employee-friendly stock program
• Unlimited PTO for time to recharge
• Distributed-first mindset and culture
• Home office, co-working, and internet stipend
• Full benefits coverage for employees, with additional coverage available for dependents
• Up to 16 weeks of paid parental leave, regardless of path to parenthood
• Annual development allowance

Location

Remote

Note: We cannot currently sponsor or support visa transfers at this time. In compliance with applicable law, all persons hired will be required to verify identity and eligibility to work.

Recruitment scam warning: All communications will come from an @honeycomb.io email address. Our recruiting process will never ask for financial or sensitive personal information such as Social Security or tax identification numbers, credit card numbers, or bank account information.

Diversity & accommodations: Honeycomb is committed to building a diverse, inclusive, and equitable workplace and provides reasonable accommodations as needed.

Privacy notice: If your application is unsuccessful (or you withdraw or decline an offer), Honeycomb will retain your information for a period of time in accordance with local laws. For more details, see the Honeycomb Privacy Notice. To request changes to retention or to update information, contact [email protected].