Staff Software Engineer, Agentic Systems

The Staff Software Engineer role will help develop the runtime infrastructure for Moveworks agent systems within ServiceNow. The position emphasizes distributed systems engineering for agent orchestration, session management, event-driven pipelines, structured concurrency, and production-grade observability and caching.

Key Responsibilities

• Agent orchestration engine: build a state machine that manages long-running agent sessions, coordinating planning, execution, and user interaction across multiple LLM calls and tool invocations.
• Distributed session management: implement lease-based ownership using DynamoDB conditional writes, heartbeat protocols for liveness, and crash recovery through checkpointing.
• Event-driven message pipeline: design ordered delivery and processing with SQS FIFO queues, handle event processing with Kafka consumers, and support real-time streaming via gRPC and Socket.IO.
• Structured concurrency: use Python asyncio TaskGroups to run multiple concurrent tasks per session, including message polling, lease heartbeats, output publishing, and orchestrator execution. Apply fail-fast semantics and graceful cancellation.
• Observability infrastructure: deliver OpenTelemetry instrumentation with distributed trace context propagation across async boundaries, including custom span lifecycle management for sessions that span minutes.
• Caching and state layers: build caching and persistent state using Redis and DynamoDB KV stores with per-org and per-bot scoping, batch read optimization, and hot-reload configuration.

Required Qualifications

• Deep experience in at least 3 of the following areas: distributed systems consistency models, idempotency, exactly-once delivery, distributed locking/leasing.
• Deep experience in at least 3 of the following areas: concurrent or async programming (Python asyncio, Go goroutines, structured concurrency, cancellation handling).
• Deep experience in at least 3 of the following areas: event-driven architectures (SQS, Kafka, pub/sub, backpressure, delivery guarantees).
• Deep experience in at least 3 of the following areas: database systems for infrastructure (DynamoDB conditional writes and transactions; Redis connection pooling and pub/sub).
• Deep experience in at least 3 of the following areas: observability (OpenTelemetry, distributed tracing, span context propagation, Prometheus metrics).
• Deep experience in at least 3 of the following areas: gRPC/protobuf (streaming RPCs, service interface design, error handling patterns).
• Strong in Python or Go (ideally both).
• Experience designing and operating systems that handle real traffic at scale.
• Comfort with ambiguity for novel problems without textbook solutions.

Technologies

Python, Go, DynamoDB, SQS FIFO, Kafka, gRPC, Socket.IO, Python asyncio TaskGroups, OpenTelemetry, Redis, Prometheus, gRPC/protobuf, LLM calls, and proprietary models.

Location and Work Persona

Location: Mountain View, CA (onsite).

Work personas are categories assigned to ServiceNow employees depending on the nature of work and assigned work location. ServiceNow may confirm the distance between your primary residence and the closest ServiceNow office using a third-party service to determine eligibility for a work persona.

Equal Opportunity Employer

ServiceNow is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, national origin or nationality, ancestry, age, disability, gender identity or expression, marital status, veteran status, or any other category protected by law. Applicants with arrest or conviction records will also be considered in accordance with legal requirements.

Accommodations

If you require a reasonable accommodation to complete any part of the application process, or are unable to use the online application and need an alternative method to apply, contact [email protected].

Export Control Regulations

For positions requiring access to controlled technology subject to export control regulations, including the U.S. Export Administration Regulations (EAR), ServiceNow may be required to obtain export control approval from government authorities for certain individuals. Employment is contingent upon ServiceNow obtaining any export license or other approval that may be required by relevant export control authorities.