Software Engineer II - AI Performance

Disney Entertainment and ESPN Product & Technology is hiring a Software Engineer II to strengthen reliability for operational AI systems. On the Reliability Insights team, you will design and build data and reporting pipelines that turn telemetry, AI agent interactions, and operational logs into measurable results for real-world performance, precision, and business impact.

This onsite role in New York, NY focuses on bridging backend engineering with data-driven evaluation. You will connect AI signals to incident lifecycles, quantify reliability outcomes, and deliver dashboards and metrics that help leadership and technical teams act on model health over time.

What you will do

• Design, build, and maintain automated feedback pipelines that capture operational outcomes (confirmed incidents, false positives, dismissed alerts) and correlate them directly to AI model predictions and agent actions.
• Create programmatic evaluation frameworks to track model precision, recall, and drift over time, including automated threshold alerts to trigger model tuning and reduce alert fatigue.
• Build end-to-end data pipelines that link AI outputs to incident lifecycles, measuring business impact using metrics such as Service to Incident correlation, notification success rates, and Mean Time to Mitigation (MTTM).
• Design and deploy scalable APIs, metrics layers, and single-pane model health dashboards that communicate AI ROI, operational health, and performance narratives to both leadership and technical stakeholders.
• Implement mechanisms to capture structured end user feedback and feed real-world usage signals back into ongoing model improvement cycles.

Qualifications

• 3+ years of applicable experience in backend and data pipeline engineering, including building scalable APIs (for example, FastAPI, Flask) and metrics or insight generation platforms.
• Strong data platform engineering experience with distributed processing tools such as PySpark, SQL, Pandas, Databricks, and Snowflake for telemetry processing and analysis.
• Practical knowledge of model evaluation concepts including precision, recall, model drift, and ground-truth labeling systems.
• Hands-on experience with modern development workflows: GitHub (version control), Docker (containerization), and cloud-native deployments on AWS and EKS.
• Proficiency with AI-assisted development tools such as Cursor and Claude Code to increase engineering velocity.
• Strong analytical skills to translate complex system telemetry and operational logs into reliability metrics and business insights.
• Excellent cross-functional collaboration and communication, including partnership with operational teams such as SRE and DevOps to turn raw data into actionable insights.

Technologies

• FastAPI, Flask, PySpark, SQL, Pandas, Databricks, Snowflake
• GitHub, Docker, AWS, EKS
• Cursor, Claude Code

Compensation

• The hiring range for this position in New York, NY is $124,900.00 to $167,400.00 per year.
• Base pay may vary based on internal equity and the candidate’s geographic region, job-related knowledge, skills, and experience.
• A bonus and/or long-term incentive units may be included in the compensation package, along with medical, financial, and/or other benefits, depending on the level and position offered.

Education: Bachelor’s degree in Computer Science, Engineering, or equivalent experience.