DeveloperJobs.io
← Back to all jobs

Job Description

As an AI/ML Model Quality engineer, you will design and run systems that improve machine learning data reliability and model performance across Wallet, Payments, and Commerce. The work spans validation frameworks, synthetic data generation, and telemetry-driven monitoring for both conventional ML and generative AI workflows.

Role Focus

Build and maintain intelligent systems for machine learning data and model quality, including validation frameworks, synthetic data generation models, and monitoring pipelines. Own data ecosystem health by defining observability metrics and leading telemetry analysis for workflows used in Wallet, Payments, and Commerce.

Responsibilities

  • Develop and maintain intelligent systems, validation frameworks, synthetic data generation models, and monitoring pipelines that support a healthy data ecosystem, ensuring models are trained, evaluated, and deployed on data teams can trust.
  • Work closely with Data Engineering, Privacy, and Legal teams.
  • Lead data ecosystem health activities that underpin ML and generative AI features across Wallet, Payments, and Commerce, including building validation frameworks, defining observability metrics, and driving telemetry analysis.
  • Define and analyze observability metrics for ML and GenAI systems.
  • Build models to support calibration and robustness measurement.
  • Lead telemetry analysis across generative AI workflows.

Requirements

  • Bachelor’s degree with exceptional hands-on experience in ML/AI model quality or applied research, or an M.S. or Ph.D. in Machine Learning, Computer Science, Data Science, Statistics, Mathematics, Engineering, or a related quantitative field (strongly preferred).
  • 3+ years of experience defining and tracking ML model health metrics in production, including model performance monitoring, feature drift detection, and observability instrumentation.
  • 3+ years in machine learning, data science, or a closely related analytical role, with a strong focus on data quality, model evaluation, or ML observability in production environments.
  • Proficiency in Python (Pandas, NumPy, Scikit-learn), plus common ML frameworks such as PyTorch and SQL for complex data analysis, metric creation, and validation.
  • Experience querying and analyzing large-scale datasets using distributed computing frameworks such as PySpark, Spark, or distributed SQL.
  • Solid understanding of statistical methods including hypothesis testing, distribution analysis, data drift detection, and statistical process control.
  • Experience with GenAI or LLM systems, including quality failure modes, output evaluation approaches, and telemetry instrumentation.
  • Strong communication skills to translate model health and data quality findings into clear, actionable insights for engineering and non-technical stakeholders.

Technologies

  • Python, Pandas, NumPy, Scikit-learn
  • PyTorch, SQL
  • PySpark, Spark
  • Tableau, Apache Superset
  • Databricks
  • LangSmith
  • MLflow, Weights & Biases

Preferred Qualifications

  • Experience with data visualization and dashboarding tools (e.g., Tableau, Apache Superset, Databricks) to present complex ML telemetry.
  • Familiarity with LLM evaluation frameworks (e.g., LangSmith) or techniques such as LLM-as-a-judge.
  • Experience with Bayesian or causal graph-based approaches to synthetic data generation.
  • Familiarity with confidence calibration techniques and uncertainty quantification.
  • Experience with ML monitoring or observability platforms (e.g., MLflow, Weights & Biases, or equivalent).
  • Experience working with privacy-constrained data or under regulatory compliance frameworks (GDPR, DMA).
  • Background in financial services, fintech, or consumer payment products.

Benefits

  • Comprehensive medical and dental coverage
  • Retirement benefits
  • A range of discounted products and free services
  • Reimbursement for certain educational expenses, including tuition
  • Opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs
  • Eligibility for discretionary restricted stock unit awards
  • Ability to purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan

Compensation and Location

Location: New York, NY (onsite)

Salary: USD 184,700 - 324,800 per year

Minimum experience: 3 years

Similar Jobs