Lead Data Engineer - Enterprise Data & Analytics

Responsibilities

• Lead the design, prototyping, and development of data pipeline architectures and production-grade pipelines.
• Drove internal process improvements by automating manual workflows and optimizing data delivery.
• Redesign infrastructure to scale with growing data needs and changing requirements.
• Perform root-cause analyses on internal and external processes and data to identify opportunities for improvement.
• Apply strong analytics to unstructured datasets and inform data strategy.
• Understand the overall architecture, collaborate as a team player, lead technical discussions, and clearly communicate outcomes.
• Serve as a senior individual contributor within the data or software engineering teams.
• Participate in the Technical Review Board alongside the Manager and Principal Engineer.
• Act as a technical liaison among managers, software engineers, and principal engineers.
• Collaborate with software engineers to analyze, develop, and test functional requirements.
• Provide hands-on technical leadership in designing, developing, reviewing, and optimizing production-grade data pipelines, data products, and platform capabilities.
• Contribute significantly to production codebases while establishing engineering standards, mentoring teammates, and delivering scalable, resilient solutions.
• Mentor and coach engineers to grow technical capability and performance.
• Work with team members to explore design approaches, prototype new technology, and assess feasibility.
• Operate in an Agile/Safe/Scrum environment to deliver high quality software.
• Establish architectural principles and design patterns, and mentor the team on their application.
• Facilitate cross-team communication among front-end, back-end, data, and platform engineers.
• Assume a formal engineering lead role within the area of expertise.
• Stay current with industry trends and developments to guide technical decisions.

Requirements

• Bachelor's degree in Computer Science, Engineering, or related field with 6 years of experience, or an Associate's degree with 8 years.
• Knowledge of professional software engineering practices and the full software development life cycle, including coding standards, code reviews, source control, build processes, testing, and operations.
• In-depth data engineering expertise with at least 5 years building data pipelines and exposure to data science or analytical modeling.
• Experience leading Data Engineering teams in a Continuous Integration / Continuous Delivery environment.
• Ability to build and lead data products that are highly resilient.
• Experience building and leading test automation suites, unit testing, data quality checks, monitoring, and observability.
• Minimum 5 years of experience with relational and NoSQL databases.
• Experience with cloud platforms such as Google Cloud Platform, Azure, and AWS.
• Continuous integration experience with Jenkins, GitHub Actions, or Azure Pipelines.
• Experience with cloud technologies, development, and deployment practices.
• Proficiency with collaboration and project management tools such as Jira, GitHub, SharePoint, and Azure Boards.
• Experience using advanced data processing technologies such as Apache Spark, Hive, Airflow, Kafka, and GCP Dataflow.
• Experience with big data, statistics, and data aspects of machine learning.
• Hands-on experience with Google BigQuery, FHIR APIs, and Vertex AI.
• Knowledge of workflow scheduling tools such as Apache Airflow and Google Composer in data systems.
• Experience with infrastructure as code and containerization in cloud environments (Kubernetes, Docker).

Technologies

• Python
• SQL
• Apache Spark
• Apache Flink
• Ray
• Hive
• Airflow
• Kafka
• Google Cloud Platform (GCP)
• BigQuery
• Vertex AI
• FHIR APIs
• Delta Lake
• Apache Iceberg
• Apache Hudi
• Parquet
• Avro
• ORC
• Jenkins
• GitHub Actions
• Azure Pipelines
• Jira
• SharePoint
• Azure Boards
• Kubernetes
• Docker
• Google Dataflow
• Google Composer
• AWS

Benefits

• Medical: Multiple plan options
• Dental: Delta Dental or reimbursement account for flexible coverage
• Vision: Affordable plan with national network
• Pre-Tax Savings: HSA and FSAs for eligible expenses
• Retirement: Competitive retirement package to secure your future

Why Mayo Clinic

Mayo Clinic is top ranked in more specialties than any other care provider according to U.S. News & World Report. We focus on patient needs while investing in employees with competitive compensation and comprehensive benefits. Ongoing education and advancement opportunities support a long, successful career here.

Site Description

Our workforce spans multiple locations beyond Minnesota. Employees are based at Mayo Clinic campuses in Phoenix/Scottsdale, Arizona; Jacksonville, Florida; Rochester, Minnesota; Mayo Clinic Health System sites across the Midwest; and international locations. Each campus offers a distinct environment where professional growth and work-life balance are supported.

Equal Opportunity

Mayo Clinic provides equal employment opportunities to all. All qualified applicants will be considered without regard to race, color, religion, sex, gender identity, sexual orientation, national origin, protected veteran status, or disability. E-Verify participation and Form I-9 work authorization may be required.

Recruiter

Laura Percival