Data Engineer II

Essentia Health is hiring a Data Engineer II to build and optimize scalable data pipelines for large datasets, supporting batch and near real-time analytics.

Responsibilities

• Design and optimize scalable, resilient data pipelines and workflows to support large datasets and meet functional and non-functional business requirements
• Implement advanced data processing and scheduling using frameworks such as Apache Spark, ADF, and Airflow
• Write and optimize advanced SQL queries for reporting and application use
• Ensure data quality, governance, and security via automated testing and monitoring
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements
• Provide technical guidance and mentorship to junior data engineers
• Develop infrastructure for both real-time and batch processing workflows
• Deliver actionable data products and insights for business-critical tools

Requirements

• Bachelor’s degree in computer science, data analytics, statistics, or related field
• 2-5 years of experience in data engineering or a related role
• Proficiency in SQL and experience with database management systems
• Strong programming skills in Python, Java, or Scala
• Experience building pipelines with orchestration tools such as Apache Airflow or AWS Glue for ingestion and delivery
• Experience with modern data management technologies such as data lake, lakehouse, and data catalogs
• Experience supporting batch and streaming data to enable near real-time analytics and reporting

Technologies

• Apache Spark, Airflow, ADF
• SQL
• Python, Java, Scala
• AWS Glue
• Data lake, lakehouse, data catalogs
• dbt
• Microsoft Fabric
• AI and machine learning

Preferred Qualifications

• Familiarity with applying AI and machine learning techniques to data use cases
• Experience creating robust data transformation workflows, including metadata management, dependency tracking, and workload orchestration
• Experience with dbt
• Experience with Microsoft Fabric
• Excellent problem-solving skills and attention to detail
• Strong communication and collaboration skills

Compensation and Schedule

• Location: Duluth, MN (Remote)
• FTE: 1
• Compensation range: $89,523.20 - $134,284.80 per year
• Shift: Day Rotation (United States of America)
• Shift start: 8:00 AM
• Shift end: 4:30 PM
• Holidays: No
• Call obligation: No
• Possible remote option: Remote

Location Details

• Building location: Peerless Building
• Department: 1007670 ANALYTICS AND ARCHITECTURE - EH SS

Benefits

• Medical, dental, vision, life, and disability insurance
• 401(k) plan with employer contributions
• Training, tuition reimbursement, and educational programs
• Flexible scheduling
• Generous time off
• Wellness resources focused on your physical, mental, and emotional health