Azure Data Engineer

Skywalk Global is seeking an Azure Data Engineer for an onsite role in Indianapolis, IN, focused on turning the agency’s data strategy into dependable data products. The work centers on building and optimizing batch and streaming pipelines in Azure, modeling data for reporting, and enforcing governance and security for sensitive health information.

This position supports data integration, pipeline development, data modeling, governance, and documentation while collaborating with cross-functional partners to translate business requirements into technical solutions. Local candidates are required for consideration.

What you’ll do

• Support data technology strategy efforts, including resolving strategic data integration issues related to data quality and stewardship.
• Provide direction and oversight for designing and implementing data solutions across multiple secure systems for integration and synthesis.
• Assist with defining, implementing, and maintaining new data management policies and data engineering design standards.
• Advise, train, and provide technical support for projects across the organization.
• Participate in project activities such as requirements gathering, solution reviews, and explaining technical complexities to non-technical stakeholders.
• Partner closely with solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation.
• Extract, transform, and load data from a variety of sources using data engineering tools and methods.
• Design, build, and maintain batch and streaming pipelines in Azure Databricks using PySpark, Spark SQL, and Delta Lake, following a medallion (bronze/silver/gold) lakehouse architecture.
• Develop, load, and tune dimensional data models and SQL workloads in Azure Synapse Analytics using dedicated and serverless SQL pools for reporting and dashboards.
• Orchestrate and monitor data workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery.
• Apply data governance and security controls to protect sensitive health data (PHI/PII) in line with HIPAA and agency policy, including Unity Catalog permissions, role-based access, row- and column-level security, and data masking.
• Document and demonstrate solutions through clear documentation, flowcharts, diagrams, layouts, charting, code comments, and maintainable code.
• Support operations by performing systems analysis and recommending changes to policies and procedures.

Required qualifications

• Master’s degree in data science, analytics, informatics, computer science, mathematics, engineering, or a related field (or equivalent experience as described below).
• 4+ years of relevant experience (or a Bachelor’s degree in a related field plus 2+ years relevant experience; certifications and coursework may also substitute as described).
• Developing with Databricks notebooks using PySpark and Spark SQL (including transformations, joins, aggregations, and window functions) plus experience with R.
• Experience with CID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM).
• Working with a medallion architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT.
• Using Unity Catalog including workspace catalogs, schemas, permissions, lineage, external locations, and storage credentials.
• Right-sizing clusters and SQL warehouses and diagnosing issues related to partitioning, shuffle, skew, and caching via Spark UI.
• Scheduling and managing dependencies with ADF Pipelines and Databricks Workflows (Jobs), and using Git Repos for source control.
• Designing MPP warehouse tables, including distribution strategies (hash, round-robin, replicated), columnstore indexes, and partitioning.
• Querying Parquet, Delta, and CSV in ADLS Gen2 using OPENROWSET, external tables, and views.
• Building star-schema facts and dimensions, including stored procedures and views, with support for SCD Type 1 and SCD Type 2.
• Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers).
• Working with workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing.
• Designing ADLS Gen2 folder/file formats (Parquet/Delta), access control via Entra ID, managed identities, and Key Vault.
• Using Git and CI/CD with Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles.
• Monitoring pipelines and workloads with Azure Monitor and Log Analytics, including alerting and troubleshooting failed jobs.
• Strong written and verbal communication skills.
• Working with Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks.

Note: This job description does not list all activities, duties, or responsibilities required of the employee. Other duties may be assigned at any time.