Data Engineer- Azure Databricks- Financial Domain Exp Must
Backend Developer
Application Security
Azure
Azure Data Engineer
Azure Databricks
Azure Functions
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Data Platform
Cloud Data Warehouse
Cloud Data Warehouse
Cloud Engineer
Cloud Infrastructure
Cloud Platform
Cloud Platforms
Cloud Platforms Cloud Platforms
Cloud Technology
Data
Data Analysis
Data Analytics
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Warehouse
Data Warehousing
Database
Databases
Databricks
Databricks Pyspark
Databricks Spark
DevOps
DevSecOps
Distributed Systems
Engineer
Engineering
ETL
GraphQL
Informatica
Information Technology (IT)
Infrastructure As Code
Microsoft Azure
Programming
Programming Language
Programming Languages
Security Automation
Security Devops
Snowflake
Software Architecture
Software Development
Software Engineering
Spark
SQL
Job Description
Build cloud data platforms and analytics pipelines using Azure Databricks with strong financial domain experience.
Responsibilities
- Develop services, controls, and reusable patterns that help the team deliver value safely, quickly, and sustainably in public cloud environments
- Design and develop highly scalable Azure cloud applications using microservices and Azure Functions
- Enable secure public cloud operations by applying repeatable implementation patterns
- Build and optimize ETL/ELT solutions on Azure Databricks using LakeBase and Spark
- Develop and deploy large-scale data pipelines to support machine learning, insights generation, and business intelligence dashboards
- Design, build, and manage modern data pipelines for data integration that support analytics and ML feature development
- Write advanced and complex SQL, including performance tuning and query optimization
- Build AI-based solutions to support business needs, process automation, and improved operational efficiency
- Improve data reliability, data integrity, system efficiency, and overall data quality
- Decompose new product capabilities into functional changes for verification and validation
- Work effectively with business stakeholders and third-party technical organizations
- Contribute to the architectural evolution of data engineering patterns, frameworks, systems, and platforms, including defining best practices and standards
- Mentor other data engineers by providing technical direction and teaching teams how to leverage cloud data platforms
- Gather and analyze data to support decision-making and provide detailed, realistic estimates
- Promote high-performance, collaborative technical execution to maintain quality output
Requirements
- 7+ years of experience with Python and Scala
- 5+ years with Apache Spark including PySpark and Spark SQL
- 3+ years of DevOps automation using Terraform
- 3+ years building and managing API lifecycles and designs (e.g., GraphQL, REST)
- 3+ years building and deploying cloud-based solutions on Azure Databricks with UC, Snowflake, Azure Functions, and Service Bus
- 5+ years with SQL, including designing complex data schemas and query performance optimization
- Experience building LLM integrations for workflow automation and business needs
- Experience with CI/CD and tools including GitHub Actions, GIT, Artifactory, and Sonar
Technologies
- Python, Scala
- Apache Spark, PySpark, Spark SQL
- Terraform
- GraphQL, REST
- Azure Databricks (UC), Snowflake
- Azure Functions, Service Bus
- SQL
- LLM integrations
- CI/CD, GitHub Actions, GIT, Artifactory, Sonar
- Microservices
- LakeBase, ETL/ELT, AI/ML
Compensation
- $70.00 - $80.00 per hour
Schedule
- 40.0 hours per week
Location
- Eden Prairie, MN (onsite)
Minimum experience: 3 years