Data Engineer – Azure Databricks and Cloud Data Solutions (Safety)
Analytics
Azure Data Engineer
Big Data
Bigdata
Business Intelligence
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Lake
Data Lakehouse
Data Lakes
Data Pipeline
Data Platform
Data Processing
Data Visualization
Data Warehouse
Database
Databases
Databricks
Databricks Dlt
Databricks Lakehouse
Delta Lake
Digital Marketing
ETL
Informatica
Reporting and Analytics
Spark
SQL
Visual Design
Job Description
Join Southern Company’s Operations Data Analytics team and help deliver secure, reusable, high-performing cloud data services built on the Azure Databricks lakehouse platform. This onsite role in Birmingham, AL focuses on Safety data and supports data product development with pipelines, scalable ingestion patterns, and enterprise-aligned architectures.
What You’ll Do
- Design and develop robust data pipelines that ingest, store, and process data from multiple sources to support analytics and reporting.
- Create and maintain logical and physical data models to enable efficient querying, analysis, and visualization.
- Build and maintain ETL processes that move data between systems while supporting data quality and consistency.
- Implement and support lakehouse solutions using medallion architecture, including data engineering pipelines and storage layers.
- Optimize data lake storage, retrieval, performance, and resource utilization for analytical workloads.
- Develop dynamic, metadata-driven frameworks and scalable data ingestion patterns.
- Support streaming datasets and Databricks Delta Live Tables.
- Apply lakehouse performance tuning through partitioning, aggregations, and materialized views.
- Architect, develop, implement, integrate, and support cloud technology solutions aligned to business partner expectations for quality, cost, and timeliness.
- Collaborate with developers and infrastructure teams to optimize application code, create reusable services, plan capacity, and protect application and data security.
- Act as a subject matter expert for data architecture and Databricks.
- Communicate effectively with customers, team members, and management, building strong relationships with peers and business partners.
- Develop required project documentation, follow Change Management policies and procedures, and respond quickly to operational problems.
- Build an understanding of company business challenges and goals while exhibiting Our Values.
What You Bring
- Bachelor’s degree required, preferably in Computer Science, Management Information Systems, Engineering, or another technology-related field.
- Experience with Databricks, including notebooks, clusters, and jobs.
- Experience with a major cloud platform, preferably Microsoft Azure, including Cosmos DB, Storage Application Services, and Azure Data Factory.
- Experience architecting data-oriented solutions using cloud data architecture platforms and patterns, preferably Azure.
- Experience implementing lakehouses with medallion architecture, data engineering pipelines, and data storage layers.
- Experience with dynamic metadata-driven frameworks and large-scale data ingestion patterns.
- Experience developing complex logical and physical data models.
- Experience optimizing cloud PaaS resources to follow best practices and reduce cost, plus lakehouse performance tuning using partitioning, aggregations, materialized views, or similar techniques.
- Experience with Agile, DevOps, and CI/CD methodologies.
- Familiarity with Apache Spark, Delta Lake, and MLflow, including knowledge of structured and unstructured data across databases, data warehouses, and data lakes.
- Knowledge of software development technologies, methodologies, and associated tools.
- Excellent root cause analysis and problem-solving skills.
Technologies You’ll Work With
- Microsoft Azure, Azure Databricks, Unity catalog
- Cosmos DB, Storage Application Services, Azure Data Factory
- Databricks (notebooks, clusters, jobs), medallion architecture
- ETL, Databricks Delta Live Tables
- Partitioning, aggregations, materialized views
- Apache Spark, Delta Lake, MLflow
- Agile, DevOps, CI/CD, structured and unstructured data, lakehouse
Benefits
- Competitive base salary
- Annual incentive awards for eligible employees
- Health, welfare and retirement benefits designed to support physical, financial, and emotional/social well-being
- Additional and specific details about total compensation and benefits provided during the hiring process
Similar Jobs
