Data Engineer
Python
Apache Airflow
Azure Data Factory
Azure Synapse Analytics
Big Data
Bigdata
Bigquery
Cloud
Cloud Platform
Data Architecture
Data Engineer
Data Integration
Data Lake
Data Lakehouse
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Warehouse
Data Warehousing
Database
Databases
Databricks
Delta Lake
ETL
Kafka
Microsoft Azure
Snowflake
Spark
SQL
Stream Processing
Job Description
SN Cloud Solutions seeks a Data Engineer to help shape the data backbone powering enterprise analytics in a hybrid Chicago setting. The role centers on designing, building, and maintaining scalable data pipelines and platforms using Python, PySpark, SQL, and cloud ETL/ELT frameworks to enable BI and analytics across the organization.
Responsibilities
- Design, develop, and optimize ETL/ELT pipelines to handle large-scale data processing.
- Implement scalable data solutions with Python, PySpark, and the Apache Spark ecosystem.
- Build and sustain data ingestion frameworks for structured and unstructured sources.
- Write and tune SQL queries, stored procedures, and related database objects.
- Consolidate data from multiple sources, including APIs, databases, and cloud storage.
- Establish data quality checks, validation, and monitoring processes.
- Collaborate with cloud platforms such as AWS, Azure, or Google Cloud Platform.
- Create and manage data lakes and data warehouses.
- Partner with data scientists, analysts, and business stakeholders to support analytics needs.
- Automate workflows using orchestration tools like Apache Airflow.
- Resolve production issues and optimize pipeline performance.
- Operate within Agile/Scrum frameworks and CI/CD best practices.
Requirements
- Minimum 3 years of hands-on Data Engineer experience.
- Strong proficiency with Python and PySpark.
- Hands-on experience with Apache Spark.
- Advanced SQL programming skills.
- Experience developing ETL/ELT solutions.
- Experience with cloud platforms (AWS, Azure, or GCP).
- Knowledge of data warehousing concepts.
- Experience with Apache Airflow or similar orchestration tools.
- Experience with Git, Jenkins, or Azure DevOps.
- Solid understanding of data modeling and database design.
- Experience with relational and NoSQL databases.
- Excellent problem-solving and communication skills.
Technologies
- Python
- PySpark
- Apache Spark
- SQL
- AWS
- Azure
- GCP
- Apache Airflow
- Databricks
- Kafka
- Delta Lake
- Snowflake
- Redshift
- BigQuery
- Synapse
- Azure Data Factory
- Git
- Jenkins
- Azure DevOps
Benefits
- Flexible schedule
Job Type
- Full-time
Salary
From $50.00 per year
Experience
- Azure Data Factory: 7 years (Required)
- SQL: 3 years (Required)
- Data warehouse: 4 years (Required)
Location
Hybrid remote in Chicago, IL 60618
Similar Jobs
