Senior Data Engineer
Senior
Azure
Azure Data Factory
Azure Databricks
Big Data
Bigdata
Data Engineer
Data Engineering
Data Factory
Data Factory Azure
Data Integration
Data Pipeline
Data Platform
Data Processing
Data Quality
Databricks
Databricks Pyspark
ETL
Hadoop
Microsoft Azure
Pyspark
Software Testing
Spark
SQL
Test Automation
Job Description
CGI Group, Inc. is hiring a Senior Data Engineer for a remote role based in Oregon (remote within the USA). In this position, you will help ensure the reliability of enterprise data platforms and ETL/ELT pipelines through end-to-end testing, SQL-based validation, and data quality testing, while participating in Agile/Scrum delivery activities.
What you’ll do
- Design, develop, and execute test plans, test scenarios, and test cases for enterprise data platforms, ETL/ELT pipelines, and data integration solutions.
- Run end-to-end ETL testing and validate data accuracy, completeness, transformation logic, reconciliation, and data quality across source and target systems.
- Develop and execute complex SQL queries for backend testing and data validation.
- Test large-scale data processing solutions using Databricks, Apache Spark, PySpark, Python, Hive, Hadoop/HDFS, and Azure Data Factory (ADF).
- Perform functional, integration, regression, system, and data quality testing across data pipelines and applications.
- Validate batch processing, job scheduling, dependencies, and workflow execution using Control M and Unix/Linux environments.
- Identify, document, track, and retest defects in collaboration with developers, data engineers, business analysts, and other stakeholders.
- Develop and maintain reusable test scripts and automation solutions where applicable.
- Participate in Agile/Scrum activities including sprint planning, refinement, daily stand ups, reviews, and retrospectives.
What you bring
- 8+ years of experience in software quality engineering/testing, including strong ETL testing, data validation, integration testing, and regression testing.
- Strong hands-on SQL skills, including writing complex queries for source-to-target validation, reconciliation, and data quality testing.
- Experience testing large-scale data platforms and ETL/ELT pipelines.
- Hands-on experience with Databricks, Apache Spark, PySpark, Python, and Azure Data Factory (ADF).
- Experience with Hadoop ecosystem technologies, including Hadoop/HDFS and Hive.
- Experience working with Unix/Linux environments and enterprise job scheduling tools such as Control M.
- Strong understanding of data quality concepts, data transformations, data mappings, and source-to-target validation.
- Experience with defect management and test management processes in Agile/Scrum environments.
- Strong analytical, troubleshooting, communication, and collaboration skills.
Technology focus
- Databricks
- Apache Spark
- PySpark
- Python
- SQL
- Hive
- Hadoop/HDFS
- Azure Data Factory (ADF)
- Control M
- Unix/Linux
- Agile/Scrum
Compensation and location
Location: Oregon (remote within the USA)
Salary: USD 62,900 - 139,300 per year
Minimum experience: 8 years
Benefits
- Competitive compensation including a profit participation program
- Comprehensive medical, dental, and vision benefits
- Basic life and accidental death & dismemberment insurance
- 401(k) plan with matching contributions and a CGI share purchase plan
- Paid accrued vacation leave ranging from 10 to 20 days per year based on job level, years of relevant prior experience, and years of service
- 10 paid holidays per year
- At least 80 consecutive hours of paid sick/safe leave (except where applicable state/local law requires more)
- Paid parental leave, ranging from 20 to 70 consecutive business days based on circumstances and applicable laws
- Bereavement leave, ranging from 1 to 7 days per year based on relationship
- Paid jury duty leave, up to time summoned
- Learning opportunities and tuition assistance
- Wellness and well being programs
Similar Jobs
L