Big data engineer with GCP
Job Description
Work onsite in Phoenix, AZ to build and optimize large-scale data solutions using GCP and Big Data tooling.
Responsibilities
- Design, develop, and optimize data pipelines for large-scale data processing, transformation, and analysis using Big Data and GCP technologies
- Develop and tune SQL queries and work across database systems
- Design and optimize data models with a focus on performance and scalability
- Build event processing pipelines using Kafka or GCP PubSub
- Apply knowledge of distributed, multi-tiered systems, algorithms, and relational databases
Requirements
- 6 to 8 years of Data Engineering experience in a Big Data environment
- Hands-on software development experience with Big Data and Analytics solutions
- Experience with Hadoop Hive, Spark, Python, and shell scripting
- Working knowledge of GCP Cloud services including BigQuery, Airflow, DataProc, and PubSub
- Proficiency in SQL and database systems
- Experience designing and optimizing data models for performance and scalability
- Experience building event processing pipelines with Kafka or GCP PubSub
- Experience with CI/CD pipelines, automated test frameworks, and source code management tools including XLR, Jenkins, and Git
- Good knowledge and experience with configuration management tools like GitHub
Technologies
- Hadoop Hive
- Spark
- Python
- Shell scripting
- GCP Cloud: BigQuery
- Airflow
- DataProc
- PubSub
- SQL
- Kafka
- CI/CD pipelines
- Automated test frameworks
- XLR
- Jenkins
- Git
- GitHub
Location
- Phoenix, AZ (in person)
Salary
- USD 85,700 - 104,000 per year