Senior Data Engineer
Senior
Analytics
API
APIs
Azure Data Lake
Big Data
Bigdata
Business Intelligence
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Integration
Data Lake
Data Pipeline
Data Platform
Data Processing
Data Warehouse
Data Warehousing
Database
Databases
ETL
Hadoop
Hive
Informatica
Integration
Programming Language
Programming Languages
Reporting and Analytics
Spark
SQL
Talend
Job Description
Brooksource invites applications for a Senior Data Engineer tasked with designing, building, and maintaining scalable data pipelines and architectures that empower advanced analytics and business intelligence through big data technologies, cloud platforms, and robust data modeling.
Responsibilities
- Design, develop, and optimize large scale ETL pipelines using Informatica, Talend, Apache Spark, and Apache Hive
- Construct and sustain data architectures on cloud platforms such as AWS, Azure Data Lake, and other public cloud environments
- Collaborate with cross functional teams to capture requirements and translate them into scalable data solutions
- Create data models employing dimensional modeling techniques for data warehousing and analytics
- Oversee data integration processes involving linked data, RESTful APIs, and other external sources
- Ensure data quality, security, and compliance throughout the data lifecycle
- Facilitate the deployment of machine learning models through integrated data pipelines
- Utilize SQL, Python, Java, Bash scripting, and Shell scripting to automate and enhance processes
- Participate in Agile development cycles through sprint planning, reviews, and continuous improvement efforts
- Document data workflows and architecture diagrams to promote transparency and knowledge sharing
Requirements
- Proven experience as a Data Engineer or similar role focusing on big data systems and cloud-based architectures
- Extensive knowledge of AWS services (S3, Redshift, Glue), Azure Data Lake, the Hadoop ecosystem, and other public cloud offerings
- Strong proficiency with SQL databases including Microsoft SQL Server, Oracle, and cloud-based options
- Hands-on ETL/ELT development experience using Informatica or Talend
- Demonstrated expertise in data modeling, including dimensional modeling for enterprise data warehouses
- Skilled in Python, Java, VBA, Bash scripting, and Shell scripting for automation tasks
- Familiarity with business intelligence tools such as Looker or equivalent platforms for reporting and analysis
- Understanding of RESTful API integration for linked data access
- Knowledge of analysis skills related to database design, query management, model training, and analytics methodologies
- Experience delivering value in Agile environments with iterative development
Technologies
- Informatica
- Talend
- Apache Spark
- Apache Hive
- AWS
- Azure Data Lake
- Hadoop
- S3
- Redshift
- Glue
- Microsoft SQL Server
- Oracle
- SQL
- Python
- Java
- VBA
- Bash scripting
- Shell Scripting
- Looker
- RESTful API
Compensation
Hourly rate: USD 65.00 - 85.00 per hour
Work Location
New York, NY, with remote work option
Similar Jobs
