Engineers and Constructors International Inc. seeks a Principal Data Engineer for a contract engagement anticipated to last 7 months. This onsite role is based in Houston, TX and focuses on leading Azure-based data lake and governance capabilities in high-volume environments.
Role Summary
The Principal Data Engineer will provide technical leadership for Azure data lake architecture and data governance, including implementation and optimization of Databricks Delta Live Tables, Structured Streaming, and Unity Catalog. The assignment includes roadmap planning across related data capabilities and ensuring production performance while maintaining strong governance, lineage, and metadata management.
Responsibilities
- Architect, design, and implement scalable Azure-based data lake solutions.
- Establish roadmaps for capability-building across Qlik Replicate, Databricks Delta Live Tables, and Unity Catalog.
- Provide technical leadership to ensure adherence to best practices and quality standards.
- Lead development of data transformation pipelines using Databricks Delta Live Tables and Structured Streaming.
- Optimize Databricks resources for high-volume processing while staying within budget constraints.
- Oversee usage of Databricks Unity Catalog for data governance, lineage, and metadata management.
- Develop and enforce robust data governance policies that support compliance, security, and data quality.
- Monitor data pipeline performance and proactively address production issues.
- Collaborate with cross-functional teams to align project objectives with business goals.
- Translate technical challenges into actionable insights for executive teams.
- Mentor and develop a high-performing team skilled in Azure, Databricks, and data streaming technologies.
- Facilitate knowledge transfer sessions to help internal teams adopt cutting-edge methodologies.
Required Qualifications
- Bachelor’s or Master’s degree in IT or a related discipline.
- 15-17 years of experience in the IT industry with expertise delivering cost-effective cloud-based data analytics solutions.
- Proven track record building data lake architectures and optimizing performance in big data environments.
- Certification in Azure Data Engineering or Databricks (preferred).
- Strong knowledge of Azure technologies, including ADF, ADLS, Databricks, Stream Analytics, Delta Live Tables, and Synapse Analytics.
- Advanced proficiency in SQL and programming languages such as Python, Scala, or Spark-SQL.
- Experience designing and developing scalable ETL pipelines and optimizing data workflows.
- Deep understanding of data migration from on-premises to cloud and cloud to cloud.
- Proven ability to lead diverse teams and manage end-to-end analytics projects.
- Exceptional communication and collaboration skills for engaging executives and stakeholders.
- Strong sense of ownership, accountability, adaptability, and commitment to results.
- Expertise in identifying risks and implementing effective mitigation strategies.
Technologies
Azure, Qlik Replicate, Databricks Delta Live Tables, Unity Catalog, Databricks, Structured Streaming, SQL, Python, Scala, Spark-SQL, ADF, ADLS, Stream Analytics, Synapse Analytics, Hadoop, Kafka, PowerBI, QlikSense, AI/ML, NLP, GenAI, AGILE, KANBAN, Mulesoft, Shell IDT/Downstream & Renewables IDT/Trading & Supply/TS - Crude & Products/TS - Products/TS - Products Investments, VIP (Voyage Management System), IMOS, ETRMs, MuleSoft
Nice to Have
- Familiarity with big data tools like Hadoop, Kafka, and Spark.
- Experience with visualization tools such as PowerBI or QlikSense.
- Knowledge of AI/ML, NLP, and GenAI concepts.
- Familiarity with AGILE and KANBAN methodologies.
Project Context
T& S Future Freight Platform – Jigsaw
The role will support building a new Freight Platform that replaces RADAR to serve the freight business and its customers. Work includes optimizing business processes and defining master data maintenance processes, with the Freight Platform acting as a hub to integrate necessary data and support an ecosystem of IT tools aligned to Shell’s business growth roadmap.
During the detailed design phase through Stage Gate 2, the engagement aims to produce an approved solution architecture addressing an Industry standard Voyage Management System (VIP) alongside other IT components to close capability gaps identified earlier in the Study Phase.
Data Platform and Integration Activities
- Assess how the platform supports capturing freight exposure management across the house, with the potential to use existing ETRMs if required.
- Determine which IMOS reporting capabilities to use while avoiding unnecessary functionality and evaluating internal, cost-efficient alternatives.
- Identify MI data use cases that provide Shell a competitive edge, including developing solutions in-house (for example, with Qliksense for exposure reporting, VaR, and Carbon Emissions reporting).
- Identify a middleware integration approach to replace Mulesoft to connect Shell systems to VESON VIP.
- Develop a data platform solution to ingest, host, and distribute data between multiple integration points.
Business Applications
- RADAR Bulk Upload
- RADAR Mail
- RADAR-SAFE
- RadarMsgServer
- RADAR_app
- VESON VIP T&S Pr| Data Modelling
- Data Engineering
- Data Integration
Portfolio
- Shell IDT/Downstream & Renewables IDT/Trading & Supply/TS - Crude & Products/TS - Products/TS - Products Investments
Eligibility and Review Notes
- Eligibility: Applicants must be a US Citizen or Permanent US Resident to qualify.
- Application review: The ECI Group uses AI-assisted tools in early stages of application review, with decisions guided by human oversight.
- Compliance: Review and processing are fully compliant with GDPR and ECI Group’s applicable data protection policies and Privacy Notice.
