Data Engineer II
Big Data
Clickhouse
Cloud
Cloud Infrastructure
Cloud Native
Cloud Native Technologies
Cloud Platform
Cloud Platforms
Cloud Platforms Cloud Platforms
Cloud Technology
Data
Data Analysis
Data Analytics
Data Architecture
Data Build Tool
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Management
Data Modeling
Data Operations
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Transforms
Data Warehouse
Database
Databases
DevOps
Devops Tools
DevSecOps
Engineer
Engineering
Engineering Software
ETL
Exasol
Informatica
Kubernetes
Lakehouse
Platform Engineering
Programming Languages
Reporting and Analytics
Security Devops
Software Development
SQL
Job Description
Vylor is hiring a full-time Data Engineer II based onsite in Des Moines, IA (commuting to Johnston, IA Pioneer Hi-Bred/Vylor office). In this role, you will design and deliver scalable lakehouse and analytics data solutions spanning Seeds and Biotech domains, along with support for an enterprise data platform and the pipelines that power analytics and research.
You will work closely with Analytics, Data Science, R&D, Data Engineering, and other cross-functional partners to build data pipelines, ETL/ELT workflows, data products, integrations, and data-quality and observability frameworks.
What you’ll do
- Design, develop, and maintain Lakehouse data pipelines, APIs, and data products for seed models, biotech research assets, enterprise applications, dashboards, and analytics.
- Build scalable ETL/ELT pipelines using Python, SQL, dbt, Trino, and shell scripting to process and integrate large-scale agricultural data from enterprise systems, laboratory platforms, geographic datasets, drones, satellites, and IoT devices.
- Create and maintain Bronze, Silver, and Gold Layer data assets by integrating sources such as Exasol, ClickHouse, MSSQL Server, and other enterprise platforms, including transformation, validation, reconciliation, and quality-control steps.
- Design Lakehouse datasets and transformation pipelines supporting seed, biotech, pathology, and disease analytics, including integration of laboratory, geographic, and historical research data.
- Develop and optimize incremental and full-refresh dbt models and complex Trino SQL queries to improve performance, scalability, and maintainability of Seeds and Biotech data products.
- Modernize legacy data applications by documenting business rules and source-to-target mappings, migrating ETL logic from legacy and Exasol-based environments to Trino and Lakehouse architectures, and deploying curated datasets to ClickHouse and Gold Layer data products.
- Develop and enhance PRISM and FRIP data integrations using Trino connectors and end-to-end comparison pipelines with reusable validation and reconciliation logic across PRISM, FRIP, Lakehouse, Exasol, and downstream application datasets.
- Build reusable frameworks for logging, error handling, scheduling, automated testing, monitoring, deployment, and data-quality validation, and troubleshoot pipeline and application integration issues.
- Create analytics-ready datasets that support statistical analysis, predictive modeling, regression, classification, clustering, dashboards, and business-intelligence reporting.
Qualifications
- Master’s degree (or equivalent) in Computer Science, Statistics, or a related field, plus 2 years related experience.
- 24 months designing and developing scalable ETL/ELT pipelines using Python, SQL, and shell scripting to extract, transform, validate, reconcile, and integrate structured, semi-structured, and unstructured data for enterprise R&D applications, analytical models, dashboards, and reporting systems.
- 24 months developing ETL design mappings, source-to-target transformation rules, data-reconciliation processes, data-quality controls, and complex SQL transformations to ensure accuracy and consistency across source and target platforms.
- 24 months developing and optimizing complex Trino SQL queries and integrations among Lakehouse, Exasol, ClickHouse, MSSQL Server, and other database platforms to deliver curated, analytics-ready datasets for R&D models and enterprise applications including phenomic pillar, dashboards, and reporting.
- 24 months developing reusable Python- and Linux shell-based pipeline reliability and observability frameworks for job scheduling, structured logging, exception handling, automated testing, monitoring, data-quality validation, cross-platform reconciliation, and application integration, along with using version-control, containerization, and automated deployment tools for production data products.
- 24 months analyzing and modernizing legacy data applications by documenting business rules and data mappings, migrating ETL logic to cloud-based or Lakehouse architecture, deploying curated data products, and using Git and Docker for source control, automated deployment, and containerization.
Technologies
- Lakehouse, Python, SQL, dbt, Trino, shell scripting
- Exasol, ClickHouse, MSSQL Server
- Kubernetes, Docker, Git, Linux
Benefits
- Health benefits for you and your family on your first day of employment
- Four weeks of paid time off and two weeks of well-being pay per year, plus paid holidays
- Excellent parental leave which includes a minimum of 16 weeks for mother and father
- Competitive retirement savings plan and tuition reimbursement program
- Numerous development opportunities to build your skills
- Opportunity to contribute to making the world a better place
- Learn more about the total rewards package: Vylor Benefits
Telecommute benefit
The position includes a telecommute benefit within commuting distance to the Johnston, IA Pioneer Hi-Bred/Vylor office, as directed.