Virtues is hiring a GCP BigQuery Data Engineer for an onsite role in Irving, TX. This position focuses on building cloud-native, scalable data platforms and real-time ingestion pipelines on GCP, with an event-driven architecture designed for near real-time analytics. If you enjoy turning streaming data into dependable analytical datasets, this role combines BigQuery enterprise warehousing with Apache NiFi and Apache Kafka pipelines.
Compensation: $100,000 - $110,000 per year. Experience: 7+ years.
What you’ll be responsible for
- Design, develop, and optimize enterprise data warehouse solutions using Google BigQuery.
- Design, develop, and maintain real-time data ingestion pipelines with Apache NiFi to capture, transform, and route streaming data into Apache Kafka topics.
- Build scalable event-driven streaming architectures using Apache NiFi, Apache Kafka, and Google BigQuery for high-throughput, fault-tolerant, low-latency analytics.
- Configure and optimize Apache NiFi processors for extraction, transformation, routing, filtering, schema validation, error handling, retry mechanisms, and reliable Kafka delivery.
- Develop streaming ingestion solutions so Google BigQuery can consume Kafka event streams and transform near real-time data into analytical tables and enterprise reporting datasets.
- Design data models and implement ETL/ELT processes to move data from raw to curated and published layers.
- Design, create, and manage large-scale BigQuery datasets, including temporary/permanent and internal/external tables.
- Optimize BigQuery performance and spending using query tuning, partitioning, clustering, and cost optimization techniques.
- Monitor, troubleshoot, and optimize streaming workloads by tuning NiFi flows, Kafka topics/partitions, and BigQuery streaming ingestion to maintain high availability, data integrity, minimal latency, and cost-efficient processing.
- Build scalable Data Lake frameworks and ingestion pipelines using cloud-native GCP technologies.
- Develop reporting and visualization solutions using Looker, Looker Studio (Data Studio), Connected Sheets, and other BigQuery reporting tools.
- Collaborate with data architects, modelers, developers, DevOps engineers, project managers, and business stakeholders to deliver scalable enterprise analytics and continuous platform improvements.
What you’ll need
- 7+ years of experience in Data Engineering, Data Warehousing, or Big Data platforms.
- Must-have skills: Apache NiFi, Apache Kafka.
- Strong hands-on experience with Google Cloud Platform (GCP), including BigQuery, Cloud Dataflow, Pub/Sub, and Google Cloud Storage (GCS).
- Hands-on experience designing and supporting real-time streaming pipelines using Apache NiFi, Apache Kafka, and Google BigQuery.
- Experience using BigQuery Console/Query Editor for data management, performance tuning, and SQL development.
- Strong experience designing scalable data models, ETL/ELT pipelines, Data Lake architectures, and enterprise data warehouse solutions.
- Experience with batch and streaming ingestion using GCP services.
- Thorough understanding of BigQuery cost structure (storage, ingestion, and query costs) with experience implementing optimization strategies such as query optimization, partitioning, and clustering.
- Experience managing large-scale datasets, including temporary/permanent and internal/external BigQuery tables.
- Experience developing reporting and visualization solutions using Looker, Looker Studio (Data Studio), and Connected Sheets.
- Excellent verbal and written communication skills and the ability to collaborate effectively with technical teams, business stakeholders, senior management, and executive leadership.
- Experience working in Agile environments and collaborating cross-functionally to deliver enterprise data engineering solutions.
Technologies you’ll work with
Google BigQuery, Apache NiFi, Apache Kafka, Google Cloud Dataflow, Google Pub/Sub, SQL, ETL/ELT, Google Cloud Storage (GCS), Looker, Looker Studio (Data Studio), Connected Sheets, CI/CD, DevOps, Agile/Scrum, Python, Spark, Data Modeling, Data Warehousing, Data Lakes, Batch & Streaming Pipelines.
Preferred qualifications
- Google Cloud certifications.
- Experience delivering enterprise-scale analytics and cloud data solutions.
