Streaming Data

Streaming Data Developer Jobs

2 jobs found

Today

Data Engineer

JANSON Fairfax, VA

Fairfax, VA

Today

$135k - $135k/yr Full time Big Data Bigdata Data

Apply Now

Job Description

JANSON is seeking a Data Engineer (Engineer IV) to support a U.S. Department of Defense (DoD) initiative in Kaiserslautern, Germany. This full-time, in-person role focuses on building and managing data pipelines and data platforms, including warehouses and lakes, while supporting integration, validation, metadata, and compliance and security requirements.

What you’ll do

Extract, transform, and load (ETL) data from multiple sources, including DoD Advana and Army Vantage, into data warehouses or data lakes.
Clean, normalize, and enrich data during transformation.
Configure automated workflows to move data between systems using Apache Airflow, AWS Glue, or Azure Data Factory.
Optimize pipelines for scalability and reliability.
Implement real-time data processing and streaming pipelines using Apache Kafka, Apache Flink, or Spark Streaming.
Support relational and NoSQL design by implementing databases such as MySQL, PostgreSQL, MongoDB, or Cassandra based on project needs, including schema design and indexing.
Build and maintain data warehouses on platforms such as Snowflake, Amazon Redshift, or Google BigQuery, optimizing storage and query performance for analytics workloads.
Support management of data lakes and integration in line with organizational policies, interpreting policies to sustain long-term data lake integration.
Integrate data from APIs, databases, flat files, and third-party systems, handling formats including JSON, XML, CSV, and Parquet.
Perform data migrations between systems during upgrades or transitions, such as moving from on-premises to cloud.
Configure APIs to expose data for consumption by other systems and applications.

Data quality, governance, and security

Manage data quality and governance at the 21st TSC level in accordance with USAEUR-AF policies and guidance.
Implement checks to ensure accuracy, completeness, and consistency, and identify and resolve data anomalies or errors.
Document data lineage, definitions, and transformations for traceability.
Implement compliance and security measures, including support for regulations such as GDPR, HIPAA, and CCPA.
Apply security controls including encryption, access control, and auditing.
Coordinate with internal teams within the Group to mitigate data quality issues affecting analyses and tool configuration.
Work with PEOs and PMs of Systems of Record to understand and correct data issues feeding multiple data platforms.

Key skills and qualifications

Eighteen (18) years of recent specialized experience OR meet one of the listed education + experience combinations (e.g., 10 years with a Bachelor’s degree; 14 years with an Associate’s degree; 7 years with a Master’s degree), and/or a major IT certification with experience as specified.
IA Technical Level II certification or higher and IAT Level II or higher (required).
IA Technical certifications (required/acceptable list): CCNA-Security, CompTIA CySA+, GICSP, GSEC, Security+ CE, CND, SSCP.
Secret clearance required/preferred: Secret (Preferred).
Bachelor’s degree in computer science, information management, or a related discipline.
Recent specialized: 10 years (Required).

Location, job type, and pay

Location: Kaiserslautern, Germany (in person)
Job type: Full-time
Clearance: Secret
Salary: USD 135,000 per year

Technologies you may work with

Apache Airflow, AWS Glue, Azure Data Factory
Apache Kafka, Apache Flink, Spark Streaming
MySQL, PostgreSQL, MongoDB, Cassandra
Snowflake, Amazon Redshift, Google BigQuery
APIs, JSON, XML, CSV, Parquet
GDPR, HIPAA, CCPA; encryption, access control, auditing

View full job page →

2d ago

GCP BigQuery Data Engineer

Virtues Irving, TX

Irving, TX

2d ago

$100k - $110k/yr Full time Analytics Big Data Bigdata

Apply Now

Job Description

Virtues is hiring a GCP BigQuery Data Engineer for an onsite role in Irving, TX. This position focuses on building cloud-native, scalable data platforms and real-time ingestion pipelines on GCP, with an event-driven architecture designed for near real-time analytics. If you enjoy turning streaming data into dependable analytical datasets, this role combines BigQuery enterprise warehousing with Apache NiFi and Apache Kafka pipelines.

Compensation: $100,000 - $110,000 per year. Experience: 7+ years.

What you’ll be responsible for

Design, develop, and optimize enterprise data warehouse solutions using Google BigQuery.
Design, develop, and maintain real-time data ingestion pipelines with Apache NiFi to capture, transform, and route streaming data into Apache Kafka topics.
Build scalable event-driven streaming architectures using Apache NiFi, Apache Kafka, and Google BigQuery for high-throughput, fault-tolerant, low-latency analytics.
Configure and optimize Apache NiFi processors for extraction, transformation, routing, filtering, schema validation, error handling, retry mechanisms, and reliable Kafka delivery.
Develop streaming ingestion solutions so Google BigQuery can consume Kafka event streams and transform near real-time data into analytical tables and enterprise reporting datasets.
Design data models and implement ETL/ELT processes to move data from raw to curated and published layers.
Design, create, and manage large-scale BigQuery datasets, including temporary/permanent and internal/external tables.
Optimize BigQuery performance and spending using query tuning, partitioning, clustering, and cost optimization techniques.
Monitor, troubleshoot, and optimize streaming workloads by tuning NiFi flows, Kafka topics/partitions, and BigQuery streaming ingestion to maintain high availability, data integrity, minimal latency, and cost-efficient processing.
Build scalable Data Lake frameworks and ingestion pipelines using cloud-native GCP technologies.
Develop reporting and visualization solutions using Looker, Looker Studio (Data Studio), Connected Sheets, and other BigQuery reporting tools.
Collaborate with data architects, modelers, developers, DevOps engineers, project managers, and business stakeholders to deliver scalable enterprise analytics and continuous platform improvements.

What you’ll need

7+ years of experience in Data Engineering, Data Warehousing, or Big Data platforms.
Must-have skills: Apache NiFi, Apache Kafka.
Strong hands-on experience with Google Cloud Platform (GCP), including BigQuery, Cloud Dataflow, Pub/Sub, and Google Cloud Storage (GCS).
Hands-on experience designing and supporting real-time streaming pipelines using Apache NiFi, Apache Kafka, and Google BigQuery.
Experience using BigQuery Console/Query Editor for data management, performance tuning, and SQL development.
Strong experience designing scalable data models, ETL/ELT pipelines, Data Lake architectures, and enterprise data warehouse solutions.
Experience with batch and streaming ingestion using GCP services.
Thorough understanding of BigQuery cost structure (storage, ingestion, and query costs) with experience implementing optimization strategies such as query optimization, partitioning, and clustering.
Experience managing large-scale datasets, including temporary/permanent and internal/external BigQuery tables.
Experience developing reporting and visualization solutions using Looker, Looker Studio (Data Studio), and Connected Sheets.
Excellent verbal and written communication skills and the ability to collaborate effectively with technical teams, business stakeholders, senior management, and executive leadership.
Experience working in Agile environments and collaborating cross-functionally to deliver enterprise data engineering solutions.

Technologies you’ll work with

Google BigQuery, Apache NiFi, Apache Kafka, Google Cloud Dataflow, Google Pub/Sub, SQL, ETL/ELT, Google Cloud Storage (GCS), Looker, Looker Studio (Data Studio), Connected Sheets, CI/CD, DevOps, Agile/Scrum, Python, Spark, Data Modeling, Data Warehousing, Data Lakes, Batch & Streaming Pipelines.

Preferred qualifications

Google Cloud certifications.
Experience delivering enterprise-scale analytics and cloud data solutions.

View full job page →

Streaming Data Developer Jobs

Data Engineer

GCP BigQuery Data Engineer

Get Streaming Data Developer Jobs alerts