Data Engineer

Infinitive Inc is hiring a Data Engineer to help design and scale next-generation, event-driven data platforms. The work centers on building reliable streaming pipelines with Apache Kafka and orchestrating long-running distributed workflows with Temporal, alongside batch and near-real-time ETL/ELT.

This onsite role is based in McLean, VA, and you will collaborate across engineering teams to standardize data contracts, keep schemas validated, and deliver production-grade data model performance across modern cloud data platforms.

Responsibilities

• Architect, deploy, and maintain high-volume distributed data streams using Apache Kafka, including producers, consumers, Kafka Connect, and Schema Registry.
• Define and enforce schema design standards, versioning strategies, and automated schema validation using formats such as Apache Avro, Protocol Buffers, and JSON Schema to maintain strict data contracts across microservices, streaming consumers, and lakehouse storage.
• Design and implement durable execution workflows with Temporal to coordinate long-running distributed pipelines, apply compensation via the Saga pattern, and manage cross-system ETL tasks.
• Build end-to-end batch and near-real-time pipelines using Python, SQL, and Apache Spark / PySpark.
• Design and optimize analytical data models, including dimensional/star schema patterns, in cloud data warehouses and lakehouses such as Snowflake, BigQuery, Databricks, and Redshift.
• Implement automated testing, continuous schema validation, data drift detection, and observability across both streaming and batch workflows.
• Partner with software engineers, machine learning engineers, and analysts to define standard schema definitions, data contracts, and production-grade CI/CD release patterns.

Requirements

• 4+ years of professional experience in data engineering, backend distributed systems, or software engineering.
• Hands-on experience with Temporal (or Cadence), including durable workflows, activities, retries, signals, queries, and long-running distributed orchestration.
• Deep expertise with Apache Kafka, including message partitioning, consumer groups, offset management, and topic design.
• Demonstrated proficiency with schema definition frameworks such as Apache Avro, Protocol Buffers/gRPC, or JSON Schema.
• Practical experience managing schema evolution with compatibility modes such as backward, forward, and full, using schema registries like Confluent Schema Registry or AWS Glue Schema Registry.
• Experience enforcing data validation rules, contract testing, and data quality checks using tools such as Great Expectations, Pandera, Pydantic, and dbt tests.
• Strong programming proficiency in Python (with Go or Java as a plus), including clean code, design patterns, and unit/integration testing standards.
• Hands-on development with Apache Spark via PySpark and Spark SQL for large-scale datasets.
• Strong experience with relational databases, dimensional data modeling, and query performance tuning.

Technologies

• Apache Kafka, Kafka Connect, Schema Registry
• Temporal, Cadence
• Apache Avro, Protocol Buffers, gRPC, JSON Schema
• Confluent Schema Registry, AWS Glue Schema Registry
• Great Expectations, Pandera, Pydantic, dbt
• Python, SQL
• Apache Spark, PySpark, Spark SQL
• Snowflake, BigQuery, Databricks, Redshift

Compensation: A reasonable estimate of the current range for this role in the U.S. is $90,000 - $154,00.00 per year.