DeveloperJobs.io
← Back to all jobs

Job Description

Senior Data Engineer role supporting end-to-end batch and streaming pipelines for financial and market data on a full-time, onsite project in Dallas.

Responsibilities

  • Own data pipelines end-to-end, from ingestion and processing through validation and publication
  • Build and maintain batch and streaming data pipelines
  • Implement reliable processing patterns for idempotency, backfills, late-arriving data, and partitioning
  • Create automated data quality checks for completeness, timeliness, anomalies, and consistency across data sources
  • Monitor production pipelines and configure alerting for timely issue detection
  • Investigate and resolve data quality and pipeline reliability incidents
  • Optimize pipeline performance and resource usage
  • Onboard new datasets and data providers
  • Handle large volumes of financial and market-related data
  • Communicate with data consumers about incidents, data issues, changes, and availability
  • Collaborate on APIs and internal libraries that enable data access and distribution
  • Improve reliability, maintainability, and observability of the data platform

Requirements

  • 5+ years of professional experience in Data Engineering
  • Strong hands-on experience with Python
  • Strong knowledge of SQL
  • Strong hands-on experience with Apache Spark
  • Experience with Hadoop / Hive or modern lakehouse formats such as Delta Lake or Apache Iceberg
  • Experience building and maintaining batch and streaming pipelines
  • Strong understanding of idempotent data processing
  • Strong understanding of backfills
  • Strong understanding of late-arriving data
  • Strong understanding of partitioning strategies
  • Experience implementing automated data quality checks
  • Experience with monitoring and alerting for production data pipelines
  • Experience working with large volumes of data
  • Strong understanding of reliable data ingestion and publication processes
  • Strong analytical and troubleshooting skills
  • High attention to detail and data correctness
  • Strong ownership and ability to independently manage data engineering tasks
  • Strong communication skills with data consumers and technical stakeholders
  • Valid US work authorization
  • Ability to work on-site full-time in Dallas, TX (McKinney, TX location noted)

Technologies

  • Python, SQL
  • Apache Spark
  • Hadoop, Hive
  • Delta Lake, Apache Iceberg
  • Kafka, Airflow
  • Snowflake, Databricks
  • Java, Scala

Nice to Have

  • Experience working with market data, including equities, macroeconomic data, or market data vendors
  • Experience with Kafka or other streaming technologies
  • Experience with Airflow or another workflow orchestration platform
  • Experience with Snowflake or Databricks
  • Experience developing APIs or reusable libraries for data distribution
  • Experience with Java or Scala
  • Previous experience in Investment Management / Financial Services

Similar Jobs