Senior Data Engineer
Job Description
Senior Data Engineer role supporting end-to-end batch and streaming pipelines for financial and market data on a full-time, onsite project in Dallas.
Responsibilities
- Own data pipelines end-to-end, from ingestion and processing through validation and publication
- Build and maintain batch and streaming data pipelines
- Implement reliable processing patterns for idempotency, backfills, late-arriving data, and partitioning
- Create automated data quality checks for completeness, timeliness, anomalies, and consistency across data sources
- Monitor production pipelines and configure alerting for timely issue detection
- Investigate and resolve data quality and pipeline reliability incidents
- Optimize pipeline performance and resource usage
- Onboard new datasets and data providers
- Handle large volumes of financial and market-related data
- Communicate with data consumers about incidents, data issues, changes, and availability
- Collaborate on APIs and internal libraries that enable data access and distribution
- Improve reliability, maintainability, and observability of the data platform
Requirements
- 5+ years of professional experience in Data Engineering
- Strong hands-on experience with Python
- Strong knowledge of SQL
- Strong hands-on experience with Apache Spark
- Experience with Hadoop / Hive or modern lakehouse formats such as Delta Lake or Apache Iceberg
- Experience building and maintaining batch and streaming pipelines
- Strong understanding of idempotent data processing
- Strong understanding of backfills
- Strong understanding of late-arriving data
- Strong understanding of partitioning strategies
- Experience implementing automated data quality checks
- Experience with monitoring and alerting for production data pipelines
- Experience working with large volumes of data
- Strong understanding of reliable data ingestion and publication processes
- Strong analytical and troubleshooting skills
- High attention to detail and data correctness
- Strong ownership and ability to independently manage data engineering tasks
- Strong communication skills with data consumers and technical stakeholders
- Valid US work authorization
- Ability to work on-site full-time in Dallas, TX (McKinney, TX location noted)
Technologies
- Python, SQL
- Apache Spark
- Hadoop, Hive
- Delta Lake, Apache Iceberg
- Kafka, Airflow
- Snowflake, Databricks
- Java, Scala
Nice to Have
- Experience working with market data, including equities, macroeconomic data, or market data vendors
- Experience with Kafka or other streaming technologies
- Experience with Airflow or another workflow orchestration platform
- Experience with Snowflake or Databricks
- Experience developing APIs or reusable libraries for data distribution
- Experience with Java or Scala
- Previous experience in Investment Management / Financial Services