DeveloperJobs.io
← Back to all jobs

Job Description

SkillStart is hiring a Senior Data Engineer for a hybrid role based in New York, NY. In this position, you will architect, build, and optimize batch and streaming data pipelines for a Capital Markets post-trade ecosystem, with an emphasis on financial data modeling, ETL/ELT workflows, and data governance requirements.

You will work across real-time event streaming, cloud data warehousing, and end-to-end operational monitoring, supporting reconciliation across reporting, risk, finance, and settlement platforms.

Responsibilities

  • Design, develop, and maintain high-throughput batch and streaming pipelines (ETL/ELT workflows) using Python, SQL, Spark, and streaming platforms.
  • Process and integrate post-trade data flows covering trade allocations, confirmations, settlement, risk management, finance, and regulatory reporting.
  • Design and optimize relational data models, including database schemas, security master structures, reference data models, and time-series financial datasets.
  • Implement and manage scalable data repositories across Data Warehouses, Data Lakes, and Data Lakehouses using cloud platforms, preferably AWS with Snowflake, Redshift, and Databricks.
  • Establish data quality rules with data lineage, automated validation checks, and reconciliation across complex data flows.
  • Integrate real-time messaging architectures, including Kafka, RabbitMQ, and IBM MQ, to ingest and distribute high-volume transactional trade data.
  • Implement data security, privacy controls, and regulatory compliance standards across ingestion and storage layers.
  • Use monitoring and observability tools such as Datadog, Prometheus, and Grafana to maintain pipeline health, support SLA compliance, and enable fast incident response.

Requirements

  • Strong understanding of post-trade data flows including allocations, confirmations, settlement, risk, finance, and regulatory reporting.
  • In-depth knowledge of securities data modeling and financial data structures, including hands-on experience with reference data, security master data, and transactional trade data.
  • Strong SQL skills, with experience designing, tuning, and optimizing relational data models and database schemas.
  • Proficiency in Python, including data processing libraries such as Pandas and distributed frameworks such as PySpark.
  • Operational understanding of Data Warehouse, Data Lake, and Data Lakehouse architectures and best practices.
  • Hands-on experience with PostgreSQL, SQL Server, Snowflake, Redshift, Databricks, or similar platforms on cloud infrastructure, preferably AWS.
  • Experience with stream and batch processing technologies such as Kafka, Spark, Flink, or similar platforms.
  • Proven track record building robust batch and streaming ETL/ELT pipelines using Python and SQL.
  • Strong understanding of data quality assurance, data lineage tracking, data validation frameworks, and automated reconciliation processes.
  • Experience with messaging and event-driven platforms such as Kafka, RabbitMQ, and IBM MQ.
  • Clear understanding of data security principles, data privacy guidelines, and financial regulatory compliance requirements.
  • Familiarity with observability tools including Datadog, Prometheus, and Grafana.

Technologies

  • Python, SQL, Spark, Pandas, PySpark
  • AWS, Snowflake, Redshift, Databricks
  • PostgreSQL, SQL Server
  • Kafka, RabbitMQ, IBM MQ, Flink
  • Datadog, Prometheus, Grafana

Preferred & Bonus Qualifications

  • Understanding of time-series data management and financial market data is a strong plus.
  • Exposure to data science, machine learning, and AI/GenAI data preparation concepts is a plus.

Compensation

USD 122,000 - 148,000 per year

Work Location

Hybrid remote in New York, NY 10114

Similar Jobs