S
Senior Data Engineer
Backend Developer
Senior
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Data Platform
Cloud Data Warehouse
Cloud Data Warehouse
Cloud Operations
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Lakehouse
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Warehouse
Data Warehousing
Database
Databases
Databricks
Engineer
ETL
Event Driven Architecture
Informatica
Information Technology (IT)
Kafka
Programming Language
Programming Languages
Pyspark
Snowflake
Software Development
Spark
SQL
Stream Processing
Streaming Data
Job Description
SkillStart is hiring a Senior Data Engineer for a hybrid role based in New York, NY. In this position, you will architect, build, and optimize batch and streaming data pipelines for a Capital Markets post-trade ecosystem, with an emphasis on financial data modeling, ETL/ELT workflows, and data governance requirements.
You will work across real-time event streaming, cloud data warehousing, and end-to-end operational monitoring, supporting reconciliation across reporting, risk, finance, and settlement platforms.
Responsibilities
- Design, develop, and maintain high-throughput batch and streaming pipelines (ETL/ELT workflows) using Python, SQL, Spark, and streaming platforms.
- Process and integrate post-trade data flows covering trade allocations, confirmations, settlement, risk management, finance, and regulatory reporting.
- Design and optimize relational data models, including database schemas, security master structures, reference data models, and time-series financial datasets.
- Implement and manage scalable data repositories across Data Warehouses, Data Lakes, and Data Lakehouses using cloud platforms, preferably AWS with Snowflake, Redshift, and Databricks.
- Establish data quality rules with data lineage, automated validation checks, and reconciliation across complex data flows.
- Integrate real-time messaging architectures, including Kafka, RabbitMQ, and IBM MQ, to ingest and distribute high-volume transactional trade data.
- Implement data security, privacy controls, and regulatory compliance standards across ingestion and storage layers.
- Use monitoring and observability tools such as Datadog, Prometheus, and Grafana to maintain pipeline health, support SLA compliance, and enable fast incident response.
Requirements
- Strong understanding of post-trade data flows including allocations, confirmations, settlement, risk, finance, and regulatory reporting.
- In-depth knowledge of securities data modeling and financial data structures, including hands-on experience with reference data, security master data, and transactional trade data.
- Strong SQL skills, with experience designing, tuning, and optimizing relational data models and database schemas.
- Proficiency in Python, including data processing libraries such as Pandas and distributed frameworks such as PySpark.
- Operational understanding of Data Warehouse, Data Lake, and Data Lakehouse architectures and best practices.
- Hands-on experience with PostgreSQL, SQL Server, Snowflake, Redshift, Databricks, or similar platforms on cloud infrastructure, preferably AWS.
- Experience with stream and batch processing technologies such as Kafka, Spark, Flink, or similar platforms.
- Proven track record building robust batch and streaming ETL/ELT pipelines using Python and SQL.
- Strong understanding of data quality assurance, data lineage tracking, data validation frameworks, and automated reconciliation processes.
- Experience with messaging and event-driven platforms such as Kafka, RabbitMQ, and IBM MQ.
- Clear understanding of data security principles, data privacy guidelines, and financial regulatory compliance requirements.
- Familiarity with observability tools including Datadog, Prometheus, and Grafana.
Technologies
- Python, SQL, Spark, Pandas, PySpark
- AWS, Snowflake, Redshift, Databricks
- PostgreSQL, SQL Server
- Kafka, RabbitMQ, IBM MQ, Flink
- Datadog, Prometheus, Grafana
Preferred & Bonus Qualifications
- Understanding of time-series data management and financial market data is a strong plus.
- Exposure to data science, machine learning, and AI/GenAI data preparation concepts is a plus.
Compensation
USD 122,000 - 148,000 per year
Work Location
Hybrid remote in New York, NY 10114