Senior Data Engineer
Job Description
Senior Data Engineer role in Boston, MA supporting Colorado’s Statewide Longitudinal Data System (SLDS/COdata).
Responsibilities
- Design, develop, test, and deploy automated ETL/ELT pipelines using Snowflake and Informatica
- Integrate data from state agencies, educational institutions, and third-party source systems
- Build complex SQL queries, transformations, stored procedures, and data processing workflows
- Create and maintain source-to-target mappings and data integration processes
- Ingest and transform structured data into standardized longitudinal data models
- Develop and maintain Informatica workflows for ingestion, cleansing, and transformation automation
- Develop and maintain Snowflake objects including tables, views, schemas, and transformation components
- Optimize Snowflake queries, warehouse utilization, and end-to-end pipeline performance
- Monitor scheduled workloads, resolve performance bottlenecks, and troubleshoot failed jobs and ingestion issues
- Implement scalable, cost-efficient data processing solutions
- Maintain RAW, transformation, and analytics data layers
- Implement automated data quality checks:
- Structural and format validation
- Data completeness and duplicate detection
- Referential integrity and business logic validation
- Perform data cleansing, reconciliation, and exception handling
- Implement record linkage and identity matching across agency datasets
- Maintain data lineage, metadata, audit trails, and governance standards
- Ensure compliance with FERPA, COPPA, ADA, and applicable state data privacy regulations
- Implement data suppression and de-identification requirements
- Reconcile source and target record counts; investigate discrepancies and support reloads for late/corrected/malformed files
- Maintain operational runbooks, incident logs, and pipeline documentation
- Transition production pipeline responsibilities from a third-party implementation partner and gradually assume independent ownership
- Develop and maintain CI/CD pipelines using Azure DevOps
- Use Git for source control, branching, pull requests, and code reviews
- Implement Infrastructure as Code (IaC) using Terraform
- Promote code across Development, Test, UAT, and Production environments
- Develop and execute manual and automated test cases using Azure Test Plans
- Participate in peer reviews, defect resolution, and release documentation
- Collaborate with Data Architects on data models and architectural standards
- Work with Data Scientists and Visualization Designers to prepare datasets for Tableau dashboards and analytics
- Coordinate with Operations Engineers on Snowflake warehouse management, permissions, and deployment processes
- Participate in knowledge-transfer sessions with third-party implementation teams
Project-specific responsibilities
- Develop and maintain data ingestion pipelines for Colorado Department of Higher Education (CDHE) course-level data
- Support conditional ingestion of postsecondary demographic, Pell Grant eligibility, and disability-related data
- Integrate PSAT/SAT assessment data and perform student identity matching
- Support existing data feeds, including NSC, K-3, CDEC, and Phase 1 integrations
- Participate in knowledge transfer and operational transition January through April 2027
- Support analytics-layer stabilization, data reliability improvements, and performance tuning
- Develop de-identified and privacy-compliant data extracts for participating higher education institutions
Requirements
- Minimum 5 years experience developing, managing, and optimizing ETL/ELT pipelines and data transformations within Snowflake
- Minimum 1 year hands-on experience with Informatica Cloud Data Governance and Catalog (CDGC), including metadata management, data profiling, cataloging, and data quality validation
- Minimum 1 year experience with Infrastructure as Code (IaC), including provisioning and managing cloud infrastructure
- Strong experience developing and maintaining automated CI/CD pipelines for build, testing, and deployment
- Strong proficiency with version control, branching strategies, pull requests, merge management, and code reviews
- Strong SQL development and data transformation skills
- Experience designing and implementing complex data ingestion and integration pipelines
- Experience with data quality, data validation, cleansing, and reconciliation processes
- Ability to troubleshoot production data pipelines and manage operational support activities
Preferred qualifications
- Experience working with state government, public-sector, or education data systems
- Familiarity with Statewide Longitudinal Data Systems (SLDS) or longitudinal student and workforce data
- Experience integrating data from multiple government agencies and external partner systems
- Experience with record linkage, entity resolution, and deterministic or probabilistic identity matching
- Knowledge of FERPA, COPPA, student data privacy, and data suppression standards
- Experience with Informatica ETL workflows and metadata-driven data integration
- Experience with Azure Test Plans and structured software testing processes
- Familiarity with Tableau data integration and analytics-layer development
- Experience working with education and workforce datasets from organizations such as CDE, CDHE, CDLE, CDEC, and National Student Clearinghouse
Technologies
- Snowflake
- Informatica
- Informatica Cloud Data Governance and Catalog (CDGC)
- Terraform
- Azure DevOps
- Git
- Azure Test Plans
- SQL
- Tableau
Expected deliverables
- Production-ready Informatica ETL workflows and Snowflake transformation pipelines
- Automated data cleansing and validation modules
- Identity matching and record-linkage routines
- Snowflake analytics-layer tables, views, and data models
- Data mapping, lineage, and dependency documentation
- Operational runbooks and incident management procedures
- Azure Test Plans, test execution records, and defect logs
- Release change logs and production feed reconciliation reports
Location and compensation
- Location: Boston, MA (onsite)
- Salary: USD 76 - 76 per hourly
- Experience: Minimum 5 years