DeveloperJobs.io
← Back to all jobs

Job Description

Senior Data Engineer role in Boston, MA supporting Colorado’s Statewide Longitudinal Data System (SLDS/COdata).

Responsibilities

  • Design, develop, test, and deploy automated ETL/ELT pipelines using Snowflake and Informatica
  • Integrate data from state agencies, educational institutions, and third-party source systems
  • Build complex SQL queries, transformations, stored procedures, and data processing workflows
  • Create and maintain source-to-target mappings and data integration processes
  • Ingest and transform structured data into standardized longitudinal data models
  • Develop and maintain Informatica workflows for ingestion, cleansing, and transformation automation
  • Develop and maintain Snowflake objects including tables, views, schemas, and transformation components
  • Optimize Snowflake queries, warehouse utilization, and end-to-end pipeline performance
  • Monitor scheduled workloads, resolve performance bottlenecks, and troubleshoot failed jobs and ingestion issues
  • Implement scalable, cost-efficient data processing solutions
  • Maintain RAW, transformation, and analytics data layers
  • Implement automated data quality checks:
    • Structural and format validation
    • Data completeness and duplicate detection
    • Referential integrity and business logic validation
  • Perform data cleansing, reconciliation, and exception handling
  • Implement record linkage and identity matching across agency datasets
  • Maintain data lineage, metadata, audit trails, and governance standards
  • Ensure compliance with FERPA, COPPA, ADA, and applicable state data privacy regulations
  • Implement data suppression and de-identification requirements
  • Reconcile source and target record counts; investigate discrepancies and support reloads for late/corrected/malformed files
  • Maintain operational runbooks, incident logs, and pipeline documentation
  • Transition production pipeline responsibilities from a third-party implementation partner and gradually assume independent ownership
  • Develop and maintain CI/CD pipelines using Azure DevOps
  • Use Git for source control, branching, pull requests, and code reviews
  • Implement Infrastructure as Code (IaC) using Terraform
  • Promote code across Development, Test, UAT, and Production environments
  • Develop and execute manual and automated test cases using Azure Test Plans
  • Participate in peer reviews, defect resolution, and release documentation
  • Collaborate with Data Architects on data models and architectural standards
  • Work with Data Scientists and Visualization Designers to prepare datasets for Tableau dashboards and analytics
  • Coordinate with Operations Engineers on Snowflake warehouse management, permissions, and deployment processes
  • Participate in knowledge-transfer sessions with third-party implementation teams

Project-specific responsibilities

  • Develop and maintain data ingestion pipelines for Colorado Department of Higher Education (CDHE) course-level data
  • Support conditional ingestion of postsecondary demographic, Pell Grant eligibility, and disability-related data
  • Integrate PSAT/SAT assessment data and perform student identity matching
  • Support existing data feeds, including NSC, K-3, CDEC, and Phase 1 integrations
  • Participate in knowledge transfer and operational transition January through April 2027
  • Support analytics-layer stabilization, data reliability improvements, and performance tuning
  • Develop de-identified and privacy-compliant data extracts for participating higher education institutions

Requirements

  • Minimum 5 years experience developing, managing, and optimizing ETL/ELT pipelines and data transformations within Snowflake
  • Minimum 1 year hands-on experience with Informatica Cloud Data Governance and Catalog (CDGC), including metadata management, data profiling, cataloging, and data quality validation
  • Minimum 1 year experience with Infrastructure as Code (IaC), including provisioning and managing cloud infrastructure
  • Strong experience developing and maintaining automated CI/CD pipelines for build, testing, and deployment
  • Strong proficiency with version control, branching strategies, pull requests, merge management, and code reviews
  • Strong SQL development and data transformation skills
  • Experience designing and implementing complex data ingestion and integration pipelines
  • Experience with data quality, data validation, cleansing, and reconciliation processes
  • Ability to troubleshoot production data pipelines and manage operational support activities

Preferred qualifications

  • Experience working with state government, public-sector, or education data systems
  • Familiarity with Statewide Longitudinal Data Systems (SLDS) or longitudinal student and workforce data
  • Experience integrating data from multiple government agencies and external partner systems
  • Experience with record linkage, entity resolution, and deterministic or probabilistic identity matching
  • Knowledge of FERPA, COPPA, student data privacy, and data suppression standards
  • Experience with Informatica ETL workflows and metadata-driven data integration
  • Experience with Azure Test Plans and structured software testing processes
  • Familiarity with Tableau data integration and analytics-layer development
  • Experience working with education and workforce datasets from organizations such as CDE, CDHE, CDLE, CDEC, and National Student Clearinghouse

Technologies

  • Snowflake
  • Informatica
  • Informatica Cloud Data Governance and Catalog (CDGC)
  • Terraform
  • Azure DevOps
  • Git
  • Azure Test Plans
  • SQL
  • Tableau

Expected deliverables

  • Production-ready Informatica ETL workflows and Snowflake transformation pipelines
  • Automated data cleansing and validation modules
  • Identity matching and record-linkage routines
  • Snowflake analytics-layer tables, views, and data models
  • Data mapping, lineage, and dependency documentation
  • Operational runbooks and incident management procedures
  • Azure Test Plans, test execution records, and defect logs
  • Release change logs and production feed reconciliation reports

Location and compensation

  • Location: Boston, MA (onsite)
  • Salary: USD 76 - 76 per hourly
  • Experience: Minimum 5 years

Similar Jobs