Senior Data Engineer (Customer Data Products)

NBME is seeking a Senior Data Engineer to modernize and optimize its AWS-based data platform for analytics, AI/ML, and business decision-making. The role focuses on building data lakes and scalable pipelines, working closely with technical stakeholders, and applying AI-assisted engineering practices.

Role Responsibilities

• Code, test, deploy, orchestrate, monitor, document, and troubleshoot cloud-based data engineering processes, including feature stores and vector databases, in alignment with best practices and security standards across the development lifecycle.
• Partner with data scientists, AI researchers, data and enterprise architects, and business stakeholders to identify, extract, clean, and format structured and unstructured data for AI/ML model training, fine-tuning, and feature extraction.
• Lead evaluation, research, and experimentation efforts using batch and streaming data technologies, LLM data preparation frameworks, and MLOps tools.
• Serve as a technical lead to demonstrate capabilities of emerging AI and data technologies to support broader adoption of modern data techniques across the organization.
• Contribute to the definition and refinement of processes and procedures for the data engineering practice.
• Educate and develop ETL developers on cloud-based data engineering initiatives to support transition to data engineering practices.
• Ensure the integrity and accuracy of corporate data, with emphasis on data security.
• Maintain high data quality for Data Services, Analytics, and Master Data Management.
• Coordinate technical solutions and take ownership of designs, development, testing, and delivery.
• Build automated, scalable, test-driven data pipelines.
• Apply software development practices including Git-based version control, CI/CD, and release management to enhance AWS CI/CD pipelines.
• Collaborate with Data Engineers, DevOps engineers, and architects to identify improvement opportunities for DataOps tools and frameworks.

Required Qualifications

• Bachelor's Degree
• At least 7 years of experience in application development (internship experience does not apply)
• At least 4 years of experience in big data technologies
• At least 4 years of experience with cloud computing using AWS
• 4+ years of application development experience including Python, SQL, Scala, or Java
• 4+ years of experience with distributed data and computing tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, MySQL, etc.)
• 4+ years of experience working on real-time data and streaming applications
• 4+ years of experience implementing NoSQL systems (MongoDB, Cassandra)
• 4+ years of data warehousing experience (Amazon Redshift)
• 6+ years of experience with UNIX/Linux, including basic commands and shell scripting
• 7+ years of experience with Agile engineering practices
• 7+ years of experience with SQL optimization
• 4+ years of experience with PySpark
• 3+ years of experience with process orchestration including AirFlow, KubeFlow, AWS Step Functions, or Luigi

Preferred Qualifications

• Proven experience implementing Generative AI, LLM data preparation pipelines, and vector databases such as Pinecone, Milvus, or pgvector
• Strong experience building and maintaining Feature Stores for machine learning models
• Experience building highly scalable, secure, production-ready APIs and Data-as-a-Service (DaaS) platforms
• AWS Certified Data Engineer or AWS Certified Machine Learning – Specialty
• 3+ years of experience with Machine Learning
• Experience building a Data-as-a-service platform
• Experience building APIs

Technology Stack

• AWS, Python, SQL, Scala, Java
• MapReduce, Hadoop, Hive, EMR, Kafka, Spark, MySQL
• MongoDB, Cassandra, Redshift
• UNIX/Linux, shell scripting
• Git, CI/CD, AirFlow, KubeFlow, AWS Step Functions, Luigi
• PySpark
• Feature stores, vector databases (including Pinecone, Milvus, pgvector)
• LLM data preparation frameworks, MLOps tools
• DataOps tools, APIs, Data-as-a-Service (DaaS), Amazon Redshift

Compensation

USD 127,337 - 160,000 per year.

Benefits

• Healthcare, Dental, Prescription, and Vision plans
• 401(k) with company match
• Tuition Reimbursement Plan
• Commuter Benefit: Public Transit or Parking options
• Remote Friendly Workplace

Location and Hybrid Work Details

• Location: Philadelphia, PA (hybrid)
• The role is designated as hybrid and may require occasional onsite work for team collaboration or meetings.
• NBME is prioritizing applicants within a 50-mile radius from the Philadelphia office.
• Onsite interviews and onboarding at the Philadelphia office may be required; advance notice will be provided when applicable.