Senior Data Engineer — Knowledge Ingestion
Job Description
Build connectors and an ingestion pipeline that transform raw knowledge content into a governed, permission-tagged corpus delivered to S3 and Amazon Bedrock Knowledge Bases.
Responsibilities
- Develop the iKNOW connector for Oracle B2C Service knowledge APIs using a full inventory extract followed by incremental sync based on modified date.
- Capture article status and answer version during Oracle ingestion.
- Build SharePoint, OneDrive, and Teams connectors using Microsoft Graph:
- Enumerate sites and drives.
- Use delta queries for incremental change.
- Attach sharing and group membership as permission metadata for every ingested item.
- Create the Sitecore connector using Sitecore content APIs for public web content in scope.
- Implement the ingestion pipeline:
- Land source outputs in S3 using a per-source prefix convention.
- Perform document conversion and text extraction.
- Deduplicate documents via content hash.
- Enrich metadata from taxonomy.
- Sync ingested content to Amazon Bedrock Knowledge Bases with chunking parameters defined by the architect.
- Produce inventory extracts for content disposition reviews and counts that feed the end-of-Phase-1 re-baselining.
- Implement retirement handling so items marked retired or deleted at the source are removed from the index on the next sync.
- Handle throttling and backoff for Oracle and Microsoft APIs.
- Ensure idempotent re-runs, plus failure isolation per source so one connector outage does not block the others.
- Instrument each pipeline stage with CloudWatch metrics and write ingestion runbooks.
- Pair with client IT during weeks 7–10 to support running, re-running, and extending the pipelines after transition.
Requirements
- 4+ years building data pipelines in production, including at least 2 years on AWS using Python.
- Hands-on experience with the Microsoft Graph API for SharePoint and Teams content, including delta queries, permissions, and throttling; or equivalent enterprise content platform experience.
- Experience ingesting unstructured content (HTML, Office documents, PDF, chat transcripts) into S3 and preparing it for search or vector indexing.
- Working knowledge of Amazon Bedrock Knowledge Bases data sources and sync behavior, or a comparable vector ingestion path on OpenSearch.
- Infrastructure-as-code experience with Terraform or AWS CDK, plus CI for deploying pipelines into a client-owned account.
- Ability to work from a shared backlog in client repositories while producing documentation as work is completed.
Technologies
- Oracle B2C Service knowledge APIs
- Microsoft Graph
- Amazon Bedrock Knowledge Bases
- S3
- CloudWatch
- Python
- Terraform
- AWS CDK
- OpenSearch
- AWS Glue
- Step Functions
- EventBridge
- Oracle
- SharePoint
- OneDrive
- Teams
- Sitecore
- HTML
- Office documents
Preferred Qualifications
- AWS Certified Data Engineer – Associate
- Prior integration with Oracle B2C Service or another Oracle Service Cloud product
- Prior integration with Sitecore XM or XP content APIs
- Experience using AWS Glue, Step Functions, or EventBridge for orchestration, and document parsing libraries for Office and PDF formats
Job Location & Experience
- Location: United States (onsite)
- Minimum experience: 4 years
When You Join Solvd
- Shape real-world AI-driven projects across key industries, collaborating with clients from startup innovation to enterprise transformation.
- Work with a global team with equal opportunities for collaboration across continents and cultures.
- Thrive in an inclusive environment focused on continuous learning, innovation, and ethical AI standards.