AI Developer
Job Description
Cobalt Benefits Group (CBG) is seeking an AI Developer to design, build, test, and deploy governed AI solutions that improve both operational efficiency and member experience. The role focuses on production-ready retrieval-augmented assistants, document-extraction agents, and secure AI-enabled workflows integrated with core systems.
Responsibilities
- Design, prototype, build, test, and deploy production AI applications and agents, selecting appropriate approaches (including Copilot Studio, workflow, agent, custom services, vendor solutions, or non-AI options) using Build-vs-Buy guidance.
- Develop retrieval-augmented generation (RAG) systems, including document ingestion, chunking, indexing, metadata, retrieval tuning, and grounded responses with source citations.
- Integrate AI solutions with internal and third-party systems via authenticated REST and GraphQL APIs, collaborating on document and structured-data pipelines.
- Implement least-privilege permissions, single sign-on, secrets management, PHI/PII protections, output validation, and logging and retention controls with IT Security.
- Create evaluation harnesses using golden test sets, subject-matter expert rubrics, and production sampling; monitor accuracy, citation support, completeness, latency, and transaction cost, including results by plan type.
- Add telemetry for prompts, retrieval, tool calls, and outputs; maintain runbooks, hand-off documentation, cost controls, and operational monitoring.
- Advance work through CBG AI Solution Delivery Playbook decision gates and required artifacts, including risk assessments, evaluation plans, and transition documentation.
- Partner with the AI Risk Manager, Compliance, Data Protection Officer, Security, and the AI Ethics Committee as appropriate, sharing reusable components, templates, and approved prompt patterns.
Requirements
- Bachelor’s degree in Computer Science, Information Systems, related field, or equivalent practical experience, plus 3+ years of professional software development experience.
- Hands-on generative AI development and evaluation experience, including RAG, context engineering, tool/function calling, and at least one production or production-pilot solution.
- Strong Python and/or C#/.NET development skills, with experience building production APIs and integrating via REST, along with OAuth or similar authorization and secure secrets/configuration management.
- Ability to review, test, debug, and maintain AI-generated code; working knowledge of SQL, data modeling, and data pipelines.
- Experience handling HIPAA-regulated information, including PHI masking and access controls.
- Experience with AI regression testing, monitoring, model version management, release controls, cost optimization, and guarded multi-step agents.
- Ability to document technical decisions, follow governance and change control, conduct business requirements sessions, and recommend delaying or stopping solutions that fail evaluation thresholds.
Preferred
- Experience with Microsoft Copilot Studio, Power Automate, Power Platform, Azure OpenAI, and Azure AI Search.
- Proven improvements to retrieval or extraction systems using before-and-after evaluation results.
- Experience with AI coding agents and Model Context Protocol (MCP).
- Familiarity with healthcare, third-party administration, employee benefits, claims, eligibility, plan documents, or EDI.
Technology Environment
Work will leverage Microsoft and Azure tools, with technology selection varying by project and risk tier. Environment includes: Microsoft 365 Copilot, Copilot Studio, Power Automate, Power Apps, Teams; Azure OpenAI / Azure AI Foundry, Azure AI Search; Azure API Management, Entra ID, Key Vault; Azure SQL, Cosmos DB, Azure Data Lake Storage, Blob Storage, Dataverse; GitHub and CI/CD pipelines. Specific technologies vary by project and risk tier.
Measures of Success
- First 90 days: Complete AI governance onboarding, ship an enhancement through Production Readiness Review, and establish a golden-test-set evaluation harness.
- First 6 months: Release a new approved AI solution, publish reusable components adopted by another project, and enable monitoring dashboards.
- First 12 months: Meet approved business KPIs, demonstrate reusable delivery patterns, and maintain current inventory, risk classification, evaluation, and monitoring records.
Compliance and Responsible AI
- Solutions must meet CBG governance and technical security requirements, with privacy, security, and human oversight proportionate to risk.
- Automated claim denials without human review and the use of PHI in public generative AI models are prohibited.
- Work must align with HIPAA, HITRUST CSF, NIST AI Risk Management Framework, and applicable OWASP guidance.
Collaboration and Personal Attributes
- Collaborative communicator who can translate business processes into technical requirements and partner across business, engineering, security, and compliance teams.
- Analytical, detail-oriented problem solver focused on measurable quality, documentation, and continuous improvement.
- Sound judgment with accountability in privacy, fairness, human oversight, and responsible AI delivery.
- Hands-on builder comfortable selecting between low-code and custom engineering approaches.
Compensation and Location
Location: Boston, MA (onsite). Compensation: USD 120,000 - 155,000 per yearly.
Benefits
- Fantastic medical, dental, and vision insurance
- Twice annual employer HSA contributions, covering 50% of the HDHP plan’s annual deductible
- Company provided Basic Life and AD&D
- Company paid Short-Term and Long-Term Disability
- Flexible Spending Accounts
- 401(k) Retirement Plan with up to a 6% employer-match (100% fully vested after 3 years)
- 10+ paid holidays
- Fully Paid half day Summer Fridays
- Generous paid vacation and sick time
- Annual Paid Volunteer Day
- Annual Tuition Reimbursement
- Annual Health and Wellness Reimbursement
- Lots of fun company events
Technology Skills
- Copilot Studio, Power Automate, Power Platform, Azure OpenAI, Azure AI Search, Power Apps
- Azure API Management, Entra ID, Key Vault
- Azure SQL, Cosmos DB, Azure Data Lake Storage, Blob Storage, Dataverse
- GitHub, CI/CD pipelines
- REST, GraphQL, OAuth
- Python, C#, .NET, SQL
- Microsoft 365 Copilot, Teams, Azure AI Foundry