// Open Role at Keywords Studios
This opportunity is available across all Player Engagement locations and is intended for internal hiring only.
French translation of the job description will be available soon.
Role Overview
As Keywords continues to expand Helpshift Care AI across Player Engagement, quality will become one of the most important factors in successful AI adoption.
The AI Ops Quality Manager plays a critical role in ensuring Care AI performs accurately, safely, and in alignment with each client’s tone, standards, and player experience expectations.
This role supports AI Ops Managers and Platform Leads by owning the quality discipline around live AI performance. As more clients move into launch, hypercare, BAU, and future AI feature adoption, this role ensures AI agent performance is not only monitored, but continuously improved through structured review, feedback, root cause analysis, and measurable action.
The AI Ops Quality Manager sits at the intersection of Operations, Product, Development, Account Management, Marketing, and Client Success. This is a role for someone who can protect quality while helping the business move faster.
What are we looking for?
Quality & Governance
· Design, maintain, and continuously improve the QA framework for Helpshift Care AI across launch, hypercare, live client environments, and future AI product capabilities.
· Create clear quality standards for AI agent responses, including accuracy, completeness, tone alignment, escalation behavior, safety, and player experience.
· Define scoring rubrics, audit methods, review cadences, and reporting formats that allow AI performance to be measured consistently across clients.
· Develop QA criteria, test scenarios, audit checks, and feedback loops for new AI features to ensure they are accurate, stable, client-ready, and aligned with player experience expectations.
· Help ensure that as the Care AI roadmap expands, Keywords has a structured quality discipline in place to support new features without creating unnecessary risk for clients, players, or live operations.
Analytics
· Conduct regular audits of AI agent interactions on live accounts to identify performance trends, defects, risks, and improvement opportunities.
· Assess whether AI responses are accurate, helpful, aligned to the client’s voice, and appropriate for the player’s issue type.
· Identify recurring gaps such as incorrect answers, weak source content, poor escalation logic, low-confidence responses, hallucination risk, or tone mismatch.
· Use a structured DMAIC approach to improve AI agent performance on live accounts.
· Define the quality issue, measure the current performance, analyze the root cause, implement improvements, and control the process through ongoing monitoring.
· Interpret quality data, identify trends, and recommend practical actions that improve AI performance and player experience.
Communication
· Act as a key guardian of client tone and player experience within Care AI deployments.
· Evaluate whether AI agent responses reflect each client’s brand, communication style, support philosophy, and player expectations.
· Partner with AI Ops Managers, Platform Leads, and client teams to ensure AI performance supports trust rather than simply increasing automation volume.
· Serve as a proactive feedback partner to Product and Development teams by translating live account quality findings into actionable insights.
· Document recurring AI performance gaps, platform limitations, client-impacting issues, and enhancement opportunities.
· Provide quality insights, performance summaries, risk flags, and improvement updates to AI Ops leadership and key stakeholders.
· Help translate detailed QA findings into clear business language for Account Management, Client Success, Marketing, Product, and executive audiences.
Business
· Partner with Product, Development, Platform Leads, and Operations to convert quality findings into practical fixes, content improvements, workflow changes, and product feedback.
· Help prioritize improvements based on client impact, player experience risk, commercial value, and operational urgency.
· Support the QA process for new Helpshift Care AI capabilities and upcoming AI product feature sets as they move through the product roadmap.
· Partner with Product, Development, AI Ops Managers, and Platform Leads to understand intended feature behavior, client use cases, operational impact, and quality risks before new capabilities are released into live environments.
· Validate new feature sets against real support workflows, client tone requirements, escalation paths, and operational use cases so AI Ops can confidently support adoption after release.
· Document findings, identify defects or performance gaps, and provide clear recommendations to Product and Development teams before broader rollout.
· Highlight wins, client-ready proof points, and measurable improvements that can support adoption, expansion, and commercial storytelling.
Leadership
· Partner with AI Ops Managers and Platform Leads to ensure quality learnings are turned into reusable playbooks, templates, calibration guides, and best practices.
· Support internal enablement for Operations, Account Management, Marketing, Product, and client-facing teams by explaining what good AI performance looks like and how to recognize risk.
· Help raise the organization’s AI quality maturity as Care AI scales across more clients and more use cases.
· Raise risks early, challenge assumptions constructively, and help protect the player experience.
· Support the responsible scaling of Care AI while maintaining quality, trust, and client confidence.
Requirements
You'd be a great fit for this role if you have:
· Experience in Quality Assurance, Player Support, Customer Support Operations, AI Operations, Knowledge Management, Process Improvement, or client-facing support environments.
· Strong understanding of support quality principles, player experience, client tone, escalation behavior, and operational risk.
· Ability to review AI or agent interactions with strong attention to detail, fairness, consistency, and practical judgment.
· Comfortable identifying quality defects, root causes, content gaps, workflow issues, and improvement opportunities.
· Experience using structured improvement methods such as DMAIC, root cause analysis, calibration, audit sampling, or continuous improvement frameworks.
· Strong communication skills, with the ability to explain quality findings clearly to both technical and non-technical stakeholders.
· Ability to work cross-functionally with Product, Development, AI Ops, Platform Leads, Operations, Account Management, Marketing, and Client Success.
· Analytical mindset with the ability to interpret quality data, identify trends, and recommend practical actions.
· Comfortable working in a fast-moving environment where AI performance, client expectations, and product capabilities continue to evolve.
· Strong ownership mentality with the confidence to raise risks early, challenge assumptions constructively, and protect the player experience.
· Genuine interest in AI, automation, Helpshift, gaming support, and the future of customer experience.
· Passion for helping Keywords scale AI responsibly while maintaining quality, trust, and client confidence.
Nice-to-haves:
· Experience supporting AI-powered products, conversational AI, virtual agents, or automation programs.
· Familiarity with Helpshift and knowledge management systems
· Background in gaming, player support, or digital customer experience environments.
· Experience developing quality scorecards, audit frameworks, calibration programs, or operational reporting.
· Passion for gaming, AI innovation, and improving customer experiences through responsible automation.
What Success Looks Like
· Care AI does not simply go live—it gets better after it goes live.
· AI agent performance is reviewed, measured, improved, and controlled through a clear quality discipline.
· Client tone is protected, player trust is preserved, and quality risks are identified before they become escalations.
· Product and Development teams receive actionable feedback from real client environments.
· New AI product features are reviewed with the same quality discipline applied to live AI agent performance.
· Account Management, Marketing, and Client Success teams have stronger proof points because AI performance is being improved and documented in a credible way.
· Keywords scales Care AI with confidence while maintaining quality, trust, and operational excellence.
· As the number of live clients increases and the product roadmap expands, AI Ops has the quality backbone needed to support growth, protect the player experience, and build long-term client trust in Helpshift Care AI.
Benefits
Based on location-specific benefits.