Structured QA for enterprise generative AI, from gateway to copilot

Websiteinvenci.com ↗

Client

Invenci AI

Industry

AI Platform · Enterprise generative AI

Engagement

Ongoing · 1+ year

QA team

3 full-stack QA engineers

Platforms

Desktop Web · Mobile Web · PWA · API · Database

About the project

Invenci designs and delivers form-fitting generative AI for enterprises in retail, power and utilities, financial services, healthcare and manufacturing: chatbots and copilots grounded in company data, RAG pipelines, private and fine-tuned LLMs, and an AI Gateway that puts governance, access control and usage policy in front of every model call.

Because client solutions are built on shared platform components, one change to the gateway, a prompt template or a retrieval step can ripple into several customer deployments at once. Release confidence has to cover the platform and every solution built on top of it.

QAonTOP joined in 2025 with three full-stack QA engineers embedded across Invenci’s delivery teams, covering web, mobile web, PWA, API and data layers.

What made this hard

Generative AI changes what “expected result” means. Invenci needed QA that could judge answers, not just compare strings — and prove that governance rules hold when users push against them.

1. CHALLENGE 01

   Non-deterministic by design

   The same prompt can return different wording every time. Tests had to decide whether an answer is acceptable, grounded and policy-compliant, not whether it matches a stored string.

2. CHALLENGE 02

   Governance has to hold under pressure

   Gateway rules — PII masking, role-based model access, blocked topics, usage limits — must hold for everyday use and for users deliberately trying to get around them.

3. CHALLENGE 03

   Multi-step AI workflows

   Automation flows chain retrieval, model calls, tool actions and human approvals. A failure at step three can look like success by step five.

4. CHALLENGE 04

   Enterprise UX on every screen

   Business users move between desktop, phone and an installed PWA. Streaming responses, file uploads and long conversations have to stay usable on all of them.

How QAonTOP tested Invenci AI

1. 01

   A risk map per solution

   Catalogued every client-facing solution and the shared components beneath it, then rated each flow by business impact and model exposure. That map decides where automation, manual depth and AI evaluation go first.

   Tools  Jira · GitHub

2. 02

   Evaluation sets for AI output

   Built prompt libraries per use case: expected facts, required source, tone and refusal behaviour. Responses are graded on a rubric for accuracy, grounding, completeness and policy — by script first, by a QA engineer on anything borderline.

   Tools  LLMs · JavaScript

3. 03

   Automate the AI workflows

   Playwright in JavaScript covers chat, copilot and admin journeys end to end, including streamed responses. PostQode and Postman drive the API layer: gateway policies, model routing, rate limits and the output of each workflow step.

   Tools  Playwright · PostQode · Postman

4. 04

   Try to break the guardrails

   Adversarial prompt sets probe prompt injection, jailbreaks, PII leakage and cross-tenant data access. Every bypass we find becomes a permanent regression case.

   Tools  LLMs · Postman

5. 05

   Load and usability

   JMeter tests concurrent sessions and token-heavy requests through the gateway. Task-based usability sessions run on desktop, mobile web and the installed PWA, with Axe DevTools checks on core screens.

   Tools  JMeter · Axe DevTools

The engagement in numbers

Before and after the engagement

What we covered