Phixe tests AI systems, not just models. A model can be well-aligned in isolation and still fail in production because of the retrieval layer feeding it, the tools it can call, or the tenant boundary around it. Our methodology exists to find those failures first — with evidence you can reproduce, hand to an enterprise reviewer, and act on.
The stance
We build and we break. The people who run your assessment have also shipped and operated AI in production, including our own product, faben. That matters because breaking is only half the work — the finding is worthless without a fix path that survives contact with your architecture and your deadline.
We test the whole system: the model, the prompt and context assembly, the retrieval pipeline, the agent’s tool surface, and the platform underneath. Attackers do not respect your component boundaries, so neither does our test plan. And we test against a known adversary model rather than improvising clever tricks — a finding that maps to a documented technique is one your team can reason about, prioritize, and defend against systematically.
Frameworks we map to
Every finding is anchored to public, versioned frameworks so it connects to a shared vocabulary instead of our private opinion.
| Framework | What it covers | How Phixe uses it |
|---|---|---|
| OWASP LLM Top 10 | The critical risks of LLM applications — LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, and the rest | The baseline coverage map for any LLM feature. Each assessment reports which LLM0x categories were in scope, what we tested, and what we found. |
| OWASP Agentic Security Initiative (Agentic Top 10) | Risks specific to tool-using, planning, and memory-holding agents — unbounded tool invocation, privilege escalation, memory poisoning | Applied whenever the system plans, calls tools, or persists state. It drives agent test cases well beyond single-turn chat. |
| MITRE ATLAS | Adversary tactics and techniques against ML and AI systems, structured like ATT&CK | We phrase attacker behavior as ATLAS techniques so a finding reads as part of a known kill chain, not a one-off stunt. |
| NIST AI RMF | Governance functions — Govern, Map, Measure, Manage — for AI risk | We frame the engagement so your reviewers and enterprise buyers can see what was mapped, what was measured, and what remains to manage. |
What we test, per surface
LLM behavior
Prompt injection (direct and indirect), jailbreaks that bypass system-prompt controls, and sensitive information disclosure — training-data leakage, system-prompt extraction (LLM07, System Prompt Leakage, in the 2025 list), and output that reveals more than the user’s authorization allows. This is the LLM01, LLM02, and LLM07 core, tested with adversarial inputs that mimic real user and content channels. See the LLM red-teaming guide and prompt injection testing guide.
RAG
Retrieval-augmented generation adds an attack surface most demos never exercise. We test corpus poisoning (malicious content that steers answers once retrieved), cross-tenant retrieval bleed (one customer’s documents surfacing for another), and retrieval abuse (prompts crafted to pull back context the caller should never see). Details in the RAG security guide.
Agents
When the system can act, the stakes change. We test tool-call abuse (coercing the agent into invoking tools with attacker-chosen arguments), privilege escalation across tool boundaries, and memory manipulation that persists a malicious instruction into future sessions. The AI agent security guide covers the failure classes in depth.
Platform
The AI feature sits on ordinary infrastructure, and ordinary infrastructure still breaks. We check authentication and authorization on every AI endpoint, multi-tenancy isolation, secrets handling, rate limiting and cost controls on model calls, and whether you have the monitoring to notice an attack in progress. The production readiness checklist is our starting line here.
The evidence standard
Every finding carries reproducible evidence — request traces, not screenshots of a vibe. Concretely, each one includes:
- Reproduction steps — the exact sequence to trigger it, deterministic where the model allows.
- Request and response traces — the actual payloads, so your engineers can confirm it without taking our word.
- Severity with reasoning — not a bare label, but why this rates where it rates given your data, users, and exposure.
- Fix guidance — a concrete remediation direction, tied to the surface and the framework category.
You can see the shape of this in our sample assessment report.
Evals as method
Red-teaming finds the failure; evals prove it stays fixed. We build a test harness with golden sets — curated inputs with expected, checkable behavior — that turn a subjective “the model seems better now” into a measurable pass or fail. Wired into your CI as regression gates, the same harness catches a prompt tweak or model upgrade that quietly reopens a closed finding. The LLM evals guide walks through building one.
Retest is part of the engagement
An assessment that ends at the report is only half a service. After you ship fixes, we retest the specific findings to confirm the remediation holds and did not shift the failure somewhere adjacent. Verification of the fix is included, not a separate upsell.
The independence rule
We never issue an independent verdict on a system we built for the same client. Build for one, assess for another — and if we productionize or build and run your product and you then need an independent assessment for a buyer or insurer, a third party re-verifies it. Within any engagement, a finding is reported on its merits, never softened because it is inconvenient to whoever wrote the code, including us. Start with an AI product readiness assessment, and see recent work for how this plays out in practice.