How We Audit Our Own Platform: 79 Findings, 7 Repos, One Sprint

Production readiness is not a phase. It is a practice. At Arbitex, we run structured platform audits before every major release — not as a checklist gatekeeping exercise, but as the engineering discipline that defines what “ready” actually means for a regulated AI governance platform.

This post walks through the methodology we used for our most recent audit cycle: how we structured the work, what we found, and how we closed it.

Why a Structured Audit

Most engineering organizations discover production issues in one of two ways: customer reports or incidents. Both are expensive. Customer reports damage trust. Incidents, depending on severity, can damage more.

The alternative is structured self-assessment: treating your own platform as the adversary would, before the adversary does.

We decided early that our audit framework would have three properties:

MECE categorization. Every finding belongs to exactly one category. Mutually exclusive, collectively exhaustive. If a finding could belong to two categories, the category definitions are wrong. This forces precision in issue taxonomy — and precision in issue taxonomy forces precision in issue resolution.

Severity stratification. Not all findings are equal. A critical security finding and a low-severity documentation gap should not compete for the same engineering bandwidth. Stratification lets us sequence work correctly: critical first, then high, then medium and low as carry-forward.

Complete closure, not mitigation. A finding is closed when the root cause is addressed, not when a workaround is in place. Mitigations are documented separately and tracked as technical debt. We do not close findings against workarounds.

The Audit Structure

We organized the audit across seven functional dimensions:

• Copy and voice — Does the public-facing language accurately represent what we ship? No vaporware, no overclaiming, no certification language we haven’t earned.
• Accessibility — WCAG 2.1 AA compliance across all user-facing surfaces. Focus management, keyboard navigation, ARIA semantics, color contrast.
• Browser compatibility — Do all features work across the browsers our enterprise customers actually use?
• Performance — Page weight, asset delivery, time to interactive. Regulated enterprise customers often operate in constrained network environments.
• SEO and discoverability — Structured metadata, canonical URLs, sitemap correctness, robots.txt hygiene.
• Links and references — Dead links, stale external references, internal routing correctness.
• Security posture — Dependency audit, configuration review, header policies, CSP correctness.

Each dimension was audited independently and produced a structured finding report with severity, category, description, and resolution path.

What We Found

Across seven repositories and seven audit dimensions, the cycle surfaced 79 findings. The distribution was roughly:

• A small set of critical and high findings — these went to the top of the queue immediately.
• A larger body of medium findings — addressed in the same sprint where capacity allowed.
• A long tail of low findings — documented, triaged, and scheduled for follow-up cycles.

No single dimension was clean. That is normal and expected. The goal of the audit is not to find zero issues — it is to find issues systematically rather than reactively. An audit that finds nothing is almost always an audit that did not look hard enough.

The findings that required the most attention were concentrated in two dimensions: copy/voice accuracy and accessibility. Copy accuracy issues tend to accumulate quietly — a number gets stale, a claim gets ahead of what shipped, a certification is implied where only audit-readiness exists. Accessibility issues tend to be structural — a missing aria-label, an inconsistent focus indicator, a color contrast ratio that passes at the component level but fails in combination with a background.

The Resolution Approach

We ran the fix cycle as a parallel sprint with non-overlapping file assignments. Each fix agent owned a specific set of files and worked independently — no merge conflicts, no coordination overhead.

Critical and high findings were closed in the first wave. Before any finding was marked closed, the resolution was verified against the original finding description. Not “addressed” — closed. The criterion: does the issue described in the finding still exist? If yes, it is not closed.

Medium and low findings were batched by dimension and worked in sequence. Where a medium finding required architectural changes beyond what a single sprint could absorb, it was formally documented as carry-forward with a specific resolution path — not closed, not forgotten.

The sprint closed with zero open critical or high findings. Dependency audit: zero vulnerabilities in production scope.

What This Process Demonstrates

Running a structured audit is not primarily about finding bugs. It is about establishing a disciplined relationship between your engineering team and the question “is this production-ready?”

For a regulated AI governance platform, that question has a higher threshold than for most software. Our customers are using Arbitex to prove compliance to HIPAA auditors, PCI-DSS assessors, and GDPR data protection authorities. The audit trail we produce, the policy enforcement we execute, and the accuracy metrics we publish need to be reliable in ways that matter to regulators — not just in ways that are good enough for most use cases.

The structured audit methodology is how we maintain that threshold systematically rather than reactively.

The Discipline, Not Just the Output

The most important outcome of an audit cycle is not the finding count or the closure rate. It is the discipline of treating your own platform as a production system that must be audited, not as a development artifact that will be polished later.

Production readiness is not a state you reach. It is a practice you maintain. The audit cycle is how we maintain it.

We publish our DLP accuracy metrics — DLP detection accuracy metrics across all supported entity types — because the same discipline that drives our audit practice drives our transparency about what our platform actually delivers. If you are evaluating AI governance platforms for a regulated environment, we recommend asking every vendor for their per-entity accuracy metrics. The answer is informative.