Rust-Powered Core Loops for Performance and Code Safety

The Arbitex DLP pipeline runs on every request. Every prompt, every response, every file upload passes through pattern matching, overlap resolution, and checksum validation before the platform makes an enforcement decision. The orchestration layer — policy resolution, compliance pack selection, audit logging — is Python. That is the right choice for a system that coordinates async services, manages configuration, and integrates with dozens of external providers.

But the hot loops are a different problem.

The Problem: Python in the Inner Loop

Pattern matching, checksum validation, and overlap resolution are tight loops over byte arrays. They run on every request, often multiple times per request when content is long or multiple compliance packs are active. Python’s interpreter overhead — object allocation, dynamic dispatch, the GIL — adds latency that compounds at scale.

The orchestration layer does not need to be fast in the same way. It makes decisions, calls services, writes logs. The inner loops need to process bytes. Thousands of pattern matches per second, each validated against checksum algorithms, each deduplicated against overlapping spans. The workload is CPU-bound, deterministic, and has no external dependencies.

This is the kind of work where language choice directly affects throughput.

Why Rust, Not C or C++

For a security product that inspects sensitive data, the implementation language of the inspection engine matters. A memory corruption bug in the DLP scanner would be an ironic security vulnerability — the system designed to catch data leaks, leaking data through its own implementation flaw.

Rust provides memory safety without garbage collection. No segfaults, no buffer overflows, no use-after-free — guaranteed at compile time by the borrow checker. The compiler rejects entire categories of bugs before they reach production. C and C++ can achieve the same raw performance, but they require the developer to manage memory correctness manually. In a security-critical code path, “the developer will get it right” is not a sufficient guarantee.

Rust also produces predictable performance. No garbage collection pauses, no runtime overhead from a managed environment. The compiled output is a native shared library with the same performance characteristics as C.

The Native Extension Bridge

The compiled code produces a native Python extension module. The platform imports the native scanner like any other Python module — no FFI boilerplate, no subprocess spawning, no network hop to a sidecar service. Function calls cross the Python-Rust boundary in microseconds.

This matters for deployment simplicity. The native scanner is not a separate service to manage, monitor, and scale independently. It is a library linked into the existing Python process. The same test suite validates both backends. The same API contract applies.

A single configuration setting selects the compiled backend. The interpreted fallback is always available. In development, either backend produces identical results — same entity types, same spans, same confidence scores, same checksum outputs. The accuracy harness validates zero detection deltas between backends.

What Runs in Rust Today

Three components have moved to Rust:

Compiled regex engine. All Tier 1 patterns compile into a single-pass multi-pattern automaton. Instead of evaluating patterns sequentially — one pass over the text per pattern — the compiled engine evaluates all patterns simultaneously in one pass. For requests with dozens of active patterns across multiple compliance packs, this eliminates redundant text scanning.

Checksum validators. Luhn validation, format-specific checksums, and structural validators run natively. Each candidate match is validated in the same call that found it, with no round-trip back to Python between matching and validation.

Overlap resolution. When multiple patterns match overlapping regions of text, the resolution logic handles conflicts: same-type overlaps keep the highest-confidence match, cross-type overlaps keep both with a confidence penalty. This runs in a single pass over the sorted match list.

The result is 10-50x faster execution on the hot path compared to the pure Python implementation. The speedup varies by workload — short prompts with few patterns see less benefit than long documents with many active compliance packs.

What Is Coming

The unified ScanEngine in arbitex-core extends this architecture beyond the DLP pipeline. Content extraction — PDF text extraction, OCR post-processing, format detection — uses the same pattern: Python orchestrates, Rust handles the byte-level work.

The unified content inspection engine that powers every channel (AI gateway, email relay, file uploads, cloud storage passthrough) will use the same Rust-backed hot loops for content extraction and pattern matching. Adding a new file format means adding one handler. Every channel benefits from the same optimized code path.

Air-Gap Safe

Rust compiles to a .so (shared object) baked into the Docker image at build time. There are no runtime downloads, no cloud dependencies, no JIT compilation step. The binary is static and deterministic — the same image produces the same behavior on every deployment.

This is a requirement for air-gapped environments. Customers running the Outpost in regulated or disconnected networks cannot depend on runtime package fetches or compilation steps. The native scanner works identically whether the host has internet access or not.

For the performance characteristics of the compiled pattern matching engine, read How We Optimized the DLP Pattern Matching Engine. To see what the platform is building next with the unified inspection engine, visit What We’re Working On.

Book a demo to see the DLP pipeline inspect your AI traffic.