Tokenization for LLM Inputs: How AI Reads What It Doesn’t See

What Structure-Preserving Substitution Actually Does in This Context

The transformation step that sits between an enterprise document and an external LLM is, conceptually, simple: identify the elements that can’t cross the boundary, replace them with placeholders that preserve their structural role, then send the result to the model. In practice, the simplicity hides a set of architectural decisions that determine whether the approach works at production scale or breaks under load.

This article walks through those decisions. It is not a tutorial on a specific library or product. It is the set of choices any team adopting pre-LLM substitution has to make explicit, along with the trade-offs each choice carries.

One clarification up front. This article covers the data-protection technique: replacing sensitive values with non-sensitive stand-ins that can be mapped back, not the NLP operation that splits text into subword units before a model reads it. The two ideas share a name in the field but almost nothing else. Throughout, the technique is described as structure-preserving substitution.

In CUBIG’s architecture, substitution is the core mechanism inside a broader context-preserving data layer that also includes detection, format preservation, and optional statistical protections. This article focuses on the substitution mechanism specifically; the design decisions are largely the same across implementations, whatever the field chooses to call them.

When an enterprise document is prepared for an external LLM, the goal is that the model sees a version of the document that keeps everything it needs for the task and leaves out everything the boundary was set up to hold in. Substitution is the mechanism that achieves the second half: it identifies sensitive elements and swaps in stand-ins.

A useful frame: the LLM doesn’t need to know that the customer is named Marlene Schmidt. It needs to know that there is a customer, that the customer is referenced in three different places in the document, and that the references all point to the same entity. A stand-in like CUST-7F2A carries the same information, a referenceable entity that appears consistently in multiple places, without carrying the identity.

That is the core property substitution provides: referential integrity without semantic disclosure. The model can reason about “the customer” across a document because the stand-in threads through the document consistently. The model cannot recover the identity because the stand-in does not encode it.

Everything else in this article is a variation on how that property is implemented, and what additional properties layer on top of it.

Deterministic vs Randomised Substitution

The first architectural decision is whether a given sensitive value always produces the same stand-in, or a different one each time.

Deterministic Substitution

Deterministic substitution means Marlene Schmidt always becomes CUST-7F2A, in every document and every workflow. The stand-in is a function of the value, usually combined with a secret key.

The benefit is consistency across documents. If two tickets reference the same customer, the LLM sees the same stand-in in both, and analytics that depend on cross-document linkage keep working. For workflows that aggregate or compare across documents, such as fraud-detection patterns, customer-history summaries, and cohort analysis, deterministic substitution is usually the only viable choice.

The cost is that determinism creates a re-identification surface. An attacker who observes enough transformed documents and has side information about which customers appear where can correlate stand-ins to identities. The risk is real for high-volume workflows, or wherever the same entity appears in many outputs over time.

Randomised Substitution

Randomised substitution generates a different stand-in for each occurrence, even of the same value. Marlene Schmidt might become CUST-7F2A in one document and CUST-3B91 in another.

The benefit is that no cross-document linkage is exposed. Each transformed document is a closed system.

The cost is that cross-document analytics break. The LLM cannot tell that two stand-ins refer to the same customer, because at the structural level they do not. For workflows that do not need cross-document linkage, such as summarising a single document or extracting clauses from a single contract, randomisation is fine. For workflows that do, randomisation forces the linkage to be reconstructed after the LLM responds, which adds complexity.

The Hybrid Pattern Most Production Deployments Use

Most production deployments end up with a hybrid: deterministic within a workflow scope, so a multi-turn conversation about a customer stays coherent, and randomised across workflow scopes, so analytics from one workflow cannot be cross-referenced with another. The boundary of the scope is itself a design decision, whether by session, user, document, or tenant, and it is one of the things a team has to settle before the architecture goes live.

Format-Preserving Substitution: Why Placeholder Strings Aren’t Enough

A naive implementation replaces sensitive values with generic placeholders: [CUSTOMER], [ACCOUNT_NUMBER], [DATE]. The LLM sees a document littered with these markers and tries to reason about it.

This works poorly in practice, for a specific reason: the LLM’s reasoning is shaped by the surface form of the input. A document that reads “Customer Marlene Schmidt called on 2026-03-15 about account 4471-9028” is, to the model, a coherent operational record. The same document with placeholders, “Customer [CUSTOMER] called on [DATE] about account [ACCOUNT_NUMBER]”, reads as a template or a redaction notice. Models are sensitive to that signal, and their outputs degrade accordingly: summaries become more abstract, extraction becomes less precise, and the model occasionally lapses into commentary about the redaction itself.

Format-preserving substitution generates stand-ins that look like the values they replace. A name becomes a plausible-looking name, such as Lyra Vesper. A date becomes a real date in a plausible range. An account number becomes a number of the same length and format that is not a real account number.

The document the LLM sees then reads as a coherent operational record with anonymous-but-realistic stand-ins. The model’s outputs come back at the quality it can actually produce, rather than degraded by the perception that it is reasoning about a template.

Format preservation has its own design choices: how plausible to make the stand-ins, whether to draw from a fixed pool of fake names or generate them on the fly, and how to handle dates and numerics where the value itself carries analytical meaning (a date in 2019 versus 2024 may matter to the analysis even when the exact date is sensitive). The general rule is that the stand-in has to preserve whatever analytical property the original value carried, no more and no less.

Where the Mapping Lives

Substitution only keeps the original values inside the boundary when the mapping, the table that connects stand-ins to original values, stays inside the enterprise environment. This is the part of the architecture that most consistently determines whether the approach delivers on its promise.

Three properties of the mapping have to hold:

1. It stays under the enterprise’s exclusive control. The mapping is, in effect, the key that re-identifies the data. If it leaves the environment, the safeguard collapses to whatever the new location provides. For workflows where data must stay in the EU region or other defined boundaries, the mapping has to live within that same boundary, colocated with the source systems rather than with the AI endpoint.
2. It is integrity-protected. Tampering with the mapping changes what gets reconstructed when the LLM’s response comes back. An attacker who can modify the mapping can substitute identities in the output. Standard practice is to apply integrity checks to the mapping itself, such as signed entries and audit logs of access, so that any tampering is detectable.
3. It is access-controlled separately from the LLM workflow. The team that operates the LLM integration does not need read access to the mapping. The reconstruction step pulls from the mapping programmatically; it does not require humans to see the original values. Separating those two access paths means the mapping can be governed under stricter controls than the LLM workflow itself.

Storage technology is secondary to these properties. The mapping can live in a dedicated database, a key-value store, an encrypted file, or a hardware-backed secure store; the right choice depends on volume, latency requirements, and existing infrastructure. What matters is that the three properties above are non-negotiable design constraints, not configurable options.

Consistency: Same Entity, Same Stand-in

A subtler design problem is ensuring that the same entity gets the same stand-in, consistently, even when it is referenced in different ways across a document.

A service ticket might mention “the customer,” then “Mr Schmidt,” then “Marlene,” then “the subscriber,” all referring to the same person. A naive implementation sees four different mentions and produces four different stand-ins, which breaks the LLM’s ability to track that they refer to one entity. The summary that comes back may treat them as four people.

Resolving this requires entity resolution before substitution: identifying which mentions in a document refer to the same underlying entity, and ensuring they all map to the same stand-in. This is a non-trivial problem in general, since entity resolution is a research field of its own, but in practice it is tractable because enterprise documents carry structural cues: a customer ID in the header tying together free-text mentions, formal naming conventions in operational logs, schema-defined relationships in structured records.

The other half of consistency is across documents within a workflow scope. If two tickets reference the same customer and the workflow needs to treat them as related, substitution has to produce the same stand-in for the customer in both. This is where the deterministic-versus-randomised choice from earlier interacts: deterministic-within-scope is what lets the LLM see “the same customer appears in three tickets” without learning who the customer is.

A well-designed substitution layer handles both kinds of consistency, within-document and within-scope, as part of the transformation rather than as an afterthought. Teams that retrofit consistency onto a per-mention approach usually find the workflow degrades in ways that look like model-quality problems but are really data-preparation problems.

Additional Protection Layers: When Substitution Alone Isn’t Enough

Substitution handles the swap. For most workflows, a well-implemented substitution layer with the mapping under the enterprise’s exclusive control is sufficient. For some workflows, an additional layer is worth adding on top.

The case for an additional layer arises when the residual risk is not in the stand-ins themselves but in the patterns they form. A transformed document may carry enough structural information, such as frequencies, co-occurrences, sequences, and ratios, that a sophisticated correlator could re-identify entities even without the raw values. This risk is particularly relevant for high-cardinality data, long time series, and workflows where many outputs accumulate over time.

The standard responses are differential privacy, k-anonymity, and similar statistical protections applied to the transformed data. Each adds noise or aggregation in a controlled way that limits how much an attacker can learn from the output, at the cost of some analytical precision. Whether the trade-off is worth it depends on the threat model and the workflow’s tolerance for noise.

For most enterprise AI workflows this layer is optional. For workflows where the data is highly sensitive, the volume is high, or the data posture demands defence in depth, it is worth the complexity. The decision is best made workflow by workflow, not as a global setting.

What Not to Substitute

A final design question often gets answered by accident: what not to substitute.

Substituting the wrong things degrades the AI’s output without improving anything. An approach that replaces every proper noun produces unreadable documents. One that replaces every numeric field destroys analytical signal. The temptation is to be aggressive, to “substitute everything that could conceivably be sensitive,” but the cost shows up immediately in output quality.

The disciplined approach is to define sensitivity explicitly, in the enterprise’s own terms, and substitute only those elements. Generic PII categories are a starting point, not a complete list. Internal project codes, customer-segment identifiers, sector-specific references, whatever the enterprise’s data posture treats as protected, go on the list. Everything else stays.

The list has to be versioned, because what counts as sensitive changes over time. It also has to be auditable, because an audit review of the workflow will want to know what was substituted, when, and under which definition. The definition layer is where most of the long-term operational cost of this architecture lives, and where most teams under-invest at the start.

The Next Step in the Workflow

Substitution prepares the document for the external model. The model processes the transformed document and returns a transformed response. The response on its own is not yet useful to the workflow: the stand-ins have to be mapped back to original values inside the enterprise environment before the output reaches the user.

That reconstruction step is the subject of the next article in this series. For the broader pattern this article is part of, see the pillar overview on running external LLMs on sensitive enterprise data. For why masking and redaction do not substitute for this approach in operational workflows, see the article on why AI workflows stall on operational data.