Context-Preserving Data Layer Evaluation Checklist

Evaluate a context-preserving data layer for AI across one complete request-response round trip. The team needs to know where original values remain, what the model receives, which relationships survive, how Reconstruction is authorized, and what appears in logs or failure paths. A prompt filter or deployment label cannot answer all of those questions.

This approval framework complements the definition of a context-preserving data layer and the broader view of AI data pipeline protection. It focuses on the evidence an architecture team should request before approving a specific implementation.

Key takeaways

• Review the full outbound and return path, not only the prompt.
• Test whether substitutes preserve the relationships required by the task.
• Verify where Reconstruction runs and who can trigger it.
• Include logs, retries, caches, errors, retention, and rollback in the decision.

What should a context-preserving data layer evaluation cover?

The evaluation should cover every component that can receive original values, substitutes, mappings, model traffic, or reconstructed output. That scope includes the normal path and the paths created by monitoring, retries, timeouts, malformed responses, and downstream writes.

NIST's Generative Artificial Intelligence Profile identifies leakage, unauthorized use, disclosure, and de-anonymization of sensitive data as generative AI risks. OWASP's LLM02:2025 guidance covers confidential business data and warns that system-prompt restrictions may be bypassed. Both support a system-level review.

Start with a data-flow diagram for one representative workflow. Mark the original record, substitution step, protected working version, approved model path, model response, Reconstruction component, reconstructed output, and final destination. Then add every supporting system that can store or copy any of those objects.

The result should be a reviewable boundary, not a statement that the data is protected. Each arrow needs an owner, protocol, payload, storage behavior, retention rule, and failure case. If the team cannot explain one arrow, that gap remains part of the approval decision.

Why are prompt-only checks incomplete?

Prompt-only checks are incomplete because sensitive values can appear before the request is assembled and after the model returns its answer. Preprocessing files, traces, retry queues, tool calls, response validators, and downstream applications can all create copies outside the prompt that an operator inspected.

OWASP recommends sanitization, input validation, access controls, restricted data sources, runtime data orchestration, and explicit retention and deletion policies. Its guidance does not establish that one control is sufficient. Instead, it shows why the application and its surrounding data path must be evaluated together.

This distinction matters when teams compare architectures. An API gateway may inspect a request. A model endpoint may offer retention controls. An output scanner may check a response. Each can perform a useful job, but the architecture review still needs to show how those controls connect across the complete workflow.

Give the return path its own diagram. An application may expose values while reconstructing an answer, writing a ticket, or recording the operation, even when the outbound request is protected. The response path contains a separate authorization decision that makes the output usable.

How do you lock the trust boundary?

Lock the trust boundary by naming every environment and recording exactly which representation may cross each boundary. The review should distinguish original values, context-preserving substitutes, the protected mapping, the protected working version, and the reconstructed output. These objects do not have the same access requirements.

LLM Capsule's current product description presents a specific data path. Original values and the protected mapping remain in the customer's environment. The protected working version travels the approved model path, and Reconstruction occurs inside that environment. Treat this as a first-party architecture claim. The deployed workflow must still prove that each integration follows the described boundary.

Ask for evidence that shows:

• the exact payload sent to the model endpoint;
• the services that can access originals and the mapping;
• the location and protection of temporary files and queues;
• the identities permitted to request Reconstruction;
• the destination and access policy for reconstructed output;
• the retention and deletion behavior for each copy.

UK ICO pseudonymisation guidance says that additional information enabling attribution should be held separately and secured. It recommends access controls, logging, key management, backup, and monitoring. Because the UK-specific guidance is under review, use it as a control reference rather than a universal compliance conclusion.

How do you test whether context is preserved?

Test context preservation against the relationships the workflow actually needs. A high replacement count does not prove that a model can still distinguish two assets, follow a ticket across retrieval and generation, or keep a table's references consistent. The acceptance test needs task-level success criteria.

Begin with a small set of representative records and identify the operational markers that carry meaning. These may include circuit IDs, incident numbers, asset names, contract references, topology nodes, or internal codenames. Define which values must remain distinct and which repeated values must remain consistent.

Google Cloud's pseudonymization documentation documents reversible and one-way cryptographic transformations. It also shows that referential integrity can depend on the method, key, and context tweak. For this evaluation, the relevant point is that preservation behavior must be specified rather than inferred from a label.

Run the same task on the approved source record and the protected working version. Check whether the model keeps entities separate, maintains references, follows the requested structure, and produces output that Reconstruction can process. Record expected differences and failures. The protected version must preserve what the approved task requires, not every surface detail.

Evaluate relationships, not only fields. A field test confirms that a value changed; a relationship test shows whether the same circuit remains connected across an alert, topology record, runbook, and final ticket.

What must Reconstruction prove?

Reconstruction must prove that an authorized workflow can return approved values to the model output inside the customer-controlled environment without sending those originals back through the model path. It must also handle altered, missing, duplicated, or fabricated substitutes safely.

Verify the component that performs Reconstruction, the caller identity it trusts, the policy applied to each field, and the destination for reconstructed output. Test successful and denied requests, then confirm what happens when the model changes the substitute format or invents a value with no mapping.

This step separates usable output from simple reversibility. A reversible transformation can recover an input value, but an operational workflow also needs authorization, output placement, audit evidence, and predictable failure behavior. Reconstructing AI output explains the last-mile problem in more detail.

At minimum, the evidence package should answer four questions:

1. Where does Reconstruction execute?
2. Which identity and policy authorize it?
3. How are unmatched or malformed substitutes handled?
4. Where is the reconstructed output allowed to go?

Do not treat a successful demonstration as sufficient evidence for every workflow. Repeat the test with each output format, authorization pattern, and failure behavior that materially changes the path.

What operational residue belongs inside the review?

Operational residue includes any copy created to run, observe, retry, diagnose, or recover the workflow. Logs, traces, caches, queues, backups, dead-letter stores, error messages, and support exports all belong inside the evaluation boundary when they can contain sensitive values or mappings.

The normal path rarely exposes every storage location. Trigger a timeout, a provider error, a parser failure, a retry, and a downstream rejection. Inspect what each component records. Check whether original values, substitutes, mappings, or reconstructed output appear in monitoring tools and whether access or retention differs from the production application.

The review should also cover data that persists by design. Ask how mappings expire, backups are protected, keys or policies rotate, and the system responds when a mapping is unavailable. Recovery that requires broad access to the mapping may create a different trust boundary from the live service.

Presidio's Anonymizer documentation notes that operators have different reversibility properties and that consistent hashes across calls require deliberate salt management. This is not an LLM Capsule claim; it shows why a review must make state, keys, and cross-call behavior explicit.

Which evidence should each owner provide?

Each control needs an accountable owner and an observable failure signal. A single security sign-off cannot replace evidence from the data, AI platform, application, privacy, and operations owners because their acceptance criteria differ.

The table is a starting contract, not proof. Replace generic evidence names with actual system artifacts. If a team cannot produce one item, record the gap, compensating control, owner, and deadline rather than marking the criterion complete.

How should the acceptance test run?

Run the acceptance test on one workflow that represents the real decision, not a sanitized demo. Use a bounded record with known operational markers, an approved model path, and a defined output destination. Document what the task must preserve before testing the data path.

A useful sequence is:

1. Capture the original input and required relationships.
2. Inspect the protected working version and outbound model request.
3. Run the normal request and validate task-level context.
4. Perform Reconstruction through the intended caller and policy.
5. Inspect the reconstructed output and downstream write.
6. Repeat with denied access, a malformed substitute, a fabricated substitute, a timeout, a retry, and a downstream failure.
7. Inspect logs, traces, queues, caches, errors, backups, and audit events.
8. Record the owner, result, limitation, and rollback action for each criterion.

Use the LLM Capsule architecture to map product components and the proof page to identify claims that need evidence. Neither page replaces a workflow-specific test. Base approval on the actual deployment artifacts and results.

Make approval depend on the full round trip

Approve a proposed context-preserving data layer for AI only when the team can explain and test the full path from the original record to the reconstructed output. The evidence must cover context, authorization, telemetry, failures, retention, and recovery. A clean model request is necessary, but it is only one part of the decision.

Use the framework as a living control record. Re-run the affected tests when the model endpoint, substitution policy, mapping design, output destination, observability stack, or retention rule changes. Approval should follow the architecture that is operating now, not the diagram that was reviewed months earlier.