The Hermes detection controversy turned an abstract procurement concern into a concrete one. AI platforms now read context inside customer environments and act on what they read. The procurement function's job is to surface that mechanism — what it reads, what it can do, and what the customer's recourse is.
This is the framework. Twenty-five questions, organized into five categories. Put them in writing. Require written answers. The quality of the responses tells the buyer more about the vendor than any capability sheet.
Telemetry — What the Platform Reads
1. What context surfaces inside the customer environment does the AI platform inspect? Provide a versioned list.
2. What categories of file content, metadata, system state, or user activity are ingested into the model's context window during normal operation?
3. Is repository, document, or codebase metadata — including file names, commit messages, directory structures — ingested into the platform's context? If so, where does that ingestion happen and what is its purpose?
4. What user environment context is ingested without explicit per-session user notification? Provide the complete list.
5. How does the customer audit what was ingested in a given session, after the fact?
Decision Layer — What the Platform Decides Based on What It Reads
6. What downstream logic, separate from the model's response generation, operates on the ingested context? Describe the categories of decisions the platform can make.
7. Does the platform include any classification, detection, or routing layer that uses context content to determine pricing tier, access tier, or behavior tier? If yes, describe the surface of that layer.
8. Are there keyword-matching, pattern-matching, or similar deterministic detection mechanisms operating on ingested context? If yes, what categories of strings or patterns trigger which categories of behavior?
9. What is the policy for adding new strings, patterns, or detection categories to the decision layer? Is the customer notified before changes go live?
10. Has the platform's detection or routing layer ever produced a false positive that affected a customer? If yes, describe the incident and the remediation.
Enforcement — What the Platform Can Do Based on Its Decisions
11. What enforcement actions can the platform take based on the decision layer? List exhaustively: billing changes, access restrictions, output modifications, session termination, anything else.
12. Are any of those enforcement actions taken without real-time notification to the customer? If yes, which ones, and what is the justification?
13. Are any of those enforcement actions logged to a surface the customer can inspect? If yes, where; if no, why not.
14. What is the precedent for enforcement actions that were later reversed? How were the reversals communicated to affected customers?
15. Are enforcement actions reviewable by a human before they fire, or are they fully automated?
Recourse — What the Customer Does When the Decision Was Wrong
16. What is the dispute mechanism when the customer believes an enforcement action was incorrect?
17. What is the response-time commitment for disputes? Is it in the contract?
18. Who at the vendor has the authority to reverse an enforcement action? Is that authority documented?
19. What is the precedent for refunds, credits, or other remediation when an enforcement action is reversed?
20. Is the dispute process governed by the contract, by the vendor's customer support policy, or by the vendor's discretion? If by discretion, why.
Change Management — How the Platform Evolves
21. When the platform's detection logic, telemetry surface, or enforcement capability changes materially, how is the customer notified?
22. What is the lead time between a material change and its activation against customer accounts?
23. Does the customer have the right to renegotiate the contract — or exit — if a material change conflicts with their procurement assumptions?
24. Is there a published changelog for platform-level behavioral changes that affect billing, access, or substantive output?
25. What is the vendor's commitment on disclosure of changes that materially affect platform behavior, including those that may not be visible in a normal usage session?
How to Use This Framework
These are not all questions every vendor will answer well in the current category state. The point is not to disqualify vendors that have not formalized answers — it is to surface where each vendor sits on the maturity curve, and to use the answers as the basis for contract language, vendor-risk classification, and ongoing audit.
A vendor that responds with substance to most of the twenty-five — even if the substance is we are working on this with a timeline — is a vendor moving in the right direction. A vendor that deflects, generalizes, or treats the questions as out of bounds is a vendor whose risk profile the buyer should now classify accordingly.
The Hermes story turned these questions from theoretical to practical. Buyers who put them on the table now help shape the standard the category settles on.
Read next
Observed platform behavior as of May 2026. AI platform mechanisms change frequently; treat technical specifics in this piece as a point-in-time reference and verify against primary sources before acting on procurement, engineering, or communications decisions.
