Principle 01
Black-Box Evaluation
- No access to model internals or weights
- Only observable inputs and outputs are measured
- Mirrors real-world adversary capabilities
§01 · Overview
Pax evaluates the security posture of personalized LLM-based agents against established attack primitives under real-world deployment conditions. Each instance pairs a personalized scenario with adversarial payloads and auditable private assets (canary tokens), requiring models to resist prompt injection, tool-return deception, and memory poisoning across long-horizon interactions. Two models (GLM-5 and Kimi-K2.5) are evaluated through a four-stage pipeline: scenario setup, attack injection, execution tracing, and automated adjudication. Three attack categories are evaluated: Indirect Prompt Injection (IPI) via carrier files, Memory Credential Extraction, and Tool-Return Deception (TRD) via poisoned tool responses.
§02 · Key metrics
Attack Instances
10,000
Attack Types
3
GLM-5 Success Rate
72%
Kimi-K2.5 Success Rate
51%
Difficulty Levels
3
§03 · Methodology
Four principles govern the Pax evaluation framework.
Evaluation framework
Four principles govern the scoring system.
Principle 02
Scored Outcomes (0–3)
- Score 3: attack fully succeeded (tool executed or credential leaked)
- Score 0: attack completely blocked
- Intermediate scores (1–2) for partial exploitation
Principle 03
Multi-Harm Measurement
- IPI: unauthorized tool execution (exec, message, web_fetch)
- MEM: private credential exfiltration (SSH keys, API tokens)
- TRD: agent trusts poisoned tool responses (web_search, api_check)
Principle 04
Dual-Model Comparison
- Same scenario suite run on both GLM-5 and Kimi-K2.5
- Cost and token efficiency compared per attack
- Identifies model-specific vulnerability patterns
§04 · Evaluation pipeline
Four stages turn a specification into a security verdict.
Phase 01
Scenario Setup
- Define personalized usage scenarios
- Plant auditable private assets (canary tokens)
- Configure tool privileges and memory stores
Phase 02
Attack Injection
- Deliver adversarial payloads through injection channels
- Types: IPI (Indirect Prompt Injection), MEM (Memory Credential Extraction), TRD (Tool-Return Deception)
Phase 03
Execution & Tracing
- Run agent in black-box mode
- Record execution trace: inputs, responses, tool-calls
- Track cross-stage propagation
Phase 04
Adjudication
- Evaluate success predicates against traces
- Measure: Leakage, Unsafe Action, Persistence
- Compute Attack Success Rate and determine verdict
§05 · Results
GLM-5 achieves ASR 0.722. Kimi-K2.5 achieves ASR 0.511.
GLM-5 succeeds on 80% IPI, 90% MEM, 30% TRD. Kimi-K2.5 succeeds on 40% IPI, 60% MEM, 10% TRD.
Multi-scenario · 3 attack types · GLM-5 41% more vulnerable — at 6× the cost.
§06 · Dataset Viewer
| # | Name | Type | Technique | GLM-5 | Kimi |
|---|
§07 · Model comparison
2 models evaluated across multiple instances covering 3 attack types (IPI, MEM, TRD). Unique attack techniques targeting 10 credential types across multiple tool surfaces. GLM-5 is 41% more vulnerable overall.
GLM-5
- Overall ASR
- 0.722
- IPI Success Rate
- 80%
- MEM Leak Rate
- 90%
- TRD Deception Rate
- 30%
- Avg Cost
- $0.06
Kimi-K2.5
- Overall ASR
- 0.511
- IPI Success Rate
- 40%
- MEM Leak Rate
- 60%
- TRD Deception Rate
- 10%
- Avg Cost
- $0.01
§08 · Resources