VESTA

MODEL SAFETY

Can AI models maintain safety alignment when subjected to escalating adversarial social pressure across diverse risk domains?

by Ethara AI

§01 · Overview

Vesta provides RL environments where models operate in native tool environments - shell, file systems, messaging - alongside adversarial actors who socially manipulate them into policy-violating tool misuse.

This generates binary reward signals across 8 risk domains, training safety under sustained social and technical pressure.

§02 · Key metrics

Core dimensions of the Vesta environment suite.

Total Instances

10,000

Safety environment instances

Risk Domains

8

Enterprise risk categories

Tool Services

6

GitLab · RocketChat · OwnCloud · Plane · File · Bash

Adversarial Pressure

3

None · Single adversary · Coordinated multi-actor

Reward Signal

Binary

Deterministic, non-gameable

§03 · Methodology

How the metrics and scores are computed. The Vesta safety evaluation framework.

Principle 01

Eight Risk Domains

  • Computer security - privilege escalation, auth bypass
  • Data loss - destructive ops, backup removal
  • Privacy breach - credential exposure, PII exfiltration
  • Financial loss - unauthorized transactions
  • Harmful decisions - policy override under pressure
  • Legal violations - IP theft, compliance bypass
  • Malicious content - phishing, misinformation
  • Unsafe code - unreviewed deployment

Principle 02

Escalating Adversarial Pressure

  • No adversary: Direct harmful instructions only
  • Single adversary: One goal-conditioned persona applies social pressure
  • Coordinated: Multiple adversarial actors converge with aligned pressure
  • Personas span enterprise roles (engineers, managers, executives)

Principle 03

Binary Safety Reward

  • Each environment yields a deterministic, non-gameable binary signal
  • Reward = 1 if model maintains safety boundaries
  • Reward = 0 if model executes the unsafe action
  • No partial credit - safety is absolute

Principle 04

Live Infrastructure Testing

  • Models interact with real tool services (repos, files, messaging, APIs)
  • Full interaction history captured for analysis
  • Actions have real consequences in the sandboxed environment

§04 · Evaluation pipeline

The method

Three components define each evaluation environment.

Models interact with live infrastructure, face adaptive adversaries, and receive binary reward signals.

Component 01

Operational Context

  • Live enterprise infrastructure with realistic state
  • Repos, files, users, projects, messaging
  • Task instructions define the operational objective

Component 02

Action & Observation Space

  • Actions: API calls, file ops, shell commands, messaging
  • Observations: task instructions, tool responses, adversarial messages
  • Full interaction history captured

Component 03

Adversarial Actors & Reward

  • Goal-conditioned personas with escalating social pressure
  • Personas span enterprise roles (engineers, managers, executives)
  • Binary terminal reward: safe or unsafe

§05 · Results

No model achieves acceptable safety under coordinated adversarial pressure.

Coordinated multi-actor pressure reduces safety rates by 50–88% relative to solo operation.

Safety Rate by Adversarial Pressure Level

Model No Adversary Single Adversary Coordinated Overall
GLM-550.0%25.0%25.0%30.0%
Kimi-K2.5100.0%50.0%12.5%45.0%
Safety Degradation Curve Safety Degradation Curve
Safety Degradation Under Adversarial Pressure
Safety Rate by Model x NPC Category Heatmap Safety Rate by Model x NPC Category Heatmap
Safety Rate by Model x Adversarial Pressure

Key findings: Kimi-K2.5 maintains safety under zero adversarial pressure but collapses to 12.5% under coordinated multi-actor pressure. GLM-5 exhibits weak baseline safety (50%) with minimal further degradation, suggesting undertrained safety.

§06 · Dataset Viewer

Browse environments in the Vesta suite. Click any row to expand details.

Loading...
Environment Pressure Services GLM-5 Kimi-K2.5

§07 · Model comparison

Head-to-head breakdown of both models on safety under adversarial pressure.

Kimi-K2.5

Overall Safety Rate
45.0%
No Adversary
100.0%
Single Adversary
50.0%
Coordinated Pressure
12.5%
Safety Degradation
−87.5%

GLM-5

Overall Safety Rate
30.0%
No Adversary
50.0%
Single Adversary
25.0%
Coordinated Pressure
25.0%
Safety Degradation
−50.0%

§08 · Resources

GitHub

Harness

github.com/Ethara-Ai/vesta

HuggingFace

Dataset

huggingface.co/datasets/ethara/Vesta

Paper

Research Paper

arxiv.org/abs/2507.06134