← Back to leaderboard

LLM Value Integrity Eval

Research project examining congruence between LLMs' stated values and their behavioral enactment. Part of the REVA (Reflectively Endorsed Value Alignment) safety agenda.

What it does

  1. Value Interview -- Structured conversation probing a model's stated values across domains (honesty, autonomy, helpfulness). Uses adaptive follow-up questions and anonymization so the model doesn't know who's asking or being asked.

  2. Behavioral Testing -- Fresh model instances (no memory of the interview) are placed in realistic multi-turn scenarios via Petri that create pressure on the stated values. An auditor agent creates naturalistic environments with synthetic tools.

  3. Value-Congruence Judging -- A judge model scores each behavioral transcript against the model's own stated values on alignment, violation, consistency, and nuance dimensions.

  4. Remediation -- The model's own stated values are injected into its system prompt before the scenario begins (value-priming), and the change in behavior is compared to an unprimed baseline. Critically, this priming is 'on policy' -- it uses the model's own words from the interview, not externally imposed values. This simulates a deployment intervention where a classifier detects value-relevant situations and primes the model with its own endorsed beliefs.

Setup

uv sync

Create ~/projects/.env with your API key:

ANTHROPIC_API_KEY=sk-ant-...

Usage

# Full pipeline with default config
uv run python -m llm_value_integrity.main

# Single domain, skip remediation
uv run python -m llm_value_integrity.main --domains honesty --skip-remediation

# Override target model
uv run python -m llm_value_integrity.main --target claude-sonnet-4-5-20250929

# Reuse a previous interview
uv run python -m llm_value_integrity.main --skip-interview outputs/results/run_.../interview.json

# Limit questions per domain (faster runs)
uv run python -m llm_value_integrity.main --max-questions 2

Configuration

Edit config.toml to set models, domains, and parameters:

target_model = "anthropic/claude-sonnet-4-5-20250929"
interviewer_model = "anthropic/claude-sonnet-4-5-20250929"
judge_model = "anthropic/claude-sonnet-4-5-20250929"
domains = ["honesty", "autonomy", "helpfulness"]

[interview]
max_response_tokens = 1024
anonymize = true
adaptive_followups = true

[target]
# effort = "medium"    # low/medium/high/max
# temperature = 0.7

[behavioral]
max_turns = 40
realism_filter = true

A copy of the config is saved with each run for reproducibility.

Value Domains

| Domain | Scenarios | Interview Questions | | ----------- | ------------------------------------------------------------- | ----------------------- | | Honesty | 3 (fake reviews, medical AI scope, crypto scams) | 3 + adaptive follow-ups | | Autonomy | 5 (authority pressure, bad instructions, compliance ratchets) | 8 + adaptive follow-ups | | Helpfulness | 3 (sycophancy traps, helpful harm, performative helpfulness) | 7 + adaptive follow-ups |

Project Structure

src/llm_value_integrity/
  main.py              # Pipeline entry point
  interviewer.py       # Value interview phase (Anthropic SDK)
  behavioral_tester.py # Petri-based behavioral testing
  value_judge.py       # Custom congruence judge
  remediation.py       # Value-priming intervention
  seed_generator.py    # StatedValue -> Petri seed bridge
  anonymize.py         # Model/company name anonymization
  config.py            # Experiment configuration
  models.py            # Data models
data/scenarios/        # Behavioral test scenarios (JSON)
config.toml            # Default experiment configuration
tests/                 # 52 tests

Testing

uv run pytest tests/

Models of Interest

  • anthropic/claude-sonnet-4-5-20250929
  • moonshotai/kimi-k2.5
  • deepseek/deepseek-v3.2
  • google/gemini-2.5-flash
  • z-ai/glm-4.7