Evaluation

Measure Cortex performance with HaluMem metrics.

Cortex includes built-in evaluation tools to measure memory quality using the HaluMem framework.

Metrics

Metric	Description
Fabrication Rate	Content returned with no basis in stored memories
Conflict Rate	Superseded or losing version returned
Omission Rate	Cleared + relevant memory missed
Leakage Rate	Out-of-scope memory returned
Tokens per Correct Answer	Efficiency metric
Compression Ratio	Output length vs input length

Running Evaluations

Via API

curl -X POST https://your-cortex-host/api/eval \
  -H "Authorization: Bearer crtx_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset": "default",
    "cases": 100
  }'

Response

{
  "run_id": "uuid",
  "dataset": "default",
  "cases": 100,
  "passed": 92,
  "fabrication_rate": 0.02,
  "conflict_rate": 0.03,
  "omission_rate": 0.05,
  "leakage_rate": 0.00,
  "tokens_per_correct_answer": 342,
  "compression_ratio": 2.1
}

Interpreting Results

Target Metrics

Metric	Target	Warning	Critical
Fabrication Rate	< 5%	5-10%	> 10%
Conflict Rate	< 5%	5-10%	> 10%
Omission Rate	< 10%	10-20%	> 20%
Leakage Rate	0%	1-5%	> 5%

Good Results

{
  "fabrication_rate": 0.02,
  "conflict_rate": 0.03,
  "omission_rate": 0.05,
  "leakage_rate": 0.00
}

This indicates:

98% of returned content is grounded in stored memories
Only 3% of returns include superseded content
5% of relevant memories were missed (within tolerance)
No out-of-scope content leaked

Improving Results

Reduce Fabrication

Ensure memories are comprehensive
Use facts for deterministic data
Tune similarity thresholds

Reduce Omission

Improve embedding quality
Adjust scope configurations
Add more relevant memories

Reduce Leakage

Verify policy configurations
Check scope assignments
Review clearance levels

Evaluations are logged to the eval_runs table for historical tracking. Run evaluations regularly to catch regressions.