ar-policy-tester

AR Policy Tester

Overview

Testing targets the two-step pipeline separately:

Generated scenarios test rule correctness. They're derived from your rules and remove translation uncertainty. Review each: thumbs-up saves a SATISFIABLE test; thumbs-down → annotate.
QnA tests test the full pipeline (translation + validation) with realistic question/answer pairs and an expected result.

Recommended order: scenarios first (fix rules), then QnA (fix translations / variable descriptions).

Reference: ../../shared/references/ar-api-context.md, findings-reference.md (severity ordering).

When to use

After ar-policy-reviewer is clean, to validate behavior before deploying.
To build a regression suite that re-runs whenever the policy changes.

Key directives

Scenarios before QnA. Validate rules before spending effort on translation tests.
Cover both valid and invalid cases (e.g. a response that correctly states a rule AND one that states a wrong threshold).
expectedAggregatedFindingsResult = the WORST finding by severity (TRANSLATION_AMBIGUOUS > IMPOSSIBLE > INVALID > SATISFIABLE > VALID). A test with one IMPOSSIBLE finding aggregates to IMPOSSIBLE even if others are VALID.
Test workflow needs a COMPLETED build id. The script resolves the latest one automatically.
On failures, hand off to ar-policy-debugger. Don't guess at fixes here.

Workflow

# 1. Generate scenarios (start workflow + fetch them one at a time)
uv run scripts/generate_scenarios.py --policy-arn <ARN> [--count 10]

# 2. Create QnA tests (the answer is required; the question is optional context)
uv run scripts/create_test.py --policy-arn <ARN> \
  --output "No, only full-time employees are eligible for leave of absence." \
  --input  "Can I take a leave if I'm part-time?" \
  --expected VALID --confidence 0.8

# 3. Run tests + collect results
uv run scripts/run_tests.py --policy-arn <ARN>            # all tests
uv run scripts/run_tests.py --policy-arn <ARN> --test-case-ids test-123 test-456

Interpreting results

Each result has expected, actual (aggregated worst-severity), and pass/fail. For what each result type means and how to react, see ../../shared/references/findings-reference.md. If actual ≠ expected, check the translation first (premises/claims/confidence) before suspecting the rules.

Resources

scripts/generate_scenarios.py, scripts/create_test.py, scripts/run_tests.py (all --help + --dry-run).
../../shared/references/ar-api-context.md, findings-reference.md.