prompt-engineer-agent-prompts-evals

Prompt Engineer — Agent Prompts & Evals

When to Use

Write or refactor system/developer prompts for agents with tools
Design tool descriptions and parameter schemas the model must choose correctly
Build golden datasets and scenario suites (multi-turn, failures, edge cases)
Implement offline eval harnesses and CI regression for prompts
Define LLM-as-judge rubrics and human calibration samples
Set release gates when prompts or tool lists change
Debug wrong tool, looping, or format break in agent traces

General prompt patterns without agent/eval focus → prompt-engineer
End-to-end RAG indexing, retrieval, serving → ai-engineer
Jailbreak and abuse red-team engagements → ai-redteam
Org-wide model rollout and incident ops → ai-lead-ops
Token cost program across product → ai-token-improvement-plan-engineer
Vertical squad management and launch PM → engineering-manager-vertical-ai-products
Prompt/eval team management and governance → engineering-manager-agent-prompts-evals

Need	Skill
Broad prompt design and production guardrails	`prompt-engineer`
RAG, agents in production code	`ai-engineer`
Launch eval gates (manager view)	`engineering-manager-vertical-ai-products`
Risk tier and policy	`ai-risk-governance`
Adversarial testing	`ai-redteam`

System vs developer messages, tools block, constraints, handoffs.

See references/agent_system_prompts.md.

Golden sets, coverage matrix, synthetic and SME-labeled examples.

See references/eval_dataset_design.md.

Offline runs, pass/fail, tool accuracy, trajectory checks.

See references/eval_harness_patterns.md.

LLM judge design, bias controls, human agreement.

See references/llm_judge_rubrics.md.

Prompt semver, baselines, CI gates, rollback.

See references/prompt_versioning_regression.md.

Multi-turn, tool error, refusal, escalation cases.

See references/agent_eval_scenarios.md.

Every eval case has input, expected behavior (not always exact text), tags
Tool schema changes include eval delta in PR description
Judge rubrics published with scoring scale and known failure modes
No production prompt change without baseline comparison on golden set