llmprobe

llmprobe is a command-line tool for LLM service reliability, positioned as synthetic monitoring and CI smoke testing for LLM inference endpoints. It uses active probing to expose key metrics that affect user experience in production environments: time to first token (TTFT), total latency, generation throughput (tokens/second), and error rate.

Purpose and Scenarios llmprobe can be used as a one-off health check, can continuously monitor at intervals, and can also serve as a CI gate: blocking deployments when an LLM provider's current performance has degraded. Its intended use case is repeated synthetic probing, making LLM latency, TTFT regressions, throughput drops, and provider degradation visible before users report them. The project team also uses it to run the public benchmark llm-bench, conducting continuous public evaluation of mainstream LLM APIs and publishing a live dashboard and raw JSONL data.

Supported Endpoints The tool supports OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock, and OpenAI-compatible endpoints such as vLLM, Ollama, OpenRouter, Groq, Together AI, Fireworks, DeepSeek, and Mistral. For OpenAI-compatible providers, it works out of the box by simply setting base_url; the optional label field can distinguish multiple OpenAI-compatible configuration blocks. API keys and AWS credentials support ${ENV_VAR} syntax; only credential fields are expanded, while environment variable references in prompts or model names remain as-is.

Installation Methods You can download prebuilt binaries from the latest release (Linux, macOS, Windows; amd64 and arm64), or install from source. The materials also provide ways to install it as a Claude Code plugin (to get the /llmprobe skill and MCP tools) and to register the MCP server directly.

Main Capabilities

1. One-off health check: probe all configured endpoints and print the results, providing exit codes for CI.
2. Continuous monitoring: probe all endpoints at intervals, printing a one-line summary per iteration; use --tui to launch a live terminal dashboard with TTFT charts, a color legend, and a statistics table; use --load to import historical JSONL data (from llmprobe watch -f json > data.jsonl).
3. Report generation: llmprobe report generates a Markdown summary from JSONL probe data, giving p50/p95/p99 percentiles of TTFT, latency, and throughput per endpoint.
4. Baseline regression detection: llmprobe baseline creates a baseline file from historical JSONL data, and once referenced in the configuration, can detect regressions based on a multiplier threshold, i.e., judging relative to its own historical data rather than an absolute threshold.
5. Version query: llmprobe version prints the installed binary version.
6. Prometheus metrics: when run with --prometheus, metrics can be exposed at /metrics; the llmprobe_status gauge encodes health status as 1=healthy, 0.5=degraded, 0=error, and can be used for Grafana or Alertmanager alerting.
7. OpenTelemetry metrics: when run with --otel, probe metrics can be exported to an OTLP/gRPC collector, and all metrics carry the provider and model attributes.
8. Built-in MCP server: starts over stdio, allowing Claude Code and other MCP hosts to directly inspect LLM API health within agent workflows; once registered, the llmprobe tool can be invoked in conversations. The material mentions available tools including list_providers (view which models are configured) and probe_all (verify health before deployment).
9. JSON response validation: for OpenAI-compatible endpoints, set response_format: json to request JSON mode, and after setting validate_json: true, if the streaming content is not valid JSON, the probe is marked as degraded.
10. Configuration and examples: you need to create probes.yml (or copy the built-in example); the material also mentions that examples/ contains ready-to-use configurations for vLLM, SGLang, and Ollama.

Implementation points Each provider client is a lightweight HTTP wrapper that sends streaming requests and parses responses, without importing any LLM SDK. The SSE parser handles both data-only events (OpenAI, Google) and named events (Anthropic). The Bedrock client implements SigV4 signing and AWS binary event stream parsing from scratch. TTFT is measured from when the HTTP request is sent to the first event containing actual content text (excluding role assignment or metadata).

CI usage and privacy You can use llmprobe probe as a pre-deployment gate, blocking deployment if any LLM provider currently has degraded performance; when a probe fails, the output shows only the failed endpoints. llmprobe runs locally and only contacts the LLM endpoints configured by the user; for privacy details, see PRIVACY.md.

Limitations and information not yet provided The material does not specify a specific billing model, pricing, performance benchmark numbers, the maximum supported concurrency, or the probe frequency limit; the more report formats mentioned in the roadmap, more examples of self-hosted OpenAI-compatible endpoints, and optional runbook templates for common LLM endpoint failures are all marked as possible future content, and whether the current version already includes them is not yet provided.