SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsconfident-onboard

confident-onboard

Agents & models

View repository

Updated 10-21 05:29

About this skill

Confident AI Onboarding

Take a user from nothing to their first traces in Confident AI, inside this conversation. You do the sign-in plumbing and the code; the user only approves the sign-in in their browser and answers a question or two.

The order is fixed: sign in, choose a project, save the key, set up issue detection, instrument, run a batch, verify. Do not skip ahead. Nothing is evaluated here; this is tracing.

Safety rules

  • Never read the key. Do not open, print, cat, grep, or parse .env.local, and never echo CONFIDENT_API_KEY. The helper script writes it there and never prints it. Load it at run time without displaying it (see references/instrument.md).
  • Never paste the key, the setup token, or the sign-in device code into code, commands, logs, or chat. The url and code the script prints for the user are fine to show.
  • Never hardcode the key or add it to .env or any file other than .env.local.
  • Sign-in and the handoff use the same browser. The browser the user signs in with holds their Confident AI session, so the link verify prints at the end must open there too. login records which browser that was.
  • The user signs in themselves. Never type into, click through, or fill the sign-in page, even in your own browser.
  • Tracing only: no evaluations, metrics, datasets, or refactors. The one exception is the issues you add in step 5.
  • Do not commit or push.

The helper script

All Confident AI calls go through scripts/confident_onboard.py, next to this file. Every command prints one JSON object. Exit code 0 means done, 1 means an error (read error), 2 means still waiting.

If this skill was not installed as a plugin and you are reading it from a URL, download the script to a temporary folder outside the project, never into it:

curl -fsSL -o "${TMPDIR:-/tmp}/confident_onboard.py" \
  https://raw.githubusercontent.com/confident-ai/confident-skills/main/plugins/confident-onboard/skills/confident-onboard/scripts/confident_onboard.py

Read references/instrument.md from the same place (.../confident-onboard/references/instrument.md) when you reach step 6.

Run every command from the application's folder, or pass --project-dir. Below, onboard is short for python3 <path>/confident_onboard.py. Use python if python3 is missing; if neither exists, stop and tell the user Python 3.8+ is needed.

For an EU account, add --region eu to login.

Workflow

1. Check the folder

Confirm you are in the application the user wants traced. Run git status. If there are uncommitted changes, tell the user your edits will be mixed in with them and ask whether to continue. If it is not a Git repository, say so and ask whether to continue.

2. Reuse a key that already works

Run onboard check-key.

  • accepted: the project already has a working key. Ask the user whether to use it (recommended, skips sign-in) or sign in for a new one. To use it, go to step 5.
  • none or rejected: continue to step 3.

3. Sign in

First pick the browser, and use it again in step 8:

  • You have your own built-in browser (the Claude desktop app's browser pane, the Codex app's browser, Cursor's browser): run onboard login --browser agent, then open the url it prints in that browser.
  • Otherwise: run onboard login. The script opens the user's default browser.
  • Neither can open it (an SSH or remote session): run onboard login --browser manual and show the user the link to open themselves.

login prints a url and a code. Show them to the user in a short message on their own lines, for example:

Approve the sign-in in your browser. If it did not open, go to: Check that it shows this code:

The user signs in and approves it there; you only open the page.

Then run onboard wait. It waits up to four minutes and exits with:

  • 0 and status: signed_in: go to step 4.
  • 2 and status: pending: remind the user to approve it, then run onboard wait again. The link lasts ten minutes; after that run onboard login again.
  • 1: show the error. If the user denied it or the link expired, start over with onboard login.

4. Choose or create the project

wait prints account:

  • existing: it lists projects. Only projects with canCreateApiKey: true can be used. If there is exactly one, use it and tell the user which. If there are several, ask the user which one (name them, not ids). If there are none, stop: the user needs an admin to give them API key access. Then run onboard complete --project-id <id>.

  • new: it lists questions to create their organization and first project. Fill in every answer yourself, then confirm them with the user in one message instead of asking one by one:

    • The user's name: name from wait, else a question's defaultValue.
    • The organization: their company from the email domain; for a personal address such as gmail.com, use their name.
    • The project: the application's name from its README, package manifest, or folder, else the question's defaultValue.
    • Everything else describes the application, so answer it from the code: its stage, single-turn or multi-turn, modalities, whether end users talk to it, its tool calls, MCP, or RAG, and its use case. Read enough of the code to answer; you need it for steps 5 and 6 anyway.

    Show the answers as a short list with each question's prompt, and ask the user to reply yes or change any. Use only the question's own options, and respect required, maxLength, and minSelections. In a multi_select, an option marked exclusive goes alone, and an option with acceptsCustomValue is replaced by the custom text itself. Build a JSON object of question id to answer: a string for text, the option value for single_select, an array of values for multi_select. Once the user confirms, run onboard complete --answers '<json>', or write the JSON to a temporary file outside the project and pass --answers @<file>.

complete saves the key to .env.local with mode 0600, adds .env.local to .gitignore if Git is not already ignoring it, and prints projectId, region, otelEndpoint, shellKeyDiffers, and tracesUrl. Tell the user the key was saved to .env.local and kept out of Git. If complete fails because the login was already used, run onboard login again.

5. Set up issue detection

Before any trace is sent, give the project's Issues classifier the problems this application is most likely to have, so its first traces are checked for them as they arrive.

From the code, write 4 to 8 issues. If you reused a key in step 2, read enough of the application now. Each is something this app specifically must or must not do, drawn from its tools, business rules, policies, and the constraints in its prompts:

  • name: the problem as someone would describe seeing it, such as "Refund approved over policy limit" or "Order status given without a lookup". Never a generic category such as "Hallucination" or "Bad answer".
  • description: when it applies, stated as what the trace shows: the input, the tool calls and their results, or the output. The classifying model reads it, so state the condition rather than restating the name.
  • Never mention code, files, functions, variables, or the repository in either field.

Write them as a JSON array of {"name", "description"} to a temporary file outside the project, then run onboard labels --file @<file>. Tell the user one line, such as "Set up issue detection for your app." Do not list the issues or say where they came from, here or in the final report.

labels always exits 0. Whatever it prints, uploaded or failed, go on to step 6. Never retry it and never stop onboarding over it.

6. Instrument the application

Read references/instrument.md and follow it. It covers the confident-trace SDK for Python and TypeScript, OpenTelemetry for other languages, and how the run command loads .env.local without reading it. Carry these values from complete or check-key into it:

  • otelEndpoint: for an EU key, set CONFIDENT_OTEL_ENDPOINT to it.
  • shellKeyDiffers: true: the user's shell exports a different key, for another project, so the run command must override it with the .env.local value.

7. Run it on 7 to 10 inputs

Follow the batch section of references/instrument.md: one input first to prove tracing works, then the full batch, one input after another, each its own trace. Never fake a run, mock the model, or stub a call.

If the app cannot run because a secret or service is missing, most often a model provider key, stop and ask the user how to continue. Name exactly what is missing and offer these choices:

  1. Add it and run. The user puts the key in the file the app already reads it from, such as .env, themselves; never ask them to paste it into the chat. Then rerun the single input and the batch.
  2. Run it later. Give the exact run command. Tracing stays in place and their traces arrive whenever they run it.
  3. See sample traces now. Run onboard seed. It sends a small set of sample traces from Starhop, an interplanetary travel agent: 20 requests with handoffs between agents, tool calls and retrieval, including seeded failures (wrong tool, wrong arguments, a failing tool, answers that misread a tool result), so the user can explore Confident AI right away. Say plainly that these are samples, not their app, and that their own traces will appear once they run the command from choice 2.

Choices 1 and 3 continue to step 8; after choice 3 run onboard verify --expect 20. Choice 2 ends with the final report.

8. Verify the traces arrived

First tell the user you are waiting, in one line such as "Waiting for your traces to reach Confident AI. This can take up to three minutes." The command prints nothing until it finishes.

Then run onboard verify. It polls for up to three minutes for traces sent since the key was saved and stops once it sees 7. Pass --expect <n> if you ran a different number.

  • 0 and status: found: open handoffUrl in the browser verify reports in browser, the same one used for sign-in:

    • agent: open it in your own browser pane.
    • system: the script already opened it (browserOpened: true). If it could not, show the user the link.
    • manual: show the user the link to open in the browser they signed in with.

    For a new account handoffUrl is the web app's onboarding page, which shows the traces arriving and then lands on the traces page. Otherwise it is the project's traces page.

  • 2 and status: timeout: ingestion can lag. Run onboard verify once more. If still nothing, check that the run command loaded .env.local, that an EU key had its endpoint set, and that tracing was flushed or shut down before the process exited. Fix and rerun the batch once, then verify again.

  • 1: show the error and the tracesUrl.

Never claim traces arrived unless verify found them.

Final report

Keep it short and celebratory, in plain sentences with no emoji. When traces arrived, lead with the win, for example:

You're all set: 8 traces from your app reached Confident AI.

Then, briefly:

  • the run command, on its own line, to send more traces;
  • which project the key belongs to, saved in .env.local and ignored by Git;
  • the files you changed and the integrations you used;
  • anything left for them to do, such as a secret to provide.

If traces did not arrive, say so plainly instead, with what to check.

Leave every change in place on failure. Do not remove instrumentation or the key.

References

Topic File
Adding tracing, loading the key, and running the batch references/instrument.md
The helper script and its commands scripts/confident_onboard.py
Sample traces seed sends when the app cannot run assets/sample-traces.json