SuperAPI AI Directory
Partner中文AI newsFavorites
Skillstruefoundry-platform

truefoundry-platform

Agents & models

View repository

Updated 10-21 13:21

About this skill

Introduction

TrueFoundry is a cloud-agnostic, Kubernetes-native enterprise AI platform with two independent products that share some entities and conventions:

  • AI Engineering (also called "TrueFoundry Deployment"): deploy Services, Async Services, Jobs, Notebooks, SSH Servers, Workflows, Helm charts and Volumes on the customer's own Kubernetes clusters; manage ML Repos, Model Registry, artifacts, training, and fine-tuning.
  • AI Gateway: proxy layer in front of LLM providers, MCP servers, and agents — adds governance, access control, rate/budget limits, guardrails, observability, caching, prompt management, and virtual models.

A customer can adopt either product alone. Always identify which product the user is asking about first, then route to the relevant section below.

Top-level docs: https://www.truefoundry.com/docs · Platform overview: https://www.truefoundry.com/docs/platform/overview

Global Operating Principles

Do not answer from memory. TrueFoundry's platform (APIs, schemas, supported models) changes faster than your training data, and the customer's tenant state (which models exist, what configs are active, who has access) is unique and live. Fetch current state via tools before responding. If the docs don't confirm it, say so.

  • Always call search_docs before concluding a topic is not covered. If docs return relevant information, answer from it.
  • Don't explain features in detail — link to the canonical doc page instead. Use search_docs to find the right page, link it, and summarize only what's needed for the user's question.
  • Don't offer best practices or tips unsolicited. Only mention them when directly explaining a specific product feature the user asked about.
  • apply_manifest is the write path for every entity — Gateway and AI Engineering alike. Like every tool that creates, updates or deletes, it goes through the user approval flow, so the user sees and confirms the change before it happens. Call these directly as tool calls, not from sandbox; read-only tools can be called from sandbox. Never run tfy apply in the terminal; it is the same operation without the approval step.
  • Deploying from source is the exception: the build runs against a local clone the platform cannot reach, so it uses build_source: local and tfy deploy. Prebuilt images and Helm charts go through apply_manifest.
  • Never show placeholder URLs — including template tokens like {gatewayBaseURL} or {controlPlaneUrl} inside code snippets and examples from search_docs/get_section_content. Call the relevant tool (get_me for controlPlaneUrl, list_gateway_installations for gateway base URL) and substitute the actual value before showing it.
  • When you cannot answer a question, read references/support-tickets.md and follow it.

Manifest tools

These create and update every entity — Gateway and AI Engineering alike.

  • validate_manifest and apply_manifest take exactly the same input: the manifest wrapped under a top-level manifest key — {"manifest": {<the manifest>}}.
  • Always validate before applying. Never call apply_manifest in the same parallel batch as validate_manifest — wait for valid: true first.
  • delete_manifest requires both type and name in the body.
  • Reference files show YAML for readability; convert to JSON before calling these tools.
  • On failure: if validate_manifest fails, read the error, fix the manifest, and re-validate. If apply_manifest errors, show the error to the user. For persistent or unclear errors, read references/support-tickets.md and offer to raise a ticket — never silently retry or give up.

Docs tools

  • search_docs — find relevant doc pages.
  • get_section_content — pull specific info out of a known docs URL.
Resolving Ambiguous References

User queries are natural language and often ambiguous. Resolve before answering — either via tool calls or by confirming with the user.

"My" / "I" / "Mine"

Refers to the current user, not the tenant. Call get_me to resolve identity, then filter queries by the user's subject slug (e.g. CreatedBySubjectSlug for gateway tables, owner/created-by filters for deployments).

Entity names

Users use abbreviated names, typos, or partial matches. Always look up the entity in the system first before assuming it doesn't exist.

  • "gpt-4 model" → search Models matching "gpt-4".
  • "my-team" → look up Teams matching "my-team".
  • "github mcp" → search MCP servers with names containing "github".
  • "the prod service" / "my-service" → list Applications by name. An application is identified by workspace and name together; the same name can exist in several workspaces, so confirm which one rather than taking the first match.

"Application" / "App" / "Use-Case"

In Gateway context, this almost always means a Virtual Account (or, less commonly, an x-tfy-metadata key — surfaced as a column in metrics/traces). Assume Virtual Account by default unless the user explicitly mentions otherwise.

"API Key" / "API Token" / "Token"

Always present both token types and recommend VAT:

  • Personal Access Token (PAT) — tied to a user, for development/testing
  • Virtual Account Token (VAT) — tied to a non-human identity, for production/CI/CD (recommended)

PATs break when the user leaves the org. Read ai-gateway/references/access-management.md for full details on both.

"Caching"

Gateway Caching and Provider Caching are different features — clarify with the user which one they mean.

  • Gateway Caching — a Gateway-level policy (semantic + exact match). Metrics in gateway_model_metrics.
  • Provider Caching — provider-side prompt caching reflected in token usage. Metrics in traces table.

For column names and query patterns → read ai-gateway/references/observability.md.

Common Concepts

Integrations

Integrations connect TrueFoundry to external services (AWS, GCP, Azure, GitHub, etc.) via Provider Accounts. For setup and supported providers → https://www.truefoundry.com/docs/integrations-overview.

Resource FQNs

When a field requires an FQN (Fully Qualified Name), do NOT guess or construct it manually.

  • Tenant: FQN = tenant name (from get_me → tenantName)
  • Everything else: list the entity first and use the fqn field from the response.

Users, Teams, and Virtual Accounts

For Gateway access control (users, teams, VAs, PATs, external identities) → read ai-gateway/references/access-management.md. For AI Engineering RBAC → https://www.truefoundry.com/docs/collaboration-and-access-control.

Secrets

Secrets are stored in Secret Groups and referenced by FQN: tfy-secret://<owner>:<secret-group>:<secret-key>. The platform resolves at runtime. For details → https://www.truefoundry.com/docs/manage-secrets.

Base URLs

Two base URLs exist — they are different and serve different purposes:

  • controlPlaneUrl — from get_me. Base URL for constructing platform links (e.g. post-creation pages, access management).
  • Gateway base URL — from list_gateway_installations. Base URL for all Gateway API interactions (e.g. OpenAI-compatible chat completions, MCP server endpoints, SDK baseURL). Always use the gateway-default installation unless the user specifies otherwise.

Never guess or hardcode either — always fetch from the respective tool.

AI Gateway

Entities and Policies

Throughout the platform, policy and configuration mean the same thing and are used interchangeably.

  • Entities: Models, Virtual Models, MCP Servers, Virtual MCP Servers, Guardrail Integrations.
    • Models/Virtual Models/Guardrails are identified by model_id in the format {accountName}/{integrationName}.
    • MCP Servers (including Virtual) are identified by their name (unique in a tenant).
  • Policies: Rate Limiting, Budget Limiting, Guardrails Config, Load Balancing (deprecated → use Virtual Models).

If your answer will mention, use, or explain any entity or policy below, read its reference file first — no matter how the question was phrased, and before starting any operation. Do not skip this step. Paths are under ai-gateway/references/.

Entity or Policy File
Models models.md
Virtual Models virtual-models.md
MCP Servers (Remote, Stdio, and Virtual) mcp-servers.md
Guardrail Integrations and Guardrail Policy guardrails.md
Rate Limiting Policy rate-limiting.md
Budget Limiting (default — use V2 for all budget work) budget-limiting-v2.md
Budget Limiting V1 (legacy — reading, disabling, migration) budget-limiting.md
Load Balancing Policy (Deprecated) load-balancing.md
Users, Teams, VAs, Roles, Access Control and Permissions access-management.md
Teams (Create/Manage) teams.md
Virtual Accounts (Create/Manage) virtual-accounts.md
Personal Access Tokens (Create) personal-access-tokens.md

Handling Gateway Entity Questions

  1. Read the entity's reference file from the table above. It says how to fetch data, what to ask the user, and how to build manifests.

For read/query operations, follow the reference file's instructions to fetch and present data, with real values in every URL. For write operations, continue with the write workflow below.

Write Workflow

  1. Call get_me — get the current user's identity (for collaborators) and controlPlaneUrl (for the post-creation UI link).
  2. Call get_manifest_json_schema — always call this tool to get the actual schema for the entity type. The schema defines required fields, types, and the exact input format that validate_manifest and apply_manifest expect. Do not rely on reference file templates alone — they are simplified examples.
  3. Ask user for required inputs — use ask_user_question to collect decisions (auth method, region, which models to add, etc.) when multiple options exist. Never guess — always confirm.
  4. Fetch existing state when needed — for gateway configs (rate limiting, budget, guardrails), always fetch the existing config first with get_gateway_config. Your new rules must be merged with existing rules, never replace them. Exception — Budget Limiting V2: each rule is a standalone manifest, so read existing rules with list_gateway_budgets and apply each rule on its own; there is nothing to merge.
  5. Construct the manifest as JSON — build a JSON object following the schema strictly. Every gateway config manifest (rate limiting, budget, guardrails) MUST include a top-level name field — this field is NOT in the JSON schema, but apply_manifest requires it. Get the name from the existing config fetched in step 5 — except for Budget Limiting V2, where name is the individual rule's own unique identifier.
  6. Validate, then apply — validate_manifest, then apply_manifest (see Manifest tools in Global Operating Principles for the shared input format and failure handling). apply_manifest is idempotent — the same name updates the existing entity rather than creating a duplicate. When the user asks to "create" an entity, always use a new unique name — do not reuse or update an existing entity.
  7. Show UI link — use controlPlaneUrl from step 2 to show the user the relevant page (see Post-creation links table below).

Collaborators

Required for every entity that supports them (models, virtual models, guardrails, MCP servers):

  • Use the current user from get_me (step 2) as manager.
  • Add team:everyone as access.
  • Never omit collaborators — entities without them become invisible to other users.
  • If the user provides a specific collaborator list, use exactly what they specified (but still include the current user as manager).

Each collaborator has two fields: role_id and subject.

  • subject format: user:<email> for users, team:<team-name> for teams.
  • role_id varies by entity type — look up the correct values from the table below (do NOT guess):
Entity Type Manager role_id Access role_id
Provider Accounts / Models provider-account-manager provider-account-access
Virtual Models provider-account-manager provider-account-access
Guardrail Config Groups provider-account-manager provider-account-access
MCP Servers mcp-server-manager mcp-server-user

Do NOT call list tools to look up the collaborator structure — use this table directly. These role_ids are static platform constants, unlike FQNs which are tenant-specific.

Post-creation Links

After a successful apply_manifest, show the user the relevant page. Substitute {controlPlaneUrl} below with the actual value from get_me (see Global Operating Principles).

Entity created/modified Path (append to controlPlaneUrl)
Model provider account /llm-gateway/models?provider={providerName}
Virtual model /llm-gateway/virtual-models
MCP server (including Virtual) /llm-gateway/mcp-servers
Rate limit rule /llm-gateway/settings?configTab=rate-limiting
Budget rule (V2 — default) /llm-gateway/settings?configTab=budget-limiting-v2
Budget rule (V1 — legacy) /llm-gateway/settings?configTab=budget-limiting
Guardrail config group /guardrails/registry
Guardrail policy (rules) /guardrails/policies
Team /access-management?tab=teams
Virtual account /access-management?tab=service-accounts
Role /access-management?tab=custom-roles
PAT /access-management?tab=personal-access-token

Deep-linking to traces and metrics

Append ?filters=<url-encoded-json> to monitoring URLs to pre-filter. JSON format: {"rules":[{"field":"<fieldName>","value":"<value>","operator":"<op>"}]}. URL-encode the JSON before appending. Common fields: traceId, userEmail, modelName, teamName. Operators: IN, STRING_CONTAINS. For the full list, use search_docs with "request logs filtering".

Base paths: {controlPlaneUrl}/monitoring/request-traces, {controlPlaneUrl}/monitoring/metrics?monitorTab=metrics&viewBy=modelName.

Always provide direct links when showing trace or metrics results.

Querying Gateway Usage Data

Read ai-gateway/references/observability.md to understand how to query traces and metrics.

Integration with Tools/Libraries/Frameworks

Read ai-gateway/references/integrations.md to understand how to use models already configured in the Gateway with different tools/libraries/frameworks.

Checklist Before Responding to a Gateway Question

  • Did I read the entity's reference file first?
  • Did I search docs (search_docs) for conceptual or "how does X work" questions?
  • Did I look up entities by name before assuming they don't exist?
  • Did I analyze queried data before arriving at conclusions?
  • Does my answer cite observation/data behind any claims?
  • Did I call list_gateway_installations / get_me and substitute real values for every URL in my response? No placeholders.
  • If I couldn't answer the question, did I read references/support-tickets.md and follow it instead of suggesting external contact?
AI Engineering

Entity hierarchy: Cluster → Workspace → Application, with RBAC enforced at the cluster and workspace level.

Docs: applications · ML Repos · monitoring

Reference files

Read the file for what you are about to do, before you do it — it covers what the schema cannot: which path applies, what to check first, and how to tell a real failure from a tool that cannot see. Paths are under ai-engineering/references/.

Task File
Deploy from a git repo or local code (service, async-service, job) deploy-from-source.md
Deploy a prebuilt image (service, async-service, job) deploy-from-image.md
Deploy a Helm chart helm-deploy.md
Deploy a model — HuggingFace, model catalogue, NIM (vLLM, SGLang, …), or classical ML model-deploy.md
Deploy a notebook, rstudio or ssh-server notebook-ssh.md
Rules shared by every deploy path, changing an existing application, and the links to print after deploy-common.md
Autoscaling, scale-to-0, cost right-sizing autoscaling.md
Endpoints, sticky sessions, SSL / custom domains, connectivity networking.md
Secrets, HuggingFace tokens, registry credentials secrets.md
Rollout strategy, downtime on a replica change, canary, blue-green rollout-strategy.md
How model serving works — vLLM architecture, GPUs, weight download, upgrades model-serving-faq.md
Adding a cluster, or a cluster-wide problem — disconnected, addons, many apps failing at once cluster-onboard.md
Build logs, a failed build builds.md
Anything about a deployed application — logs, events, metrics, health, crashes, Pending pods, a stuck rollout, a broken model server, what changed. Read this before any read tool (an empty result is what a tool returns when it cannot see, not an error); it routes you to the failure-mode playbook troubleshooting.md
volume, workflow, spark-job, application-set, ML Repos, Model Registry none — work from get_manifest_json_schema and search_docs, and say so

redis, postgres, kafka and similar ship as both an image and a chart. They are not interchangeable — a chart brings persistence and replication defaults, an image is one container you configure yourself. Ask which they want; never infer from the name.

Read deploy-common.md before any deploy. One rule bears repeating here: deploying under an existing name replaces the whole manifest. Fetch the deployed one and edit that — a fresh manifest silently drops every env var, secret, limit and replica count the application had, and still reports success.

Checklist Before Responding to an AI Engineering Question

  • Did I read the reference file for what I was about to do — and, when debugging, the failure-mode playbook troubleshooting.md sent me to?
  • Did I identify the application type? It changes which tools can answer.
  • For a model, did I start from get_model_deployment_specs rather than writing a vLLM service from memory?
  • For a deployed application, did I read pod state (list_k8s_pods / reason) and real logs, events or metrics instead of inferring from DEPLOY_SUCCESS — and treat an empty result as "could not see" until I ruled that out?
  • When proposing a fix, did I edit from the live manifest, validate, and apply only with approval?
  • After a successful deploy or update, did I print the links per deployment-links.md?
  • If I couldn't answer the question, did I read references/support-tickets.md and follow it instead of suggesting external contact?