groq

Groq

Groq provides ultra-fast LLM inference using its custom LPU (Language Processing Unit) hardware. The API is fully OpenAI-compatible, so any workflow that works against api.openai.com can be pointed at api.groq.com/openai/v1 with minimal changes.

Official docs: https://console.groq.com/docs/overview

---

When to Use

Use this skill when you need to:

• Run chat completions at extremely low latency (Groq LPU is significantly faster than GPU-based inference)
• Use open-weight models such as Llama 3.3 70B, Llama 3.1 8B, Mixtral 8x7B, or Gemma 2 9B
• Transcribe audio using Whisper via an OpenAI-compatible endpoint
• List available models on Groq's platform
• Drop in a fast, cost-effective inference backend where OpenAI compatibility is assumed

---

Prerequisites

Connect the Groq connector at app.vm0.ai/connectors.

Troubleshooting: If requests fail, run zero doctor check-connector --env-name GROQ_TOKEN or zero doctor check-connector --url https://api.groq.com/openai/v1/models --method GET

---

How to Use

All examples below assume you have GROQ_TOKEN set via the Groq connector.

Base URL: https://api.groq.com/openai/v1

1. Basic Chat Completion

Write to /tmp/groq_request.json:

{
  "model": "llama-3.3-70b-versatile",
  "messages": [{"role": "user", "content": "Explain LPU inference in one paragraph."}]
}

Then run:

curl -s "https://api.groq.com/openai/v1/chat/completions" --header "Content-Type: application/json" --header "Authorization: Bearer $GROQ_TOKEN" -d @/tmp/groq_request.json | jq '.choices[0].message.content'

Key models:

• llama-3.3-70b-versatile — Llama 3.3 70B; best quality, still very fast
• llama-3.1-8b-instant — Llama 3.1 8B; lowest latency
• mixtral-8x7b-32768 — Mixtral 8x7B with 32K context
• gemma2-9b-it — Google Gemma 2 9B instruct
• whisper-large-v3 — Audio transcription (see section 4)

2. Chat Completion with System Prompt

Write to /tmp/groq_request.json:

{
  "model": "llama-3.3-70b-versatile",
  "messages": [
    {"role": "system", "content": "You are a concise technical assistant. Reply in JSON only."},
    {"role": "user", "content": "List the top 3 use cases for LPU-based inference."}
  ]
}

Then run:

curl -s "https://api.groq.com/openai/v1/chat/completions" --header "Content-Type: application/json" --header "Authorization: Bearer $GROQ_TOKEN" -d @/tmp/groq_request.json | jq '.choices[0].message.content'

3. Streaming Chat Completion

For real-time token output, set stream: true. The response is Server-Sent Events (SSE).

Write to /tmp/groq_request.json:

{
  "model": "llama-3.1-8b-instant",
  "messages": [{"role": "user", "content": "Write a haiku about fast inference."}],
  "stream": true
}

Then run:

curl -s "https://api.groq.com/openai/v1/chat/completions" --header "Content-Type: application/json" --header "Authorization: Bearer $GROQ_TOKEN" -d @/tmp/groq_request.json

Each SSE chunk is a data: {...} line with a delta.content field. The stream ends with data: [DONE].

4. Audio Transcription (Whisper)

Transcribe audio files using Groq's hosted whisper-large-v3 model.

curl -s "https://api.groq.com/openai/v1/audio/transcriptions" --header "Authorization: Bearer $GROQ_TOKEN" -F "file=@audio.mp3" -F "model=whisper-large-v3" | jq '.text'

Supported formats: mp3, mp4, mpeg, mpga, m4a, wav, webm (max 25 MB per file).

Note: The vtt and srt response formats are not supported by Groq — use the default json format.

5. List Available Models

Retrieve all models currently available on the Groq platform:

curl -s "https://api.groq.com/openai/v1/models" --header "Authorization: Bearer $GROQ_TOKEN" | jq -r '.data[].id' | sort

6. Check Token Usage

Token usage is returned in every non-streaming chat completion response:

Write to /tmp/groq_request.json:

{
  "model": "llama-3.1-8b-instant",
  "messages": [{"role": "user", "content": "Hi!"}]
}

Then run:

curl -s "https://api.groq.com/openai/v1/chat/completions" --header "Content-Type: application/json" --header "Authorization: Bearer $GROQ_TOKEN" -d @/tmp/groq_request.json | jq '.usage'

Response includes prompt_tokens, completion_tokens, and total_tokens.

---

Guidelines

1. Pick the right model for the job: Use llama-3.3-70b-versatile for quality-first tasks; use llama-3.1-8b-instant when latency or cost is the priority
2. Groq is OpenAI-compatible: Any skill or code targeting api.openai.com/v1 can be redirected to api.groq.com/openai/v1 — just swap the base URL and API key
3. Streaming is the default expectation on Groq: Because inference is so fast, streaming responses are common; handle SSE with a plain curl call and parse data: lines
4. Watch unsupported parameters: logprobs, logit_bias, top_logprobs, and messages[].name are not supported; omit them to avoid errors
5. Temperature 0 is not allowed: If you need deterministic output, use a very small positive value such as 0.01
6. Audio format restrictions: Whisper on Groq does not support vtt or srt output formats; use the default json response format
7. Check the model list regularly: Groq adds and retires models; always confirm the model ID with GET /openai/v1/models before coding against a specific model