dspy-parallel

Run LM Calls in Parallel with dspy.Parallel

Guide the user through using DSPy's Parallel module to execute multiple LM calls concurrently. dspy.Parallel is the built-in way to speed up batch processing and fan-out patterns without writing threading code yourself.

What is dspy.Parallel

dspy.Parallel takes a list of (module, inputs) pairs and executes them concurrently using a thread pool. It handles threading, progress bars, error limits, and timeouts so you don't have to.

Use it when you have:

• A batch of inputs to run through the same module (classify 500 tickets, summarize 100 articles)
• Multiple independent modules to run on the same input (sentiment + topics + entities at once)
• Any set of LM calls that don't depend on each other

If call B depends on the result of call A, those two calls must be sequential. Everything else can be parallel.

Step 1 — Clarify the use case

Before writing code, ask:

1. Same module or mixed? Are you running one module on many inputs (batch), or multiple different modules on the same input (fan-out)? Same-module batches can use either dspy.Parallel or the simpler .batch() method (see below). Fan-out requires dspy.Parallel.
2. How many items? Fewer than ~5 items — just call the module directly; parallelism overhead is not worth it. 5–100 items — dspy.Parallel with 4-8 threads. 100+ items — increase threads up to your provider rate limit.
3. Any data dependencies? If any call needs the output of a previous call, those must stay sequential. Confirm all pairs are truly independent before parallelizing.
4. Error tolerance? If any failure should abort the batch, use max_errors. If you want partial results, set return_failed_examples=True.

Basic usage

Pass a list of (module, inputs) pairs. Each pair is one unit of work:

import dspy

lm = dspy.LM("openai/gpt-4o-mini")  # or any LiteLLM-supported provider
dspy.configure(lm=lm)

# A module to run on every input
classify = dspy.Predict("text -> label: str")

# A batch of inputs
texts = [
    "I love this product!",
    "Terrible experience, want a refund.",
    "It's okay, nothing special.",
    "Best purchase I've made this year.",
]

# Build execution pairs: (module, inputs_dict)
exec_pairs = [(classify, {"text": t}) for t in texts]

# Run them all in parallel
parallel = dspy.Parallel(num_threads=4)
results = parallel(exec_pairs)

for text, result in zip(texts, results):
    print(f"{text[:30]:30s} -> {result.label}")

results is a list in the same order as exec_pairs, so results[i] corresponds to exec_pairs[i].

Constructor options

dspy.Parallel(
    num_threads=4,               # number of concurrent threads (default: settings.num_threads)
    max_errors=5,                # stop after this many failures (default: settings.max_errors)
    return_failed_examples=False,# if True, return failures separately instead of raising
    provide_traceback=None,      # include tracebacks in error output (None = use settings default)
    disable_progress_bar=False,  # suppress the tqdm progress bar
    timeout=120,                 # max seconds per task before timeout
)

Parameter	Type	Default	Purpose
num_threads	int | None	None	Number of concurrent threads. Falls back to dspy.settings.num_threads.
max_errors	int | None	None	Stop execution after this many errors. Falls back to dspy.settings.max_errors.
access_examples	bool	True	Unpack Example objects via .inputs(). Set False to pass raw Examples.
return_failed_examples	bool	False	When True, return failed examples separately instead of raising.
provide_traceback	bool | None	None	Include Python tracebacks for failed examples.
disable_progress_bar	bool	False	Suppress the progress bar.
timeout	int	120	Max seconds per individual task.
straggler_limit	int	3	Threshold for flagging slow-running tasks.

.batch() — simpler alternative for same-module batching

Every dspy.Module has a built-in .batch() method that handles the most common case: running the same module on many inputs. If you do not need to mix different modules per item, .batch() is less code than dspy.Parallel:

classify = dspy.Predict("text -> label: str")
texts = ["Great!", "Terrible.", "Meh."]

# Build a list of Examples
examples = [dspy.Example(text=t).with_inputs("text") for t in texts]

# Run in parallel -- same as dspy.Parallel under the hood
results = classify.batch(examples, num_threads=4)
for r in results:
    print(r.label)

.batch() accepts all the same concurrency parameters as dspy.Parallel:

results = classify.batch(
    examples,
    num_threads=4,
    max_errors=5,
    return_failed_examples=True,
    provide_traceback=None,
    timeout=120,
)
# With return_failed_examples=True: returns (results, failed_examples, exceptions)

When to use .batch() vs dspy.Parallel:

Need	Use
Same module, many inputs	.batch() — simpler API
Different modules per input (fan-out)	dspy.Parallel — required
Mix modules and share an exec loop	dspy.Parallel — more control

Configuring concurrency

Start with a thread count that matches your rate limits, not your CPU cores. LM calls are I/O-bound (waiting on HTTP responses), so you can safely use many threads:

# Conservative -- good starting point
parallel = dspy.Parallel(num_threads=4)

# Aggressive -- if your provider allows high concurrency
parallel = dspy.Parallel(num_threads=16)

# Match your provider's rate limit
# e.g., 60 requests/min = ~1/sec, so 4-8 threads keeps the pipeline full
parallel = dspy.Parallel(num_threads=8)

If you hit rate-limit errors (HTTP 429), reduce num_threads or add retry logic in your LM configuration.

Input formats

Parallel accepts inputs as dictionaries, dspy.Example objects, or tuples:

module = dspy.Predict("question -> answer")

# Dict inputs (most common)
pairs = [(module, {"question": "What is DSPy?"})]

# dspy.Example inputs
example = dspy.Example(question="What is DSPy?").with_inputs("question")
pairs = [(module, example)]

# Both work the same way
parallel = dspy.Parallel(num_threads=2)
results = parallel(pairs)

Aggregating results

Results come back as a list. Aggregate however your application needs:

import dspy

classify = dspy.Predict("text -> label: str, confidence: float")
texts = ["Great!", "Terrible.", "Meh.", "Amazing!", "Awful."]

parallel = dspy.Parallel(num_threads=4)
results = parallel([(classify, {"text": t}) for t in texts])

# Count labels
from collections import Counter
label_counts = Counter(r.label for r in results)
print(label_counts)  # Counter({'positive': 2, 'negative': 2, 'neutral': 1})

# Filter by confidence
high_confidence = [
    (text, r.label)
    for text, r in zip(texts, results)
    if r.confidence > 0.8
]

# Build a summary dict
output = [
    {"text": t, "label": r.label, "confidence": r.confidence}
    for t, r in zip(texts, results)
]

Error handling

By default, Parallel raises an exception after max_errors failures. To handle errors gracefully, use return_failed_examples=True:

parallel = dspy.Parallel(
    num_threads=4,
    max_errors=10,
    return_failed_examples=True,
    provide_traceback=True,
)

results, failed_examples, exceptions = parallel(exec_pairs)

When return_failed_examples=True, the return value is a 3-tuple:

• results -- list of successful predictions (same length as successes)
• failed_examples -- list of (module, inputs) pairs that failed
• exceptions -- list of exceptions corresponding to each failure

Handle failures after the batch completes:

results, failed, errors = parallel(exec_pairs)

print(f"Succeeded: {len(results)}, Failed: {len(failed)}")

# Retry failures with a fallback module
if failed:
    fallback = dspy.ChainOfThought("text -> label: str")
    retry_pairs = [(fallback, inputs) for _, inputs in failed]
    retry_results = parallel(retry_pairs)

Setting an error budget

Use max_errors to fail fast when too many calls are failing (e.g., provider outage):

# Stop the whole batch if more than 5 calls fail
parallel = dspy.Parallel(num_threads=4, max_errors=5)

try:
    results = parallel(exec_pairs)
except Exception as e:
    print(f"Batch aborted: {e}")

Timeouts

The timeout parameter sets a per-task time limit in seconds. Tasks that exceed this are terminated:

# Give each task up to 60 seconds
parallel = dspy.Parallel(num_threads=4, timeout=60)

Using different modules per item

Each pair can use a different module. This is useful for fan-out patterns where you run multiple analyses on the same input:

import dspy

sentiment = dspy.Predict("text -> sentiment: str")
topics = dspy.Predict("text -> topics: list[str]")
summary = dspy.ChainOfThought("text -> summary: str")

text = "DSPy is a framework for programming language models..."

# Fan out: three different modules, same input
exec_pairs = [
    (sentiment, {"text": text}),
    (topics, {"text": text}),
    (summary, {"text": text}),
]

parallel = dspy.Parallel(num_threads=3)
results = parallel(exec_pairs)

combined = {
    "sentiment": results[0].sentiment,
    "topics": results[1].topics,
    "summary": results[2].summary,
}

When to use Parallel vs a sequential loop

Scenario	Use	Why
Process 100+ items through the same module	Parallel	Massive speedup from concurrent HTTP requests
Run 3 independent analyses on one input	Parallel	All three calls happen at once
Pipeline where step 2 needs step 1's output	Sequential loop	There's a data dependency
Single LM call	Neither	No benefit from parallelism
Processing 2-3 items	Either works	Overhead is negligible either way

Sequential loop (before)

# Slow: each call waits for the previous one to finish
results = []
for text in texts:
    result = classify(text=text)
    results.append(result)

Parallel (after)

# Fast: all calls run concurrently
parallel = dspy.Parallel(num_threads=8)
results = parallel([(classify, {"text": t}) for t in texts])

For a batch of 100 items with ~1 second per LM call:

• Sequential: ~100 seconds
• Parallel (8 threads): ~13 seconds

Parallel inside a module

Wrap Parallel usage inside a dspy.Module for clean composition:

class BatchClassifier(dspy.Module):
    def __init__(self, num_threads=4):
        self.classify = dspy.Predict("text -> label: str, confidence: float")
        self.num_threads = num_threads

    def forward(self, texts: list[str]):
        parallel = dspy.Parallel(num_threads=self.num_threads)
        exec_pairs = [(self.classify, {"text": t}) for t in texts]
        results = parallel(exec_pairs)

        return dspy.Prediction(
            labels=[r.label for r in results],
            confidences=[r.confidence for r in results],
        )

# Usage
classifier = BatchClassifier(num_threads=8)
result = classifier(texts=["Great!", "Terrible.", "Meh."])
print(result.labels)  # ["positive", "negative", "neutral"]

This keeps the parallelism as an implementation detail. Callers don't need to know about threading -- they just pass a list and get a list back.

Gotchas

1. Claude writes a for loop instead of using dspy.Parallel. When asked to process a batch of inputs, Claude defaults to a sequential loop. For any batch of 5+ independent LM calls, use dspy.Parallel — it is dramatically faster because LM calls are I/O-bound.
2. Claude sets num_threads to match CPU cores. LM calls are network-bound (waiting on HTTP responses), not CPU-bound. Thread count should match your provider rate limit, not your CPU count. 8-16 threads is typical even on a 4-core machine.
3. Claude forgets that return_failed_examples=True changes the return type. Without it, parallel(pairs) returns a flat list. With it, it returns a 3-tuple (results, failed_examples, exceptions). Destructure accordingly or the code will break.
4. Claude nests Parallel inside Parallel without considering total concurrency. An inner Parallel(num_threads=3) inside an outer Parallel(num_threads=4) creates up to 12 concurrent LM calls. This can exceed provider rate limits. Calculate the total: outer_threads * inner_threads.
5. Claude uses dspy.Parallel for 1-2 items. The threading overhead is not worth it for fewer than ~5 items. Just call the module directly.
6. Skipping result order verification. dspy.Parallel preserves input order — results[i] corresponds to exec_pairs[i]. Verify this with a quick sanity check: zip results with your original inputs and confirm the first few items align before processing the full batch.

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• Modules are the building blocks you pass to Parallel -- see /dspy-modules
• Batch classification (sort items into categories at scale) -- see /ai-sorting
• Multi-step pipelines that combine sequential and parallel stages -- see /ai-building-pipelines
• Async API serving (non-blocking DSPy in FastAPI / asyncio) -- see /ai-serving-apis
• Evaluation uses its own threading via num_threads -- see /dspy-evaluate
• For worked examples (batch classification, multi-aspect analysis), see examples.md
• Install /ai-do if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Additional resources

• dspy.Parallel API docs
• For constructor signatures and method reference, see reference.md
• For worked examples (batch classification, multi-aspect analysis), see examples.md