This article presents a reference architecture for the emerging LLM application stack, showcasing the systems, tools, and design patterns most commonly used by AI startups and established technology companies.
Purpose
Large language models are a powerful new primitive for building software, but because they are novel and behave differently from conventional compute resources, how to use them is not always obvious. This article aims to provide a useful reference for developers who are working with LLMs.
Key Capabilities
In-Context Learning Design Pattern
The core idea is to use a ready-made LLM directly (without fine-tuning), guiding its behavior through clever prompting and conditioning on private "context" data. The workflow can be divided into three stages:
- Data Preprocessing/Embedding: Private data (such as legal documents) is chunked, passed through an embedding model, and stored in a vector database for later retrieval.
- Prompt Construction/Retrieval: When a user submits a query, the application constructs a series of prompts to submit to the language model. The compiled prompt typically combines a developer-hardcoded prompt template, few-shot examples, necessary information retrieved from external APIs, and relevant documents retrieved from the vector database.
- Prompt Execution/Inference: The prompt is submitted to a pretrained LLM for inference, including proprietary model APIs and open source or self-trained models. Some developers also add operational systems such as logging, caching, and validation at this stage.
Context Data
This includes text documents, PDFs, and structured formats such as CSV or SQL tables. Data loading and transformation solutions vary by developer. Most use traditional ETL tools such as Databricks or Airflow. Some use document loaders in orchestration frameworks, such as LangChain (powered by Unstructured) and LlamaIndex (powered by Llama Hub).
Embeddings
Most developers use the OpenAI API, especially the text-embedding-ada-002 model. Some large enterprises are also exploring Cohere. Developers who prefer open source can use Hugging Face's Sentence Transformers library.
Vector Databases
The most common market choice is Pinecone, because it is fully cloud-hosted, easy to get started with, and has the production features large enterprises need (such as large-scale performance, SSO, and uptime SLAs). Other options include:
- Open source systems such as Weaviate, Vespa, and Qdrant: usually mentionedOffers excellent single-node performance, can be customized for specific applications, and is popular among experienced AI teams.
- Local vector management libraries such as Chroma and Faiss: good developer experience, easy to get started for small applications and development experiments, but not necessarily a replacement for large-scale full databases.
- OLTP extensions such as pgvector: suitable for enterprises that prefer Postgres or purchase data infrastructure from a single cloud provider.
Prompting Strategies
Most developers start with simple prompts, including direct instructions (zero-shot prompting) or some example outputs (few-shot prompting). More advanced prompting strategies aim to ground model responses in real sources and provide external context that the model was not trained on. The Prompt Engineering Guide lists at least 12 advanced prompting strategies, including chain-of-thought, self-consistency, generated knowledge, tree of thoughts, directional stimulus, etc.
Orchestration Frameworks
Orchestration frameworks such as LangChain and LlamaIndex abstract away many details of prompt chaining, interacting with external APIs, retrieving contextual data from vector databases, and maintaining memory across multiple LLM calls. They also provide templates for common applications. LangChain is the leader, currently at version 0.0.201. ChatGPT is also available as an API and performs some of the same functions as other orchestration frameworks, such as abstracting custom prompts, maintaining state, and retrieving contextual data through plugins or APIs.
Language Models
OpenAI is the leader in the language model space. Almost all developers use the OpenAI API to launch new LLM applications, usually with the gpt-4 or gpt-4-32k models. When projects move into production and scale, common options include:
- Switching to gpt-3.5-turbo: about 50 times cheaper than GPT-4 and faster.
- Experimenting with other proprietary vendors (especially Anthropic's Claude models): offering fast inference, GPT-3.5-level accuracy, more customization options, and context windows of up to 100k.
- Offloading some requests to open-source models: particularly effective in high-traffic B2C use cases. Relevant platforms include Databricks, Anyscale, Mosaic, Modal, and RunPod. Inference options for open-source models include simple API interfaces from Hugging Face and Replicate, raw compute resources from major cloud providers, etc..
How to Use
The article provides a reference architecture diagram of the LLM application stack, with links to each project for quick reference. A more comprehensive and continuously updated version is available in the LLM App Stack repository on GitHub.
Applicable Scenarios
Suitable for developers who are building applications with LLMs, especially developers who are just getting started with LLMs. The article is based on conversations with founders and engineers at AI startups, including input from Ted Benson, Harrison Chase, Ben Firshman, Ali Ghodsi, Raza Habib, Andrej Karpathy, Greg Kogan, Jerry Liu, Moin Nadeem, Diego Oppenheimer, Shreya Rajpal, Ion Stoica, Dennis Xu, Matei Zaharia, and Jared Zoneraich.
Limitations
The stack is still in its early stages and may change significantly as the underlying technology advances. Open questions remain about how embeddings and vector databases will evolve as context windows expand, and whether vector databases will consolidate like their OLTP and OLAP counterparts.
