Nature of the Article
This article is the Chinese translation of the technical article "One Year of Insights on Building LLM Applications (Part One)". The original was co-authored by Eugene Yan, Bryan Bischof, Charles Frye, Hamel Husain, Jason Liu, and Shreya Shankar, originally published on May 28, 2024, and translated by the translator "ian" and published on a personal blog. The article is divided into three parts: tactics, operations, and strategy. This translation is the first part, "Tactics", focusing on specific practical details when building applications with LLMs.
The translator notes that the first draft was translated by Deepseek V2, but many technical terms were incorrect; after multiple rounds of proofreading, some terms and vocabulary were standardized. The translator has sent an email to the original authors requesting translation authorization and has received consent from multiple original authors. The translator plans to continue translating subsequent parts of this series, but as of the time the materials were provided, subsequent translations have not yet been provided.
Background and Positioning
The article points out that over the past year, LLMs have become "good enough" for practical applications. The widespread availability of LLMs has enabled people beyond machine learning experts to integrate intelligence into products. Although the barrier to building AI products has been lowered, creating effective products that go beyond demos remains difficult. The authors come from different backgrounds, including independent consultants, researchers studying how ML/AI teams work, leaders of applied AI teams, and deep learning educators. They have compiled the lessons learned over the past year building real applications on top of LLMs, hoping to make this article a practical handbook for building successful products around LLMs.
Main Content of the Tactics Section
The article is organized around the core components of the emerging LLM technology stack, specifically including the following aspects.
Prompting
The article suggests starting with prompting techniques, arguing that they are both easily underestimated and easily overestimated. They are underestimated because proper use of prompting techniques can achieve twice the result with half the effort; they are overestimated because even prompt-based applications require a great deal of engineering work around prompts. Several techniques mentioned in the article include: in-context learning in multi-turn conversations, chain-of-thought reasoning, and providing relevant resources. For multi-turn examples, the article suggests targeting n >= 5; examples should represent the expected input distribution; it is not necessarily required to provide complete "input-output" pairs; and if using an LLM that supports tool calling, the examples should also include the tools you want the Agent to use. For chain-of-thought, the article encourages the model to articulate its thought process before giving the final answer, and points out that the more specific the chain-of-thought, the more effective it is; adding one or two sentences of detailed explanation can usually greatly reduce the hallucination rate. The article also mentions that some research questions the actual effectiveness of this technique, and the academic community's specificits role remains disputed. In terms of providing relevant resources, the article points out that this is usually achieved through retrieval-augmented generation (RAG), but merely including resources is not enough; the model also needs to be instructed to prioritize their use, cite them directly, and state when the resources are insufficient.
Structured Inputs and Outputs
The article argues that structured inputs and outputs can help models understand input content more accurately and generate outputs that can be reliably integrated into downstream systems. Regarding structured outputs, the article mentions that Instructor and Outlines are ideal choices: choose Instructor when using an LLM API SDK, and choose Outlines when self-hosting models. The article also reminds that different LLM families have different format preferences; for example, Claude tends toward XML, while GPT favors Markdown and JSON.
Keep Prompts Concise
Borrowing the "God Object" anti-pattern from software, the article explains that the same problem exists in prompt engineering: prompts start with simple instructions and a few examples, then gradually expand while handling edge cases, and may eventually evolve into a 2,000-token "monster" that actually performs worse on common and straightforward inputs. The article recommends splitting all-encompassing prompts into multiple simple, focused, and easy-to-understand prompts so that each prompt can be iterated on and evaluated separately. In addition, the article suggests re-examining the amount of context sent to the Agent, carefully reading the final prompt on a blank page to identify redundant content, self-contradictory statements, and inappropriate formatting, and pondering how to construct the context and highlight the connections between its parts.
Information Retrieval/RAG
The article points out that the effectiveness of RAG depends largely on the relevance, information density, and level of detail of the retrieved documents. Relevance is usually quantified by ranking metrics such as mean reciprocal rank (MRR) or normalized discounted cumulative gain (NDCG). In terms of information density, among equally relevant documents, more concise documents with less redundant information should be preferred. Regarding level of detail, the article uses generating SQL queries from natural language as an example, explaining that including extra details such as column descriptions and representative values can help the LLM better understand table semantics.
The article also emphasizes not forgetting keyword search, recommending it as a baseline and for hybrid search, and cites the views of the CEO of Perplexity.ai and the CTO of Sourcegraph to explain that nearest-neighbor search using embeddings alone may produce noisy results, and that hybrid methods work best in most cases.
Regarding the choice between RAG and fine-tuning, the article cites research showing that RAG may be more advantageous, whether for knowledge encountered during training or entirely new knowledge, RAG consistently outperforms fine-tuning; at the same time, RAG has practical advantages in maintaining retrieval index updates, deleting problematic documents, and partitioned control over retrieval indexes. The article also discusses whether long-context models will make RAG obsolete, arguing that claims of RAG's demise are greatly exaggerated, for reasons including the continued need to select information to feed into the model, the lack of convincing data showing that models can effectively reason over extremely large contexts, and the cost issues arising from Transformer inference costs growing with context length.
Adjusting and Optimizing Workflows
The article points out that prompting an LLM is only the beginning, and that to extract maximum value it is necessary to go beyond a single prompt and embrace workflows. Using AlphaCodium as an example, the article explains that converting from a single prompt to a multi-step workflow can increase GPT-4's accuracy (pass@5) on CodeContests from 19% to 44%. The workflow includes iterative solutions, explicit planning steps, rewriting user prompts into Agent prompts, designing Agent behavior as a linear chain/DAG/state machine, plan validation, prompt engineering with fixed upstream states, and more. The article also discusses how the nondeterministic nature of AI Agents makes deployment a challenge, suggesting that Agent systems generate deterministic plans and then execute those plans in a structured, repeatable way, and points out that the most successful Agent builders may be those with extensive experience managing junior engineers.
Obtaining More Diverse Outputs Through Means Other Than Temperature
The article discusses the failure modes that may arise from increasing the temperature parameter, for example, certain products may never be output, or the temperature may be too high, causing references to nonexistent products. The article suggests increasing output diversity by adjusting elements in the prompt, such as shuffling the order of the item list each time it is inserted into the prompt, keeping a short list of recent outputs to prevent repetition, and changing the wording used in the prompt.
Caching
The article points out that caching saves costs and eliminates generation latency by removing the need to recompute responses for identical inputs; if responses have previously been reviewed, these vetted responses can also be provided, reducing the risk of providing harmful or inappropriate content. One straightforward method is to use the unique ID of the item being processed; for open queries, techniques from the search field can be borrowed, and features such as autocomplete and spelling correction also help normalize user input and improve cache hit rates.
When to Fine-Tune
The article points out that if reliable, high-quality outputs still cannot be obtained even after extensive prompt engineering, it may be necessary to fine-tune the model for a specific task. It gives two examples: Honeycomb's natural language query assistant, through fine-tuning the model on the syntax and rules of a domain-specific languageOn the other hand, better output was produced; ReChat's Lucy needs to generate responses in a canonical format that combines structured and unstructured data, and fine-tuning is crucial for it to work consistently. The article also reminds that fine-tuning comes with huge costs, including annotating fine-tuning data, fine-tuning and evaluating models, self-hosting, etc., and mentions that costs can be reduced by generating and fine-tuning on synthetic datasets or by guided training on open-source datasets.
Evaluation and Monitoring
The article argues that rigorous and thoughtful evaluation is crucial, and offers the following lessons: create assertion-based unit tests from real input/output samples; LLM-as-Judge is useful but not a silver bullet; "intern tests" generated for evaluation; overemphasizing certain evaluations may harm overall performance; simplify annotation into binary tasks or pairwise comparisons; (reference-free) evaluation and guardrails can be used interchangeably; LLMs will return output when they should not; hallucination is a stubborn problem.
The article specifically explains assertion-based unit tests, execution-based evaluation (ODEX), and precautions for LLM-as-Judge (use pairwise comparisons, control position bias, allow ties, use chain of thought, control response length), etc. The article also mentions that traditional classifiers and reward models can achieve higher accuracy than LLM-as-Judge with lower cost and latency. The "intern test" refers to giving the exact input to a language model, including context, to an ordinary college student in the relevant major, and seeing whether they can succeed and how long it takes, thereby judging task difficulty and whether the context is sufficient.
The article uses the "Needle-in-a-Haystack" (NIAH) evaluation as an example to explain that when a measure becomes a target, it is no longer a good measure. Overemphasizing the NIAH evaluation may lead to performance degradation on extraction and summarization tasks. The article also suggests simplifying annotation into binary classification and pairwise comparison to reduce annotators' cognitive burden and improve inter-rater consistency.
Applicable Scenarios and Limitations
The article is aimed at practitioners using LLMs to build products as well as enthusiasts of "weekend project development," and is especially suitable for developers with some experience to browse the headings and quickly locate sections of interest for reading. The article repeatedly reminds that because the content is long, careful reading takes a long time. The text also points out that some translated sentences are slightly awkward because English conventions differ from Chinese conventions, and there may be formatting errors. The article is the first in a series, and the subsequent "Operations" and "Strategy" articles are not yet provided in this material. Regarding the external tools mentioned in the article (such as Instructor, Outlines, AlphaCodium, BM25, etc.), their specific functions, fees, prices, platforms, orusage steps, this material does not provide detailed information.
