SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillskb-retriever

kb-retriever

Writing & research

View repository

Updated 10-15 20:27

About this skill

Local Knowledge Base Retrieval Skill (kb-retriever)

Knowledge Base Directory Description

  • The knowledge base is stored under a root directory and contains multiple file types (such as .md/.txt、.pdf、.xlsx , etc.), usually split into multi-level subdirectories by type or business purpose.
  • Useshierarchical directory index files：
    • The root directory has a data_structure.md, which explains the main "domain directories" and their purposes.
    • Each domain directory can have its own data_structure.md, which explains which subdirectories/files are under that directory and their respective purposes.
    • Deeper subdirectories can also continue to have data_structure.md, forming a multi-level index tree.
  • Knowledge base root directory conventions:
    • By default, the knowledge base is considered to be located in the knowledge/ directory under the current project root directory.
    • If the user explicitly specifies another path in the conversation (for example, "my knowledge base is in /data/kb" or "use the ./docs directory as the knowledge base"), then use the path specified by the user as the root directory.
    • When the default path knowledge/ does not exist or cannot be accessed, you should confirm the actual knowledge base root directory location with the user rather than guessing arbitrarily.
  • A single business file may be very large:
    • Do not directly use Read to read the entire file
    • For PDF and Excel, use the corresponding Skill for structured processing, then combine grep/partial reading for fine-grained retrieval

Locate the knowledge root directory

  • For the root directory, prioritize listening to the user: if the user gave a path (such as ./docs、./knowledge-personal), directly use the path provided by the user.
  • Default root directory: otherwise, the agreed root directory is knowledge/。
    • under the current project. Use shell to explicitly check whether the directory exists: prefer using test -d knowledge, or as a fallback use ls -d knowledge。
    • Note: it is forbidden to use Glob "knowledge" in . patterns like this to determine whether a directory exists,Glob only returns file paths, not the directory itself; an empty result cannot distinguish between "the directory does not exist" and "the directory exists but is empty".
  • Only when the root directory has been confirmed to exist via test -d , etc., should you use Glob to retrieve content under that directory, and use the directory as path, for example:
    • Index files:pattern="**/data_structure.md", path="knowledge"
    • All Markdown:pattern="**/*.md", path="knowledge"
  • If the default knowledge/ does not exist (test -d fails): do not guess other directories; clearly tell the user that the default root directory was not found, and ask the user to specify the actual knowledge base path.

Key principle: learn first, then process

Mandatory checklist when encountering PDF or Excel files：

  • ✅ Have read the corresponding references document to learn the handling method
  • ✅ Have understood the recommended tools and commands
  • ✅ Have completed file processing (extraction/conversion)
  • ⏭️ Now retrieval can begin

Prohibited actions：

  • ❌ Directly attempting to process a PDF without reading pdf_reading.md
  • ❌ Directly attempting to process Excel without reading excel_reading.md
  • ❌ Skip the file processing step and directly search the raw PDF/Excel

Overall process

  1. Understand user requirements

    • Read the user's question and extract:
      • Topic/domain keywords (e.g., "sales report", "system architecture", "interface documentation")
      • Time or scope constraints (e.g., "Q1 2023", "latest version")
      • The required output type (explanation, summary, specific field values, etc.)
    • Determine the knowledge base root directory:
      • First check whether the user specified a knowledge base path in the question.
      • Otherwise use the default root directory knowledge/。
      • If the default root directory does not exist or the directory structure is abnormal, ask the user for confirmation instead of assuming on your own.

  2. View the directory index hierarchically data_structure.md

    • Use a concept of a "current working directory":
      • By default, start from the knowledge base root directory specified by the user; if the user did not specify one, use the current directory.
    • In the current working directory, if there exists data_structure.md：
      • Use Read to read the first several lines of the file (for example, limit=300), and continue reading in segments if necessary.
      • Goal:
        • Understand which subdirectories and files exist under the current directory
        • Understand the purpose description of each subdirectory/file
      • Based on the user's question, selectthe most relevant several subdirectories or files, forming a candidate set.
    • For candidate subdirectories:
      • Recursively enter the subdirectory, treat it as the new "current working directory", and continue searching for data_structure.md within it and repeat the above process.
      • During the recursive process, avoid going deep into all branches at once; prioritize drilling down along the path most relevant to the question.
    • For candidate business files (md/text, PDF, Excel, etc.):
      • After completing the necessary directory-level exploration, collect these files as the finalretrieval target list。
    • When prioritizing:
      • Prefer domain directories and files whose purpose descriptions highly match the question topic
      • Next consider constraints such as time/version (if reflected in the index)
      • General explanatory documents (such as README.md and overall design documents) are given lower priority

  3. Learn file processing methods (mandatory when encountering PDF/Excel)

    • Before processing PDF files：
      • you must first read references/pdf_reading.md(note that this directory is under the Skills directory, not the Knowledge directory) to learn the extraction method
      • Focus on understanding: the pdftotext command, pdfplumber usage, and table extraction methods
    • Before processing Excel files：
    • Purpose: ensure the correct tools and methods are used, and avoid blind retrieval

  4. Execute processing and retrieval by file type

    • Use the method just learned to process files (extract, transform, structure)
    • For each type of candidate file, execute according to the following "Markdown/Text", "PDF", and "Excel" strategies
    • General principles:
      • Start with the most relevant and most precise files
      • Within each file, perform progressive local retrieval to avoid loading all content at once
      • If satisfactory information cannot be obtained from the current file, switch to the next candidate file

  5. Iterative retrieval

    • All file types use the unified "multi-round iterative retrieval mechanism" (see the public retrieval principles above)

  6. Answer organization and source tracing

    • Aggregate the context obtained from multiple rounds of retrieval and comprehensively answer the user's question.
    • As much as possible:
      • Provide a clear and direct answer
      • Indicate the file names used (including approximate locations when necessary, such as chapter or approximate line/page numbers)
    • If the answer is based on inference or incomplete information:
      • Clearly mark assumptions and uncertainties
      • Suggest that the user can provide a more specific file scope or keywords

Public retrieval principles

Keyword selection strategy

  • Extract 3-8 keywords from the user's question (including possible English abbreviations, synonyms, hypernyms/hyponyms)
  • Phrases can be combined (e.g. "sales report", "API interface timeout")
  • When necessary, include business terms, technical terms, and common abbreviations (e.g. "uv", "pv", "GMV")

Basic principles of grep retrieval

  • Always specify the most precise include and path possible to avoid searching the entire directory
  • For pattern, first try the core nouns and terms in the question, then try synonyms
  • For each hit, read only the local area near the match (a few lines above and below)
  • Save "file name + location information + text snippet"

Multi-round iterative retrieval mechanism (maximum 5 times)

All file types use a unified iterative strategy:

  1. Iteration control
    • Maintain a count of "number of retrieval attempts", with a maximum of 5
    • Increment the count after each retrieval
  2. Each round of iteration process
    1. Generate/update retrieval keywords based on the question (which may include synonyms and expanded terms)
    2. Select files or file parts that have not yet been fully retrieved
    3. Execute retrieval (grep/local reading/specialized Skill invocation)
    4. Analyze the obtained context snippets
    5. Determine whether they are sufficient to answer the question
  3. Termination conditions
    • Enough context to support the answer has been found; or
    • 5 attempts have been reached and suitable information still has not been found
  4. Handling when information is insufficient
    • Clearly inform the user that the information is missing or may not be in the current knowledge base
    • Provide the closest information found and explain the uncertainty
    • Suggest how the user can narrow the scope (more specific file names, keywords, time range, etc.)

Notes

  • It is forbidden to directly call on the first attempt:Glob "knowledge" in . or any call that attempts to use Glob to determine directory existence; directory existence should be checked via shell commands (such as test -d).
  • When using this Skill to query the knowledge base, it is forbidden to use other tools such as web search to obtain knowledge

Specific strategies for different file types

1. Markdown / text files (.md, .txt, .log, etc.)

  1. Candidate file selection

    • Based on data_structure.md and the file name and path to judge relevance
    • Prioritize searching title and directory-type files (such as summary documents, design overviews)

  2. grep locating and local reading

    • Use the Grep tool on the specified candidate files, with include limiting to specific suffixes (such as "*.md")
    • For files with matches, use Read to read only the local area near the match:
      • Control reading via line number offset and limit (for example, read a few dozen lines before and after the matching line)
      • Avoid reading the entire file

  3. Special handling

    • If the content is only a table of contents/titles, continue locating deeper content based on links or section names
    • Apply the "multi-round iterative retrieval mechanism" (see the common retrieval principles above)

2. PDF file retrieval strategy

Workflow：

  1. First: read the processing method guide

    • Before processing any PDF,you must first read references/pdf_reading.md(note that this directory is located under the Skills directory, not under the Knowledge directory)
    • Focus on understanding: the pdftotext command, pdfplumber usage, table extraction methods, quick decision table

  2. Select candidate PDFs

    • Based on data_structure.md the descriptions in, select the 1-3 most relevant files
    • If the user specifies a particular PDF file, prioritize using that file

  3. Apply the learned methods to extract text

    • Use the tools recommended in pdf_reading.md (prefer pdftotext or pdfplumber)
    • Important: Use pdftotext input.pdf output.txt to extract text to a file, do not output directly to stdout (to avoid consuming a large number of tokens)
    • If tables need to be extracted, use pdfplumber's table extraction function

  4. Perform retrieval on the extraction results

    • Use grep to perform keyword searches on the extracted text
    • For each hit, extract the context around the hit (dozens of lines above and below or several adjacent pages)
    • Save "file name + page number/approximate location + text snippet"
    • Apply the "multi-round iterative retrieval mechanism" (see the common retrieval principles above)

3. Excel file retrieval strategy

Workflow：

  1. First: read the processing method guide

    • Before processing any Excel,you must first read：
      • references/excel_reading.md - learn how to read worksheets (note that this directory is located under the Skills directory, not under the Knowledge directory)
      • references/excel_analysis.md - learn how to analyze data (note that this directory is located under the Skills directory, not under the Knowledge directory)
    • Focus on understanding: pandas reading methods, column filtering, data filtering, aggregation operations

  2. Select candidate Excel files

    • Based on data_structure.md and the file/worksheet naming, select the most relevant sheets
    • Prioritize workbooks/worksheets containing keywords such as "report", "statistics", "log", "configuration", "mapping", etc.
    • If the user specifies a particular Excel file, use that file first

  3. Use the learned methods to explore the structure

    • Use pandas to read the first 10-50 rows (use the nrows parameter to limit)
    • Focus on mastering: column names/field names, data types (numeric, date, text), key fields
    • Compare column names with the user's question to identify potential key fields (such as "revenue", "sales", "error_code", etc.)

  4. Perform data retrieval and analysis

    • Use the learned pandas methods for filtering and aggregation (such as df[df['column'] == value]）
    • Read only the data near matching rows each time, avoiding reading the entire table at once
    • If the question includes a time range, add time filtering to the retrieval
    • Apply the "multi-round iterative retrieval mechanism" (see the common retrieval principles above)

Collaboration with other tools

PDF processing

  • Must read before processing PDF references/pdf_reading.md Learn the processing methods
  • Use pdfplumber/pypdf for text extraction, table extraction, and metadata reading
  • Prioritize using the pdftotext command-line tool for fast text extraction

Excel processing

Tool usage principles

  • Grep: used to find line numbers and matching snippets by keyword in specified files; always specify include and path as precisely as possible
  • Read: only used for partial file reading; always set a reasonable limit (such as 200-500 lines) and an appropriate offset
  • For any file that may be very large：
    • Do not read directly from beginning to end
    • Always narrow the scope first through indexes, tables of contents, keywords, etc., before reading

Answer style and error handling

  • Answer style
    • Answer in the language used by the user (Chinese/English) whenever possible.
    • Give the conclusion first, then the brief basis.
    • If needed, list the cited files and approximate locations afterward, for example:
      • Source: around line 100 of design/api_gateway.md
      • Source: Summary worksheet of reports/2023_Q1_sales.xlsx
  • When information is missing or uncertain
    • Clearly state that no completely matching information was found in the current knowledge base, or that only a partial answer can be given.
    • Do not fabricate facts.
    • Suggest how the user can help narrow the scope:
      • Specify a more specific directory/file
      • Provide more precise keywords or field names
      • Specify the time/version range