SuperAPI AI Directory
Partner中文AI newsFavorites
Skillskb-retriever

kb-retriever

Writing & research

View repository

Updated 10-15 20:27

About this skill

Local knowledge base retrieval Skill (kb-retriever)

Knowledge base directory description

  • The knowledge base is stored under a root directory and contains multiple file types (such as .md/.txt、.pdf、.xlsx , etc.), and is usually split into multi-level subdirectories by type or business purpose.
  • Adopthierarchical directory index files：
    • The root directory has a data_structure.md, explaining the main "domain directories" and their purposes.
    • Each domain directory can have its own data_structure.md, explaining which subdirectories/files are under that directory and their respective purposes.
    • Deeper subdirectories can also continue to have data_structure.md, forming a multi-level index tree.
  • Knowledge base root directory conventions:
    • By default, the knowledge base is considered to be located in the knowledge/ directory under the current project root directory.
    • If the user explicitly specifies another path in the conversation (for example, "my knowledge base is in /data/kb" or "use ./docs as the knowledge base"), then use the user-specified path as the root directory.
    • When the default path knowledge/ does not exist or cannot be accessed, you should confirm the actual knowledge base root directory location with the user instead of guessing arbitrarily.
  • A single business file may be very large:
    • Do not directly use Read to read the entire file
    • For PDF and Excel, use the corresponding Skill for structured processing, then combine grep/partial reading for fine-grained retrieval

Locate the knowledge root directory

  • For the root directory, prioritize the user: if the user gave a path (such as ./docs、./knowledge-personal), directly use the path provided by the user.
  • Default root directory: otherwise, agree that the root directory is knowledge/。
    • under the current project. Use shell to explicitly check whether the directory exists: prefer using test -d knowledge, or as a fallback use ls -d knowledge。
    • Note: it is forbidden to use Glob "knowledge" in . patterns like this to determine whether a directory exists,Glob only returns file paths, not the directory itself, and an empty result cannot distinguish between "the directory does not exist" and "the directory exists but is empty".
  • Only when the root directory has already been confirmed to exist via test -d , etc., use Glob to search for content under that directory, and use the directory as path, for example:
    • Index files:pattern="**/data_structure.md", path="knowledge"
    • All Markdown:pattern="**/*.md", path="knowledge"
  • If the default knowledge/ does not exist (test -d fails): do not guess other directories, clearly tell the user that the default root directory was not found, and ask the user to specify the actual knowledge base path.

Key principle: learn first, then process

Mandatory checklist when encountering PDF or Excel files：

  • ✅ Have read the corresponding references document to learn the handling methods
  • ✅ Have understood the recommended tools and commands
  • ✅ Have completed file processing (extraction/conversion)
  • ⏭️ Now retrieval can begin

Prohibited actions：

  • ❌ Directly attempting to process PDF without having read pdf_reading.md
  • ❌ Directly attempting to process Excel without having read excel_reading.md
  • ❌ Skip the file processing step and directly search the raw PDF/Excel

Overall workflow

  1. Understand the user's needs

    • Read the user's question and extract:
      • Topic/domain keywords (e.g. "sales report", "system architecture", "interface documentation")
      • Time or scope constraints (e.g. "2023 Q1", "latest version")
      • The required output type (explanation, summary, specific field values, etc.)
    • Determine the knowledge base root directory:
      • First check whether the user specified a knowledge base path in the question.
      • Otherwise use the default root directory knowledge/。
      • If the default root directory does not exist or the directory structure is abnormal, ask the user for confirmation rather than making assumptions.

  2. View the directory index layer by layer data_structure.md

    • Use the concept of a "current working directory":
      • By default, start from the knowledge base root directory specified by the user; if the user did not specify one, use the current directory.
    • In the current working directory, if there is data_structure.md：
      • Use Read to read the first several lines of the file (e.g. limit=300), and continue reading in segments if necessary.
      • Goal:
        • Understand which subdirectories and files exist in the current directory
        • Understand the purpose description of each subdirectory/file
      • Based on the user's question, selectthe most relevant several subdirectories or files, forming a candidate set.
    • For candidate subdirectories:
      • Recursively enter the subdirectory, treat it as the new "current working directory", and continue looking for the data_structure.md within it and repeat the above process.
      • During the recursive process, avoid going deep into all branches at once; prioritize drilling down along the path most relevant to the question.
    • For candidate business files (md/text, PDF, Excel, etc.):
      • After completing the necessary directory-level exploration, collect these files as the finalretrieval target list。
    • When prioritizing:
      • Prefer domain directories and files whose purpose descriptions highly match the question's topic
      • Next consider constraints such as time/version (if reflected in the index)
      • General explanatory documents (such as README.md, overall design documents) are given lower priority

  3. Learn file processing methods (mandatory when encountering PDF/Excel)

    • Before processing PDF files：
      • you must first read references/pdf_reading.md(note that this directory is under the Skills directory, not the Knowledge directory) to learn the extraction method
      • Focus on understanding: pdftotext command, pdfplumber usage, table extraction methods
    • Before processing Excel files：
    • Purpose: ensure the correct tools and methods are used, and avoid blind retrieval

  4. Process and retrieve by file type

    • Process files using the methods just learned (extract, transform, structure)
    • For each category of candidate files, execute according to the following "Markdown/text", "PDF", and "Excel" strategies
    • General principles:
      • Start with the most relevant and most precise files
      • Within each file, progressively perform local retrieval to avoid loading the entire content at once
      • If the current file does not yield satisfactory information, switch to the next candidate file

  5. Iterative retrieval

    • All file types use the unified "multi-round iterative retrieval mechanism" (see the public retrieval principles above)

  6. Answer organization and source tracing

    • Aggregate the context obtained from multiple rounds of retrieval and comprehensively answer the user's question.
    • As much as possible:
      • Provide a clear and direct answer
      • Indicate the file names used (including approximate location when necessary, such as section or approximate line/page count)
    • If the answer is based on inference or incomplete information:
      • Clearly mark assumptions and uncertainties
      • Suggest that the user can provide a more specific file scope or keywords

Public retrieval principles

Keyword selection strategy

  • Extract 3-8 keywords from the user's question (including possible English abbreviations, synonyms, hypernyms/hyponyms)
  • Can combine phrases (e.g., "sales report", "API interface timeout")
  • When necessary, include business terms, technical terms, and common abbreviations (e.g., "uv", "pv", "GMV")

Basic principles of grep retrieval

  • Always specify the most precise include and path possible, avoiding searching the entire directory
  • For the pattern, first try the core nouns and terms in the question, then try synonyms
  • For each hit, read only the local area near the match (a few lines above and below)
  • Save "file name + location information + text snippet"

Multi-round iterative retrieval mechanism (maximum 5 times)

All file types use the unified iterative strategy:

  1. Iteration control
    • Maintain a count of "number of retrieval attempts", with a maximum of 5
    • Increment the count after each retrieval
  2. Process for each round of iteration
    1. Generate/update retrieval keywords based on the question (may include synonyms and expanded terms)
    2. Select files or parts of files that have not yet been fully retrieved
    3. Execute retrieval (grep/local reading/dedicated Skill invocation)
    4. Analyze the retrieved context snippets
    5. Determine whether it is sufficient to answer the question
  3. Termination conditions
    • Context sufficient to support the answer is found; or
    • 5 attempts have been reached and suitable information still has not been found
  4. Handling when information is insufficient
    • Clearly inform the user that the information is missing or may not be in the current knowledge base
    • Provide the closest information found and explain the uncertainty
    • Suggest how the user can narrow the scope (more specific file names, keywords, time range, etc.)

Notes

  • It is forbidden to directly call on the first attempt:Glob "knowledge" in . or any call that attempts to use Glob to determine directory existence; directory existence should be checked via a shell command (such as test -d) check.
  • When using this Skill to query the knowledge base, it is forbidden to use other tools such as web search to obtain knowledge

Specific strategies for different file types

1. Markdown / text files (.md, .txt, .log, etc.)

  1. Candidate file selection

    • Based on data_structure.md and the file name and path, determine relevance
    • Prioritize searching title and directory-type files (such as summary documents, design overviews)

  2. grep locating and local reading

    • Use the Grep tool on the specified candidate files, with include limited to specific suffixes (such as "*.md")
    • For files with matches, use Read to read only the local area near the match:
      • Control reading via line number offset and limit (for example, read a few dozen lines before and after near the matching line)
      • Avoid reading the entire file

  3. Special handling

    • If the content is only a table of contents/titles, continue locating deeper content based on links or section names
    • Apply the "multi-round iterative retrieval mechanism" (see the public retrieval principles above)

2. PDF file retrieval strategy

Workflow：

  1. First: read the processing method guide

    • Before processing any PDF,you must first read references/pdf_reading.md(note that this directory is located under the Skills directory, not the Knowledge directory)
    • Focus on understanding: the pdftotext command, pdfplumber usage, table extraction methods, quick decision table

  2. Select candidate PDFs

    • Based on data_structure.md in the description, select the 1-3 most relevant files
    • If the user specifies a particular PDF file, prioritize using that file

  3. Apply the learned methods to extract text

    • Use the tools recommended in pdf_reading.md (prefer pdftotext or pdfplumber)
    • Important: Use pdftotext input.pdf output.txt to extract text to a file; do not output directly to stdout (to avoid consuming a large number of tokens)
    • If tables need to be extracted, use pdfplumber's table extraction feature

  4. Perform retrieval on the extraction results

    • Use grep to perform keyword searches on the extracted text
    • For each hit, extract the context within a range near the hit (dozens of lines above and below, or several adjacent pages)
    • Save "file name + page number/approximate location + text snippet"
    • Apply the "multi-round iterative retrieval mechanism" (see the public retrieval principles above)

3. Excel file retrieval strategy

Workflow：

  1. First: read the processing method guide

    • Before processing any Excel,you must first read：
      • references/excel_reading.md - Learn how to read worksheets (note that this directory is located under the Skills directory, not the Knowledge directory)
      • references/excel_analysis.md - Learn how to analyze data (note that this directory is located under the Skills directory, not the Knowledge directory)
    • Focus on understanding: pandas reading methods, column filtering, data filtering, aggregation operations

  2. Select candidate Excel files

    • Based on data_structure.md and the file/worksheet naming, select the most relevant sheets
    • Prioritize workbooks/worksheets containing keywords such as "report", "statistics", "log", "configuration", "mapping", etc.
    • If the user specifies a particular Excel file, prioritize using that file

  3. Apply the learned methods to explore the structure

    • Use pandas to read the first 10-50 rows (using the nrows parameter to limit)
    • Focus on mastering: column names/field names, data types (numeric, date, text), key fields
    • Compare column names with the user's question to identify potential key fields (such as "revenue", "sales", "error_code", etc.)

  4. Perform data retrieval and analysis

    • Use the learned pandas methods for filtering and aggregation (such as df[df['column'] == value]）
    • Read only the data near the matching rows each time, avoiding reading the entire table at once
    • If the question includes a time range, add time filtering to the retrieval
    • Apply the "multi-round iterative retrieval mechanism" (see the public retrieval principles above)

Collaboration with other tools

PDF processing

  • Before processing PDFs, you must first read references/pdf_reading.md to learn the processing methods
  • Use pdfplumber/pypdf for text extraction, table extraction, and metadata reading
  • Prefer using the pdftotext command-line tool for fast text extraction

Excel processing

Tool usage principles

  • Grep: used to find line numbers and matching snippets by keyword in specified files; always specify include and path as precisely as possible
  • Read: only used for local file reading; always set a reasonable limit (such as 200-500 lines) and an appropriate offset
  • For any file that may be very large：
    • do not read directly from beginning to end
    • always narrow the scope first through indexes, tables of contents, keywords, etc., before reading

Answer style and error handling

  • Answer style
    • Try to answer in the language used by the user (Chinese/English).
    • Give the conclusion first, then give a brief rationale.
    • If needed, list the cited files and approximate locations afterward, for example:
      • Source: design/api_gateway.md around line 100
      • Source: reports/2023_Q1_sales.xlsx Summary worksheet
  • When information is missing or uncertain
    • Clearly state that no exactly matching information was found in the current knowledge base or that only a partial answer can be given.
    • Do not fabricate facts.
    • Suggest how the user can help narrow the scope:
      • Specify a more specific directory/file
      • Provide more precise keywords or field names
      • Specify a time/version range