SuperAPI AI Directory

ov-gguf

Agents & models

View repository

Updated 10-16 09:49

About this skill

Paths are relative to src/frontends/gguf/. Read only the route the task needs. First identify the input path: a native .gguf file or a supplied GgufDecoder (llama.cpp cgraph). They share converters; builders and validation differ. Registration or conversion alone never establishes accuracy.

Task Read
Is a model supported? supported_models.md or mmproj.md; quote the checkpoint result and limitations, not just the catalog entry
Load a model, make it stateful/GenAI/PA-ready, tokenizer, IR export runtime.md
Add or port an architecture architectures.md; extensions.md when shipping it as a plugin
Add or replace an mmproj branch extensions.md, then the graph contracts in mmproj.md
Missing ggml operation or op_case how_to_add_op.md
Wrong numbers debugging_accuracy.md
GenAI/mmproj token, media or PA mismatch multimodal integration bisection
Unsupported weight type quantization.md; the fix belongs in src/quant/, not a builder
Build, run tests, regenerate fixtures testing.md
Run or resume GGUF validation across OpenVINO, GenAI and llama.cpp validation.md; use existing GHA workflows and focused local tests, preserving revisions, coverage and failures

Easy to miss

  • Which layer is missing. An unclaimed general.architecture needs a catalog row or ArchitectureExtension; a new mmproj type needs a ProjectorExtension; a structurally different use of an existing op needs an op_case. Only a genuinely new GGML op needs a translator.
  • Registration timing. Architecture/projector extensions before load(); converters and passes before convert(), on the same frontend instance. GGUFMakeStateful and AdaptToGenAI are C++ only.
  • Test build. New translator or builder sources go into the explicit FRONTEND_SRCS list in tests/CMakeLists.txt, or the test binary fails to link. Run ov_gguf_frontend_tests unfiltered before finishing: the op-coverage gate runs in teardown, so check the exit status. GGUFArchConversion skips without generated headers unless OV_GGUF_REQUIRE_ARCH_FIXTURES=1 requires them; generate them with gen_arch_fixtures.py --fetch. Check actual selected counts and skips; registration, conversion, decoder numerics and GenAI media/API integration establish different levels of support.
  • References. Layout-sensitive expectations come from real ggml CPU via a committed oracle under tests/, with several heads/tokens and unequal Q/KV dimensions; NumPy or single-head tests can share the bug. Do not tighten expect_near tolerances from an x86-only run.
  • Comparisons. Use full logits or features on identical inputs and token histories, with INFERENCE_PRECISION_HINT=f32, DYNAMIC_QUANTIZATION_GROUP_SIZE=0 and the reference's KV precision. Set OV_GGUF_Q4_K_ZP_F16=1 before the process starts. Report quantized-CPU and represented-F32 comparisons separately and keep failed results. GGML_OPENVINO_* variables configure only the llama.cpp backend. Inspect tensor-type inventory rather than checkpoint names. Identical quantized weights do not rule out CPU arithmetic differences; exact expansions isolate them but do not waive the original parity gate. Shared exported-model layout does not imply identical media geometry.
  • Shared changes. Translator, VIEW/op_case or shared block changes rerun GGUFArchitectureAccuracy, GGUFArchConversion and the mmproj suites, not just the target model.
  • Long runs. Redirect output to a file, keep the launched PID and stop only that process. Large Q2 MoE checkpoints expand at CPU compilation; check memory before starting one.
  • Ineffective rebuilds. Verify graph attributes, plugin/executor parameters and the linked, loaded CPU variants before repeating a sweep. Use build and runtime identity for overlays or multiple worktrees; version strings and ldd alone do not prove execution identity.
  • Validation batches. Extend shared-path regression checks to tokenizer, mask and PA changes with representative SDPA/PA/media/chat cases. The validation guide separates numerical parity, model quality and Optimum compatibility; a known arithmetic difference remains a failed parity check unless the user changes the acceptance contract.