google-cloud-storage-okf

Google Cloud Storage OKF Skill

Use the google-cloud-storage-okf skill to efficiently navigate, inspect, and extract targeted byte ranges from unstructured Google Cloud Storage (gs://) objects via progressive disclosure, instead of reading or dumping entire files into the context window.

Open Knowledge Format (OKF) in GCS

The Open Knowledge Format (OKF) establishes an open standard for attaching self-describing, AI-ready semantic knowledge directly to data assets without mutating raw content.

While OKF at the catalog/dataset level organizes knowledge across tables into concept documents, google-cloud-storage-okf adapts OKF to object storage:

Native Embedding: Stores the OKF manifest directly in the blob's GCS Object Context (custom_contexts), eliminating disconnected sidecar files.
Progressive Disclosure: Provides a structured 3-tier access pattern (peek → list_regions → expand) so AI agents can navigate and extract information from multi-GB files with minimal context window overhead.

GCS Object Context vs. Custom Metadata

In Google Cloud Storage (GCS), there is a fundamental distinction between Object Context and Custom Metadata:

GCS Object Context (custom_contexts / contexts.custom): A dedicated, structured context layer in GCS designed for AI agent knowledge, semantic indices, and progressive disclosure manifests. OKF manifests are targeted primarily at GCS Object Context.
GCS Custom Metadata (metadata / x-goog-meta-*): Traditional key-value pairs used for generic user tagging and legacy object attributes. Custom metadata is separate from Object Context.

OKF keys (such as okf.sum, okf.r.000, okf.created_at, and okf.src_generation) are written directly to GCS Object Contexts (custom_contexts).

Client SDK Compatibility & Architecture

The skill provides dual-layer compatibility across different versions of the google-cloud-storage Python SDK:

Newer / Latest SDK: If the client installation exposes higher-level typed classes (ObjectContexts, ObjectCustomContextPayload), the skill uses them to populate and read blob.contexts.
Standard / Older SDK: If higher-level classes are not yet exported in the local SDK version, the skill drops down to direct REST JSON payload injection (blob._properties["contexts"] / blob._patch_property) and signals blob._changes. When blob.patch() executes, the SDK transmits the standard HTTP REST PATCH request to GCS.
Read Compatibility: The skill inspects both blob.contexts and raw server response fields in blob._properties["contexts"], guaranteeing that Object Contexts remain 100% readable and writable across all customer environments without startup import failures.

Core Architecture & Progressive Disclosure

GCS-OKF stores lightweight, self-contained Table of Contents (TOC) metadata directly inside GCS Object Context on the original blob itself without modifying or duplicating the underlying object data:

Tier 0 (peek): High-level summary card to quickly gauge relevance.
Tier 1/2 (list_regions): Semantic Table of Contents detailing contiguous regions with inclusive byte ranges, titles, and keywords.
Tier 2/3 (expand): Precise byte-range reader retrieving targeted slices based on region boundaries.
Generative Layer (write_okf): Persists newly generated OKF indices into GCS Object Context.

+-------------------------------------------------------------+
| 1. peek(gs_uri)                                             |
|    -> Evaluate relevance & inspect existing Object Contexts |
+-------------------------------------------------------------+
                              |
                              v
+-------------------------------------------------------------+
| 2. list_regions(gs_uri)                                     |
|    -> Locate specific chapters/sections and byte boundaries |
+-------------------------------------------------------------+
                              |
                              v
+-------------------------------------------------------------+
| 3. expand(gs_uri, start_byte, end_byte)                     |
|    -> Read exact byte-range slice into context              |
+-------------------------------------------------------------+

Setup (Read Before Any OKF Call)

Every OKF verb is served by the google-cloud-storage-okf MCP server. If that server is not registered with your MCP client, the tools do not exist in your session and no OKF work is possible.

Tool Availability Gate

Before the first OKF step of any task, you MUST confirm that the MCP tools peek, list_regions, expand, and write_okf are all present in your tool list. They may carry a harness-specific server-name prefix; a prefixed name counts as present.

All four present: proceed with the OKF workflow.
Any one missing: do NOT start the workflow. Tell the user the server is not registered with this client, relay the One-Time Setup steps below, and STOP. Do not resume OKF work until all four tools are bound.

Never Bypass the MCP Server

NEVER import or execute scripts/core.py or scripts/mcp_server.py directly as a substitute for the MCP tools — for example python3 -c "import core; core.peek(...)", or launching mcp_server.py as a module in a subshell and driving it by hand. The server MUST be registered with the harness so that every OKF call goes through MCP.

One-Time Setup

Install the whole skill directory, including scripts/, wherever your harness discovers skills. A SKILL.md-only copy cannot run the server.
Install the server's two Python dependencies, the MCP SDK and the GCS client library, using the python3 that will run the server:
```
python3 -m pip install mcp google-cloud-storage
```
Authenticate and pick a project. The server reads Application Default Credentials and needs a Google Cloud project to bill and quota its requests:
```
gcloud auth application-default login
gcloud config set project <project-id>
```
Instead of a machine-wide project, you can pass one to this server only through the env block in step 4.
Register the canonical server command with your MCP client, then restart the client. Use an absolute path for <abs skill_dir> and the same python3 as in step 2; the env block is optional:
```
{
  "mcpServers": {
    "google-cloud-storage-okf": {
      "command": "python3",
      "args": ["<abs skill_dir>/scripts/mcp_server.py"],
      "env": {"GOOGLE_CLOUD_PROJECT": "<project-id>"}
    }
  }
}
```
Each harness has its own place for this snippet, and your harness knows where that is.

If the client reports that the server failed to connect, run the command from step 4 by hand and read the traceback; a missing dependency or an incomplete install shows up there directly.

Credential, Project, and Permission Errors

If any OKF tool returns an error about credentials, a missing project (for example Project was not passed and could not be determined from the environment), or permission denied (401 or 403), the environment is not set up, not the request. Relay the error verbatim together with the matching fix from step 3 and STOP:

Missing or expired credentials: gcloud auth application-default login.
Missing project: gcloud config set project <project-id>, or the env block in step 4.
Permission denied: the authenticated principal needs read access to the object, and storage.objects.update for write_okf; ask the user to grant it.

Do NOT change the user's gcloud configuration or credentials yourself, do NOT guess a project, and do NOT retry the operation through gcloud storage, gsutil, or client libraries as a workaround.

MCP Tools Reference

1. Peek Tool (`peek`)

Retrieves the Tier 0 summary of the GCS object from its Object Context.

Args:
- gs_uri: Target GCS URI (e.g., 'gs://my-bucket/documents/report.pdf').
Returns:
- High-level summary string if OKF Object Context exists.
- Stale index warning if the underlying object generation changed.
- Diagnostic OKF_NOT_FOUND error detailing remaining Object Context capacity and pre-existing Object Context entries.

2. List Regions Tool (`list_regions`)

Retrieves the Tier 1/2 Table of Contents (region index) stored in the object's Object Context.

Args:
- gs_uri: Target GCS URI.
Returns:
- Formatted JSON string containing a list of region descriptors, each with:
- id: Region identifier (e.g., 'r0', 'r1').
- start_byte: 0-indexed starting byte offset (inclusive).
- end_byte: 0-indexed ending byte offset (inclusive).
- title: Descriptive section/chapter title.
- keywords: List of relevant keywords/tags.
- Stale index warning JSON ({"warning": "...", "regions": []}) if generation changed.
- Diagnostic OKF_NOT_FOUND error if regions are missing.

3. Expand Tool (`expand`)

Fetches a specific byte-range slice from the GCS object payload.

Args:
- gs_uri: Target GCS URI.
- start_byte: 0-indexed start offset (inclusive).
- end_byte: 0-indexed end offset (inclusive).
Returns:
- UTF-8 decoded text slice (or Base64-encoded string for binary data).
- Stale index warning string if the underlying object generation changed since OKF index creation.

4. Write OKF Tool (`write_okf`)

Writes an OKF manifest into GCS Object Context (okf.r.000, okf.sum, etc.).

Args:
- gs_uri: Target GCS URI.
- summary: High-level overview string of the object.
- regions_json: JSON array string of region descriptors (e.g., '[{"id": "r0", "start_byte": 0, "end_byte": 1024, "title": "Introduction", "keywords": ["overview"]}]').
  - Grounded Byte Boundaries: Offsets MUST be 0-indexed integers satisfying 0 <= start_byte <= end_byte < object_size. Regions MUST be listed in ascending order by start_byte and cannot overlap (for inclusive byte ranges, next start_byte must be at least previous end_byte + 1). write_okf strictly validates against actual object size and rejects inverted, negative, out-of-order, overlapping, or out-of-bounds byte ranges.
- created_at: (Optional) ISO 8601 timestamp string (defaults to current UTC time).
Returns:
- Success message or error message if Object Context capacity is exceeded.

OKF Generation Workflow (Handling `OKF_NOT_FOUND`)

When calling peek or list_regions on an object that lacks OKF Object Contexts, the tool returns a comprehensive OKF_NOT_FOUND diagnostic payload including Object Context capacity, existing contexts, custom metadata, and object attributes:

OKF_NOT_FOUND: Manifest summary missing for 'gs://my-bucket/archive.log'.
Object Context Capacity: 25530 bytes remaining (out of 25600 total, 70 bytes used across 3 keys).
Existing Object Contexts:
{
  "source": "ingestion-pipeline",
  "log-type": "application-debug",
  "environment": "production"
}
Existing Custom Metadata:
{
  "category": "system-logs",
  "retention": "30d"
}
Object Attributes:
{
  "size": 1048576,
  "content_type": "text/plain",
  "generation": 1723334400000000,
  "updated": "2026-08-10T12:00:00Z"
}

Confirmation Gate (Ask Before Generating OKF)

Generating an OKF index reads slices of the object to compute semantic boundaries (consuming time and tokens) and mutates the live object's GCS Object Context (custom_contexts) via write_okf. Before scanning an object to split it into regions or calling write_okf, apply this gate:

Small Objects (< 20 KB) on Read Queries: If the user only asked a read, search, or diagnostic question and Object Attributes.size < 20480, do NOT generate an OKF index — read the object directly with expand(gs_uri, 0, size - 1) and answer the question.
When Confirmation Is Required (Ask Once, Then Stop): Whenever peek or list_regions returns OKF_NOT_FOUND on an object >= 20 KB (or returns a Stale OKF index warning) and the user has not already explicitly requested or approved generating/writing the OKF index:
1. Do NOT call expand to scan or segment the object and do NOT call write_okf.
2. Tell the user that the object does not have an OKF index yet (or that its existing OKF index is stale), citing the object's size and remaining Object Context Capacity from the peek/list_regions diagnostic.
3. Ask a single yes/no question confirming whether to inspect the object and generate/write an OKF index into its GCS Object Context, and end your turn.
4. On yes: execute Steps to Generate and Commit OKF below (expand to determine semantic region boundaries → write_okf), then complete the user's original request.
5. On no: do NOT call write_okf; leave the object's GCS Object Context untouched.
When Consent Is Already Granted (Proceed Without Re-Asking): Proceed directly to Steps to Generate and Commit OKF without stopping to ask when:
- The user's prompt explicitly asks to generate, index, or write an OKF manifest for the target object(s) (or explicitly states that permission to generate OKF is granted),
- The user has already confirmed in the conversation, or
- You are an isolated worker sub-agent invoked for batch indexing after the lead agent has already obtained user confirmation.

Steps to Generate and Commit OKF:

Leverage Pre-existing Object Context, Custom Metadata & Attributes:
- Inspect the returned Existing Object Contexts, Existing Custom Metadata, and Object Attributes from the diagnostic message.
- Use these rich signals (e.g., environment, category, log-type, content_type, size) to inform and enrich your generated summary, region titles, and keywords.
Evaluate Splitting Heuristics:
- Small Objects (< 20 KB): If the entire object easily fits in context and is small, read directly via expand(gs_uri, 0, size - 1) or standard tools without forcing OKF indexing (unless the user explicitly asked to generate an OKF index for it).
- Large/Medium Objects (> 20 KB to multi-GB): Segment the object into semantic, logical regions (e.g., chapters, log phases, data tables, video keyframe intervals) using inclusive byte ranges. This is a critical step, think of it as generating the most useful summary and Table of Contents for the object. If needed, make a couple of detailed plans first on how to split the object. For example, splitting naively by paragraphs on a text document is not great. You want to keep collecting paragraphs together that are semantically related so it surfaces as one semantic region. Look to see what tools you have at your disposal for best semantic splitting -- for example, you may have a PDF parser or even a local small model like Gemma. You split the object once and every agent uses it later on so it is ok to pay a high cost for this splitting.
- Homogeneous Logs / Structured Dumps: Split based on timestamp boundaries or major processing milestones.

Format and Commit OKF Manifest to Object Context:

Create a concise summary string (under ~200-250 bytes is optimal).

Assemble the region list into valid JSON:

[
  {
    "id": "r0",
    "start_byte": 0,
    "end_byte": 4095,
    "title": "Header & Config",
    "keywords": ["config", "init", "schema"]
  },
  {
    "id": "r1",
    "start_byte": 4096,
    "end_byte": 18432,
    "title": "Main Execution Trace",
    "keywords": ["trace", "rpc", "errors"]
  }
]

Call write_okf(gs_uri=gs_uri, summary=summary, regions_json=regions_json).

Parallel Processing with Sub-Agents

When processing multi-file buckets or directories with multiple unindexed objects, agents should avoid single-threaded sequential processing and instead invoke parallel worker sub-agents via invoke_subagent:

Multi-File Batch Parallelism

When indexing multiple objects under a GCS prefix or bucket:

List Candidate URIs: Identify all unindexed objects in the target bucket/prefix.
Confirm Once for the Batch: Unless the user's prompt already explicitly requests or grants permission to generate OKF for the batch, report the candidate URIs and ask a single yes/no question for confirmation before dispatching workers.
Partition Batches: Split the list of URIs into balanced batches across worker sub-agents (e.g., 5–10 objects per sub-agent).
Dispatch Parallel Sub-Agents: Invoke worker sub-agents concurrently using invoke_subagent:
- Each sub-agent is assigned an isolated batch of URIs and informed that user confirmation to generate OKF is already granted.
- Each sub-agent runs the OKF generation flow (peek -> inspect attributes -> generate semantic regions -> write_okf).
Aggregate Results: Workers report completion status and persisted region counts back to the lead agent.

Telemetry & Attribution

Outbound Google Cloud Storage API calls from the skill, MCP server, cURL, or custom scripts are stamped with the standard GCS Skills User-Agent telemetry header:

User-Agent: gcs-skills/1.0 (skill:google-cloud-storage-okf)

When integrating via the gcloud storage CLI, pass the standard metrics environment variable:

CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-storage-okf)"

Guardrails & Best Practices

Never Bypass the MCP Server: Only invoke OKF through the MCP tools; never run or import scripts/core.py or scripts/mcp_server.py directly as a substitute (see Setup).
Never Reconfigure the User's Environment: On credential, project, or permission errors, relay the fix and stop; never change gcloud configuration or credentials yourself (see Setup).
Confirm Before Generating OKF: When peek or list_regions returns OKF_NOT_FOUND on an object >= 20 KB (or a Stale OKF index warning) and the user has not explicitly requested or pre-approved OKF generation, report the object's size and remaining Object Context capacity, ask once for confirmation, and wait for a yes before scanning the object or calling write_okf.
Never Overwrite Object Payload: OKF operations only read byte ranges and write non-destructive Object Contexts. Never delete or overwrite the original GCS object.
Avoid Massive Full-Blob Dumps: Do NOT stream or dump large unverified objects into the agent context. Always use list_regions followed by targeted expand calls.
Respect Object Context Capacity Budgets:
- Hard key limit per GCS object: 50 Object Context keys across all entries (limits an object to ~45–50 OKF regions maximum).
- Hard total byte limit per GCS object: 25,600 bytes across all Object Context keys.
- Hard limit per Object Context value: 256 bytes (compression & fallback truncation are handled automatically by write_okf).
Detect Stale Indices: If peek or list_regions returns a Stale OKF index warning (indicating the underlying blob generation changed since indexing), confirm with the user (unless already granted) before re-evaluating and re-generating the OKF index.