SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsschema-coverage

schema-coverage

Agents & models

View repository

Updated 10-15 21:37

About this skill

Schema Coverage (Terraform resource vs acceptance tests)

Goal

Given a Terraform resource, compare its schema attributes/blocks to what the acceptance tests configure and assert, then highlight opportunities to improve coverage.

Report findings in this order:

  1. Attributes with no coverage
  2. Attributes with poor coverage

Inputs (infer if not provided)

  • Resource name (e.g. elasticstack_elasticsearch_security_api_key)
  • Or the schema file / acceptance test file path
  • Or the Go package directory containing the resource and *_acc_test.go

If the user has an acceptance test open, infer the resource under test from:

  • resource.Test(...) names like resourceName := "elasticstack_..."
  • resource.ParallelTest(...) steps referencing a single resource
  • config builder function names like testAcc...Resource...

Workflow

1) Locate and parse the schema

Find the resource schema definition and capture all schema keys:

  • Top-level attributes in Schema: map[string]*schema.Schema{ ... }
  • Nested block schemas in:
    • Elem: &schema.Resource{ Schema: ... }
    • lists/sets/maps of resources and objects
    • nested blocks referenced via helpers

For each attribute/block, record:

  • Path: terraform attribute path (top-level foo, nested block.0.bar, etc.)
  • Schema metadata: Required/Optional/Computed, ForceNew, type (TypeString, TypeList, etc.), MaxItems/MinItems

2) Locate acceptance tests and collect attribute usage

Scan the acceptance tests for two independent signals:

  • Configured attributes: attributes explicitly set in HCL test configs (including nested blocks).
  • Asserted attributes: attributes referenced in checks, including:
    • value assertions (e.g. TestCheckResourceAttr)
    • set-only assertions (e.g. TestCheckResourceAttrSet)
    • absence assertions (e.g. TestCheckNoResourceAttr)
    • collection assertions (e.g. type-set element checks, list length checks)

Also detect update coverage by identifying multiple resource.TestStep{ Config: ... } steps for the same resource and whether attribute values change between steps.

3) Build a coverage matrix

For each schema attribute path, compute:

  • Configured? (ever set in any test config)
  • Asserted? (ever referenced by any check)
  • Assertion quality:
    • value-specific: asserts the exact value (preferred)
    • set-only: only checks “is set” (weaker)
    • absence: checks unset / removed (good for optional fields)
  • Value diversity: count distinct values across steps/configs (e.g. name="a" vs name="b")
  • Optional-unset coverage: optional field has a test step where it is deliberately omitted, plus an assertion that it is absent (or defaults appropriately)
  • Empty-collection coverage: collection has a step where it is empty (or omitted if optional), plus assertions validating the empty state
  • Update coverage: attribute value changes across steps, and a post-update assertion verifies the new value

4) Produce the report (strict ordering)

Use the template below.

Report template

## Schema coverage report: <resource>

### Scope
- **Schema**: <file(s) or function(s)>
- **Acceptance tests**: <test file(s)>

### 1) Attributes with no coverage
These schema attributes/blocks are not referenced in acceptance tests (neither configured nor asserted):
- `<attr_path>`: <Required/Optional/Computed>, <type>. **Gap**: not configured, not asserted.

### 2) Attributes with poor coverage
These attributes appear in tests but the coverage is weak:
- `<attr_path>`: <schema flags/type>
  - **Observed**: <how it’s currently used (configured/asserted), example values>
  - **Gaps**:
    - <one or more of: set-only assertion, single value only, no unset coverage, no empty collection coverage, no update coverage>
  - **Suggested improvements**:
    - <concrete test step or assertion to add>

### Suggested next steps (smallest diffs first)
1. Add value-specific assertions for set-only checks
2. Add an “unset optional” step + `TestCheckNoResourceAttr`
3. Add an “empty collection” step + collection assertions
4. Add an “update” step changing the attribute + post-update assertions

Rules of thumb (for prioritization)

  • Prefer value-specific assertions over “is set”.
  • For Optional attributes, include at least one step where the attribute is omitted, and assert absence/default behavior.
  • For collections (list/set/map), add a case for empty (or omitted if optional), and assert the expected empty state.
  • For Update behavior, ensure there is at least one multi-step test where the attribute changes and the test asserts the new state.
  • For Computed-only attributes, it’s acceptable to use set-only assertions when exact values are not deterministic, but prefer deterministic assertions when possible.

Notes / limitations

  • Some schemas are built via helpers; follow helper references until all attribute keys are accounted for.
  • Attribute paths in checks often use block.0.attr indexing; normalize consistently when matching to schema blocks.
  • Don’t mark an attribute as “covered” solely because it appears in raw HCL—prefer it being asserted. Treat “configured but never asserted” as poor coverage, not good coverage.