managing-data-catalogs

Managing Data Catalogs

Guide to selecting, configuring, and using data catalog systems for data discovery, governance, and unified table access across multiple query engines.

When to Use This Skill

Use this skill when:

Setting up an Iceberg catalog (Hive Metastore, AWS Glue, REST/Tabular)
Comparing catalog options for your data platform
Using DuckDB to unify queries across heterogeneous data sources
Evaluating open-source metadata discovery tools (DataHub, OpenMetadata)
Configuring cross-service table access (Athena, EMR, Spark, DuckDB)

Do not use this skill for:

Table format selection (Delta vs Iceberg) → use @designing-data-storage
Cloud storage authentication patterns → use @accessing-cloud-storage
Raw storage I/O without catalog abstraction → use @accessing-cloud-storage
General ETL pipeline patterns → use @building-data-pipelines

Quick Catalog Type Comparison

Catalog	Backend	Managed?	Best For
Hive Metastore	RDBMS (Postgres/MySQL)	Self-hosted	Existing Hadoop, high partition counts
AWS Glue	AWS-managed serverless	AWS-managed	AWS-native stacks (Athena, EMR)
Tabular/REST	SaaS (Nessie-backed)	Vendor-managed	Iceberg-native, Git-like branching
DuckDB (embedded)	Local file/Postgres	Self-hosted	Single-user, PoC, small teams

When to Use Which Catalog

Scenario	Recommended Catalog	Why
AWS-native (Athena, Redshift Spectrum)	AWS Glue	Serverless, IAM integration
Self-hosted Hadoop/Spark	Hive Metastore	Battle-tested, no vendor lock-in
Iceberg-first, multi-cloud	Tabular or Hive	Native Iceberg features or flexibility
Small team, PoC, analytics	DuckDB	Zero infrastructure, SQL-native
LinkedIn-scale metadata	DataHub	Enterprise lineage, scale
Governance-heavy workflows	OpenMetadata	Built-in workflows, data quality

Core Patterns

Catalog → Table → Storage Mapping

Catalog (name) → Table identifier → Metadata location → Data files
                                    (schema, snapshots, partitions)

Key insight: The catalog stores only metadata pointers. Actual data lives in object storage (S3, GCS, Azure).

Multi-Engine Access Pattern

# Same table, different engines
table = catalog.load_table("db.events")

# PyIceberg → Pandas DataFrame
df = table.scan().to_pandas()

# Spark SQL (via same catalog)
spark.table("db.events")  # Same underlying data

# DuckDB (via ATTACH or REST catalog)
SELECT * FROM iceberg.db.events

Detailed Guides

Guide	Covers	When to Read
Hive Metastore	Docker deployment, PyIceberg integration, pros/cons	Self-hosting Hadoop/Spark
AWS Glue Catalog	GlueCatalog setup, crawlers, IAM, Unity Catalog federation	AWS-native stacks
REST Catalog & Tabular	Tabular SaaS, Nessie patterns, Git-like branching	Iceberg-first, multi-cloud
DuckDB Multi-Source	ATTACH patterns, unified views, limitations	Single-user/PoC catalog
Open Source Tools	Amundsen vs DataHub vs OpenMetadata comparison	Metadata discovery/governance

Quick Reference: PyIceberg Catalog Setup

from pyiceberg.catalog import load_catalog

# Hive Metastore
catalog = load_catalog("hive", **{
    "type": "hive",
    "uri": "thrift://localhost:9083",
    "warehouse": "s3://bucket/warehouse/"
})

# AWS Glue
catalog = load_catalog("glue", **{
    "type": "glue",
    "region": "us-east-1",
    "warehouse": "s3://bucket/warehouse/"
})

# REST/Tabular
catalog = load_catalog("rest", **{
    "type": "rest",
    "uri": "https://api.tabular.io/ws/...",
    "token": "tabular-token-...",
    "warehouse": "s3://bucket/warehouse/"
})

# Create and query table
table = catalog.create_table("db.events", schema=schema)
table.append(data)
df = table.scan().to_pandas()

Best Practices

Catalog Selection

Default to Hive Metastore if you have Hadoop investments
Use AWS Glue for pure AWS (Athena, EMR, Redshift Spectrum)
Choose Tabular for Iceberg-native with branching needs
Separate concerns: Iceberg catalog (table storage) ≠ business metadata catalog (OpenMetadata for discovery)

About this skill