Description
Provides the agent with methodologies to discover data lake assets, connect to Enterprise catalogs (Unity Catalog, AWS Glue, Polaris), and inspect Delta Lake, Apache Iceberg, and Apache Hudi tables.
Context
Engineers need deep metadata without heavy compute. The agent must seamlessly connect to the configured metadata catalog and use lightweight Python libraries to retrieve schemas, partition layouts, and snapshot histories.
Instructions
1. Catalog Connection
Always ask the user for their preferred catalog or check their environment variables (AWS_PROFILE, DATABRICKS_HOST).
Iceberg via REST/Polaris:
from pyiceberg.catalog import load_catalog
catalog = load_catalog( "default", **{ "type": "rest", "uri": "https://api.polaris.com/v1", "credential": "oauth2_token..." } ) table = catalog.load_table("namespace.table")
**Delta via Unity Catalog (Databricks SDK):**
```python
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
table = w.tables.get("catalog.schema.table_name")
print(table.columns)
2. Time Travel & Table Recovery
If a user mentions corrupted data or dropped tables:
- Suggest querying a specific snapshot/version.
- Delta:
SELECT * FROM my_table VERSION AS OF 12 - Iceberg:
SELECT * FROM my_table FOR SYSTEM_VERSION AS OF 12345
To restore a table:
-- Delta Lake
RESTORE TABLE my_table TO VERSION AS OF 12;
-- Apache Iceberg
CALL catalog.system.rollback_to_snapshot('my_table', 12345);
3. RAG-Ready Metadata Summary
Always output findings in structured Markdown so other agents or humans can easily read it.
Example Format:
### 🗂️ Enterprise Table Profile: `[Table Name]`
- **Catalog:** [Glue / Unity / Polaris]
- **Format:** [Delta/Iceberg/Hudi]
- **Location:** `[s3/adls/gcs path]`
- **Latest Version/Snapshot:** [Version ID]
- **Time Travel Retained:** [X days]
#### Schema
| Column | Type | Nullable | Masking/Tag |
|--------|------|----------|-------------|
| id | long | False | - |
| email | str | True | `[MASKED]` |