datapulse-my

DataPulse is an open-source, read-only verification layer for Malaysian public data, designed to address errors that occur when AI cites data due to outdated underlying data, unclear licensing, or inability to verify. It continuously observes official Malaysian open datasets and publishes machine-readable evidence indicating whether sources are reachable, content freshness, applicable licenses, whether structure or record counts have changed, and observation signature timestamps.

Purpose DataPulse provides verifiable data health signals for AI builders and agent developers, researchers, analysts, journalists, compliance and regulatory monitoring teams, as well as civic technologists and public servants. Its core goal is not to replace official sources or to prove that data is semantically correct, but to make data source conditions observable and observations reproducible.

Key Capabilities

• Continuous monitoring: scheduled probing according to the update frequency declared by datasets, recording reachability, true freshness signals, schema shape, record counts, and collection characteristics.
• Ten health states: fresh, aging, stale, discontinued, degraded, browser-dependent, unreachable, unknown, unknown-freshness, reference. Anything that cannot be proven fresh is marked as unknown-freshness rather than defaulting to healthy.
• Signed evidence and immutable logs: each observation is signed and recorded in a public log, making observation times and records verifiable as untampered.
• Machine-readable first: the entire catalog can be discovered through a single index and queried through a read-only MCP server.
• Read-only MCP server: provides 19 tools, including search_datasets, get_dataset, get_data_passport, find_stale, find_anomalies, find_deteriorating, find_recovering, find_unreliable, find_schema_drift, check_reconciliation, get_provenance, get_evidence, verify_dataset, get_freshness_summary, verify_evidence, trust_verdict, verify_attestation, find_by_licence, usage_summary. Endpoint:https://mcp.data-pulse.my/mcp (Streamable HTTP, no authentication required).
• Coverage: 440 official datasets from 45 publishers, including 30 GTFS transit feeds (KTMB, Prasarana, BAS.MY). Current distribution: 144 fresh, 121 aging, 126 stale, 1 discontinued, 3 degraded, 3 unreachable, 6 unknown-freshness, 36 reference.
• Browser-dependent probing: for sources that require client-side JavaScript rendering, probing uses self-hosted Camofox (a patched headless Chromium sidecar). The current health summary shows 0 browser-dependent sources.
• External verification: provides tools such as verify_external.py, which can independently check published Ed25519 dataset envelopes, GitHub source parity, and Rekor/Sigstore health witnesses. Observation receipts can be verified offline using scripts/observation_verify.py.
• RSS subscriptions and status badges.

Usage AI agents can connect to compatible clients such as Claude Desktop, Cursor, and Cline through an MCP configuration block (no API key required). For example, after configuring it in claude_desktop_config.json and restarting Claude Desktop, confirm that the tool icon shows "datapulse-my" and the read-only tool list. It can also be queried directly through the public endpoint. The dataset catalog can be discovered from datapulse.json, and each health report link provides a plain-language assessment. Non-GTFS datasets also have machine-readable report envelopes under data/json/; the 30 GTFS transit feeds do not provide envelopes, but instead publish health reports and GTFS samples.

Use Cases

• AI agents check freshness and licenses before citing Malaysian data.
• Researchers, analysts, and journalists provide verifiable timeliness and licensing grounds for coursework, papers, dashboards, or published data.
• Compliance and regulatory monitoring teams retain tamper-evident inspection trails.
• Civic technologists and public servants obtain transparent, reproducible views of public data discoverability and description reliability.

Limitations DataPulse does not replace officialProvenance does not prove that upstream government data is semantically correct; it only proves the completeness and timing of its observation. It does not bypass authentication, CAPTCHA, or terms-of-service restrictions; sources that cannot be probed are marked as unreachable or browser-dependent and are not silently scraped. All scraping is rate-limited and identifies the User-Agent. No credentials, accounts, or personal data are collected. To adjust probe frequency, exclude a dataset, or remove it from the manifest, you can submit a GitHub issue to contact the maintainers.