Data architecture

Identity model

unified_credit_code is the canonical organization identifier. The first two characters normally indicate the legal-person form:

The explicit organization type returned by the registry is authoritative. The 2026-07-29 snapshot contains 459 valid-code exceptions to the usual prefix convention; validation counts them instead of silently reclassifying them.

Historical RICF identifiers are not discarded. legacy_id_links.parquet retains every row-level match, its method and confidence. organization_crosswalk.parquet aggregates all historical identifiers for a current credit code. legacy_identity_conflicts.parquet and legacy_identity_unmatched.parquet retain unresolved cases. identity/source_index/ contains five content-addressed imports from raw_data/index/ (43,943 rows), and legacy_name_changes.parquet contains only the 12 deterministic transitions supported by those and direct-USCC evidence. Fourteen ambiguous name-change groups remain conflicts. foundation_master.parquet selects field values under the tracked source precedence. foundation_field_evidence.parquet retains every candidate value and source, while foundation_field_conflicts.parquet exposes disagreements instead of hiding them.

When a complete type-3 registry is present, the crosswalk automatically uses all registry foundations. Before that checkpoint it falls back to foundations in the current annual-report index.

Layers

Raw artifacts are content-addressed by source, endpoint, and a canonical request payload. CAPTCHA values and browser cookies are excluded from the raw request envelope and cache key. Each processed live-source row carries _source, _retrieved_at, _snapshot_id, and _raw_artifact.

The tracked legacy/raw_data/ directory predates this pipeline and is not the same as the ignored data/raw/<snapshot>/ runtime layer.

Main current tables

• charitable_organization_report_index.parquet preserves the full KBE242 response, including non-foundation records.
• foundation_report_index.parquet is the prefix-53 foundation subset.
• foundation_profiles.parquet contains current detailed registry profiles.
• social_organization_registry/ preserves every official source row, partitioned by organization type and page, with a manifest listing the exact parts.
• social_organization_registry_entities.parquet is the canonical one-row-per-valid-credit-code analytical view. Exact page overlaps and rows without a valid code remain in social_organization_registry_duplicate_rows.parquet and social_organization_registry_unkeyed_rows.parquet.
• annual_report_sections.parquet maps filings to report sections.
• annual_<component>.parquet contains source-native annual-report component rows with filing and organization keys added. The component registry now includes supervision KBE172–174, investment KBE177–181/KBE225, organization-building KBE182–190/KBE192, and other explanations KBE219. These opaque new raw tables remain internal unless a reviewed derivative is separately catalogued.
• current_governance_roster.parquet is the public privacy-transformed derivative of KBE185: 340,322 board/supervisor rows for 2019–2025, with no DOB column and whole-year age produced only through privacy.py.
• annual_report_party_subsections.parquet and annual_report_party_branches.parquet retain both levels of party-building data.
• legacy/ contains lossless Parquet migrations of the tracked public TSVs.
• companion/ contains manifest-selected imports from the two unchanged sibling repositories, plus a 117,788-entry content-addressed archive inventory. Corrupt evidence is flagged and cannot contribute panel values.
• historical_html/ contains the internal file inventory, long form-control and table-cell extractions, and independent financial verification for 67,449 source records. Those raw long tables remain excluded from release; their evidence-mapped, privacy-safe financial projection contributes 3,751,523 observations to the canonical panel.
• organization_panel.parquet is the canonical harmonized long panel described below. foundation_panel.parquet is a byte-identical deprecated compatibility alias, not a second data product.
• panel_duplicate_observations.parquet retains every duplicate-key source row, its deterministic rank, disagreement class, and identity quarantine.
• panel_restatements.parquet records where a later filing disagrees with a year’s own filing.
• supplemental/ contains the exact 2014 survey reconciliation and a content-addressed 653-document evidence index; both remain internal pending human/privacy review.

The raw MCA field names remain in each table. Human-readable aliases are added instead of replacing source fields so that later codebook revisions remain auditable.

Two of the registry entity table’s source-native columns now carry an alias in the published lookup: aaae0129 as start_up_fund and aaae0114 as registration_division_code. Neither name comes from the platform, which labels neither column, so both rest on an identity test in src/ricf/evidence.py that reruns on every build — the start-up fund against the same organization’s filed akbe1061, the division against the division its own credit code encodes. The source columns are unchanged in the table; the alias exists only where the data is published. Nine further columns of the same table remain unaliased because nothing establishes what they hold.

Harmonized panel

The published legacy tables use three header conventions across four years: 2013 and 2016 use RICF letter codes, 2014 and 2015 use Chinese labels, and the 2015/2016 financial tables add _begin/_end and _last/_this suffixes. The current MCA financial tables are long-format and carry the official Chinese line-item label in the row. Nothing joined these vocabularies, so a variable could not be read as one series across both eras.

metadata/variable_crosswalk.json is the tracked, hand-reviewable mapping that does. It records, per component:

• every historical letter code with its Chinese and English labels;
• which legacy column in which file carries it, and at what measure;
• which current source label and value columns carry it; and
• the evidence grade that justified each mapping.

build-panel reads only that document. Its grain is one measurement per (unified_credit_code, data_year, component, variable, measure, source row).

Financial-activities rows key on the base variable, with the restriction carried in measure, exactly as the balance sheet uses begin and end.

See Harmonization method for the evidence behind each mapping and the validation results.

A prior_period or opening_balance row is emitted only for a year with no filing of its own. Where the year does have a filing, its own value is preferred and the disagreement, if any, is written to panel_restatements.parquet. Nothing is silently dropped.

Duplicate rows are also never discarded. Exactly one row per duplicate key has precedence_rank == 1; identity-conflicted rows remain quarantined. The canonical filed-value view filters observation_kind == "reported", precedence_rank == 1, and identity_conflict == false. The underlying 11,159,082-row, 2004–2025 table is a retained-evidence table, while this filter is the analytical default. It covers 16,712 organizations; its 1,496,548 non-foundation rows remain available in bulk data and registry search, while longitudinal profile pages deliberately remain foundation-only.

Showcase foundation panels

The static showcase joins annual evidence only after canonical processing:

1. Current filing and component rows link directly by unified_credit_code. Current governance rows use the same direct key and are accepted only when the source/derived relationship, mapping evidence, and privacy summary all validate.
2. Historical rows with a valid credit code use it directly.
3. Other historical rows link by the deterministic (source_year, historical_ricf_id) evidence in legacy_id_links.parquet; the imported 2017 basic table can additionally supply its recorded credit code.
4. Ambiguous identity links and codes outside the canonical foundation index are excluded from public profile shards and counted in foundations/manifest.json.

The output grain is one foundation detail record per unified credit code, containing a map of available organization-years. Within a year, source evidence remains traceable, while harmonized panel consumers select rank 1 and exclude identity-conflicted rows. A record can therefore retain multiple historical sources without plotting or aggregating them twice. The foundation index publishes separate historical_years and report_years arrays plus their deterministic union, available_years.

This layer is a browser-optimized derivative, not a canonical processed table. The detail manifest records every contributing source-table path, row count, byte size, and SHA-256 hash.

site/data/panel/ publishes the harmonized panel in the same sharding scheme, plus a variable legend. The site loads one shard per profile view, so a continuous trend costs one extra request and no change to first paint.

Registry totals

The platform’s homepage total and the registry universe have different denominators. On 2026-07-29 the homepage reported 778,576 organizations, while blank type-specific searches include inactive records. The snapshot’s canonical entity table holds:

The 1,199,440 captured source rows reconcile to 1,198,950 unique valid credit codes, 425 exact adjacent-page overlaps, and 65 rows without a valid credit code. Counts are live and drift while the partitions are collected, so the type totals a blank search returns can exceed the entity table by a few hundred rows. Every snapshot records all observed totals, row and page counts, and completeness flags. Consumers must not substitute the homepage counter for the registry universe.

Release model

Releases are immutable and named by snapshot. A release is refused unless the snapshot satisfies the tracked conditions in metadata/release_gate.json: a present and valid validation report, no failed checks, no validation warning outside the reviewed list, and a minimum row count for each key table. The evaluation is recorded in the release manifest. --skip-gate releases anyway and records the failure rather than hiding it. Each processed Parquet file is copied into the release and converted to deterministic gzip-compressed UTF-8 TSV. Releases built by the current pipeline record path, format, byte size, SHA-256 hash, row count, columns, processed source path, the CC BY-NC 4.0 identifier and URL, and the required citation in manifest.json. Raw network responses and browser sessions are never included.

The gate also names internal processed families that are not release payloads: companion/, historical_html/, the source-native opaque probe tables, the 2014 remarks derivative, and the supplemental evidence index. KBE185 is released only through current_governance_roster.parquet, never through its raw DOB-bearing table.

The release builder uses a temporary sibling directory and an atomic rename. It refuses to overwrite an existing snapshot directory.