A Data Vault Source is a Data Catalog record definition of type DV_SOURCE. It describes a logical feed that supplies rows to hubs, links, and satellites. Model tables reference the source name, not a raw database connection.

Each definition combines:

  • DV semantics — record source indicator, optional group for partial workflow loads

  • Physical layout — database table, CSV / Parquet / Iceberg, composite multi-table query, JSON extraction, or pipeline feed from a source model (.hsm)

Sources live under namespace hop/{project}/sources in the FILE catalog storage directory (retail example: work/edw-catalog/) and open in the Data Catalog perspective. See Data Catalog for folder layout and refresh behavior.

Field layout is stored on structured homes (dvSource.fields and, for table-backed sources, physical table fields). Do not invent a parallel layout blob in catalog JSON.

Composite sources

A COMPOSITE DV_SOURCE is published from a source-model source query. It stores:

  • Projected field list (aliases used when mapping hub/sat attributes)

  • Path to the .hsm file and the query name (generation prefers live SQL from the model)

  • Optional cached SQL when the model file is unavailable

Data Vault Update builds either a single-connection Table Input or a Merge Join pipeline depending on the query’s generation mode. See Source modeler overview.

JSON sources

A JSON DV_SOURCE is published from a source-model Source JSON object. It stores:

  • Projected field list (pass-through keys + JsonPath extractions)

  • Path to the .hsm file and the Source JSON name

  • Parent reference (table / query / nested JSON source) and the JSON field to parse

At load time, Data Vault Update injects the generated parent + JsonInput graph, then:

  • Adds a Constant (or rename) for the vault record-source column when the feed uses a static sourceIndicator

  • For hubs, sorts and distincts identity fields so MergeRowsPlus CDC matches database/file sources on incremental loads

Retail sample: feed_order_shipment_tracking over order_shipment_event.payload → hub_order_shipment / link / sat.

Create and edit a source

  1. Open the Hop project.

  2. Open the Data Catalog perspective.

  3. Under hop/{project}/sources, create or open a DV_SOURCE record.

  4. Set connection, table, and fields (database-backed sources).

  5. Configure record source indicator and optional group.

The model toolbar Import sources action can also create catalog entries from live database metadata.

JSON structure (file catalog)

Each source is a JSON file. Important fields:

Field Description

namespace

Must match folder path (for example hop/retail-example/sources)

name

Unique within namespace; used in hub Record sources and satellite Default record source

type

DV_SOURCE

physicalTable

Database connection name, schema, table (databaseMetaName, schemaName, tableName)

dvSource.sourceIndicator

Optional fixed record source value for every row (for example CRM, ERP)

dvSource.sourceIndicatorField

Optional source column for per-row record source (used when indicator is empty)

dvSource.group

Optional tag for Record source group on Data Vault Update

dvSource.fields

Authoritative column layout (types, lengths, primary-key / FK flags, CSV options). Location refs (physicalTable / physicalFile) do not store the contract.

Example path: retail-example/work/edw-catalog/hop/retail-example/sources/E2E-customer-hub.json.

Record source indicator at runtime

The plugin applies this precedence:

  1. If source indicator is non-empty (after variable resolution), that static value is used for all rows.

  2. Otherwise, if source indicator field is set, the pipeline reads that column and aliases it to the model’s record source column name.

  3. If neither is set, no indicator is added from this source.

The target column name comes from .hdv model configuration unless a hub or link overrides it.

Database field layout

Database-backed sources store per-column metadata used by Load from source, hashing, and type checking. See Database source details for the field editor and Import from live schema.

Keeping the field list accurate is essential for Check model and detailed data type checking.

Groups and partial loads

Set group on a source when only some feeds should run in a batch (for example nightly vs hourly). On Data Vault Update, set Record source group to match; sources with a different or empty group are skipped for that run.

Validation

  • Catalog validation — feed-level checks, proposals, acknowledgements; see Resource definition validation

  • Check model — verifies every referenced source exists and fields are present; optional live schema type checks

When detailed type checking is active, database sources resolve types from the live table. Stored catalog metadata remains the baseline; drift from the live schema produces warnings.

Practical advice

  • Name sources after the business feed, not only the physical table.

  • Agree on record source value conventions across the program.

  • Use group for scheduled partial loads without maintaining separate models.

  • Prefer a static source indicator when all rows share one feed identity.

  • Refresh the catalog (F5) after editing JSON files on disk.

Catalog type PIPELINE points at a Hop pipeline (.hpl) and an output transform. Field layout is the declared contract (from the source modeler pipeline card or catalog fields). Data Vault Update injects the pipeline with MetaInject using those declared output fields.

Typical authoring path:

  1. Add a Source pipeline card on a .hsm model.

  2. Design convert/filter/aggregate transforms; set the output transform.

  3. Get fields from transform (or edit the declared field list), mark logical keys, Publish to catalog.

  4. Bind hubs/links/satellites to the published feed name.

See Source modeler overview.