Why it matters

A warehouse that binds to physical table names breaks the first time a table is renamed. Data Hopper EDW avoids that by keeping the contract in one place: the Data Catalog. A source model (.hsm) imports the structure of a source system and publishes it to the catalog as feeds; hubs, links, satellites and every later layer bind to the feed name. Rename a column in the catalog and every model that uses it knows — rename it only in the database and nothing does.

This is the first step of the build order, before any vault model exists. It shows what a catalog feed is, how a source model produces one, and how to keep the two in step. No warehouse load is needed; you read the retail-example project and publish feeds of your own. Budget 30-40 minutes.

Concepts

The catalog is the contract bus

The catalog is a bus with two directions. In: source feeds (types DV_SOURCE, COMPOSITE, JSON, PIPELINE) published from a source model. Out: published DV / BV / DM table layouts, lineage siblings and operations stubs — written when Update resource definition group (or a per-model update) runs with catalog publish enabled. Harvest does not rewrite working-tree contracts; apply-harvest and refresh-from-source are explicit actions.

The Data Catalog is the contract bus

Namespaces, and where a feed lives

A feed is a JSON record definition under a FILE catalog storage directory. The path after the storage root must match the record’s namespace, and the catalog key is namespace/name. You browse feeds in the Data Catalog perspective and refresh with F5 when files change on disk. The default sources namespace is derived from the open project folder: hop/{project-folder-name}/sources, so retail-example/ publishes under hop/retail-example/sources.

Models publish into further namespaces: hop/{project}/models/{model-name}/ for raw vault table layouts, hop/{project}/business-vault/{model-name}/ for BV targets, hop/{project}/dimensional/ for dimensional tables, and derived hop/{project}/lineage/…​ records. So “the catalog lost my feeds after a load” is usually a look in the wrong place: publish writes targets under models, while your source feeds stay under sources.

Record definitions: the DV_SOURCE feed

A Data Vault Source is a catalog record definition of type DV_SOURCE. It describes a logical feed that supplies rows to hubs, links and satellites, and it combines two things:

  • DV semantics — a record source indicator and an optional group for partial loads.

  • Physical layout — a database table, CSV / Parquet / Iceberg, a composite multi-table query, a JSON extraction, or a pipeline feed.

Model tables reference the source name, not a raw database connection; that name is what appears in a hub’s Record sources and a satellite’s Default record source. Column layout lives in the record’s dvSource.fields[] — the authoritative contract. A physicalTable (or physicalFile) reference only says where the rows come from; it does not hold the columns. Keep the field list accurate, because that is what model checks and detailed data type checking read.

Table cards, and query, JSON and pipeline cards

A source model draws one card per feed, and the card type decides the catalog type that Push to catalog writes:

Card Catalog type What it adds beyond a physical table

Source table

DATABASE

The table with its columns and primary keys.

Source query

COMPOSITE

A named multi-table projection: driving table, ordered joins, column list, optional WHERE.

Source JSON

JSON

A named JSON extraction over a parent table, query or JSON source: JsonPath fields plus pass-through keys.

Source pipeline

PIPELINE

A feed that already exists as a Hop pipeline, with declared output fields.
The retail source model with an All customer info query dialog and its generated SQL

The retail example uses all of them. models/source-tables-crm.hsm models the CRM landing tables plus a query All customer info (feed_customer_enriched), a Source JSON order_shipment_tracking (feed_order_shipment_tracking) and a pipeline source asn-package-lines.

A Source JSON dialog — parent source

Import schema brings in more than columns

Import schema asks for an RDBMS connection and an optional schema filter, then for a multi-select of tables. It imports the columns and the primary keys, and it discovers the database’s foreign keys as relationship edges on the canvas. If the database has no foreign keys, the tables and primary keys still arrive; you draw the relationships yourself with Shift+drag (or middle-button drag) between cards.

Every card also has a Data type mapping tab, where project profiles and per-field overrides (rename, type, length, conversion mask, locale, time zone) shape the layout that publish and load actually use:

A source table’s Data type mapping tab with profiles

Catalog version tags

A catalog version freezes the source record definitions a resource definition group’s models use, under a semantic tag such as v1.0.0 or 2026-Q3-Release. Tags are a design-time baseline: the schema gate compares a live source or the working catalog against a frozen tag, and can do that on a build machine with no database at all. Snapshots are immutable (editing the working catalog after tagging does not change them), duplicate tags are rejected, and the model files themselves (.hdv / .hbv / .hdm) are not copied into a snapshot — those stay in Git. Only the FILE catalog implementation supports versions.

Two checks, two questions

Check model on the .hsm validates the model itself — unique table names, relationship endpoints, query driving tables, a non-empty projection — and, for any card that already carries a catalog feed name, that the published feed still matches the card’s effective layout. A stale catalog is an error, because the canvas is not what vault loads read. Catalog validation is the other question: feed-level contract checks such as a field added or removed, type or length drift, or a source that models use but the working catalog lacks. A passing model check still requires accurate definitions in the catalog; the two layers complement each other.

Guided walkthrough

Find a published feed

  1. Start Hop GUI and open the project folder retail-example/ (File → Open project).

  2. Open the Data Catalog perspective and expand the namespace hop/retail-example/sources.

  3. Double-click one feed, for example the customer feed the retail vault binds to, and read its fields: each row is a column with a Hop type, a length and a primary-key position. These are the contracts hubs and satellites bind to by name.

  4. Find the physical location reference on the record — the database connection, schema and table the feed reads. That pointer is an implementation detail of the feed; the field list is the contract.

  5. Press F5 (or the refresh action) after editing catalog JSON files on disk.

Read a source model

  1. Open models/source-tables-crm.hsm in the source modeler.

  2. Select a table card and use toolbar Browse. The canvas centers that card and draws only its neighbourhood one or two relationships deep, without rewriting the saved layout — the fastest way to read part of a source model you did not draw.

    The source modeler canvas with tables

  3. Double-click a table card to open it: columns, primary-key positions, and on the Data type mapping tab the profiles and overrides that shape the published layout.

    A source table dialog with columns and primary-key positions

  4. Open a query card and use Generate SQL / Preview data / Validate to see the join the feed will load. On the General tab, a Catalog feed name overrides the name used when publishing (empty means: use the query name).

Bring in your own tables

  1. Create a new source model (.hsm) in your own project and save it, for example models/crm.hsm.

  2. Edit model: set the Catalog connection to local-catalog and the Configuration to source-model.

  3. Toolbar Import schema: pick the CRM connection, select two tables that relate to each other, and import. Columns and primary keys arrive, and a database foreign key becomes a relationship edge.

  4. If no relationship appeared, Shift+drag from the child card to the parent card. Open the relationship dialog to check the child and parent multiplicity (the crow’s-foot symbols) if it is not what you expect.

  5. Toolbar Push to catalog. A multi-select dialog lists every publishable object — tables with columns as DATABASE, queries with a projection as COMPOSITE, JSON sources as JSON, pipeline sources as PIPELINE — all preselected. Deselect what you do not want and confirm. Feeds that already exist under the same name are overwritten.

  6. Refresh the Data Catalog (F5) and find your feeds under hop/{your-project-folder}/sources.

  7. Optional: run Check model on the .hsm. It reports unique names, relationship endpoints, query driving tables, and whether the published feeds still match the canvas. Fix what it reports, re-publish, check again.

Try it yourself

In the catalog, open one of the feeds you just published and read its field list. Then:

  1. Note the record’s namespace and name; together they form the catalog key that models will bind to.

  2. Change nothing yet, and press Tag version (Tag catalog version) on the catalog connection: choose the resource definition group that defines the source scope, then enter a tag such as v0.1.0. This freezes the current source contracts.

  3. Expand Catalog sources versions under that connection and open your tag read-only. Editing the working catalog now does not change it.

Checkpoint: your namespace lists at least two feeds, and each feed’s field list matches the columns you imported. If it does not, re-open the model, confirm the catalog connection under Edit model, and Push to catalog again.

Questionnaire

Answer the seven questions. Every answer comes with its reasoning, and Copy my results gives you a text summary to keep.

1. What is a catalog feed, and where does it live?

2. What does a DV_SOURCE record definition describe?

3. What happens on Push to catalog when a feed with the same name already exists?

4. Besides columns, what does Import schema bring into the model?

5. Why do hubs bind to a feed name instead of a physical table?

6. Scenario: a source table gains a nullable column, and nothing has been re-published since. What should you expect?

7. What is a catalog version tag for?

Where to go next

← Back to the training index