Reference walkthrough for advanced Data Vault and Business Vault patterns in the integration-tests/ project: multi-satellite Customer 360, external read-only raw vault tables, and golden-dataset regression suites.

To build an EDW from scratch, start with Getting started — building an EDW. To tour the finished retail sample (CSV → DV → BV → dimensional), see Getting started — retail example.

Estimated time: half a day with Hop GUI, or follow along read-only first.

Prerequisites

  1. Register integration-tests/ as a Hop project named hopper-edw.

  2. Configure database connections CRM and Vault in project metadata (metadata/rdbms/CRM.json, metadata/rdbms/Vault.json).

  3. Install Data Hopper EDW plugin in Apache Hop 2.20.0-SNAPSHOT.

  4. Open the project in Hop GUI so ${PROJECT_HOME} resolves.

Full environment details: Integration tests Hop project.

Chapter 1 — Data Vault concepts

Data Vault 2.0 uses three building blocks:

  • Hub — a core business entity (Customer, Product) identified by business keys

  • Link — a relationship between hubs

  • Satellite — descriptive history attached to a hub or link; insert-only when attributes change

Record sources in the Data Catalog describe where staging data lives (connection, table, columns, record-source indicator).

The plugin stores hubs, links, and satellites in a Data Vault model (.hdv file) with embedded configuration (hashing, target database, column names).

Chapter 2 — Explore sources and the basic model

  1. In Hop GUI, open integration-tests/tests/basic/vault1.hdv.

  2. Click Edit model and note the target database (Vault) and hashing settings.

  3. Inspect hubs, links, and satellites on the canvas.

  4. Open a hub dialog — review business keys and Record sources.

Sources for this model live in the data catalog under catalog-data/hop/integration-tests/sources/ (CRM-customer, CRM-order, etc.).

Optional: run integration-tests/run-tests.sh tests/basic/update-vault1.hwf from the command line to see a full load without using the GUI.

Chapter 3 — Validate and inspect pipelines

On the vault1.hdv toolbar:

  1. Click Check model — fix any errors before continuing.

  2. Click Generate DDL — preview CREATE/ALTER for the Vault database.

  3. Select sat_customer, then Debug (or Show update pipeline) — inspect the generated insert-only satellite pipeline.

The same generation logic runs in the Data Vault Update workflow action.

Reference: Plugin overview, Data Vault Update action.

Chapter 4 — Run a raw vault load

  1. Open workflow integration-tests/tests/basic/update-vault1.hwf.

  2. Run it (or run via integration-tests/run-tests.sh).

The workflow creates CRM tables, loads CSVs, drops vault tables, runs Data Vault Update twice (initial + incremental), and validates against golden datasets.

You now have raw satellite history in the Vault database — the input for Business Vault.

Chapter 5 — Introduce Business Vault (single satellite)

  1. Open integration-tests/tests/basic/vault1.hbv.

  2. Note Data Vault model file → ${PROJECT_HOME}/tests/basic/vault1.hdv.

  3. Open the SCD2 table sat_customer_hb — it references satellite sat_customer.

  4. Click Edit model on the BV toolbar — target database, valid from/to defaults, sentinels.

  5. Check model, then Show build pipeline on the SCD2 table.

This is the simplest BV pattern: one satellite → one functional SCD2 table.

Reference: Business Vault overview, Business Vault SCD2.

Chapter 6 — Multi-satellite Customer 360

  1. Open integration-tests/tests/multi-satellite-bv/customer-360.hdv — one hub, four satellites.

Customer 360 Data Vault model — hub and four satellites

  1. Open integration-tests/tests/multi-satellite-bv/customer-360.hbv.

Customer 360 Business Vault model — customer_360_bv SCD2 table

  1. Open SCD2 table customer_360_bv:

    • General — parent hub hub_customer, build mode Full rebuild

    • Derivatives — four satellites

    • Field mappings — rename attributes to cust_* columns

    • Satellite settings — per-satellite source indicator

SCD2 table dialog — General tab
SCD2 table dialog — Derivatives tab with sat_customer_demo
SCD2 table dialog — Field mappings tab
SCD2 table dialog — Satellite settings tab

  1. Check model — multi-satellite rules require explicit mappings.

  2. Reload DV model (toolbar) if you edited the .hdv on disk.

  3. Show build pipeline — expect multiple TableInput branches, SortedSchemaMerge, RepeatFields, analytic validity, TableOutput.

Run integration-tests/tests/multi-satellite-bv/update-customer-360.hwf for end-to-end DV load + BV rebuild + validation.

For incremental SCD2, open customer-360-incremental.hbv (same mappings, buildMode=INCREMENTAL) and run update-customer-360-incremental.hwf. That workflow runs the same DV load waves, then Business Vault Update with the incremental pipeline, and validates against the same golden current-state dataset.

For a large-table Full rebuild split, open customer-360-partitioned.hbv (hashKeyPartitionCount=4) and run update-customer-360-partitioned.hwf. Same golden current-state dataset; the target is truncated once, then four hash-key slices are loaded.

Chapter 7 — External read-only raw vault tables

When dbt or another tool loads raw vault tables, Hop can still model them for Business Vault.

  1. Open integration-tests/tests/multi-satellite-bv/customer-360-external.hdv.

  2. Open each satellite dialog → Options tab → Integration mode = External read-only.

  3. Canvas shows (ext) on those tables.

Customer 360 external Data Vault model — satellites marked ext

  1. Open customer-360-external.hbv — same BV SCD2 definition as the hop-managed variant.

  2. Check model and Show build pipeline — BV reads satellite history from the Vault database using metadata only; Data Vault Update skips DDL/pipelines for external tables.

Reference: Data Vault integration modes.

Chapter 8 — Production workflow pattern

Typical batch:

  1. Data Vault Update on .hdv — load raw vault (skip external tables automatically)

  2. Business Vault Update on .hbv — rebuild SCD2 tables

Configure pipeline run configuration, parallel copies, optional DDL and catalog publish on each action.

Reference: Business Vault Update action.

Chapter 9 — Troubleshooting

Symptom What to check

Cannot load linked .hdv

${PROJECT_HOME} set; path on .hbv correct; use Reload DV model

SCD2 pipeline: target database required

Vault connection in project metadata; target database on both .hdv and .hbv; run Check model

Validation passes but pipeline fails

Reload DV model before generate; stale cache was a common cause

External satellite empty in BV

Warehouse table must exist and match modeled attributes; Vault connection must read it

Type check errors on hub/satellite

Run Check model with detailed type checking; align source catalog with live CRM schema

Chapter 10 — Next steps

Topic Document

Primary tutorial (retail)

getting-started-retail

Feature overview

feature-overview

Plugin overview

datavault-plugin

Business Vault reference

business-vault-overview

CI / regression project

integration-tests/PROJECT.md

Executive summary

hop-data-vault-overview slides

Run the full test orchestrator when ready:

cd integration-tests && ./run-tests.sh