Data Vault configuration is project metadata that many .hdv files can share. Put hashing, naming, target database, catalog, and load settings in one object (Metadata perspective → EDW → Data Vault configuration) and point each model at it with Configuration on the Edit model dialog.

Older models may still embed a <configuration> block. Those files keep working. Use Extract to project configuration… when you want to stop copying the same settings across hub and link models.

There is one target database per configuration. Hashing and naming rules apply to every hub, link, and satellite in models that reference that configuration, unless a table dialog provides a specific column-name override (for example a per-hub hash key field name).

The same pattern exists for source models, Business Vault, and dimensional models.

Configure EDW setup

A new Hop project is offered the standard catalog and the four configuration objects the first time you open a source, Data Vault, Business Vault, or dimensional model.

If you dismissed that prompt, or want to create only the elements that are still missing, open Tools → Configure EDW setup…​. The same dialog is shown even when “Do not show this message again” was checked. Existing elements stay selected and disabled; missing ones can still be created. Uncheck Do not show this message again to restore the automatic first-time prompt.

The dialog does not create a resource definition group. After you have a .hdv, create the group yourself — Getting started, chapter 3.

Configure EDW setup — standard catalog and model-configuration elements

Edit model dialog

Open the configuration from the visual model editor with the Edit model toolbar button. The dialog has a name, description, and a Configuration picker at the top. When the picker is empty (legacy embedded model), the same tabs edit the copy stored in the .hdv file. When a project configuration is selected, edit the shared object from the Metadata perspective (or the picker’s New / Edit buttons).

Edit Data Vault Model — vault1 uses configuration data-vault-vault1

When a project configuration is selected, the hashing and load tabs below are edited on that configuration (Metadata perspective → EDW → Data Vault configuration, or the picker buttons). The screenshots use configuration data-vault-vault1.

General tab

Target database and hashing strategy.

Data Vault configuration data-vault-vault1 — General tab
Option Description

Target database

The Hop database connection where Data Vault tables live. Hop-managed models create and load tables here. Read-only existing vaults only read from this connection (Business Vault and dimensional loads).

Read-only existing vault

When enabled, every .hdv that uses this configuration documents an already-built raw vault. Hop refuses Data Vault Update, Generate DDL, and Debug. Model check keeps referential integrity only (hash keys, parent hubs/links, names, attributes). Use a dedicated configuration object — do not share it with Hop-managed models. Mixed tables in one file still use per-table External read-only.

Data catalog

Optional catalog connection used when resolving DV_SOURCE feeds for models that use this configuration.

Hash algorithm

Algorithm for surrogate hash keys (MD5, SHA-256, SHA-1, SHA-512). Affects computation and maximum key length.

Hash key data type

Physical storage for hash keys: HEX (default), STRING, or BINARY. BINARY is the most compact form and sorts correctly on the required Hop 2.19.0+ runtime (see Hop issue 7346).

Hash content casing

Upper-case, lower-case, or preserve business key casing before hashing. UPPER is the usual Data Vault choice.

Business key delimiter

String used to join multipartite business key parts for hashing, and to compose stored composite hub BK strings (for example `

` or #). Also used for link hash inputs over hub hashes.

Hash content prefix / suffix

Optional text prepended or appended to the concatenated key content before hashing. Supports variables. A trailing suffix is common for VaultSpeed-style recipes and is applied to the hash input only — it is not written into a composed hub business-key column.

Null placeholder

Value substituted for NULL key parts during hashing and during stored composite BK composition (default ^^).

Trim business keys

Remove leading and trailing whitespace before hashing and before composing stored composite hub BKs (recommended, on by default).

Hash composed business key

When enabled, hub hashing for composite (single-column) business keys uses the composed vault field only. Default is off: hash ordered source parts (better VaultSpeed trailing-suffix parity). Satellites and links always map source parts for parent/hub hash calculation; they do not store the composed BK column.

Unknown records tab

Controls generation of the classic Data Vault "unknown" / ghost sentinel row in hubs and links.

Data Vault configuration data-vault-vault1 — Unknown records tab
Option Description

Generate unknown record

When enabled, the model expects an unknown sentinel row in hubs (and corresponding link entries). The Data Vault Update action can insert missing sentinel rows before loading when Ensure unknown and invalid records is enabled.

Unknown business key value

Business key value stored in unknown hub rows. Also used to compute the hub hash when Unknown hash key value is empty.

Unknown hash key value

Optional fixed hash key for unknown hub rows. Supports variables and Hop hexadecimal expressions. When empty, the hash is derived from the unknown business key value.

Unknown link hash key value

Optional fixed hash key for unknown link rows. When empty, derived from participating hub unknown hash values.

Unknown record source

RECORD_SOURCE value written into unknown sentinel rows (default UNKNOWN).

Invalid records tab

Controls generation of an "invalid" / error sentinel row for malformed or rejected keys.

Data Vault configuration data-vault-vault1 — Invalid records tab
Option Description

Generate invalid record

When enabled, the model expects an invalid sentinel row in hubs and links. Missing rows can be inserted by the Data Vault Update action.

Invalid business key value

Business key value for invalid hub rows. Used for hash computation when Invalid hash key value is empty.

Invalid hash key value

Optional fixed hash key for invalid hub rows (default all FF bytes for MD5-style keys).

Invalid link hash key value

Optional fixed hash key for invalid link rows.

Invalid record source

RECORD_SOURCE value for invalid sentinel rows (default INVALID).

Orphan handling tab

Optional policy when a child feed (link or satellite) arrives before its parent hub, or with a null business key. Default is PASS (same as previous releases): generated loads hash the key and insert the child even if the hub row does not exist yet.

This is not a new warehouse layer. It is a model policy plus generated pipeline fragments. Late-arriving dimensions (facts before dim members) are a separate dimensional feature (issue #109).

Data Vault configuration data-vault-vault1 — Orphan handling tab
Option Description

Orphan handling policy

PASS — current behavior (dangling hashes if FKs are off). INFER — insert a placeholder hub row with the real business key and RECORD_SOURCE=INFERRED. SENTINEL — remap null or missing parents to the unknown ghost row. QUARANTINE — write the child row to the quarantine table and skip the load. FAIL — abort the pipeline.

Inferred record source

Value stamped on placeholder hub rows (INFERRED by default).

Store inferred flag

When enabled, hub tables gain a boolean column (default is_inferred). Off by default so existing DDL does not change.

Inferred flag field

Column name for that flag.

Quarantine table

Shared table on the vault database (dv_orphan_quarantine by default). Created on first Data Vault Update when any table uses QUARANTINE.

Link and satellite dialogs can override the model policy (value INHERIT keeps the model default). The hub dialog has Allow inferred inserts (on for managed hubs; infer is refused for external hubs).

The cheapest fix for valid child-before-parent keys is still to also load the parent hub from the child feed: use Also load parent hubs…​ on the link (or satellite) dialog, or enable Also load parent hubs from child/link feeds when generating a vault from a source model. That reuses the existing hub→link→sat load order.

Standard columns tab

Names and patterns for columns present on every Data Vault table.

Data Vault configuration data-vault-vault1 — Standard columns tab
Option Description

Load date field name

Column for the batch load timestamp (default LOAD_DATE). Every generated pipeline writes the same load date for all rows in one action run.

Load end date field name

Column name for an optional load end timestamp on satellites when end-dating is used.

Record source field name

Column that identifies the originating feed (default RECORD_SOURCE).

Record source field length

Character length of the record source column in target tables. Supports variables (default 100).

Store load cycle ID

When enabled (off by default), every Data Vault table includes an integer load cycle ID column. Each Data Vault Update allocates the next value from a durable control table on the target database and stamps that same ID on all rows written in the run (including special/sentinel rows).

Load cycle ID field name

Column name for the load cycle ID (default LOAD_CYCLE_ID). Supports variables. Used only when Store load cycle ID is enabled.

Load cycle control table

Physical table on the model target database that holds the counter (default dv_load_cycle). Created automatically on first allocate. Supports variables. The allocated value is also published as workflow variable DV_LOAD_CYCLE_ID.

Use load end date pattern

Documents that satellites use an end-dating approach. Generated pipelines today perform insert-only change detection; this flag is mainly for documentation and future extensions.

Generate primary keys in DDL

When enabled, generated CREATE TABLE statements include a PRIMARY KEY on table identity columns: hub hash key; link hash key; satellite parent hash key + multi-active driving key (if any) + load date. Off by default. Does not retrofit constraints onto existing tables via ALTER. Enabling foreign keys also adds primary keys on parent hubs/links so references are valid. When that primary key is not generated, update DDL adds a non-unique index on the same columns (satellite grain is parent hash, driving key, and load date). A later structure update adds the index on a table that already exists. Snowflake is skipped.

Generate foreign keys in DDL

When enabled, generated CREATE TABLE statements include FOREIGN KEY constraints for model relationships (link → hubs, satellite/STS → parent hub or link). Off by default. Skipped on engines that do not support foreign keys (for example SingleStore). Applies only to CREATE TABLE, not ALTER drift. A child feed that is not listed as a hub record source is the usual Data Vault pattern and is not a model-check error. If that child can arrive before the parent, seed the hub or use an orphan policy other than PASS.

Caution (hub multi-source + bulk load)

Hubs and links with multiple record sources generate one pipeline per source. The Data Vault Update action wraps those pipelines in a serial multi-source workflow so sources for the same physical table run one after another (each subsequent source sees keys already loaded). Free pipelines (satellites, single-source hubs/links) still run in the parallel orchestrator. This is required for safe primary-key enforcement under bulk load.

Target loading tab

Options for how generated pipelines write rows to target tables.

Data Vault configuration data-vault-vault1 — Target loading tab
Option Description

Target table batch size

Commit size for Table Output transforms in generated pipelines (default 1000). Supports variables.

Target table parallel copies

Parallel copies of Table Output (default 1). A variable such as ${OUTPUT_COPIES} is stored on the transform and resolved when the pipeline runs. In staging-file bulk mode, copies write separate shard files.

Target load mode

Table Output, native bulk, or staging file. The choices depend on the target engine.

Bulk load staging folder

Folder for staging-file mode. The default is ${java.io.tmpdir}/dv2/bulk/.

Bulk load delimiter / enclosure / encoding

CSV settings for that staging file. Defaults are comma, double quote, and UTF-8.

Bulk load requires local file

Staging files must be on the local filesystem. MySQL LOAD DATA LOCAL INFILE needs this.

Sort rows in memory

Number of rows Sort Rows keeps in memory before spilling to disk in generated update pipelines (default 1000000). Used where a merge key is computed in Hop (for example satellite/link hash sort). Hub updates rely on SQL ORDER BY only (no Sort Rows). Supports variables.

Execution logging level

Log level for the generated load pipelines.

Create shard key on hash key (SingleStore)

When the target is SingleStore, CREATE TABLE can append SHARD KEY on the hash key.

Include driving keys in shard key (SingleStore)

For a multi-active satellite, include the driving key with the parent hash in the shard key.

On SQL Server (and PostgreSQL with ICU/locale collations), generated hub (and satellite target) SQL automatically inspects live source/target column collations and types. When a sort risk is detected (for example NVARCHAR vs VARCHAR, or different collations) and both legs use the same database engine, the same bridge COLLATE is applied on both source and target ORDER BY expressions so Merge Rows streams stay aligned without manual configuration. Collation names are not portable across engines: a SQL Server source with a PostgreSQL vault does not inject French_CI_AS (or similar) into Postgres ORDER BY (issue #108). Vault VARCHAR columns are created with the UTF-8 collation Latin1_General_100_CI_AS_SC_UTF8 (SQL Server 2019+) so Unicode values store without requiring NVARCHAR. Because SQL Server measures UTF-8 VARCHAR(n) in bytes while source/NVARCHAR lengths are character-oriented, generated DDL multiplies string lengths by three (and promotes to VARCHAR(MAX) when that exceeds 8000) so multi-byte values such as accented text cannot truncate on insert.

Unicode capability (EDW requirement)

Data Vault, Business Vault, and dimensional models require a Unicode-capable target database. Model check (and update actions that enforce model check) refuse to treat non-Unicode targets as valid EDW destinations by default:

  • SQL Server — database default collation must end with _UTF8 (for example Latin1_General_100_CI_AS_SC_UTF8). Requires SQL Server 2019 or later.

  • PostgreSQL — database encoding must be UTF8.

  • MySQL / MariaDB / SingleStore — schema character set must be utf8mb4 (plain utf8 / utf8mb3 or latin1 is rejected).

utf8 vs utf8mb4 (MySQL family)

On MySQL, MariaDB, and SingleStore, the historical charset name utf8 is not full Unicode: it is a 3-byte encoding (BMP only, historically utf8mb3). Supplementary-plane characters (many emoji, some rare CJK and symbols) need 4 bytes and therefore utf8mb4. Everyday Western European text often fits in both; full EDW Unicode storage does not.

If the vault database is unreachable during check, a warning is raised so offline design can continue; when connectivity is available and the encoding is wrong, an error blocks the update when enforcement is enabled (see plugin-wide options below).

Generated artifacts tab

Where generated update pipelines and bulk-load workflows are named and saved.

Data Vault configuration data-vault-vault1 — Generated artifacts tab
Option Description

Generated artifact folder

When set, each generated update pipeline (.hpl) and bulk-load workflow (.hwf) is saved in this folder before execution. Supports variables (for example ${PROJECT_HOME}/generated-pipelines). Leave empty to run in memory only.

Hub pipeline name prefix

Prefix for hub pipeline names (default hub-). Full name: prefix + target table name + - + source name.

Link pipeline name prefix

Prefix for link pipelines (default link-).

Satellite pipeline name prefix

Prefix for satellite pipelines (default sat-).

Bulk workflow name prefix

Name prefix of the master workflow when target load mode is staging file (default `DV Bulk Update - ` plus the model name).

Plugin-wide GUI options

In addition to the per-model configuration above, Hop GUI exposes global options under Configuration → Hopper EDW:

Hopper EDW options in the Hop configuration perspective
Option Description

Draw hash keys in model

When enabled, hash key column names are drawn on table cards in the visual model editor.

Enforce Unicode-capable target databases (EDW)

When enabled (default), model check reports an error if the target fails the Unicode capability probe (for example SingleStore/MySQL schema charset utf8 instead of utf8mb4). When disabled, the probe still runs and failures become warnings only — useful for controlled migrations onto legacy charsets. Prefer recreating the EDW schema as utf8mb4 (or the engine-equivalent full Unicode setting) when operations allow.

Warn when timestamp fractional precision is reduced

When enabled (default), model check reports a warning if a source date/time has more fractional-second digits than the target layout or engine allows (for example nanosecond timestamps stored as SingleStore/MySQL DATETIME(6) microseconds — the best available mapping). Never an error. Disable when truncation is accepted.

Remind to publish catalog feeds when saving a source model

When enabled (default), saving a .hsm file compares published catalog feeds to the canvas (effective layout, including data type mappings). If any published feed is stale, a dialog asks whether to publish those feeds. Don’t remind me after save turns this option off; re-enable it here.

Maximum undo/redo operations kept in memory

Number of gzip-compressed XML snapshots kept for undo/redo on .hdv files (default 200). Older snapshots are discarded when the limit is exceeded.

Load runs shown in the model metrics panel

How many recent executions the panel to the right of a model lists (default 20).

These settings are stored in the Hop GUI configuration, not inside individual .hdv files.

Per-table column overrides

While the model configuration drives hashing and most column naming, individual table dialogs still allow targeted overrides where that makes sense:

  • Hub – Optional hash key field name and record source field name overrides.

  • Link – Optional link hash key field name and record source field name overrides.

  • Satellite – Uses the parent hub’s hash key column name for the foreign key in the satellite table.

There is no longer a separate per-table "configuration picker" metadata reference; all hashing rules come from the model’s embedded configuration.

How the configuration affects daily work

  • The target database connection is required for Generate DDL, Debug, and the Data Vault Update action on Hop-managed models. Business Vault SCD2 generation also reads satellite history from this connection when a .hbv model links to the .hdv file.

  • Read-only existing vault refuses those load/DDL actions. The target database name must still be set so Business Vault and dimensional models can read the documented tables. See Data Vault integration modes.

  • Tables marked External read-only do not require Hop to load data, but the target database name must still be set so downstream features can resolve the connection.

  • Hash algorithm and data type directly influence storage size and join performance.

  • Naming settings (LOAD_DATE, RECORD_SOURCE, etc.) should be agreed early with consumers of the Data Vault.

  • Unknown and invalid sentinel settings matter when downstream facts or links must reference "missing" or "bad" keys.

  • Check model warns when no target database is configured, because DDL, hashing, and pipeline generation depend on it.

Configure these settings when you create or adopt a model. They are persisted inside the .hdv file and travel with the model between environments (subject to connection name resolution in each Hop project).