Synthetic data

Table of Contents

How a pipeline uses it
Cardinality
Retail example

The Synthetic data transform (plugin id SyntheticData, category Input) generates seeded rows for demos, samples, and tests. It replaces one-off Python generators when the shape of the data is sequences, choices, ranges, templates, and parent-child relationships.

Apache Hop’s Fake transform decorates an existing row with realistic names and addresses. Generate rows emits constants. Synthetic data covers the cases those do not: stable ids, foreign keys, a sample of existing keys plus a block of new keys, status paths, unique pairs, and an optional nested XML document.

How a pipeline uses it

Drop Synthetic data on the canvas. It has no main input.
Set Seed and Cardinality, then add output fields.
For parent-child data, draw an info hop from the parent transform and set Parents transform (drawing the hop fills the name). Copy rows to every target (distribute off) when one generator also writes a file.
Write flat files with Text file output. The transform itself does not write CSV.

Each copy keeps its own random sequence. Use one copy, and give every copy its own seed expression such as '${SEED}+3'. The same seed and parameters always produce the same rows. A pipeline must not assume that two transforms share one Python-style random stream.

Dialog reference: Synthetic data dialog.

Cardinality

Mode What it emits

Mode	What it emits
`COUNT`	A fixed number of rows. A count of `0` still lets Text file output write a header when the file is created at start.
`COMBINE`	The union of id populations. `RANGE` is `start .. start+count-1`. `SAMPLE` draws unique ids from `from..to` and emits them in sorted order. A population whose count expression is `0` is skipped. `SEQUENCE` with `source=population` formats that id.
`PER_PARENT`	One or more child rows for a sample of parent rows.
`PATHS`	Zero or more step rows for each selected parent. The selector field picks the first matching path. `*` is the fallback. An empty step list emits nothing.
`UNIQUE_PAIRS`	Distinct pairs from two info streams, or from two integer ranges, capped at the cartesian product.
`HIERARCHY`	Flat rows at the leaf of a parent/child join, with optional splitting into groups. The same rows can be written as nested XML.

COUNT

A fixed number of rows. A count of 0 still lets Text file output write a header when the file is created at start.

COMBINE

The union of id populations. RANGE is start .. start+count-1. SAMPLE draws unique ids from from..to and emits them in sorted order. A population whose count expression is 0 is skipped. SEQUENCE with source=population formats that id.

PER_PARENT

One or more child rows for a sample of parent rows.

PATHS

Zero or more step rows for each selected parent. The selector field picks the first matching path. * is the fallback. An empty step list emits nothing.

UNIQUE_PAIRS

Distinct pairs from two info streams, or from two integer ranges, capped at the cartesian product.

HIERARCHY

Flat rows at the leaf of a parent/child join, with optional splitting into groups. The same rows can be written as nested XML.

Expressions are numeric: integers, variables, + - * /, parentheses, max, min, div (integer division), mod, round, and ifEq(left, right, whenEqual, whenNot). dateDiff(later, earlier) is a day count. Hop variables are resolved first. ${field} that is not a variable reads a row field.

Retail example

retail-example/pipelines/generate-retail-data.hpl builds the CRM CSV wave and files/asn_${RETAIL_CSV_WAVE}.xml. Workflows run-retail-initial-setup-db and run-retail-update-db-update run that pipeline instead of generate-retail-data.py.

Initial defaults: seed 42, 10,000 customers, 1,000 products, 50 warehouses, 25 sales reps, wave initial, load date 2024/01/01 00:00:00.000. The order scale parameter ORDERS stays 100000. Each wave, including the initial load, emits max(50, ORDERS/100) orders (1,000 at that default), so the opening month matches a monthly update. When ORDERS is smaller than that sample, the initial wave is capped at ORDERS. Later order ids still start at ORDERS+1. pipelines/next-retail-load.hpl reads the latest row in retail_load_log and adds MONTHS with Calculator Date A + B Months (0 on initial, 1 on update). That date is LOAD_DATE. RETAIL_CSV_WAVE is initial or yyyy-MM. WAVE_INDEX is how many loads are already in the log, so the first update is 1 and the March update is 2. New ids do not overlap across waves (start = base + (wave-1)*sample + 1, sample max(50, n/100)). Shipment message_id values mix in the wave index so later months do not repeat earlier UUIDs. Hubs emit new ids only. Customer satellites emit a sample of existing ids plus the new ids.

The generated files follow the same names and columns as before. They are not byte-identical to the historical Python output, because each transform has its own seed instead of one shared random.Random. Do not treat the committed retail-example/files/*_initial.csv snapshot as an oracle for this transform. A full initial run overwrites those files locally.

asn_demo.xml stays a hand-written fixture. Generation writes asn_${WAVE}.xml only.