Scenario A/B testing
A WOO PostgreSQL warehouse case study applying control, treatment, isolated-variable, and delta logic to financial planning scenarios.
Decision question
The failure mode in planning workflows is changing multiple assumptions at once or comparing against a stale baseline. This scenario engine treats each what-if plan like a controlled experiment over a reconciled cost warehouse.
Anchored the control condition to a reconciled monthly fact view instead of a fresh spreadsheet pull.
Modeled scenarios as first-class data rows in analytics.scenario, with multipliers and active date windows.
Seeded a baseline row and an isolated vendor-spend treatment row for controlled comparison.
Joined scenarios to the baseline monthly fact view so comparisons stay aligned by month.
Computed total_cost_monthly_scn and delta_monthly directly in SQL.
Added validation queries for scenario existence, row counts, annual totals, and ranked deltas.
Kept the method honest: this is financial scenario testing, not live traffic-split product experimentation.
Experiment frame
The framework borrows A/B testing discipline and applies it to financial scenario modeling. It does not claim live product traffic splitting.
Control
The reconciled, QA-checked monthly cost baseline used for executive reporting. It is the trusted comparison condition.
Treatment
A named scenario row with vendors_multiplier = 0.90 while payroll and commission remain at 1.0, isolating one lever.
Effect
A computed SQL output showing treatment minus control at the monthly grain, then rollable into annual planning views.
Scenario engine SQL
1CREATE TABLE IF NOT EXISTS analytics.scenario (2 scenario_id bigserial PRIMARY KEY,3 scenario_name text UNIQUE NOT NULL,4 payroll_multiplier numeric NOT NULL DEFAULT 1.0,5 vendors_multiplier numeric NOT NULL DEFAULT 1.0,6 commission_multiplier numeric NOT NULL DEFAULT 1.0,7 start_month date NOT NULL DEFAULT '2025-01-01'::date,8 end_month date NOT NULL DEFAULT '2025-12-01'::date,9 created_at timestamptz NOT NULL DEFAULT now(),10 updated_at timestamptz NOT NULL DEFAULT now()11);1213INSERT INTO analytics.scenario (14 scenario_name,15 payroll_multiplier,16 vendors_multiplier,17 commission_multiplier,18 start_month,19 end_month20)21VALUES22 ('Baseline (1.0x)', 1.0, 1.0, 1.0, '2025-01-01'::date, '2025-12-01'::date),23 ('Test: Vendors -10%', 1.0, 0.90, 1.0, '2025-01-01'::date, '2025-12-01'::date)24ON CONFLICT (scenario_name)25DO UPDATE SET26 payroll_multiplier = EXCLUDED.payroll_multiplier,27 vendors_multiplier = EXCLUDED.vendors_multiplier,28 commission_multiplier = EXCLUDED.commission_multiplier,29 start_month = EXCLUDED.start_month,30 end_month = EXCLUDED.end_month,31 updated_at = now();Leadership can ask what-if questions without waiting for a bespoke spreadsheet model.
Each test keeps assumptions visible: scenario name, multiplier values, and date window.
Changing one lever no longer risks accidentally changing multiple assumptions at once.
The same framework can support vendor, payroll, commission, or combined planning scenarios.
Scope and method note
This is scenario / what-if A/B testing applied to financial planning: a defined control, an isolated treatment, and a computed effect. It is not live user-facing product experimentation, traffic splitting, or statistical significance testing. Raw payroll/vendor data is intentionally excluded for privacy.