OpenHTF vs pytest for Hardware Testing

A practical comparison of OpenHTF and pytest for testing physical products — execution model, measurements vs assertions, fixtures vs plugs, operator interaction, output records — and when to use each or both.

Teams that already write Python ask this first. Both are Python, both are open source, both run "tests". They are built for different jobs.

One-line answer

pytest runs many independent test functions against software and reports pass/fail per function. OpenHTF runs one physical unit through an ordered sequence of steps and records measured values with limits, per unit, for the life of the product. Use pytest to test your code (including your plugs); use OpenHTF to test your hardware.

Side by side

	OpenHTF	pytest
Unit of execution	One DUT through one `Test` (a run)	A collection of `test_*` functions
Ordering	Explicit, sequential; flow control via `PhaseResult`, groups, checkpoints, branches	Unordered by design; ordering needs plugins
Result granularity	Measurements with value, limits, unit, marginal flag; phase outcome; test outcome	Assertion pass/fail; values are lost unless printed
Hardware access	Plugs: one instance per run, `tearDown` guaranteed, injected by decorator	Fixtures: scoped setup/teardown, injected by name
Operator	Built-in Operator UI and prompts	None; needs custom code
Serial-number tracking	`dut_id` on every record, prompted or set programmatically	None built in
Output	Structured test record → JSON, database via callbacks	Console, JUnit XML, plugins
Retries / conditional steps	`REPEAT`, `repeat_limit`, `run_if`, `BranchSequence`, `Subtest`	`pytest-rerunfailures`, `skipif`
Background sampling	Monitors	None
Typical consumer of results	Manufacturing, quality, repair — per unit, over months	Developers, CI — per commit

Where the difference shows

Measurements vs assertions

pytest

def test_supply_voltage(dmm):
    v = dmm.measure_vdc()
    assert 4.95 <= v <= 5.05, f"Vout out of range: {v}"

The value v exists only in the failure message. Six months later, "is Vout drifting upward across the lot?" cannot be answered from pytest output.

OpenHTF

@htf.plug(dmm=Dmm)
@htf.measures(htf.Measurement("vout").in_range(4.95, 5.05).with_units(units.VOLT))
def measure_supply(test, dmm):
    test.measurements.vout = dmm.measure_vdc()

Every run stores vout = 5.012 V, limits 4.95–5.05, PASS. Yield, Cpk and drift are queries over the records — see Manufacturing Test Analytics.

Fixtures vs plugs

pytest fixtures and OpenHTF plugs both manage setup and teardown of shared resources. Plugs add: a logger wired into the record, a tearDown that always runs after the run, state shared across every phase of one run by construction, and configuration binding for per-station addresses. Fixtures are more flexible in scope (function, module, session); plugs are always per run, which is what a station wants.

The operator

A production test asks a human to scan a barcode, press a button, read an LED. pytest has no concept of this. OpenHTF's UserInput plug and Operator UI make it a one-liner — see Device Under Test.

Sequencing

Hardware tests have order: power on before measuring, calibrate before burn-in, release the fixture last. pytest deliberately does not guarantee order. OpenHTF's phase groups, checkpoints and subtests exist for exactly this.

Where pytest is the better tool

Unit-testing your plugs and helpers. Mock the instrument, assert the SCPI strings. pytest, plain and simple.
Firmware/software on the DUT that you can test without a fixture — protocol parsers, CLI tools.
CI on every commit with hundreds of quick checks and rich failure diffs.

Many teams do both: pytest in CI for the test code, OpenHTF on the station for the product.

Using both

TofuPilot runs pytest suites on stations as well, mapping each test_* function to a phase and promoting assert lo <= x <= hi, "label" to a measurement — a pragmatic path when a team wants OpenHTF-style records from an existing pytest suite. See Pytest on TofuPilot. The structural advantages above still favour OpenHTF for tests written from scratch for a production line.