Bench Harness

benchmatrix.bench_harness

Build pytest-benchmark matrices and attach benchmatrix metadata.

pytest-benchmark remains the measurement engine and source of truth for calibration, timing, statistics, reporting, and JSON export. This module orchestrates metric-by-implementation-by-case matrices, attaches strict JSON-safe metadata to benchmark.extra_info, and streams lightweight invocation progress records. Detailed timing statistics should be read from pytest-benchmark's terminal report, CSV output, saved runs, or JSON output.

Target functions must be synchronous callables that complete the work to be measured before returning. Async functions are not supported. Lazy return values such as generators, lazy dataframe expressions, query objects, futures, and deferred computation graphs are not forced by this harness; if such objects are returned, the benchmark may measure only object construction.

TargetFunction module-attribute

TargetFunction: TypeAlias = Callable[..., object]

Synchronous callable measured by pytest-benchmark through benchmatrix.

Target functions must perform the work being measured before returning. Async functions are rejected. Lazy return values are not forced by the harness.

BenchmarkLifecycleHook module-attribute

BenchmarkLifecycleHook: TypeAlias = Callable[
    [BenchmarkHookContext], None
]

Synchronous setup or cleanup hook for one benchmark invocation.

BenchmarkResultValidator module-attribute

BenchmarkResultValidator: TypeAlias = Callable[
    [BenchmarkHookContext, object], None
]

Synchronous correctness hook for one benchmark invocation result.

BenchmarkFixture

Bases: Protocol

pytest-benchmark fixture surface used by benchmatrix.

Attributes:

Name	Type	Description
extra_info	MutableMapping[str, object]	Mutable metadata attached to pytest-benchmark output.

Source code in src/benchmatrix/bench_harness.py

class BenchmarkFixture(Protocol):
    """pytest-benchmark fixture surface used by benchmatrix.

    Attributes:
        extra_info: Mutable metadata attached to pytest-benchmark output.
    """

    extra_info: MutableMapping[str, object]

    def __call__(self, target: Callable[..., T], *args: object, **kwargs: object) -> T:
        """Benchmark ``target`` with pytest-benchmark automatic calibration."""
        ...

    def pedantic(
        self,
        target: Callable[..., T],
        *,
        args: Sequence[object] | None = None,
        kwargs: Mapping[str, object] | None = None,
        setup: Callable[[], tuple[Sequence[object], Mapping[str, object]]] | None = None,
        teardown: Callable[..., object] | None = None,
        rounds: int = _DEFAULT_PEDANTIC_ROUNDS,
        warmup_rounds: int = _DEFAULT_WARMUP_ROUNDS,
        iterations: int = _DEFAULT_PEDANTIC_ITERATIONS,
    ) -> T:
        """Benchmark ``target`` with pytest-benchmark pedantic mode."""
        ...

__call__

__call__(
    target: Callable[..., T],
    *args: object,
    **kwargs: object,
) -> T

Benchmark target with pytest-benchmark automatic calibration.

Source code in src/benchmatrix/bench_harness.py

def __call__(self, target: Callable[..., T], *args: object, **kwargs: object) -> T:
    """Benchmark ``target`` with pytest-benchmark automatic calibration."""
    ...

pedantic

pedantic(
    target: Callable[..., T],
    *,
    args: Sequence[object] | None = None,
    kwargs: Mapping[str, object] | None = None,
    setup: Callable[
        [], tuple[Sequence[object], Mapping[str, object]]
    ]
    | None = None,
    teardown: Callable[..., object] | None = None,
    rounds: int = _DEFAULT_PEDANTIC_ROUNDS,
    warmup_rounds: int = _DEFAULT_WARMUP_ROUNDS,
    iterations: int = _DEFAULT_PEDANTIC_ITERATIONS,
) -> T

Benchmark target with pytest-benchmark pedantic mode.

Source code in src/benchmatrix/bench_harness.py

def pedantic(
    self,
    target: Callable[..., T],
    *,
    args: Sequence[object] | None = None,
    kwargs: Mapping[str, object] | None = None,
    setup: Callable[[], tuple[Sequence[object], Mapping[str, object]]] | None = None,
    teardown: Callable[..., object] | None = None,
    rounds: int = _DEFAULT_PEDANTIC_ROUNDS,
    warmup_rounds: int = _DEFAULT_WARMUP_ROUNDS,
    iterations: int = _DEFAULT_PEDANTIC_ITERATIONS,
) -> T:
    """Benchmark ``target`` with pytest-benchmark pedantic mode."""
    ...

BenchmarkConfig dataclass

Configuration passed from benchmatrix to pytest-benchmark.

Parameters:

Name	Type	Description	Default
pedantic_rounds	int	Number of pedantic benchmark rounds to request.	_DEFAULT_PEDANTIC_ROUNDS
warmup_rounds	int	Number of pedantic warmup rounds to request.	_DEFAULT_WARMUP_ROUNDS
pedantic_iterations	int	Number of function calls per pedantic round when inputs are reused. This value is intentionally ignored when BenchmarkCase.fresh_inputs is true because pytest-benchmark setup mode is used to keep input construction outside the timed target-function body.	_DEFAULT_PEDANTIC_ITERATIONS
stream_progress	bool	Whether benchmark helpers should print one progress line per benchmark invocation.	True
before_benchmark	BenchmarkLifecycleHook | None	Optional synchronous hook called immediately before pytest-benchmark starts an invocation.	None
validate_result	BenchmarkResultValidator | None	Optional synchronous correctness hook called with the result returned by pytest-benchmark. The hook should raise when the result is invalid.	None
after_benchmark	BenchmarkLifecycleHook | None	Optional synchronous hook called after result validation, or during cleanup if benchmarking or validation raises.	None

Attributes:

Name	Type	Description
pedantic_rounds	int	Number of pedantic benchmark rounds to request.
warmup_rounds	int	Number of pedantic warmup rounds to request.
pedantic_iterations	int	Number of function calls per pedantic round when inputs are reused.
stream_progress	bool	Whether benchmark helpers should print one progress line per benchmark invocation.
before_benchmark	BenchmarkLifecycleHook | None	Optional untimed setup hook for a benchmark invocation.
validate_result	BenchmarkResultValidator | None	Optional untimed correctness hook for the returned target result.
after_benchmark	BenchmarkLifecycleHook | None	Optional untimed cleanup hook for a benchmark invocation.

Raises:

Type	Description
TypeError	If a timing control has the wrong type, progress output is not boolean, or a configured hook is not callable or is asynchronous.
ValueError	If rounds or iterations are not positive, or if warmup rounds are negative.

Warning

For tail_latency benchmarks, setting pedantic_iterations above one means raw samples are per-round averages of multiple calls rather than individual-call latency samples. The harness emits a runtime warning for this configuration.

Source code in src/benchmatrix/bench_harness.py

@dataclass(frozen=True, slots=True)
class BenchmarkConfig:
    """Configuration passed from benchmatrix to pytest-benchmark.

    Args:
        pedantic_rounds: Number of pedantic benchmark rounds to request.
        warmup_rounds: Number of pedantic warmup rounds to request.
        pedantic_iterations: Number of function calls per pedantic round when
            inputs are reused. This value is intentionally ignored when
            ``BenchmarkCase.fresh_inputs`` is true because pytest-benchmark
            setup mode is used to keep input construction outside the timed
            target-function body.
        stream_progress: Whether benchmark helpers should print one progress
            line per benchmark invocation.
        before_benchmark: Optional synchronous hook called immediately before
            pytest-benchmark starts an invocation.
        validate_result: Optional synchronous correctness hook called with the
            result returned by pytest-benchmark. The hook should raise when the
            result is invalid.
        after_benchmark: Optional synchronous hook called after result
            validation, or during cleanup if benchmarking or validation raises.

    Attributes:
        pedantic_rounds: Number of pedantic benchmark rounds to request.
        warmup_rounds: Number of pedantic warmup rounds to request.
        pedantic_iterations: Number of function calls per pedantic round when
            inputs are reused.
        stream_progress: Whether benchmark helpers should print one progress
            line per benchmark invocation.
        before_benchmark: Optional untimed setup hook for a benchmark
            invocation.
        validate_result: Optional untimed correctness hook for the returned
            target result.
        after_benchmark: Optional untimed cleanup hook for a benchmark
            invocation.

    Raises:
        TypeError: If a timing control has the wrong type, progress output is
            not boolean, or a configured hook is not callable or is
            asynchronous.
        ValueError: If rounds or iterations are not positive, or if warmup
            rounds are negative.

    Warning:
        For ``tail_latency`` benchmarks, setting ``pedantic_iterations`` above
        one means raw samples are per-round averages of multiple calls rather
        than individual-call latency samples. The harness emits a runtime
        warning for this configuration.
    """

    pedantic_rounds: int = _DEFAULT_PEDANTIC_ROUNDS
    warmup_rounds: int = _DEFAULT_WARMUP_ROUNDS
    pedantic_iterations: int = _DEFAULT_PEDANTIC_ITERATIONS
    stream_progress: bool = True
    before_benchmark: BenchmarkLifecycleHook | None = None
    validate_result: BenchmarkResultValidator | None = None
    after_benchmark: BenchmarkLifecycleHook | None = None

    def __post_init__(self) -> None:
        """Validate benchmark configuration after initialization."""
        if isinstance(self.pedantic_rounds, bool) or not isinstance(self.pedantic_rounds, int):
            raise TypeError("BenchmarkConfig.pedantic_rounds must be an integer.")
        if self.pedantic_rounds <= 0:
            raise ValueError("BenchmarkConfig.pedantic_rounds must be positive.")

        if isinstance(self.warmup_rounds, bool) or not isinstance(self.warmup_rounds, int):
            raise TypeError("BenchmarkConfig.warmup_rounds must be an integer.")
        if self.warmup_rounds < 0:
            raise ValueError("BenchmarkConfig.warmup_rounds must be non-negative.")

        if isinstance(self.pedantic_iterations, bool) or not isinstance(self.pedantic_iterations, int):
            raise TypeError("BenchmarkConfig.pedantic_iterations must be an integer.")
        if self.pedantic_iterations <= 0:
            raise ValueError("BenchmarkConfig.pedantic_iterations must be positive.")

        if not isinstance(self.stream_progress, bool):
            raise TypeError("BenchmarkConfig.stream_progress must be a boolean.")

        _validate_hook(self.before_benchmark, field="before_benchmark")
        _validate_hook(self.validate_result, field="validate_result")
        _validate_hook(self.after_benchmark, field="after_benchmark")

__post_init__

__post_init__() -> None

Validate benchmark configuration after initialization.

Source code in src/benchmatrix/bench_harness.py

def __post_init__(self) -> None:
    """Validate benchmark configuration after initialization."""
    if isinstance(self.pedantic_rounds, bool) or not isinstance(self.pedantic_rounds, int):
        raise TypeError("BenchmarkConfig.pedantic_rounds must be an integer.")
    if self.pedantic_rounds <= 0:
        raise ValueError("BenchmarkConfig.pedantic_rounds must be positive.")

    if isinstance(self.warmup_rounds, bool) or not isinstance(self.warmup_rounds, int):
        raise TypeError("BenchmarkConfig.warmup_rounds must be an integer.")
    if self.warmup_rounds < 0:
        raise ValueError("BenchmarkConfig.warmup_rounds must be non-negative.")

    if isinstance(self.pedantic_iterations, bool) or not isinstance(self.pedantic_iterations, int):
        raise TypeError("BenchmarkConfig.pedantic_iterations must be an integer.")
    if self.pedantic_iterations <= 0:
        raise ValueError("BenchmarkConfig.pedantic_iterations must be positive.")

    if not isinstance(self.stream_progress, bool):
        raise TypeError("BenchmarkConfig.stream_progress must be a boolean.")

    _validate_hook(self.before_benchmark, field="before_benchmark")
    _validate_hook(self.validate_result, field="validate_result")
    _validate_hook(self.after_benchmark, field="after_benchmark")

BenchmarkCase dataclass

Named input case and metadata for a pytest-benchmark matrix.

Warning

If fresh_inputs is false, pytest-benchmark may call the target function repeatedly with the same argument objects. That is appropriate only when the target function treats its inputs as immutable or when reuse reflects the workload you want to measure.

If fresh_inputs is true, this harness uses pytest-benchmark pedantic setup so input construction is setup work rather than timed target-function work. That avoids accidentally timing input creation, but it also means the benchmark is not an end-to-end measurement that includes input construction. To benchmark construction cost, put that construction inside the target function itself.

When fresh_inputs is true, BenchmarkConfig.pedantic_iterations is ignored because pytest-benchmark setup mode is used. The harness emits a runtime warning when a non-default value is ignored.

Parameters:

Name	Type	Description	Default
name	str	Human-readable case name used in parameter IDs and metadata.	required
make_args	Callable[[], tuple[object, ...]]	Factory returning positional arguments for the target function.	_empty_args
make_kwargs	Callable[[], dict[str, object]]	Factory returning keyword arguments for the target function.	_empty_kwargs
work_units	float | Callable[[], float] | None	Positive logical amount of work performed by one target call. This can represent items, rows, bytes, tokens, records, events, or any other domain-specific unit.	None
work_unit_name	str	Name of the logical work unit, such as "items", "rows", "bytes", or "tokens". Use a base unit name without spaces, slashes, or "/s"; display code appends "/s" for throughput.	_DEFAULT_WORK_UNIT_NAME
fresh_inputs	bool	Whether each benchmark round needs newly created inputs.	False
metadata	Mapping[str, object]	Additional strict-JSON-renderable metadata describing the case. Reasonable scalar types such as paths, datetimes, enums, and NumPy scalars are coerced; unsupported values raise MetadataSerializationError.	_empty_metadata()

Attributes:

Name	Type	Description
name	str	Human-readable case name used in parameter IDs and metadata.
make_args	Callable[[], tuple[object, ...]]	Factory returning positional arguments for the target function.
make_kwargs	Callable[[], dict[str, object]]	Factory returning keyword arguments for the target function.
work_units	float | Callable[[], float] | None	Positive logical amount of work performed by one target call.
work_unit_name	str	Name of the logical work unit.
fresh_inputs	bool	Whether each benchmark round needs newly created inputs.
metadata	Mapping[str, object]	Strict JSON-safe metadata describing the case.

Source code in src/benchmatrix/bench_harness.py

@dataclass(frozen=True, slots=True)
class BenchmarkCase:
    """Named input case and metadata for a pytest-benchmark matrix.

    Warning:
        If ``fresh_inputs`` is false, pytest-benchmark may call the target
        function repeatedly with the same argument objects. That is appropriate
        only when the target function treats its inputs as immutable or when
        reuse reflects the workload you want to measure.

        If ``fresh_inputs`` is true, this harness uses pytest-benchmark
        pedantic setup so input construction is setup work rather than timed
        target-function work. That avoids accidentally timing input creation,
        but it also means the benchmark is not an end-to-end measurement that
        includes input construction. To benchmark construction cost, put that
        construction inside the target function itself.

        When ``fresh_inputs`` is true, ``BenchmarkConfig.pedantic_iterations``
        is ignored because pytest-benchmark setup mode is used. The harness
        emits a runtime warning when a non-default value is ignored.

    Args:
        name: Human-readable case name used in parameter IDs and metadata.
        make_args: Factory returning positional arguments for the target
            function.
        make_kwargs: Factory returning keyword arguments for the target
            function.
        work_units: Positive logical amount of work performed by one target
            call. This can represent items, rows, bytes, tokens, records,
            events, or any other domain-specific unit.
        work_unit_name: Name of the logical work unit, such as ``"items"``,
            ``"rows"``, ``"bytes"``, or ``"tokens"``. Use a base unit name
            without spaces, slashes, or ``"/s"``; display code appends ``"/s"``
            for throughput.
        fresh_inputs: Whether each benchmark round needs newly created inputs.
        metadata: Additional strict-JSON-renderable metadata describing the
            case. Reasonable scalar types such as paths, datetimes, enums, and
            NumPy scalars are coerced; unsupported values raise
            ``MetadataSerializationError``.

    Attributes:
        name: Human-readable case name used in parameter IDs and metadata.
        make_args: Factory returning positional arguments for the target
            function.
        make_kwargs: Factory returning keyword arguments for the target
            function.
        work_units: Positive logical amount of work performed by one target
            call.
        work_unit_name: Name of the logical work unit.
        fresh_inputs: Whether each benchmark round needs newly created inputs.
        metadata: Strict JSON-safe metadata describing the case.
    """

    name: str
    make_args: Callable[[], tuple[object, ...]] = _empty_args
    make_kwargs: Callable[[], dict[str, object]] = _empty_kwargs
    work_units: float | Callable[[], float] | None = None
    work_unit_name: str = _DEFAULT_WORK_UNIT_NAME
    fresh_inputs: bool = False
    metadata: Mapping[str, object] = field(default_factory=_empty_metadata)

    def __post_init__(self) -> None:
        """Validate benchmark case fields after initialization."""
        object.__setattr__(self, "name", _validate_name(self.name, field="case name"))

        _validate_case_callable(self.make_args, field="make_args")
        _validate_case_callable(self.make_kwargs, field="make_kwargs")

        object.__setattr__(self, "work_unit_name", _validate_work_unit_name(self.work_unit_name))

        if self.work_units is not None:
            if callable(self.work_units):
                _validate_case_callable(self.work_units, field="work_units")
            else:
                _ = _validate_work_units(self.work_units)

        if not isinstance(self.fresh_inputs, bool):
            raise TypeError("BenchmarkCase.fresh_inputs must be a boolean.")

        if not isinstance(self.metadata, Mapping):
            raise TypeError("BenchmarkCase.metadata must be a mapping.")

        coerced_metadata = _coerce_json_mapping(
            self.metadata,
            path="BenchmarkCase.metadata",
        )
        reserved_metadata = sorted(_RESERVED_CASE_METADATA_KEYS.intersection(coerced_metadata))
        if reserved_metadata:
            formatted = ", ".join(repr(key) for key in reserved_metadata)
            raise ValueError(f"BenchmarkCase.metadata uses reserved key(s): {formatted}.")
        object.__setattr__(self, "metadata", coerced_metadata)

    def make_call(self) -> tuple[tuple[object, ...], dict[str, object]]:
        """Return positional and keyword arguments for one target invocation.

        Returns:
            A tuple containing positional arguments and keyword arguments.
        """
        args = self.make_args()
        if not isinstance(args, tuple):
            raise TypeError("BenchmarkCase.make_args must return a tuple.")

        kwargs = self.make_kwargs()
        if not isinstance(kwargs, dict):
            raise TypeError("BenchmarkCase.make_kwargs must return a dictionary.")
        if any(not isinstance(key, str) for key in kwargs):
            raise TypeError("BenchmarkCase.make_kwargs must return a dictionary with string keys.")

        return args, kwargs

    def work_unit_count(self) -> float | None:
        """Return the logical work-unit count for throughput metrics.

        Returns:
            The logical work-unit count, or ``None`` when the case has no work
            unit count.

        Raises:
            ValueError: If the work-unit count is not positive or finite.
        """
        if self.work_units is None:
            return None

        value = self.work_units() if callable(self.work_units) else self.work_units
        return _validate_work_units(value)

    @classmethod
    def from_values(
        cls,
        name: str,
        *args: object,
        work_units: float | Callable[[], float] | None = None,
        work_unit_name: str = _DEFAULT_WORK_UNIT_NAME,
        fresh_inputs: bool = False,
        copier: Callable[[object], object] | None = None,
        metadata: Mapping[str, object] | None = None,
        **kwargs: object,
    ) -> BenchmarkCase:
        """Create a benchmark case from concrete argument values.

        Args:
            name: Case name.
            *args: Positional arguments for the target function.
            work_units: Positive logical amount of work performed by one target
                call.
            work_unit_name: Name of the logical work unit, such as ``"items"``,
                ``"rows"``, ``"bytes"``, or ``"tokens"``. Use a base unit name
                without spaces, slashes, or ``"/s"``.
            fresh_inputs: Whether target invocations need fresh inputs. When
                true and ``copier`` is omitted, a shallow copy is made for each
                argument value.
            copier: Optional copy function applied to each argument value. Use
                ``deep_copy`` or a domain-specific copy function when shallow
                copies are not fresh enough for the benchmarked workload.
            metadata: Optional strict-JSON-renderable case metadata.
            **kwargs: Keyword arguments for the target function.

        Returns:
            A configured benchmark case.
        """

        if not isinstance(fresh_inputs, bool):
            raise TypeError("BenchmarkCase.fresh_inputs must be a boolean.")
        if copier is not None:
            _validate_case_callable(copier, field="copier")

        effective_copier = shallow_copy if fresh_inputs and copier is None else copier

        def make_args() -> tuple[object, ...]:
            """Return case positional arguments."""
            if effective_copier is None:
                return args

            return tuple(effective_copier(arg) for arg in args)

        def make_kwargs() -> dict[str, object]:
            """Return case keyword arguments."""
            if effective_copier is None:
                return dict(kwargs)

            return {key: effective_copier(value) for key, value in kwargs.items()}

        return cls(
            name=name,
            make_args=make_args,
            make_kwargs=make_kwargs,
            work_units=work_units,
            work_unit_name=work_unit_name,
            fresh_inputs=fresh_inputs or effective_copier is not None,
            metadata={} if metadata is None else dict(metadata),
        )

__post_init__

__post_init__() -> None

Validate benchmark case fields after initialization.

Source code in src/benchmatrix/bench_harness.py

def __post_init__(self) -> None:
    """Validate benchmark case fields after initialization."""
    object.__setattr__(self, "name", _validate_name(self.name, field="case name"))

    _validate_case_callable(self.make_args, field="make_args")
    _validate_case_callable(self.make_kwargs, field="make_kwargs")

    object.__setattr__(self, "work_unit_name", _validate_work_unit_name(self.work_unit_name))

    if self.work_units is not None:
        if callable(self.work_units):
            _validate_case_callable(self.work_units, field="work_units")
        else:
            _ = _validate_work_units(self.work_units)

    if not isinstance(self.fresh_inputs, bool):
        raise TypeError("BenchmarkCase.fresh_inputs must be a boolean.")

    if not isinstance(self.metadata, Mapping):
        raise TypeError("BenchmarkCase.metadata must be a mapping.")

    coerced_metadata = _coerce_json_mapping(
        self.metadata,
        path="BenchmarkCase.metadata",
    )
    reserved_metadata = sorted(_RESERVED_CASE_METADATA_KEYS.intersection(coerced_metadata))
    if reserved_metadata:
        formatted = ", ".join(repr(key) for key in reserved_metadata)
        raise ValueError(f"BenchmarkCase.metadata uses reserved key(s): {formatted}.")
    object.__setattr__(self, "metadata", coerced_metadata)

make_call

make_call() -> tuple[tuple[object, ...], dict[str, object]]

Return positional and keyword arguments for one target invocation.

Returns:

Type	Description
tuple[tuple[object, ...], dict[str, object]]	A tuple containing positional arguments and keyword arguments.

Source code in src/benchmatrix/bench_harness.py

def make_call(self) -> tuple[tuple[object, ...], dict[str, object]]:
    """Return positional and keyword arguments for one target invocation.

    Returns:
        A tuple containing positional arguments and keyword arguments.
    """
    args = self.make_args()
    if not isinstance(args, tuple):
        raise TypeError("BenchmarkCase.make_args must return a tuple.")

    kwargs = self.make_kwargs()
    if not isinstance(kwargs, dict):
        raise TypeError("BenchmarkCase.make_kwargs must return a dictionary.")
    if any(not isinstance(key, str) for key in kwargs):
        raise TypeError("BenchmarkCase.make_kwargs must return a dictionary with string keys.")

    return args, kwargs

work_unit_count

work_unit_count() -> float | None

Return the logical work-unit count for throughput metrics.

Returns:

Type	Description
float | None	The logical work-unit count, or None when the case has no work
float | None	unit count.

Raises:

Type	Description
ValueError	If the work-unit count is not positive or finite.

Source code in src/benchmatrix/bench_harness.py

def work_unit_count(self) -> float | None:
    """Return the logical work-unit count for throughput metrics.

    Returns:
        The logical work-unit count, or ``None`` when the case has no work
        unit count.

    Raises:
        ValueError: If the work-unit count is not positive or finite.
    """
    if self.work_units is None:
        return None

    value = self.work_units() if callable(self.work_units) else self.work_units
    return _validate_work_units(value)

from_values classmethod

from_values(
    name: str,
    *args: object,
    work_units: float | Callable[[], float] | None = None,
    work_unit_name: str = _DEFAULT_WORK_UNIT_NAME,
    fresh_inputs: bool = False,
    copier: Callable[[object], object] | None = None,
    metadata: Mapping[str, object] | None = None,
    **kwargs: object,
) -> BenchmarkCase

Create a benchmark case from concrete argument values.

Parameters:

Name	Type	Description	Default
name	str	Case name.	required
*args	object	Positional arguments for the target function.	()
work_units	float | Callable[[], float] | None	Positive logical amount of work performed by one target call.	None
work_unit_name	str	Name of the logical work unit, such as "items", "rows", "bytes", or "tokens". Use a base unit name without spaces, slashes, or "/s".	_DEFAULT_WORK_UNIT_NAME
fresh_inputs	bool	Whether target invocations need fresh inputs. When true and copier is omitted, a shallow copy is made for each argument value.	False
copier	Callable[[object], object] | None	Optional copy function applied to each argument value. Use deep_copy or a domain-specific copy function when shallow copies are not fresh enough for the benchmarked workload.	None
metadata	Mapping[str, object] | None	Optional strict-JSON-renderable case metadata.	None
**kwargs	object	Keyword arguments for the target function.	{}

Returns:

Type	Description
BenchmarkCase	A configured benchmark case.

Source code in src/benchmatrix/bench_harness.py

@classmethod
def from_values(
    cls,
    name: str,
    *args: object,
    work_units: float | Callable[[], float] | None = None,
    work_unit_name: str = _DEFAULT_WORK_UNIT_NAME,
    fresh_inputs: bool = False,
    copier: Callable[[object], object] | None = None,
    metadata: Mapping[str, object] | None = None,
    **kwargs: object,
) -> BenchmarkCase:
    """Create a benchmark case from concrete argument values.

    Args:
        name: Case name.
        *args: Positional arguments for the target function.
        work_units: Positive logical amount of work performed by one target
            call.
        work_unit_name: Name of the logical work unit, such as ``"items"``,
            ``"rows"``, ``"bytes"``, or ``"tokens"``. Use a base unit name
            without spaces, slashes, or ``"/s"``.
        fresh_inputs: Whether target invocations need fresh inputs. When
            true and ``copier`` is omitted, a shallow copy is made for each
            argument value.
        copier: Optional copy function applied to each argument value. Use
            ``deep_copy`` or a domain-specific copy function when shallow
            copies are not fresh enough for the benchmarked workload.
        metadata: Optional strict-JSON-renderable case metadata.
        **kwargs: Keyword arguments for the target function.

    Returns:
        A configured benchmark case.
    """

    if not isinstance(fresh_inputs, bool):
        raise TypeError("BenchmarkCase.fresh_inputs must be a boolean.")
    if copier is not None:
        _validate_case_callable(copier, field="copier")

    effective_copier = shallow_copy if fresh_inputs and copier is None else copier

    def make_args() -> tuple[object, ...]:
        """Return case positional arguments."""
        if effective_copier is None:
            return args

        return tuple(effective_copier(arg) for arg in args)

    def make_kwargs() -> dict[str, object]:
        """Return case keyword arguments."""
        if effective_copier is None:
            return dict(kwargs)

        return {key: effective_copier(value) for key, value in kwargs.items()}

    return cls(
        name=name,
        make_args=make_args,
        make_kwargs=make_kwargs,
        work_units=work_units,
        work_unit_name=work_unit_name,
        fresh_inputs=fresh_inputs or effective_copier is not None,
        metadata={} if metadata is None else dict(metadata),
    )

BenchmarkHookContext dataclass

Identity and inputs available to benchmark lifecycle hooks.

Attributes:

Name	Type	Description
metric_name	MetricName	Metric requested for this benchmark invocation.
implementation_name	str	Name of the implementation under test.
case_name	str	Matrix case name under test.
function	TargetFunction	Synchronous target function under test.
case	BenchmarkCase	Benchmark case definition used by the invocation.

Source code in src/benchmatrix/bench_harness.py

@dataclass(frozen=True, slots=True)
class BenchmarkHookContext:
    """Identity and inputs available to benchmark lifecycle hooks.

    Attributes:
        metric_name: Metric requested for this benchmark invocation.
        implementation_name: Name of the implementation under test.
        case_name: Matrix case name under test.
        function: Synchronous target function under test.
        case: Benchmark case definition used by the invocation.
    """

    metric_name: MetricName
    implementation_name: str
    case_name: str
    function: TargetFunction
    case: BenchmarkCase

BenchmarkInvocationRecord dataclass

Lightweight record returned after one benchmark invocation.

This record is not a timing result. Timing results come from pytest-benchmark's report, saved runs, CSV output, or JSON output.

Attributes:

Name	Type	Description
metric_name	MetricName	Metric requested for this benchmark invocation.
implementation_name	str	Name of the implementation under test.
case_name	str	Name of the input case under test.
extra_info	Mapping[str, object]	Strict JSON-safe metadata attached to pytest-benchmark output. Values are limited to JSON primitives, lists, and string-keyed mappings after metadata coercion. The metadata includes benchmatrix producer and schema-version markers.

Source code in src/benchmatrix/bench_harness.py

@dataclass(frozen=True, slots=True)
class BenchmarkInvocationRecord:
    """Lightweight record returned after one benchmark invocation.

    This record is not a timing result. Timing results come from
    pytest-benchmark's report, saved runs, CSV output, or JSON output.

    Attributes:
        metric_name: Metric requested for this benchmark invocation.
        implementation_name: Name of the implementation under test.
        case_name: Name of the input case under test.
        extra_info: Strict JSON-safe metadata attached to pytest-benchmark
            output. Values are limited to JSON primitives, lists, and
            string-keyed mappings after metadata coercion. The metadata includes
            benchmatrix producer and schema-version markers.
    """

    metric_name: MetricName
    implementation_name: str
    case_name: str
    extra_info: Mapping[str, object]

benchmark_single_call_latency

benchmark_single_call_latency(
    benchmark: BenchmarkFixture,
    implementation_name: str,
    function: TargetFunction,
    case_name: str,
    case: BenchmarkCase,
    *,
    config: BenchmarkConfig | None = None,
    stream: TextIO | None = None,
) -> BenchmarkInvocationRecord

Benchmark single-call latency for one implementation and case.

Parameters:

Name	Type	Description	Default
benchmark	BenchmarkFixture	Pytest-benchmark fixture instance.	required
implementation_name	str	Name of the implementation under test.	required
function	TargetFunction	Synchronous function implementation to benchmark. The function must complete the measured work before returning.	required
case_name	str	Name of the input case under test.	required
case	BenchmarkCase	Benchmark input case.	required
config	BenchmarkConfig | None	Benchmark harness configuration. Defaults to BenchmarkConfig().	None
stream	TextIO | None	Stream used for progress output. Defaults to sys.stdout when progress output is enabled.	None

Returns:

Type	Description
BenchmarkInvocationRecord	A lightweight invocation record containing metadata attached to the
BenchmarkInvocationRecord	benchmark. This is not a timing result.

Raises:

Type	Description
TypeError	If function is an async function.

Warning

This measures completed target-function work only. Input construction, lazy-result consumption, and other setup are excluded unless they occur inside function.

Source code in src/benchmatrix/bench_harness.py

def benchmark_single_call_latency(
    benchmark: BenchmarkFixture,
    implementation_name: str,
    function: TargetFunction,
    case_name: str,
    case: BenchmarkCase,
    *,
    config: BenchmarkConfig | None = None,
    stream: TextIO | None = None,
) -> BenchmarkInvocationRecord:
    """Benchmark single-call latency for one implementation and case.

    Args:
        benchmark: Pytest-benchmark fixture instance.
        implementation_name: Name of the implementation under test.
        function: Synchronous function implementation to benchmark. The
            function must complete the measured work before returning.
        case_name: Name of the input case under test.
        case: Benchmark input case.
        config: Benchmark harness configuration. Defaults to
            ``BenchmarkConfig()``.
        stream: Stream used for progress output. Defaults to ``sys.stdout``
            when progress output is enabled.

    Returns:
        A lightweight invocation record containing metadata attached to the
        benchmark. This is not a timing result.

    Raises:
        TypeError: If ``function`` is an async function.

    Warning:
        This measures completed target-function work only. Input construction,
        lazy-result consumption, and other setup are excluded unless they occur
        inside ``function``.
    """
    resolved_config = _resolve_config(config)
    metric_name = METRIC_SINGLE_CALL_LATENCY
    extra_info: dict[str, object] = _make_base_extra_info(
        metric_name,
        implementation_name,
        case_name,
        case,
    )
    final_extra_info = _set_extra_info(benchmark, extra_info)
    _ = _run_target_with_hooks(
        benchmark,
        metric_name,
        implementation_name,
        function,
        case_name,
        case,
        config=resolved_config,
        force_pedantic=False,
    )

    record = BenchmarkInvocationRecord(
        metric_name=metric_name,
        implementation_name=implementation_name,
        case_name=case_name,
        extra_info=final_extra_info,
    )
    _maybe_display_invocation_record(record, config=resolved_config, stream=stream)
    return record

benchmark_batch_throughput

benchmark_batch_throughput(
    benchmark: BenchmarkFixture,
    implementation_name: str,
    function: TargetFunction,
    case_name: str,
    case: BenchmarkCase,
    *,
    config: BenchmarkConfig | None = None,
    stream: TextIO | None = None,
) -> BenchmarkInvocationRecord

Benchmark batch throughput for one implementation and case.

Parameters:

Name	Type	Description	Default
benchmark	BenchmarkFixture	Pytest-benchmark fixture instance.	required
implementation_name	str	Name of the implementation under test.	required
function	TargetFunction	Synchronous function implementation to benchmark. The function must complete the measured work before returning.	required
case_name	str	Name of the input case under test.	required
case	BenchmarkCase	Benchmark input case. If case.work_units is provided, throughput is later derived as work units per second; otherwise it is derived as calls per second.	required
config	BenchmarkConfig | None	Benchmark harness configuration. Defaults to BenchmarkConfig().	None
stream	TextIO | None	Stream used for progress output. Defaults to sys.stdout when progress output is enabled.	None

Returns:

Type	Description
BenchmarkInvocationRecord	A lightweight invocation record containing metadata attached to the
BenchmarkInvocationRecord	benchmark. This is not a timing result.

Raises:

Type	Description
TypeError	If function is an async function.
ValueError	If case.work_units is not positive or finite.

Warning

Throughput is derived from one synchronous target invocation. It does not model concurrency, saturation, queueing, or service request load. case.work_units must accurately describe work completed by each target call.

Source code in src/benchmatrix/bench_harness.py

def benchmark_batch_throughput(
    benchmark: BenchmarkFixture,
    implementation_name: str,
    function: TargetFunction,
    case_name: str,
    case: BenchmarkCase,
    *,
    config: BenchmarkConfig | None = None,
    stream: TextIO | None = None,
) -> BenchmarkInvocationRecord:
    """Benchmark batch throughput for one implementation and case.

    Args:
        benchmark: Pytest-benchmark fixture instance.
        implementation_name: Name of the implementation under test.
        function: Synchronous function implementation to benchmark. The
            function must complete the measured work before returning.
        case_name: Name of the input case under test.
        case: Benchmark input case. If ``case.work_units`` is provided,
            throughput is later derived as work units per second; otherwise it
            is derived as calls per second.
        config: Benchmark harness configuration. Defaults to
            ``BenchmarkConfig()``.
        stream: Stream used for progress output. Defaults to ``sys.stdout``
            when progress output is enabled.

    Returns:
        A lightweight invocation record containing metadata attached to the
        benchmark. This is not a timing result.

    Raises:
        TypeError: If ``function`` is an async function.
        ValueError: If ``case.work_units`` is not positive or finite.

    Warning:
        Throughput is derived from one synchronous target invocation. It does
        not model concurrency, saturation, queueing, or service request load.
        ``case.work_units`` must accurately describe work completed by each
        target call.
    """
    resolved_config = _resolve_config(config)
    metric_name = METRIC_BATCH_THROUGHPUT
    extra_info: dict[str, object] = _make_base_extra_info(
        metric_name,
        implementation_name,
        case_name,
        case,
    )
    work_unit_count = case.work_unit_count()

    if work_unit_count is None:
        extra_info[KEY_THROUGHPUT_UNIT] = THROUGHPUT_UNIT_CALLS_PER_SECOND
    else:
        extra_info[KEY_WORK_UNITS] = work_unit_count
        extra_info[KEY_WORK_UNIT_NAME] = case.work_unit_name
        extra_info[KEY_THROUGHPUT_UNIT] = THROUGHPUT_UNIT_WORK_UNITS_PER_SECOND

    final_extra_info = _set_extra_info(benchmark, extra_info)
    _ = _run_target_with_hooks(
        benchmark,
        metric_name,
        implementation_name,
        function,
        case_name,
        case,
        config=resolved_config,
        force_pedantic=False,
    )

    record = BenchmarkInvocationRecord(
        metric_name=metric_name,
        implementation_name=implementation_name,
        case_name=case_name,
        extra_info=final_extra_info,
    )
    _maybe_display_invocation_record(record, config=resolved_config, stream=stream)
    return record

benchmark_tail_latency

benchmark_tail_latency(
    benchmark: BenchmarkFixture,
    implementation_name: str,
    function: TargetFunction,
    case_name: str,
    case: BenchmarkCase,
    *,
    config: BenchmarkConfig | None = None,
    stream: TextIO | None = None,
) -> BenchmarkInvocationRecord

Benchmark latency distribution for one implementation and case.

Parameters:

Name	Type	Description	Default
benchmark	BenchmarkFixture	Pytest-benchmark fixture instance.	required
implementation_name	str	Name of the implementation under test.	required
function	TargetFunction	Synchronous function implementation to benchmark. The function must complete the measured work before returning.	required
case_name	str	Name of the input case under test.	required
case	BenchmarkCase	Benchmark input case.	required
config	BenchmarkConfig | None	Benchmark harness configuration. Defaults to BenchmarkConfig().	None
stream	TextIO | None	Stream used for progress output. Defaults to sys.stdout when progress output is enabled.	None

Returns:

Type	Description
BenchmarkInvocationRecord	A lightweight invocation record containing metadata attached to the
BenchmarkInvocationRecord	benchmark. This is not a timing result.

Raises:

Type	Description
TypeError	If function is an async function.

Warning

This uses pedantic mode. Tail percentiles should be calculated from pytest-benchmark JSON data values. This is an implementation-comparison metric, not production p95/p99 latency under load.

If case.fresh_inputs is false and config.pedantic_iterations is greater than one, raw samples are per-round averages of multiple calls, not individual-call latency samples. The harness emits a runtime warning for that configuration.

Source code in src/benchmatrix/bench_harness.py

def benchmark_tail_latency(
    benchmark: BenchmarkFixture,
    implementation_name: str,
    function: TargetFunction,
    case_name: str,
    case: BenchmarkCase,
    *,
    config: BenchmarkConfig | None = None,
    stream: TextIO | None = None,
) -> BenchmarkInvocationRecord:
    """Benchmark latency distribution for one implementation and case.

    Args:
        benchmark: Pytest-benchmark fixture instance.
        implementation_name: Name of the implementation under test.
        function: Synchronous function implementation to benchmark. The
            function must complete the measured work before returning.
        case_name: Name of the input case under test.
        case: Benchmark input case.
        config: Benchmark harness configuration. Defaults to
            ``BenchmarkConfig()``.
        stream: Stream used for progress output. Defaults to ``sys.stdout``
            when progress output is enabled.

    Returns:
        A lightweight invocation record containing metadata attached to the
        benchmark. This is not a timing result.

    Raises:
        TypeError: If ``function`` is an async function.

    Warning:
        This uses pedantic mode. Tail percentiles should be calculated from
        pytest-benchmark JSON ``data`` values. This is an
        implementation-comparison metric, not production p95/p99 latency under
        load.

        If ``case.fresh_inputs`` is false and ``config.pedantic_iterations`` is
        greater than one, raw samples are per-round averages of multiple calls,
        not individual-call latency samples. The harness emits a runtime
        warning for that configuration.
    """
    resolved_config = _resolve_config(config)
    metric_name = METRIC_TAIL_LATENCY
    _warn_for_tail_latency_iteration_semantics(case, resolved_config)

    extra_info: dict[str, object] = _make_base_extra_info(
        metric_name,
        implementation_name,
        case_name,
        case,
    )
    extra_info[KEY_TAIL_LATENCY_NOTE] = (
        "Use pytest-benchmark JSON data to compute p50/p90/p95/p99. "
        "This is not production p95/p99 under load. If pedantic_iterations is "
        "greater than one, samples are per-round averages of multiple calls."
    )
    extra_info[KEY_TAIL_PERCENTILES] = list(TAIL_PERCENTILES)

    final_extra_info = _set_extra_info(benchmark, extra_info)
    _ = _run_target_with_hooks(
        benchmark,
        metric_name,
        implementation_name,
        function,
        case_name,
        case,
        config=resolved_config,
        force_pedantic=True,
    )

    record = BenchmarkInvocationRecord(
        metric_name=metric_name,
        implementation_name=implementation_name,
        case_name=case_name,
        extra_info=final_extra_info,
    )
    _maybe_display_invocation_record(record, config=resolved_config, stream=stream)
    return record

run_benchmark_metric

run_benchmark_metric(
    benchmark: BenchmarkFixture,
    metric_name: MetricName,
    implementation_name: str,
    function: TargetFunction,
    case_name: str,
    case: BenchmarkCase,
    *,
    config: BenchmarkConfig | None = None,
    stream: TextIO | None = None,
) -> BenchmarkInvocationRecord

Run one benchmark metric for one implementation and case.

Parameters:

Name	Type	Description	Default
benchmark	BenchmarkFixture	Pytest-benchmark fixture instance.	required
metric_name	MetricName	Metric to benchmark.	required
implementation_name	str	Name of the implementation under test.	required
function	TargetFunction	Synchronous function implementation to benchmark.	required
case_name	str	Name of the input case under test.	required
case	BenchmarkCase	Benchmark input case.	required
config	BenchmarkConfig | None	Benchmark harness configuration. Defaults to BenchmarkConfig().	None
stream	TextIO | None	Stream used for progress output. Defaults to sys.stdout when progress output is enabled.	None

Returns:

Type	Description
BenchmarkInvocationRecord	A lightweight invocation record containing metadata attached to the
BenchmarkInvocationRecord	benchmark. This is not a timing result.

Raises:

Type	Description
TypeError	If function is an async function.
ValueError	If metric_name is unsupported.

Source code in src/benchmatrix/bench_harness.py

def run_benchmark_metric(
    benchmark: BenchmarkFixture,
    metric_name: MetricName,
    implementation_name: str,
    function: TargetFunction,
    case_name: str,
    case: BenchmarkCase,
    *,
    config: BenchmarkConfig | None = None,
    stream: TextIO | None = None,
) -> BenchmarkInvocationRecord:
    """Run one benchmark metric for one implementation and case.

    Args:
        benchmark: Pytest-benchmark fixture instance.
        metric_name: Metric to benchmark.
        implementation_name: Name of the implementation under test.
        function: Synchronous function implementation to benchmark.
        case_name: Name of the input case under test.
        case: Benchmark input case.
        config: Benchmark harness configuration. Defaults to
            ``BenchmarkConfig()``.
        stream: Stream used for progress output. Defaults to ``sys.stdout``
            when progress output is enabled.

    Returns:
        A lightweight invocation record containing metadata attached to the
        benchmark. This is not a timing result.

    Raises:
        TypeError: If ``function`` is an async function.
        ValueError: If ``metric_name`` is unsupported.
    """
    resolved_config = _resolve_config(config)
    resolved_metric_name = _validate_metric_name(metric_name)

    if resolved_metric_name == METRIC_SINGLE_CALL_LATENCY:
        return benchmark_single_call_latency(
            benchmark,
            implementation_name,
            function,
            case_name,
            case,
            config=resolved_config,
            stream=stream,
        )

    if resolved_metric_name == METRIC_BATCH_THROUGHPUT:
        return benchmark_batch_throughput(
            benchmark,
            implementation_name,
            function,
            case_name,
            case,
            config=resolved_config,
            stream=stream,
        )

    if resolved_metric_name == METRIC_TAIL_LATENCY:
        return benchmark_tail_latency(
            benchmark,
            implementation_name,
            function,
            case_name,
            case,
            config=resolved_config,
            stream=stream,
        )

    raise ValueError(f"Unsupported benchmark metric: {resolved_metric_name!r}")

make_benchmark_parameters

make_benchmark_parameters(
    implementations: Mapping[str, TargetFunction],
    cases: Mapping[str, BenchmarkCase]
    | Iterable[BenchmarkCase],
    *,
    metrics: Iterable[MetricName] | None = None,
) -> list[object]

Create pytest parameters for a metric-by-implementation-by-case matrix.

Parameters:

Name	Type	Description	Default
implementations	Mapping[str, TargetFunction]	Mapping from implementation name to target function.	required
cases	Mapping[str, BenchmarkCase] | Iterable[BenchmarkCase]	Mapping or iterable of benchmark input cases.	required
metrics	Iterable[MetricName] | None	Metrics to include in the parameter matrix. Defaults to all supported benchmatrix metrics.	None

Returns:

Type	Description
list[object]	A list of values suitable for pytest.mark.parametrize.

Source code in src/benchmatrix/bench_harness.py

def make_benchmark_parameters(
    implementations: Mapping[str, TargetFunction],
    cases: Mapping[str, BenchmarkCase] | Iterable[BenchmarkCase],
    *,
    metrics: Iterable[MetricName] | None = None,
) -> list[object]:
    """Create pytest parameters for a metric-by-implementation-by-case matrix.

    Args:
        implementations: Mapping from implementation name to target function.
        cases: Mapping or iterable of benchmark input cases.
        metrics: Metrics to include in the parameter matrix. Defaults to all
            supported benchmatrix metrics.

    Returns:
        A list of values suitable for ``pytest.mark.parametrize``.
    """
    resolved_metrics = _metric_items(metrics)
    implementation_items = _implementation_items(implementations)
    case_items = _case_items(cases)
    pytest = _load_pytest()
    entries: list[tuple[MetricName, str, TargetFunction, str, BenchmarkCase]] = []

    for metric_name in resolved_metrics:
        for implementation_name, function in implementation_items:
            for case_name, case in case_items:
                entries.append((metric_name, implementation_name, function, case_name, case))

    collection_order = _collection_order_environment()
    if collection_order is not None:
        random_seed, order_index = collection_order
        indices = balanced_order_indices(
            [
                (implementation_name, case_name, metric_name)
                for metric_name, implementation_name, _function, case_name, _case in entries
            ],
            order_index=order_index,
            random_seed=random_seed,
        )
        entries = [entries[index] for index in indices]

    parameters: list[object] = []
    for metric_name, implementation_name, function, case_name, case in entries:
        parameters.append(
            pytest.param(
                metric_name,
                implementation_name,
                function,
                case_name,
                case,
                id=f"{metric_name}::{implementation_name}::{case_name}",
            )
        )

    return parameters

make_benchmark_test

make_benchmark_test(
    implementations: Mapping[str, TargetFunction],
    cases: Mapping[str, BenchmarkCase]
    | Iterable[BenchmarkCase],
    *,
    metrics: Iterable[MetricName] | None = None,
    config: BenchmarkConfig | None = None,
) -> Callable[..., None]

Create a pytest test function for a complete benchmark matrix.

Assign the returned function to a module-level name beginning with test_ so pytest collects it.

Parameters:

Name	Type	Description	Default
implementations	Mapping[str, TargetFunction]	Mapping from implementation name to target function.	required
cases	Mapping[str, BenchmarkCase] | Iterable[BenchmarkCase]	Mapping or iterable of benchmark input cases.	required
metrics	Iterable[MetricName] | None	Metrics to include in the parameter matrix. Defaults to all supported benchmatrix metrics.	None
config	BenchmarkConfig | None	Benchmark harness configuration. Defaults to BenchmarkConfig().	None

Returns:

Type	Description
Callable[..., None]	A parametrized pytest test function ready for module-level assignment.

Source code in src/benchmatrix/bench_harness.py

def make_benchmark_test(
    implementations: Mapping[str, TargetFunction],
    cases: Mapping[str, BenchmarkCase] | Iterable[BenchmarkCase],
    *,
    metrics: Iterable[MetricName] | None = None,
    config: BenchmarkConfig | None = None,
) -> Callable[..., None]:
    """Create a pytest test function for a complete benchmark matrix.

    Assign the returned function to a module-level name beginning with
    ``test_`` so pytest collects it.

    Args:
        implementations: Mapping from implementation name to target function.
        cases: Mapping or iterable of benchmark input cases.
        metrics: Metrics to include in the parameter matrix. Defaults to all
            supported benchmatrix metrics.
        config: Benchmark harness configuration. Defaults to
            ``BenchmarkConfig()``.

    Returns:
        A parametrized pytest test function ready for module-level assignment.
    """
    resolved_config = _resolve_config(config)
    parameters = make_benchmark_parameters(implementations, cases, metrics=metrics)

    def benchmark_test(
        benchmark: BenchmarkFixture,
        metric_name: MetricName,
        implementation_name: str,
        function: TargetFunction,
        case_name: str,
        case: BenchmarkCase,
    ) -> None:
        """Run one entry in the generated benchmark matrix."""
        _ = run_benchmark_metric(
            benchmark,
            metric_name,
            implementation_name,
            function,
            case_name,
            case,
            config=resolved_config,
        )

    pytest = _load_pytest()
    return pytest.mark.parametrize(
        ("metric_name", "implementation_name", "function", "case_name", "case"),
        parameters,
    )(benchmark_test)

shallow_copy

shallow_copy(value: object) -> object

Return a shallow copy of value.

Parameters:

Name	Type	Description	Default
value	object	Value to copy.	required

Returns:

Type	Description
object	A shallow copy of value.

Source code in src/benchmatrix/bench_harness.py

def shallow_copy(value: object) -> object:
    """Return a shallow copy of ``value``.

    Args:
        value: Value to copy.

    Returns:
        A shallow copy of ``value``.
    """
    return copy.copy(value)

deep_copy

deep_copy(value: object) -> object

Return a deep copy of value.

Parameters:

Name	Type	Description	Default
value	object	Value to copy.	required

Returns:

Type	Description
object	A deep copy of value.

Source code in src/benchmatrix/bench_harness.py

def deep_copy(value: object) -> object:
    """Return a deep copy of ``value``.

    Args:
        value: Value to copy.

    Returns:
        A deep copy of ``value``.
    """
    return copy.deepcopy(value)