Metrics

The metrics package has dataclasses with with to store and serialize output results.

Dataclasses

Measurements are the result of a benchmark run.

This module influenced by https://rtnl.link/hMNNI90KBGj

class construe.metrics.metrics.Metric(label=None, sub_label=None, description=None, device=None, env=None)

Bases: object

Container information used to define a benchmark measurement.

This class is similar to a pytorch TaskSpec.

Attributes:

description

device

env

label

sub_label

title

Best effort attempt at a string label for the metric.

Methods

summarize()

Builds a summary string for printing the metric.

label: Optional[str] = None

sub_label: Optional[str] = None

description: Optional[str] = None

device: Optional[str] = None

env: Optional[str] = None

property title: str

Best effort attempt at a string label for the metric.

summarize()

Builds a summary string for printing the metric.

Return type:

str

class construe.metrics.metrics.Measurement(metric, raw_metrics, per_run=1, units=None, metadata=None)

Bases: object

The result of a benchmark measurement.

This class stores one or more measurements of a given statement. It is similar to the pytorch measurement and provides convienence methods and serialization.

Attributes:

env

has_warnings

iqr

mean

median

metadata

metrics

row_name

title

units

Methods

merge(measurements)

Merge measurement replicas into a single measurement.

meets_confidence
to_array

metric: Metric

raw_metrics: List[float]

per_run: int = 1

units: Optional[str] = None

metadata: Optional[Dict[Any, Any]] = None

property metrics: List[float]

property median: float

property mean: float

property iqr: float

property has_warnings: bool

property title: str

property env: str

property row_name: str

meets_confidence(threshold=0.1)

Return type:

bool

to_array()

static merge(measurements)

Merge measurement replicas into a single measurement.

This method will extrapolate per_run=1 and will not transfer metadata.

Return type:

List[Measurement]

construe.metrics.metrics.select_duration_unit(t)

Determine how to scale a duration to format for human readability.

Return type:

Tuple[str, float]

construe.metrics.metrics.humanize_duration(u)

Return type:

str

Serialization

Handles serialization and deserialization of metrics.

class construe.metrics.serialize.MetricsJSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: JSONEncoder

Methods

default(o)	Encode first looks to see if there is a dump method and uses that, otherwise it attempts to serialize a dataclass; and the last step is perform the default JSON serialization of primitive types.
encode(o)	Return a JSON string representation of a Python data structure.
iterencode(o[, _one_shot])	Encode the given object and yield each string representation as available.

default(o)

Encode first looks to see if there is a dump method and uses that, otherwise it attempts to serialize a dataclass; and the last step is perform the default JSON serialization of primitive types.

class construe.metrics.serialize.MetricsJSONDecoder(*args, **kwargs)

Bases: JSONDecoder

Methods

decode(s[, _w])	Return the Python representation of s (a str instance containing a JSON document).
raw_decode(s[, idx])	Decode a JSON document from s (a str beginning with a JSON document) and return a 2-tuple of the Python representation and the index in s where the document ended.

object_hook

classmap = {'Measurement': <class 'construe.metrics.metrics.Measurement'>, 'Metric': <class 'construe.metrics.metrics.Metric'>}

object_hook(data)