Benchmark Library
The benchmark library contains the runner for executing benchmarks and the abstract base class that all benchmarks must inherit from (as well as some other helper functionality).
Runner
Benchmark ABC and global benchmark runner.
- class construe.benchmark.runner.BenchmarkRunner(benchmarks, device=None, env=None, n_runs=1, limit=None, data_home=None, model_home=None, use_sample=True, cleanup=True, verbose=True)[source]
Bases:
objectExecutes one or more benchmarks, configuring them with a top-level config and collecting all measurements, merging as necessary and outputing them to a file.
- Attributes:
- is_complete
Methods
execute
run
run_benchmark
save
- property is_complete
- execute(idx, benchmark, progress)[source]
- Return type:
Iterable[Measurement]
- class construe.benchmark.runner.Results(n_runs, benchmarks, started, errors=<class 'list'>, limit=None, duration=None, env=None, device=None, options=None, successes=0, failures=0, measurements=None)[source]
Bases:
objectA result of all runs of a Benchmark including benchmarking information.
- Attributes:
- device
- duration
- env
- limit
- measurements
- options
Methods
alias of
list-
n_runs:
int
-
benchmarks:
List[str]
-
started:
str
- errors
alias of
list
-
limit:
Optional[int] = None
-
duration:
Optional[float] = None
-
env:
Optional[str] = None
-
device:
Optional[str] = None
-
options:
Optional[Dict] = None
-
successes:
Optional[int] = 0
-
failures:
Optional[int] = 0
-
measurements:
Optional[List[Measurement]] = None
Benchmark ABC
Defines the base class for all Benchmarks.
- class construe.benchmark.base.Benchmark(**kwargs)[source]
Bases:
ABCAll benchmarks must subclass this class to ensure all properties and methods are correctly set for generic benchmarks to be run correctly.
- Attributes:
- data_home
- description
- metadata
- model_home
- options
- use_sample
Methods
after([cleanup])This method is called after the benchamrk is run; if cleanup is True the class should delete any cached datasets or models.
before()This method is called before the benchmark runs and should cause it to setup any datasets and models needed for the benchmark to run.
inference(instance)This represents the primary inference of the benchmark and is measured for latency and memory usage to add to the metrics.
instances([limit])This method should yield all instances in the dataset at least once.
preprocess(instance)Any preprocessing that must be performed on an instance is handled with this method.
total(**kwargs)For progress bar purposes should report the total number of instances in one run of the Benchmark.
- abstract static total(**kwargs)[source]
For progress bar purposes should report the total number of instances in one run of the Benchmark. Generally this should be hard-coded but can also be computed if necessary.
- property data_home: str
- property model_home: str
- property use_sample: bool
- property metadata: Dict
- property options: Dict | None
- abstract property description
- abstract before()[source]
This method is called before the benchmark runs and should cause it to setup any datasets and models needed for the benchmark to run.
- abstract after(cleanup=True)[source]
This method is called after the benchamrk is run; if cleanup is True the class should delete any cached datasets or models.
- abstract instances(limit=None)[source]
This method should yield all instances in the dataset at least once.
- Return type:
Generator[Any,None,None]
Limit Utility
Handles limiting the output of generators.