Benchee Files

1.5.0 (2025-10-21)

Finally max_sample_size is here to alleviate a variety of woes resulting from gathering too many samples! Also finally, Benchee lets you exclude outliers!

Features (User Facing)

• Introduce max_sample_size which guides how many samples will be gathered at most for a given scenario. This avoids a variety of issues when scenarios gather too many samples (memory consumption, statistics taking long to calculate, formatters hanging/not working). Defaults to 1_000_000, setting it to nil gathers unlimited samples again (behavior before this version).
• Introduce exclude_outliers option which when set to true will automatically exclude outliers from the samples gathered. Especially important for run time, you can remove samples caused by garbage collection or external factors. Defaults to false. Shout out to @NickNeck who implemented this long wished for feature over in Statistex.
• Display input_name entries in Livebook/Table.Reader protocol. Thanks @madlep!

Bugfixes (User Facing)

• fixed a bug where if times were supplied as 0 instead of 0.0 we'd sometimes gather a single measurement
• elixir 1.19 compilation warnings have been fixed

Features (Plugins)

• The %Benchee.Statistics{} struct now comes with values to accompany the outlier exclusion feature:
• outliers - if outlier exclusion was enabled, may include any samples of outliers that were found, empty list otherwise
• lower_outlier_bound - value below which values are considered an outlier
• upper_outlier_bound - value above which values are considered an outlier