scikit-learn benchmarks#
scikit-learn’s team has started to develop a benchmark located here: scikit-learn_benchmarks I replicate here the steps I used to run and publish it from a local machine.
Installation#
I followed the steps
git clone https://github.com/scikit-learn/scikit-learn.git
cd asv_benchmarks
Run a benchmark#
I then ran a first benchmark with my current installation of Python.
asv run -b LinearRegression --no-pull
The tests do not store any result with option
--option=<python_path>
.
Publish a benchmark#
I then published it on a local directory.
asv publish -o html
Server to display the content#
I created a key
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365
And then a small application:
from starlette.routing import Router, Mount
from starlette.staticfiles import StaticFiles
app = Router(routes=[
Mount('/', app=StaticFiles(directory='html'), name="html"),
])
And a server:
uvicorn webapp:app ---host <host> -port 8877 --ssl-keyfile=./key.pem --ssl-certfile=./cert.pem
Other benchmarks#
The first benchmark extends the official scikit-learn benchmarks available at scikit-learn_benchmarks. The results can be seen at this Scikit-Learn/ONNX benchmark with AirSpeedVelocity.
The second benchmark is produced using an automated way implemented in mlprodict. The sources are available at asv-skl2onnx and displayed at Prediction with scikit-learn and ONNX benchmark. A subset of these models is available at Prediction with scikit-learn and ONNX benchmark (SVM + Trees).
The last benchmark is a standalone benchmark only comparing onnxruntime and scikit-learn. The sources are available at scikit-onnx-benchmark and displayed at onnxruntime vs scikit-learn for comparison.
I also created two mini benchmark to get a sense of what the previous ones look like: mlprodict model of benchmark, mlprodict model applied to linear models.