ONNX CI Pipelines#

  • CI pipelines matrix:

    When it runs

    Config

    Test

    Linux-CI

    Every PR

    • Ubuntu-18.04
    • DEBUG=1 or 0
    • ONNX_USE_LITE_PROTO=OFF
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_BUILD_TESTS=1
    • ONNX_WERROR=ON
    • ONNX_ML=1 or 0

    • ONNX C++ tests
    • Style check (flake8, mypy, and clang-format)
    • Test doc generation
    • Test proto generation
    • Verify node test generation

    Windows-CI

    Every PR

    • windows-2019
    • ONNX_USE_LITE_PROTO=ON
    • ONNX_USE_PROTOBUF_SHARED_LIBS=ON
    • ONNX_BUILD_TESTS=1
    • ONNX_WERROR=ON
    • ONNX_ML=1 or 0

    • Test building ONNX in conda environment
    • Test doc generation
    • Test proto generation
    • Verify node test generation

    Mac-CI

    Every PR

    • macOS-10.15
    • DEBUG=1
    • ONNX_USE_LITE_PROTO=ON or OFF
    • ONNX_ML=1 or 0
    • ONNX_BUILD_TESTS=1
    • ONNX_WERROR=ON

    • ONNX C++ tests
    • Test doc generation
    • Test proto generation
    • Verify node test generation

    Windows_No_Exception CI

    Every PR

    • vs2019-winlatest
    • ONNX_DISABLE_EXCEPTIONS=ON
    • ONNX_USE_LITE_PROTO=ON
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1
    • ONNX_USE_MSVC_STATIC_RUNTIME=ON
    • ONNX_DISABLE_STATIC_REGISTRATION=ON or OFF

    • Only ONNX C++ tests
    • Test selective schema loading

    Lint / Lint Python

    Every PR

      ubuntu-latest

    • Not required – it shows lint warnings for suggestions in PR
    • flake8 (but required by style.sh in another CI)
    • pyflakes
    • misspell
    • shellcheck
    • pylint
    • mypy (but required by style.sh in another CI)

    Lint / Enforce style

    Every PR

      ubuntu-latest

    • flake8
    • isort
    • black
    • mypy
    • clang-format
    • unix line endings
    • c++ namespace rules
    • Auto-generated files are up to date

    WindowsRelease

    • Main branch
    • Release branch
    • Weekly(1)

    • Latest Windows
    • x86 and x64
    • ONNX_USE_LITE_PROTO=ON
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1
    • ONNX_USE_MSVC_STATIC_RUNTIME=OFF

    • Release Windows wheel
    • Release onnx-weekly package
    • Verify with different dependency versions - latest and min supported numpy version, latest and min supported protobuf version(2)
    • Verify ONNX with the latest ONNX Runtime PyPI package(3).

    LinuxRelease_aarch64

    • Main branch
    • Release branch
    • Weekly

    • Latest manylinux2014_aarch64
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1
    • ONNX_USE_LITE_PROTO=ON

    • Release Linux aarch64 wheel
    • Release onnx-weekly package
    • Verify with different dependency versions - latest numpy version, latest and min supported protobuf version
    • Verify ONNX with the latest ONNX Runtime PyPI package

    LinuxRelease_x86_64

    • Main branch
    • Release branch
    • Weekly

    • Latest LinuxRelease_x86_64
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1
    • ONNX_USE_LITE_PROTO=ON

    • Release Linux x86_64 wheel
    • Release onnx-weekly package
    • Test TEST_HUB=1(4)
    • Verify with different dependency versions - latest numpy version, latest and min supported protobuf version
    • Verify ONNX with the latest ONNX Runtime PyPI package.

    MacRelease

    • Main branch
    • Release branch
    • Weekly

    • macos-11
    • MACOSX_DEPLOYMENT_TARGET=10.12(5)
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1
    • ONNX_USE_LITE_PROTO=ON

    • Release Mac wheel
    • Release onnx-weekly package
    • Verify with different dependency versions - latest numpy version, latest and min supported protobuf version
    • Verify ONNX with the latest ONNX Runtime PyPI package.
    • Test source distribution generation
    • Test build with source distribution
    • Release onnx-weekly source distribution

    Weekly CI with latest onnx.checker

    weekly(6)

    • macos-latest
    • MACOSX_DEPLOYMENT_TARGET=10.12
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1

    • Test latest ONNX checker
    • Test latest ONNX shape inference
    • With all models from onnx/models(7)

    • (1) When the release CIs will run:

      • After a PR has been merged into main/rel-* branch

      • Run weekly (Sunday midnight) and release Python wheel to onnx-weekly package on TestPyPI.

      • Any PR targeting rel-* branch

      • To manually run them, add a PR label “run release CIs” (only maintainers have permission).

    • (2) Minimum supported versions are listed here.

    • (3) Test ONNX Python wheel with onnxruntime.InferenceSession from latest ONNXRuntime. Please note that ONNX Runtime does not support Windows-x86 thus its verification is skipped.

    • (4) TEST_HUB=1 will test onnx.hub by using this API to download an ONNX model from onnx/models. This test is restricted to only 1 pipeline for saving quota usage.

    • (5) Although the build environment is macos-11, use MACOSX_DEPLOYMENT_TARGET=10.12 and -p macosx_10_12_x86_64 to force the wheel to support 10.12+.

    • (6):

      • The ONNX Model Zoo test will run weekly (Sunday midnight)

      • To manually trigger it, add a PR label “test ONNX Model Zoo” (only maintainers have permission). Please note that it will need a lot of download bandwidth from onnx/models so use it with caution.

    • (7) Some old deprecated models (opset-1) are skipped.