module jenkinshelper.jenkins_server

Inheritance diagram of pyquickhelper.jenkinshelper.jenkins_server

Short summary

module pyquickhelper.jenkinshelper.jenkins_server

Extends Jenkins Server from python-jenkins.

source on GitHub

Classes

class

truncated documentation

JenkinsExt

Extensions for the Jenkins server based on module python-jenkins.

Properties

property

truncated documentation

Engines

Static Methods

staticmethod

truncated documentation

get_cmd_custom

Custom script for Jenkins.

hash_string

Hashes a string.

Methods

method

truncated documentation

__init__

_setup_jenkins_server_job_iteration

_setup_jenkins_server_modules_loop

adjust_scheduler

Adjusts the scheduler to avoid having two jobs starting at the same time, jobs are delayed by an hour, two hours, …

create_job_template

Adds a job to the Jenkins server.

delete_all_jobs

Deletes all jobs permanently.

delete_job

Deletes Jenkins job permanently.

extract_requirements

Extracts the requirements for a job.

get_cmd_standalone

Custom command for Jenkins (such as updating conda)

get_engine_from_job

Extracts the engine from the job definition, it should be like [engine].

get_jenkins_job_name

Infers a name for the jenkins job.

get_jenkins_script

Builds the Jenkins script for a module and its options.

get_jobs

Gets the list of all jobs recursively to the given folder depth, see get_all_jobs. …

jenkins_open

Overloads the same method from module python-jenkins to replace string by bytes.

process_options

Postprocesses a script inserted in a job definition.

setup_jenkins_server

Sets up many jobs in Jenkins.

Documentation

Extends Jenkins Server from python-jenkins.

source on GitHub

class pyquickhelper.jenkinshelper.jenkins_server.JenkinsExt(url, username=None, password=None, timeout=<object object>, mock=False, engines=None, platform=None, pypi_port=8067, fLOG=<function noLOG>, mails=None)[source][source]

Bases: jenkins.Jenkins

Extensions for the Jenkins server based on module python-jenkins.

Some useful Jenkins extensions:

The whole class can define many different engines. A job can send a mail at the end of the job execution.

source on GitHub

Parameters
  • url – url of the server

  • username – username

  • password – password

  • timeout – timeout

  • mock – True by default, if False, avoid talking to the server

  • engines – list of Python engines {name: path to python.exe}

  • platform – platform of the Jenkins server

  • pypi_port – pypi port used for the documentation server

  • mails – (str) list of mails to contact in case of a mistaje

  • fLOG – logging function

If platform is None, it is replace by the value returned by get_platform.

source on GitHub

property Engines[source]

the available engines

source on GitHub

Type

return

__init__(url, username=None, password=None, timeout=<object object>, mock=False, engines=None, platform=None, pypi_port=8067, fLOG=<function noLOG>, mails=None)[source][source]
Parameters
  • url – url of the server

  • username – username

  • password – password

  • timeout – timeout

  • mock – True by default, if False, avoid talking to the server

  • engines – list of Python engines {name: path to python.exe}

  • platform – platform of the Jenkins server

  • pypi_port – pypi port used for the documentation server

  • mails – (str) list of mails to contact in case of a mistaje

  • fLOG – logging function

If platform is None, it is replace by the value returned by get_platform.

source on GitHub

_artifacts = '\n <hudson.tasks.ArtifactArchiver>\n <artifacts>__PATTERN__</artifacts>\n <allowEmptyArchive>true</allowEmptyArchive>\n <onlyIfSuccessful>true</onlyIfSuccessful>\n <fingerprint>false</fingerprint>\n <defaultExcludes>true</defaultExcludes>\n <caseSensitive>true</caseSensitive>\n </hudson.tasks.ArtifactArchiver>\n'[source]
_cleanup_repo = '\n<hudson.plugins.ws__cleanup.PreBuildCleanup plugin="ws-cleanup@0.37">\n <deleteDirs>true</deleteDirs>\n <cleanupParameter></cleanupParameter>\n <externalDelete></externalDelete>\n <disableDeferredWipeout>true</disableDeferredWipeout>\n</hudson.plugins.ws__cleanup.PreBuildCleanup>\n'[source]
_config_job = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<project>\n <actions />\n <description>__DESCRIPTION__</description>\n <logRotator class="hudson.tasks.LogRotator">\n <daysToKeep>__KEEP__</daysToKeep>\n <numToKeep>__KEEP__</numToKeep>\n <artifactDaysToKeep>-1</artifactDaysToKeep>\n <artifactNumToKeep>-1</artifactNumToKeep>\n </logRotator>\n <keepDependencies>false</keepDependencies>\n <properties />\n __GITREPOXML__\n <canRoam>true</canRoam>\n <disabled>false</disabled>\n <blockBuildWhenDownstreamBuilding>true</blockBuildWhenDownstreamBuilding>\n <blockBuildWhenUpstreamBuilding>true</blockBuildWhenUpstreamBuilding>\n __TRIGGER__\n <concurrentBuild>false</concurrentBuild>\n __LOCATION__\n <builders>\n __TASKS__\n </builders>\n <publishers>\n __PUBLISHERS__\n </publishers>\n <buildWrappers>\n <hudson.plugins.build__timeout.BuildTimeoutWrapper plugin="build-timeout@1.19">\n <strategy class="hudson.plugins.build_timeout.impl.NoActivityTimeOutStrategy">\n <timeoutSecondsString>__TIMEOUT__</timeoutSecondsString>\n </strategy>\n <operationList>\n <hudson.plugins.build__timeout.operations.AbortOperation/>\n <hudson.plugins.build__timeout.operations.FailOperation/>\n </operationList>\n </hudson.plugins.build__timeout.BuildTimeoutWrapper>\n __BUILDWRAPPERS__\n </buildWrappers>\n</project>\n'[source]
_git_repo = '\n<scm class="hudson.plugins.git.GitSCM" plugin="git@2.3.4">\n <configVersion>2</configVersion>\n <userRemoteConfigs>\n <hudson.plugins.git.UserRemoteConfig>\n <url>__GITREPO__</url>\n __CRED__\n </hudson.plugins.git.UserRemoteConfig>\n </userRemoteConfigs>\n <branches>\n <hudson.plugins.git.BranchSpec>\n <name>*/master</name>\n </hudson.plugins.git.BranchSpec>\n </branches>\n <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>\n <submoduleCfg class="list" />\n __WIPE__\n</scm>\n'[source]
_publishers = '\n<hudson.tasks.Mailer plugin="mailer">\n <recipients>__MAIL__</recipients>\n <dontNotifyEveryUnstableBuild>false</dontNotifyEveryUnstableBuild>\n <sendToIndividuals>true</sendToIndividuals>\n</hudson.tasks.Mailer>\n'[source]
_setup_jenkins_server_job_iteration(job, get_jenkins_script, location, adjust_scheduler, add_environ, yml_engine, overwrite, prefix, credentials, github, disable_schedule, jenkins_server, update, indexes, deps, i, counts)[source][source]
_setup_jenkins_server_modules_loop(jobs, counts, get_jenkins_script, location, adjust_scheduler, add_environ, yml_engine, overwrite, prefix, credentials, github, disable_schedule, jenkins_server, update, indexes, deps)[source][source]
_task_batch_lin = '\n<hudson.tasks.Shell>\n <command>__SCRIPT__\n </command>\n</hudson.tasks.Shell>\n'[source]
_task_batch_win = '\n<hudson.tasks.BatchFile>\n <command>__SCRIPT__\n </command>\n</hudson.tasks.BatchFile>\n'[source]
_trigger_startup = '\n<triggers>\n <org.jvnet.hudson.plugins.triggers.startup.HudsonStartupTrigger plugin="startup-trigger-plugin">\n <quietPeriod>0</quietPeriod>\n <runOnChoice>ON_BOTH</runOnChoice>\n </org.jvnet.hudson.plugins.triggers.startup.HudsonStartupTrigger>\n</triggers>\n'[source]
_trigger_time = '\n<triggers>\n <hudson.triggers.TimerTrigger>\n <spec>__SCHEDULER__</spec>\n </hudson.triggers.TimerTrigger>\n</triggers>\n'[source]
_trigger_up = '\n<triggers>\n <jenkins.triggers.ReverseBuildTrigger>\n <spec></spec>\n <upstreamProjects>__UP__</upstreamProjects>\n <threshold>\n <name>__FAILURE__</name>\n <ordinal>__ORDINAL__</ordinal>\n <color>__COLOR__</color>\n <completeBuild>true</completeBuild>\n </threshold>\n </jenkins.triggers.ReverseBuildTrigger>\n</triggers>\n'[source]
_wipe_repo = '\n <extensions>\n <hudson.plugins.git.extensions.impl.WipeWorkspace />\n </extensions>\n'[source]
adjust_scheduler(scheduler, adjust_scheduler=True)[source][source]

Adjusts the scheduler to avoid having two jobs starting at the same time, jobs are delayed by an hour, two hours, three hours…

Parameters
  • scheduler – existing scheduler

  • adjust_scheduler – True to change it

Returns

new scheduler (only hours are changed)

The function uses member _scheduled_jobs. It creates it if it does not exist.

source on GitHub

create_job_template(name, git_repo, credentials='', upstreams=None, script=None, location=None, keep=30, scheduler=None, py27=False, description=None, default_engine_paths=None, success_only=False, update=False, timeout=1200, additional_requirements=None, return_job=False, adjust_scheduler=True, clean_repo=True, **kwargs)[source][source]

Adds a job to the Jenkins server.

Parameters
  • name – name

  • credentials – credentials

  • git_repo – git repository

  • upstreams – the build must run after… (even if failures), must be None in that case

  • script – script to execute or list of scripts

  • keep – number of buils to keep

  • location – location of the build

  • scheduler – add a schedule time (upstreams must be None in that case)

  • py27 – python 2.7 (True) or Python 3 (False)

  • description – add a description to the job

  • default_engine_paths – define the default location for python engine, should be dictionary { engine: path }, see below.

  • success_only – only triggers the job if the previous one was successful

  • update – update the job instead of creating it

  • additional_requirements – requirements for this module built by this Jenkins server, otherthise, we assume they are available on the installed distribution

  • timeout – specify a timeout

  • kwargs – additional parameters

  • adjust_scheduler – adjust the scheduler of a job so that it is delayed if this spot is already taken

  • return_job – return job instead of submitting the job

  • clean_repo – clean the repository before building (default is yes)

The job can be modified on Jenkins. To add a time trigger:

H H(13-14) * * *

Same trigger but once every week and not every day (Sunday for example):

H H(13-14) * * 0

Parameter success_only prevents a job from running if the previous one failed. Options success_only must be specified. Parameter update updates a job instead of creating it.

source on GitHub

delete_all_jobs()[source][source]

Deletes all jobs permanently.

Returns

list of deleted jobs

source on GitHub

delete_job(name)[source][source]

Deletes Jenkins job permanently.

Parameters

name – name of Jenkins job, str

source on GitHub

extract_requirements(job)[source][source]

Extracts the requirements for a job.

Parameters

job – job name

Returns

3-tuple job, local requirements, pipy requirements

Example:

"pyensae <-- pyquickhelper <---- qgrid"

The function returns:

(pyensae, ["pyquickhelper"], ["qgrid"])

source on GitHub

static get_cmd_custom(job)[source][source]

Custom script for Jenkins.

Parameters

job – module and options

Returns

script

source on GitHub

get_cmd_standalone(job)[source][source]

Custom command for Jenkins (such as updating conda)

Parameters

job – module and options

Returns

script

source on GitHub

get_engine_from_job(job, return_key=False)[source][source]

Extracts the engine from the job definition, it should be like [engine].

Parameters
  • job – job string

  • return_key – return the engine name too

Returns

engine or tuple(engine, name)

If their is no engine definition, the system uses the default one (key=*default*) if it was defined. Otherwise, it raises an exception.

source on GitHub

get_jenkins_job_name(job)[source][source]

Infers a name for the jenkins job.

Parameters

job – str

Returns

name

source on GitHub

get_jenkins_script(job)[source][source]

Builds the Jenkins script for a module and its options.

Parameters

job – module and options

Returns

script

Method setup_jenkins_server describes which tags this method can interpret. The method allow command such as [custom...], they will be run in a virtual environment as setup.py custom.... Parameter job can be empty, in that case, this function returns an empty string. Requirements local and from pipy can be specified by added in the job name:

  • <-- module1, module2 for local requirements

source on GitHub

get_jobs(folder_depth=0, folder_depth_per_request=10, view_name=None)[source][source]

Gets the list of all jobs recursively to the given folder depth, see get_all_jobs.

Returns

list of jobs, [ { str: str} ]

source on GitHub

static hash_string(s, le=4)[source][source]

Hashes a string.

Parameters
  • s – string

  • le – cut the string to the first l character

Returns

hashed string

source on GitHub

jenkins_open(req, add_crumb=True, resolve_auth=True)[source][source]

Overloads the same method from module python-jenkins to replace string by bytes.

Parameters

source on GitHub

process_options(script, options)[source][source]

Postprocesses a script inserted in a job definition.

Parameters
  • script – script to execute (in a list)

  • options – dictionary with options

Returns

new script

source on GitHub

setup_jenkins_server(github, modules, get_jenkins_script=None, overwrite=False, location=None, prefix='', credentials='', update=True, yml_engine='jinja2', add_environ=True, disable_schedule=False, adjust_scheduler=True)[source][source]

Sets up many jobs in Jenkins.

Parameters
  • github – github account if it does not start with http://, the link to git repository of the project otherwise, we assume all jobs in modules are located on the same account otherwise the function will have to called twice with different parameters

  • modules – modules for which to generate the

  • get_jenkins_script – see get_jenkins_script (default value if this parameter is None)

  • overwrite – do not create the job if it already exists

  • location – None for default or a local folder

  • prefix – add a prefix to the name

  • credentials – credentials to use for the job (string or dictionary)

  • update – update job instead of deleting it if the job already exists

  • yml_engine – templating engine used to process yaml config files

  • add_environ – use of local environment variables to interpret the job

  • adjust_scheduler – adjust the scheduler of a job so that it is delayed if this spot is already taken

  • disable_schedule – disable scheduling for all jobs

Returns

list of created jobs

If credentials are a dictionary, the function looks up into it by using the git repository as a key. If it does not find it, it looks for default key. If there is not found, the function assumes, there is not credentials for this git repository.

The function get_jenkins_script is called with the following parameters:

  • job

The extension Extra Columns Plugin is very useful to add extra columns to a view (the description, the output of the last execution). Here is a list of useful extensions:

Tag description:

  • [engine]: to use this specific engine (Python path)

  • [27]: run with python 2.7

  • [LONG]: run longer unit tests (files start by test_LONG_)

  • [SKIP]: run skipped unit tests (files start by test_SKIP_)

  • [GUI]: run skipped unit tests (files start by test_GUI_)

  • [custom.+]: run setup.py <custom.+> in a virtual environment

  • [UT] {-d_10}: run setup.py unittests -d 10 in a virtual environment, -d 10 is one of the possible parameters

Others tags:

  • [conda_update]: update conda distribution

  • [update]: update distribution

  • [install]: update distribution

  • [local_pypi]: write a script to run a local pypi server on port 8067 (default option)

  • pymyinstall [update_modules]: run a script to update all modules (might have to be ran a couple of times before being successful)

modules is a list defined as follows:

  • each element can be a string or a tuple (string, schedule time) or a list

  • if it is a list, it contains a list of elements defined as previously

  • if the job at position i is not scheduled, it will start after the last job at position i-1 whether or not it fails

  • the job can be defined as a tuple of 3 elements, the last one contains options

The available options are:

  • pre: defines a string to insert at the beginning of a job

  • post: defines a string to insert at the end of a job

  • script: defines a full script if the job to execute is custom

Example

modules=[  # update anaconda
        ("standalone [conda_update] [anaconda3]",
         "H H(0-1) * * 0"),
        "standalone [conda_update] [anaconda2] [27]",
        "standalone [local_pypi]",
        #"standalone [install]",
        #"standalone [update]",
        #"standalone [install] [py34]",
        #"standalone [update] [py34]",
        #"standalone [install] [winpython]",
        #"standalone [update] [winpython]",
 # pyquickhelper and others,
 ("pyquickhelper", "H H(2-3) * * 0"),
 ("pysqllike <-- pyquickhelper", None, dict(success_only=True)),
 ["python3_module_template <-- pyquickhelper",
     "pyquickhelper [27] [anaconda2]"],
 ["pyquickhelper [winpython]",
     "python3_module_template [27] [anaconda2] <-- pyquickhelper", ],
 ["pymyinstall <-- pyquickhelper", "pyensae <-- pyquickhelper"],
 ["pymmails <-- pyquickhelper", "pyrsslocal <-- pyquickhelper, pyensae"],
 ["pymyinstall [27] [anaconda2] <-- pyquickhelper", "pymyinstall [LONG] <-- pyquickhelper"],
 # update, do not move, it depends on pyquickhelper
 ("pyquickhelper [anaconda3]", "H H(2-3) * * 1"),
 ["pyquickhelper [winpython]", "pysqllike [anaconda3]",
            "pysqllike [winpython] <-- pyquickhelper",
            "python3_module_template [anaconda3] <-- pyquickhelper",
            "python3_module_template [winpython] <-- pyquickhelper",
            "pymmails [anaconda3] <-- pyquickhelper",
            "pymmails [winpython] <-- pyquickhelper",
            "pymyinstall [anaconda3] <-- pyquickhelper",
            "pymyinstall [winpython] <-- pyquickhelper"],
 ["pyensae [anaconda3] <-- pyquickhelper",
            "pyensae [winpython] <-- pyquickhelper",
            "pyrsslocal [anaconda3] <-- pyquickhelper, pyensae",
            "pyrsslocal [winpython] <-- pyquickhelper"],
 ("pymyinstall [update_modules]",
            "H H(0-1) * * 5"),
 "pymyinstall [update_modules] [winpython]",
 "pymyinstall [update_modules] [py34]",
 "pymyinstall [update_modules] [anaconda2]",
 "pymyinstall [update_modules] [anaconda3]",
 # py35
 ("pyquickhelper [py34]", "H H(2-3) * * 2"),
 ["pysqllike [py34]",
            "pymmails [py34] <-- pyquickhelper",
            "python3_module_template [py34] <-- pyquickhelper",
            "pymyinstall [py34] <-- pyquickhelper"],
 "pyensae [py34] <-- pyquickhelper",
 "pyrsslocal [py34] <-- pyquickhelper, pyensae",
],

Example:

from ensae_teaching_cs.automation.jenkins_helper import setup_jenkins_server
from pyquickhelper.jenkinshelper import JenkinsExt

engines = dict(Anaconda2=r"C:\Anaconda2",
               Anaconda3=r"C:\Anaconda3",
               py35=r"c:\Python35_x64",
               py36=r"c:\Python36_x64",
               default=r"c:\Python36_x64",
               custom=r"c:\CustomPython")

js = JenkinsExt('http://machine:8080/', "user", "password", engines=engines)

if True:
    js.setup_jenkins_server(github="sdpython", overwrite = True,
                            location = r"c:\jenkins\pymy")

Another example:

import sys
sys.path.append(r"C:\<path>\ensae_teaching_cs\src")
sys.path.append(r"C:\<path>\pyquickhelper\src")
sys.path.append(r"C:\<path>\pyensae\src")
sys.path.append(r"C:\<path>\pyrsslocal\src")
from ensae_teaching_cs.automation.jenkins_helper import setup_jenkins_server, JenkinsExt
js = JenkinsExt("http://<machine>:8080/", <user>, <password>)
js.setup_jenkins_server(location=r"c:\jenkins\pymy", overwrite=True, engines=engines)

Parameter credentials can be a dictionary where the key is the git repository. Parameter dependencies and no_dep were removed. Dependencies are now specified in the job name using <-- and they exclusively rely on pipy (local or remote). Add options for module Build Timeout Plugin.

source on GitHub

pyquickhelper.jenkinshelper.jenkins_server._modified_linux_jenkins(requirements_local, requirements_pypi, module='__MODULE__', port='__PORT__', platform=None)[source][source]
pyquickhelper.jenkinshelper.jenkins_server._modified_linux_jenkins_any(requirements_local, requirements_pypi, module='__MODULE__', port='__PORT__', platform=None)[source][source]
pyquickhelper.jenkinshelper.jenkins_server._modified_windows_jenkins(requirements_local, requirements_pypi, module='__MODULE__', port='__PORT__', platform=None)[source][source]
pyquickhelper.jenkinshelper.jenkins_server._modified_windows_jenkins_27(requirements_local, requirements_pypi, module='__MODULE__', port='__PORT__', anaconda=True, platform=None)[source][source]
pyquickhelper.jenkinshelper.jenkins_server._modified_windows_jenkins_any(requirements_local, requirements_pypi, module='__MODULE__', port='__PORT__', platform=None)[source][source]