module optim.sgd#

Inheritance diagram of mlstatpy.optim.sgd

Short summary#

module mlstatpy.optim.sgd

Implements simple stochastic gradient optimisation. It is inspired from _stochastic_optimizers.py.

source on GitHub

Classes#

class

truncated documentation

BaseOptimizer

Base stochastic gradient descent optimizer.

SGDOptimizer

Stochastic gradient descent optimizer with momentum.

Methods#

method

truncated documentation

__init__

__init__

_display_progress

Displays training progress.

_display_progress

Displays training progress.

_evaluate_early_stopping

_evaluate_early_stopping

_get_updates

_get_updates

Gets the values used to update params with given gradients.

_regularize_gradient

Applies regularization.

_regularize_gradient

Applies regularization.

iteration_ends

Performs update to learning rate and potentially other states at the end of an iteration.

iteration_ends

Performs updates to learning rate and potential other states at the end of an iteration.

loss_regularization

loss_regularization

train

Optimizes the coefficients.

train

Optimizes the coefficients.

update_coef

Updates coefficients with given gradient.

update_coef

Updates coefficients with given gradient.

Documentation#

Implements simple stochastic gradient optimisation. It is inspired from _stochastic_optimizers.py.

source on GitHub

class mlstatpy.optim.sgd.BaseOptimizer(coef, learning_rate_init=0.1, min_threshold=None, max_threshold=None, l1=0.0, l2=0.0)#

Bases : object

Base stochastic gradient descent optimizer.

Paramètres

  • coef – array, initial coefficient

  • learning_rate_init – float The initial learning rate used. It controls the step-size in updating the weights.

  • min_threshold – coefficients must be higher than min_thresold

  • max_threshold – coefficients must be below than max_thresold

The class holds the following attributes:

  • learning_rate: float, the current learning rate

  • coef: optimized coefficients

  • min_threshold, max_threshold: coefficients thresholds

  • l2: L2 regularization

  • l1: L1 regularization

source on GitHub

__init__(coef, learning_rate_init=0.1, min_threshold=None, max_threshold=None, l1=0.0, l2=0.0)#
_display_progress(it, max_iter, loss, losses=None, msg=None)#

Displays training progress.

_evaluate_early_stopping(it, max_iter, losses, early_th, verbose=False)#
_get_updates(grad)#
_regularize_gradient(grad)#

Applies regularization.

source on GitHub

iteration_ends(time_step)#

Performs update to learning rate and potentially other states at the end of an iteration.

source on GitHub

train(X, y, fct_loss, fct_grad, max_iter=100, early_th=None, verbose=False)#

Optimizes the coefficients.

Paramètres

  • X – datasets (array)

  • y – expected target

  • fct_loss – loss function, signature: f(coef, X, y) -> float

  • fct_grad – gradient function, signature: g(coef, x, y, i) -> array

  • max_iter – number maximum of iteration

  • early_th – stops the training if the error goes below this threshold

  • verbose – display information

Renvoie

loss

The method keeps the best coefficients for the minimal loss.

source on GitHub

update_coef(grad)#

Updates coefficients with given gradient.

Paramètres

grad – array, gradient

source on GitHub

class mlstatpy.optim.sgd.SGDOptimizer(coef, learning_rate_init=0.1, lr_schedule='invscaling', momentum=0.9, power_t=0.5, early_th=None, min_threshold=None, max_threshold=None, l1=0.0, l2=0.0)#

Bases : BaseOptimizer

Stochastic gradient descent optimizer with momentum.

Paramètres

  • coef – array, initial coefficient

  • learning_rate_init – float The initial learning rate used. It controls the step-size in updating the weights,

  • lr_schedule – {“constant”, “adaptive”, “invscaling”}, learning rate schedule for weight updates, “constant” for a constant learning rate given by learning_rate_init. “invscaling” gradually decreases the learning rate learning_rate_ at each time step t using an inverse scaling exponent of power_t. learning_rate_ = learning_rate_init / pow(t, power_t), “adaptive”, keeps the learning rate constant to learning_rate_init as long as the training keeps decreasing. Each time 2 consecutive epochs fail to decrease the training loss by tol, or fail to increase validation score by tol if “early_stopping” is on, the current learning rate is divided by 5.

  • momentum – float Value of momentum used, must be larger than or equal to 0

  • power_t – double The exponent for inverse scaling learning rate.

  • early_th – stops if the error goes below that threshold

  • min_threshold – lower bound for parameters (can be None)

  • max_threshold – upper bound for parameters (can be None)

  • l1 – L1 regularization

  • l2 – L2 regularization

The class holds the following attributes:

  • learning_rate: float, the current learning rate

  • velocity*: array, velocity that are used to update params

Stochastic Gradient Descent applied to linear regression

The following example how to optimize a simple linear regression.

<<<

import numpy
from mlstatpy.optim import SGDOptimizer


def fct_loss(c, X, y):
    return numpy.linalg.norm(X @ c - y) ** 2


def fct_grad(c, x, y, i=0):
    return x * (x @ c - y) * 0.1


coef = numpy.array([0.5, 0.6, -0.7])
X = numpy.random.randn(10, 3)
y = X @ coef

sgd = SGDOptimizer(numpy.random.randn(3))
sgd.train(X, y, fct_loss, fct_grad, max_iter=15, verbose=True)
print('optimized coefficients:', sgd.coef)

>>>

    0/15: loss: 8.762 lr=0.1 max(coef): 0.89 l1=0/1.1 l2=0/0.83
    1/15: loss: 3.516 lr=0.0302 max(coef): 1.2 l1=0.25/1.9 l2=0.029/1.7
    2/15: loss: 3.552 lr=0.0218 max(coef): 1.2 l1=0.13/2.4 l2=0.0067/2.1
    3/15: loss: 1.281 lr=0.018 max(coef): 0.94 l1=0.009/2.1 l2=4.7e-05/1.7
    4/15: loss: 0.6536 lr=0.0156 max(coef): 0.84 l1=0.0088/2 l2=2.9e-05/1.5
    5/15: loss: 0.4447 lr=0.014 max(coef): 0.79 l1=0.015/2 l2=8.1e-05/1.4
    6/15: loss: 0.3163 lr=0.0128 max(coef): 0.79 l1=0.0084/2 l2=4.1e-05/1.4
    7/15: loss: 0.2406 lr=0.0119 max(coef): 0.78 l1=0.041/1.9 l2=0.00071/1.3
    8/15: loss: 0.1743 lr=0.0111 max(coef): 0.77 l1=0.014/1.9 l2=0.00011/1.3
    9/15: loss: 0.1322 lr=0.0105 max(coef): 0.76 l1=0.018/1.9 l2=0.00016/1.2
    10/15: loss: 0.1067 lr=0.00995 max(coef): 0.75 l1=0.011/1.9 l2=6.3e-05/1.2
    11/15: loss: 0.09046 lr=0.00949 max(coef): 0.75 l1=0.0073/1.8 l2=2.7e-05/1.2
    12/15: loss: 0.07389 lr=0.00909 max(coef): 0.74 l1=0.0031/1.8 l2=3.7e-06/1.2
    13/15: loss: 0.06312 lr=0.00874 max(coef): 0.74 l1=0.02/1.8 l2=0.00018/1.2
    14/15: loss: 0.05491 lr=0.00842 max(coef): 0.73 l1=0.0019/1.8 l2=1.9e-06/1.2
    15/15: loss: 0.04847 lr=0.00814 max(coef): 0.73 l1=0.0065/1.8 l2=2e-05/1.1
    optimized coefficients: [ 0.399  0.668 -0.732]

source on GitHub

__init__(coef, learning_rate_init=0.1, lr_schedule='invscaling', momentum=0.9, power_t=0.5, early_th=None, min_threshold=None, max_threshold=None, l1=0.0, l2=0.0)#
_display_progress(it, max_iter, loss, losses=None, msg='loss')#

Displays training progress.

_get_updates(grad)#

Gets the values used to update params with given gradients.

Paramètres

grad – array, gradient

Renvoie

updates, array, the values to add to params

source on GitHub

iteration_ends(time_step)#

Performs updates to learning rate and potential other states at the end of an iteration.

Paramètres

time_step – int number of training samples trained on so far, used to update learning rate for “invscaling”

source on GitHub