mxnet.optimizer.Optimizer¶

class mxnet.optimizer.Optimizer(rescale_grad=1.0, param_idx2name=None, wd=0.0, clip_gradient=None, learning_rate=0.01, lr_scheduler=None, sym=None, begin_num_update=0, multi_precision=False, param_dict=None)[source]¶

The base class inherited by all optimizers.

Parameters

rescale_grad (float, optional, default 1.0) – Multiply the gradient with rescale_grad before updating. Often choose to be 1.0/batch_size.
param_idx2name (dict from int to string, optional, default None) – A dictionary that maps int index to string name.
clip_gradient (float, optional, default None) – Clip the gradient by projecting onto the box [-clip_gradient, clip_gradient].
learning_rate (float) – The initial learning rate.
lr_scheduler (LRScheduler, optional, default None) – The learning rate scheduler.
wd (float, optional, default 0.0) – The weight decay (or L2 regularization) coefficient. Modifies objective by adding a penalty for having large weights.
sym (Symbol, optional, default None) – The Symbol this optimizer is applying to.
begin_num_update (int, optional, default 0) – The initial number of updates.

multi_precision (bool, optional, default False) –

Flag to control the internal precision of the optimizer.:

False: results in using the same precision as the weights (default),
True: makes internal 32-bit copy of the weights and applies gradients
in 32-bit precision even if actual weights used in the model have lower precision.
Turning this on can improve convergence and accuracy when training with float16.

param_dict (dict of int -> gluon.Parameter, default None) – Dictionary of parameter index to gluon.Parameter, used to lookup parameter attributes such as lr_mult, wd_mult, etc. param_dict shall not be deep copied.
Properties –
---------- –
learning_rate – The current learning rate of the optimizer. Given an Optimizer object optimizer, its learning rate can be accessed as optimizer.learning_rate.

__init__(rescale_grad=1.0, param_idx2name=None, wd=0.0, clip_gradient=None, learning_rate=0.01, lr_scheduler=None, sym=None, begin_num_update=0, multi_precision=False, param_dict=None)[source]¶: Initialize self. See help(type(self)) for accurate signature.

Methods

`__init__`([rescale_grad, param_idx2name, wd, …])	Initialize self.
`create_optimizer`(name, **kwargs)	Instantiates an optimizer with a given name and kwargs.
`create_state`(index, weight)	Creates auxiliary state for a given weight.
`create_state_multi_precision`(index, weight)	Creates auxiliary state for a given weight, including FP32 high precision copy if original weight is FP16.
`register`(klass)	Registers a new optimizer.
`set_learning_rate`(lr)	Sets a new learning rate of the optimizer.
`set_lr_mult`(args_lr_mult)	Sets an individual learning rate multiplier for each parameter.
`set_lr_scale`(args_lrscale)	[DEPRECATED] Sets lr scale.
`set_wd_mult`(args_wd_mult)	Sets an individual weight decay multiplier for each parameter.
`update`(index, weight, grad, state)	Updates the given parameter using the corresponding gradient and state.
`update_multi_precision`(index, weight, grad, …)	Updates the given parameter using the corresponding gradient and state.

Attributes

`learning_rate`
`opt_registry`