mxnet.optimizer.AdaDelta¶

class mxnet.optimizer.AdaDelta(rho=0.9, epsilon=1e-05, **kwargs)[source]¶

The AdaDelta optimizer.

This class implements AdaDelta, an optimizer described in ADADELTA: An adaptive learning rate method, available at https://arxiv.org/abs/1212.5701.

This optimizer updates each weight by:

grad = clip(grad * rescale_grad + wd * weight, clip_gradient)
acc_grad = rho * acc_grad + (1. - rho) * grad * grad
delta = sqrt(acc_delta + epsilon) / sqrt(acc_grad + epsilon) * grad
acc_delta = rho * acc_delta + (1. - rho) * delta * delta
weight -= (delta + wd * weight)

This optimizer accepts the following parameters in addition to those accepted by Optimizer.

Parameters

rho (float) – Decay rate for both squared gradients and delta.
epsilon (float) – Small value to avoid division by 0.

__init__(rho=0.9, epsilon=1e-05, **kwargs)[source]¶: Initialize self. See help(type(self)) for accurate signature.

Methods

`__init__`([rho, epsilon])	Initialize self.
`create_optimizer`(name, **kwargs)	Instantiates an optimizer with a given name and kwargs.
`create_state`(index, weight)	Creates auxiliary state for a given weight.
`create_state_multi_precision`(index, weight)	Creates auxiliary state for a given weight, including FP32 high precision copy if original weight is FP16.
`register`(klass)	Registers a new optimizer.
`set_learning_rate`(lr)	Sets a new learning rate of the optimizer.
`set_lr_mult`(args_lr_mult)	Sets an individual learning rate multiplier for each parameter.
`set_lr_scale`(args_lrscale)	[DEPRECATED] Sets lr scale.
`set_wd_mult`(args_wd_mult)	Sets an individual weight decay multiplier for each parameter.
`update`(index, weight, grad, state)	Updates the given parameter using the corresponding gradient and state.
`update_multi_precision`(index, weight, grad, …)	Updates the given parameter using the corresponding gradient and state.

Attributes

`learning_rate`
`opt_registry`