mxnet.ndarray.BilinearSampler¶

mxnet.ndarray.BilinearSampler(data=None, grid=None, cudnn_off=_Null, out=None, name=None, **kwargs)¶

Applies bilinear sampling to input feature map.

Bilinear Sampling is the key of [NIPS2015] “Spatial Transformer Networks”. The usage of the operator is very similar to remap function in OpenCV, except that the operator has the backward pass.

Given \(data\) and \(grid\), then the output is computed by

\[\begin{split}x_{src} = grid[batch, 0, y_{dst}, x_{dst}] \\ y_{src} = grid[batch, 1, y_{dst}, x_{dst}] \\ output[batch, channel, y_{dst}, x_{dst}] = G(data[batch, channel, y_{src}, x_{src})\end{split}\]

\(x_{dst}\), \(y_{dst}\) enumerate all spatial locations in \(output\), and \(G()\) denotes the bilinear interpolation kernel. The out-boundary points will be padded with zeros.The shape of the output will be (data.shape[0], data.shape[1], grid.shape[2], grid.shape[3]).

The operator assumes that \(data\) has ‘NCHW’ layout and \(grid\) has been normalized to [-1, 1].

BilinearSampler often cooperates with GridGenerator which generates sampling grids for BilinearSampler. GridGenerator supports two kinds of transformation: affine and warp. If users want to design a CustomOp to manipulate \(grid\), please firstly refer to the code of GridGenerator.

Example 1:

## Zoom out data two times
data = array([[[[1, 4, 3, 6],
                [1, 8, 8, 9],
                [0, 4, 1, 5],
                [1, 0, 1, 3]]]])

affine_matrix = array([[2, 0, 0],
                       [0, 2, 0]])

affine_matrix = reshape(affine_matrix, shape=(1, 6))

grid = GridGenerator(data=affine_matrix, transform_type='affine', target_shape=(4, 4))

out = BilinearSampler(data, grid)

out
[[[[ 0,   0,     0,   0],
   [ 0,   3.5,   6.5, 0],
   [ 0,   1.25,  2.5, 0],
   [ 0,   0,     0,   0]]]

Example 2:

## shift data horizontally by -1 pixel

data = array([[[[1, 4, 3, 6],
                [1, 8, 8, 9],
                [0, 4, 1, 5],
                [1, 0, 1, 3]]]])

warp_maxtrix = array([[[[1, 1, 1, 1],
                        [1, 1, 1, 1],
                        [1, 1, 1, 1],
                        [1, 1, 1, 1]],
                       [[0, 0, 0, 0],
                        [0, 0, 0, 0],
                        [0, 0, 0, 0],
                        [0, 0, 0, 0]]]])

grid = GridGenerator(data=warp_matrix, transform_type='warp')
out = BilinearSampler(data, grid)

out
[[[[ 4,  3,  6,  0],
   [ 8,  8,  9,  0],
   [ 4,  1,  5,  0],
   [ 0,  1,  3,  0]]]

Defined in src/operator/bilinear_sampler.cc:L256

Parameters

data (NDArray) – Input data to the BilinearsamplerOp.
grid (NDArray) – Input grid to the BilinearsamplerOp.grid has two channels: x_src, y_src
cudnn_off (boolean or None, optional, default=None) – whether to turn cudnn off
out (NDArray, optional) – The output NDArray to hold the result.

Returns

out – The output of this function.

Return type

NDArray or list of NDArrays