Changes made on Keras code

[roywei]

Using this issue to track all the changes made on keras code, to avoid confusion for keras developers and future work.

1. Override Keras model with mxnet model
2. Kernel shape in Conv layers, provide in both channels last and channels first format. Need to implement that for every new Conv layer. In build() and compute_output_shape()
3. change on multi_gpu_model to pass mxnet context
4. Batch Norm
5. Embedding layers

[anirudhacharya]

@roywei please add label - "enhancement"

[roywei]

cudnn7.2 is adding support on channels last, we can remove the kernel shape, transpose, and batch norm performance hacks after that

[sandeep-krishnamurthy]

Full updated list as of v2.2.0

Changes of MXNet backend in keras-team/keras

• keras/README.md : Introducing Keras-MXNet, Installation, link to mxnet-backend/docs, configuring keras backend, support

• keras/docs/backend.md : Add MXNet in all backend related docs

• keras/setup.py : full rewrite for keras-mxnet pip package

• keras/LICENSE : Amazon contribution

• keras/ISSUE_TEMPLATE : Change to refer to awslabs/keras-apache-mxnet repo

• keras/.travis.yml : 2 env for MXNet backend (Python 2.7 and 3.6), Installing MXNet, run keras/backend/mxnet_backend.py tests

• keras/backend/init.py: Default backend is mxnet, import backend functions

• keras/engine/saving.py

In def _need_convert_kernel():

uses_correlation = {'tensorflow': True,
                    'theano': False,
                    'cntk': True,
                    'mxnet': False}

New function:

def save_mxnet_model(model, prefix, epoch=0):

• keras/engine/training_utils.py

In def weighted()

If applying sample weighting, we introduce a fix here.

# If sample_weights shape is like (100, ), we convert it to (100, 1).

# Because, MXNet treats the shape (100, ) as (100) leading to broadcast operator
# failures in below operations.

if K.backend() == 'mxnet' and weight_ndim == 1:
    weights = K.reshape(weights, shape=(weights.shape[0], 1))

• keras/engine/training.py

# We need to overload the Keras Model class to handle MXNet model building activities.
	# get_model() returns the reference to MXNetModel that inherits and extends keras.engine.Model.
	if K.backend() == 'mxnet':
	    Model = K.get_model()

• keras/engine/sequential.py

+# We need to overload the Keras Model class to handle MXNet model building activities.
+# get_model() returns the reference to MXNetModel that inherits and extends keras.engine.Model.
+if K.backend() == 'mxnet':
+    Sequential = K.get_sequential_model()

• keras/engine/models.py

+from .engine.saving import save_mxnet_model

• keras/layers/convolutional.py

MXNet backend special case the kernel shape based on data_format

In class _conv, Conv2DTranspose:

	def build:
		
		if self.data_format == 'channels_first' and K.backend() == 'mxnet':
    			kernel_shape = (self.filters, input_dim) + self.kernel_size

In class _SeparableConv

	def build:
		
		if self.data_format == 'channels_first' and K.backend() == 'mxnet':
 			   depthwise_kernel_shape = (input_dim, self.depth_multiplier) + self.kernel_size
  			  pointwise_kernel_shape = (self.filters, self.depth_multiplier * input_dim) + (1,) * self.rank

In class DepthwiseConv2D

	def build:
		        if self.data_format == 'channels_first' and K.backend() == 'mxnet':
			            depthwise_kernel_shape = (input_dim, self.depth_multiplier, self.kernel_size[0],
	        						                              self.kernel_size[1])

• keras/layers/core.py

Class Lambda

def compute_output_shape():

	# With TensorFlow and MXNet we can infer the output shape directly:
	if K.backend() == 'tensorflow' or K.backend() == 'mxnet':

• keras/layers/normalization.py

Class BatchNormalization():
	def __init__()
		
		#MXNet backend is not yet capable of automatically handling feature axis in channels_last format.
		if axis == -1 and K.image_data_format() == 'channels_first' and K.backend() == 'mxnet':
    			self.axis = 1
		else:
    			self.axis = axis

• keras/layers/embeddings.py

Class Embeddings:

	def call(self, inputs):
    if K.dtype(inputs) != 'int32':
        inputs = K.cast(inputs, 'int32')
    # Use K.embedding(mx.sym.Embedding) in MXNet backend instead of K.gather(mx.sym.take)
    # K.gather is not working with Embedding layer using MXNet backend
    # Refer to this issue: https://github.com/awslabs/keras-apache-mxnet/issues/63
    if K.backend() == "mxnet":
        out = K.embedding(inputs, self.embeddings, self.input_dim, self.output_dim)
    else:
        out = K.gather(self.embeddings, inputs)
    return out

• keras/utils/init.py

from .np_utils import to_channels_first

• keras/models.py

from .engine.saving import save_mxnet_model

• keras/optimizers.py

We monkey patch Keras optimizers with MXNet optimizers

if K.backend() == 'mxnet':
    SGD, Adagrad, Adadelta, Adam, Adamax, RMSprop, Nadam = K.get_optimizers()

• keras/losses.py

+        if K.backend() == 'mxnet':
+            warnings.warn('MXNet Backend: If you are using a custom loss function and use slice operator in custom '
+                          'loss function, please set the **_keras_shape** attribute of the loss tensor explicitly.\n\n'
+                          'Without this workaround, you may encounter shape mismatch errors in the broadcast '
+                          'operations.\n\nFor more details - '
+                          'https://github.com/awslabs/keras-apache-mxnet/issues/120\n\n')

[roywei]

Update:
#137 changed keras/layers/normalization.py line 114, when gamma and beta is None, we still create those params for mxnet backend. Until bug on mxnet side is fixed (apache/mxnet#11774)

[roywei]

Update:
#143 changed Conv1D layer init to have data_format as None instead of 'channels_last'.

Issue raised with Keras, will create PR with keras-team if needed.