updates files, updated readme

.gitignore

@@ -3,3 +3,5 @@ __pycache__
 *.pkl
 *.gz
 *.npz
+ddtn/logs
+ddtn/logs2

README.md

@@ -10,13 +10,14 @@ However, the repo also contains code for other kinds of ST-layers. The following
 transformation models are included:
 
 * Affine transformations
+* Diffiomorphic affine transformations
 * Homography transformations
 * CPAB transformations
 * Thin-Plate-Spline (TPS) transformations
 
 The code is based upon the original implementation of the CPAB transformations by
 Oren Freifeld (Github repo: [cpabDiffeo](https://github.com/freifeld/cpabDiffeo)).
-Additionall, some of the code for doing interpolation is based upon the Tensorflow
+Additionally, some of the code for doing interpolation is based upon the Tensorflow
 implementation of the Spatial Transformer Networks (Github repo: 
 [spatial-transformer-network](https://github.com/kevinzakka/spatial-transformer-network)).
 
@@ -55,35 +56,81 @@ application ect.) we request you to cite the following papers:
 * To use the GPU implementation, you need a nvidia GPU and CUDA + cuDNN installed. 
   See [Tensorflows GPU installation instructions](https://www.tensorflow.org/install/) 
   for more details
+
+The code was testes running python 3.6, tensorflow 1.8.0 and CUDA 9.0. However, 
+the code should be compatible with tensorflow from version 1.4.0.
 
-The code are only tested on Linux, but should also work on MAC. The code cannot 
-run on Windows at the moments, since the compiled dynamic libraries (.so files) 
-are only combiled for UNIX. If comes up with a way to compile these for windows, 
-please let us know.
+The code should run on all operating system with or without an GPU. If you are on
+Linux or MAC and tensorflow is able to detect your GPU then the fast GPU version
+of the CPAB transformations will be uses. If you are on windows or do have an GPU,
+an slower (and memory consuming) implementation will be used that is written in
+pure tensorflow.
 
 ## Installation
 
-Clone this reposatory to a directory of your choice
+1. Clone this reposatory to a directory of your choice
 ```
 git clone https://github.com/SkafteNicki/ddtn
 ```
-Add this directory to your PYTHONPATH
+2. Add this directory to your PYTHONPATH
 ```
 export PYTHONPATH=$PYTHONPATH:$YOUR_FOLDER_PATH/ddtn
 ```
+3. (optional) If you want to use the fast GPU version their is a good chance that
+you have to recompile the dynamic libraries where the operation is defined. Go to
+the folder 'ddtn/ddtn/cuda/' and type the following in a command prompt
+```
+make clean
+make
+```
 
-## Running code
+## Using the code
+
+Try opening a python command promt and type in the follow command
+```
+import ddtn
+```
+This should give you one of the following outputs
+```
+----------------------------------------------------------------------
+Operating system: linux
+Using the fast cuda implementation for CPAB
+----------------------------------------------------------------------
+or
+----------------------------------------------------------------------
+Operating system: linux
+Using the slow pure tensorflow implementation for CPAB
+----------------------------------------------------------------------
+```
+The reposatory comes with two scripts you can try to run
+``` 
+python play_with_transformers.py <- show what the different transformers can do
+python mnist_classifier.py <- trains a classifier on a distorted mnist dataset
+```
+for both script you can type `-h` after to get the commandline options.
 
+To use the different transformers in your own settings, there are primarily 3
+files that are important
+
+1. `ddtn.transformers.transformers` 
+
+   Defines methods called tf_'name'_transformer eg. tf_Affine_transformer, tf_CPAB_transformer ect. that as input takes a grid of points and a parametrization and outputs the transformed grid.
+
+2. `ddtn.transformers.transformer_layers`
+
+   Defines methods called ST_'name'_transformer eg. ST_Affine_transformer, ST_CPAB_transformer ect. that as input takes an image, a parametrization and output the transformed image.
+
+3. `ddtn.transformers.keras_layers` 
+
+   Defined keras layers called Spatial'name'Layer eg. SpatialAffineLayer, SpatialCPABLayer ect. that can be incorporated into keras models (main used high-level-api for tensorflow).
 
 ## Known bugs
-1 "Executor failed to create kernel. Not found: Op type not registered 
+1. *"Executor failed to create kernel. Not found: Op type not registered 
 'tf_CPAB_transformer' in binary running on HedonismeBot. Make sure the Op and 
-Kernel are registered in the binary running in this process"
-   `tf.Defun` seems to be working but still give out this error message. Should
-    probably try to find a replacement for `tf.Defun`
-   - This error has something to do with `tf.Defun`, but the code seems to run.
-   It is probably due to the fact that tensorflow sessions freeze the current
-   graph. Will try to find a replacement for `tf.Defun`. Look into `tf.RegisterGradient`
-   at some point.
+Kernel are registered in the binary running in this process"*.
 
+   `tf.Defun` seems to be working but still give out this error message. Should
+    probably try to find a replacement for `tf.Defun`. Has something to do with
+    the fact that tensorflow sessions freeze the current graph. Look into
+    `tf.RegisterGradient` at some point.
 

ddtn/helper/training_logger.py

@@ -12,63 +12,80 @@
 
 #%%
 class KerasTrainingLogger(tf.keras.callbacks.Callback):
-    """
+    """ A Keras callback class that can be given to any Keras model.fit(...)
+        method. This class does two things:
+            * Logs the loss and acc of the model after each batch
+            * Transform some input imgs after each epoch and logs these
+    
     Arguments
-        data:
+        data: 4D-`Tensor` [n_imgs, height, width, n_channel] with images that
+            are transformed in the end of each epoch
+            
+        name: string, results are stored in a folder called 'log_dir/name'. 
+            Thus to easily compare the results of different runs in tensorboard, 
+            keep the logdir the same but change the name for each run
         
-        log_dir:
+        log_dir: string, directory to store the tensorboard files
             
         trans_layer: integer determining where the transformation layer is in
-        the model
+            the model (usually translayer=0, because it is the first layer)
     """
-    def __init__(self, data, log_dir='./logs', trans_layer=1):
-        super(KerasTrainingLogger, self).__init__()
+    def __init__(self, data, name, log_dir='./logs', trans_layer=0, **kwargs):
+        super(KerasTrainingLogger, self).__init__(**kwargs)
+
+        # Initialize variables
         self.data = data
-        self.N = data.shape[0]
-        self.log_dir = log_dir
+        self.log_dir = log_dir + '/' + name
         self.tl = trans_layer
-        self.validation = None
-        self.trans_func = None
-        self.model = None
-        self.sess = None
         self.step = 0
+        self.validation = self.trans_func = self.model = self.sess = None
+
+        # Placeholders for summary
+        self.loss_p = tf.placeholder(tf.float32)
+        self.acc_p = tf.placeholder(tf.float32)
+        self.img_p = tf.placeholder(tf.float32, shape=data.shape)
+
+        # Summary operations
+        self.summ_op = tf.summary.merge([tf.summary.scalar('loss', self.loss_p),
+                                         tf.summary.scalar('acc', self.acc_p)])
+        self.summ_val_op = tf.summary.merge([tf.summary.scalar('loss_val', self.loss_p),
+                                             tf.summary.scalar('acc_val', self.acc_p)])
+
+        self.img_op = tf.summary.image('trans_img', self.img_p)
 
     def set_model(self, model):
         self.model = model
         self.sess = K.get_session()
 
     def on_train_begin(self, logs=None):
         self.validation = self.params['do_validation']
-        self.trans_func = K.function([self.model.inputs], [self.model.layers[self.tl].output])
-        self.train_writer = tf.summary.FileWriter(self.log_dir + '/train')
+        self.trans_func = K.function([self.model.input], [self.model.layers[self.tl].output])
+        self.train_writer = tf.summary.FileWriter(self.log_dir)
         if self.validation:
-            self.val_writer = tf.summary.FileWriter(self.log_dir + '/val')
-
-
+            self.val_writer = tf.summary.FileWriter(self.log_dir)
+
     def on_batch_end(self, batch, logs=None):
         logs = {} if logs is None else logs
-        tf_loss = tf.summary.scalar('loss', tf.cast(logs.get('loss'), tf.float32))
-        tf_acc = tf.summary.scalar('acc', tf.cast(logs.get('acc'), tf.float32))
-        tf_summ = tf.summary.merge([tf_loss, tf_acc])
-        summary = self.sess(tf_summ)
+        summary = self.sess.run(self.summ_op, feed_dict={
+                                self.loss_p: logs.get('loss'),
+                                self.acc_p: logs.get('acc')})
         self.train_writer.add_summary(summary, global_step=self.step)
         self.step += 1
 
     def on_epoch_end(self, epoch, logs=None):
         logs = {} if logs is None else logs
         if self.validation:
-            tf_loss = tf.summary.scalar('loss', tf.cast(logs.get('val_loss'), tf.float32))
-            tf_acc = tf.summary.scalar('acc', tf.cast(logs.get('val_acc'), tf.float32))
-            tf_summ = tf.summary.merge([tf_loss, tf_acc])
-            summary = self.sess(tf_summ)
+            summary = self.sess.run(self.summ_val_op, feed_dict={
+                                    self.loss_p: logs.get('val_loss'),
+                                    self.acc_p: logs.get('val_acc') })
             self.val_writer.add_summary(summary, global_step=self.step)
-
-        imgs = self.trans_func([self.data])
-        tf_img = tf.summary.image(tf.cast(imgs, tf.float32))
-        summary = self.sess(tf_img)
+
+        imgs = self.trans_func([self.data])[0]
+        summary = self.sess.run(self.img_op, feed_dict={self.img_p: imgs})
         self.train_writer.add_summary(summary, global_step=self.step)
 
     def on_train_end(self, logs=None):
+        #self.sess.close()
         self.train_writer.close()
         if self.validation: self.val_writer.close()
 

ddtn/helper/utility.py

@@ -18,12 +18,14 @@
 
 #%%
 def gpu_support():
+    """ Checks for GPU and CUDA support """
     test1 = check_for_gpu()
     test2 = check_cuda_support()
     return (test1 and test2)
 
 #%%
 def check_for_gpu():
+    """ Check if tensorflow has detected a GPU """
     devices = device_lib.list_local_devices()
     gpu = False
     for d in devices:
@@ -32,6 +34,7 @@ def check_for_gpu():
 
 #%%
 def check_cuda_support():
+    """ Check if tensorflow was build with CUDA """
     return tf.test.is_built_with_cuda()
 
 #%%
@@ -75,8 +78,18 @@ def load_basis():
         raise ValueError('call setup_CPAB.py first')
     return basis
 
+#%%
+def debug_printer(string):
+    """ Small debug function """
+    print('\n')
+    print(70*'-')
+    print(string)
+    print(70*'-')
+    print('\n')
+
 #%%
 def get_cat():
+    """ Get cat image """
     direc = get_dir(__file__)
     return plt.imread(direc + '/../cat.jpg')
 

ddtn/mnist_classifier.py

@@ -7,12 +7,12 @@
 
 #%% Packages
 from tensorflow.python.keras.models import Sequential
-from tensorflow.python.keras.layers import Dense, Conv2D, InputLayer, MaxPool2D, Flatten
-from tensorflow.python.keras import backend as K
+from tensorflow.python.keras.layers import Dense, Conv2D, InputLayer, MaxPool2D, Flatten, Lambda
 
 from ddtn.transformers.construct_localization_net import get_loc_net
-from ddtn.helper.transformer_util import get_keras_layer
+from ddtn.transformers.transformer_util import get_keras_layer
 from ddtn.data.mnist_getter import get_mnist_distorted
+from ddtn.helper.training_logger import KerasTrainingLogger
 
 import argparse
 
@@ -27,8 +27,8 @@ def _argument_parser():
     parser.add_argument('-bs', action="store", dest="batch_size", type=int, default = 100,
                         help = '''Batch size: Default: 100''')
     parser.add_argument('-tt', action="store", dest="transformer_type", type=str, 
-                        default='affine', help = '''Transformer type to use. 
-                        Choose between: no, affine, cpab, affine_diffio, homografy
+                        default='no', help = '''Transformer type to use. 
+                        Choose between: no, affine, cpab, affine_diffeo, homografy
                         or TPS''')
     res = parser.parse_args()
     args = vars(res)
@@ -46,10 +46,13 @@ def _argument_parser():
 
 #%%
 if __name__ == '__main__':
+    # Get input arguments
     args = _argument_parser()
 
+    # Get some data
     X_train, y_train, X_test, y_test = get_mnist_distorted()
 
+    # Data format
     input_shape=(60,60,1)
 
     # Keras model
@@ -65,6 +68,8 @@ def _argument_parser():
         transformer_layer = get_keras_layer(args['transformer_type'])
         model.add(transformer_layer(localization_net=loc_net, 
                                     output_size=input_shape))
+    else:
+        model.add(Lambda(lambda x: x)) # identity layer -> same model structure
 
     # Construct feature extraction network
     model.add(Conv2D(32, (3,3), activation='relu'))
@@ -85,19 +90,22 @@ def _argument_parser():
     # Get summary of model
     print('Global model')
     model.summary()
-    if args['transformer_type'] != 'no':
-        print('Localization net')
-        loc_net.summary()
-
+
+    # Training logger
+    direc = './logs2'
+    logger = KerasTrainingLogger(data=X_test[:10],
+                                 name=args['transformer_type'],
+                                 log_dir=direc, 
+                                 trans_layer=0)
+
     # Fit model
     model.fit(X_train, y_train,
               batch_size=args['batch_size'],
               epochs=args['num_epochs'],
-              verbose=1,
-              validation_data=(X_test, y_test))    
+              validation_data=(X_test, y_test),
+              callbacks=[logger])    
+
+    # Save weights
+    model.save_weights(direc + '/' + args['transformer_type'] + '/weights')
 
-#    # Construct transformer function
-#    trans_func = K.function([model.input], [model.layers[1].output])
-#    
-#    # Feed some data
-#    new_imgs = trans_func([X_train[:10]])[0]
+

ddtn/run_many.py

@@ -0,0 +1,29 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Created on Mon May 28 14:18:21 2018
+
+@author: nsde
+"""
+
+#%%
+import os
+
+#%%
+ne = '-ne 50 '
+bs = '-bs 100 '
+lr = '-lr 1e-5 '
+
+#%%
+os.system("PYTHONPATH='/home/nsde/Documents/ddtn' python mnist_classifier.py -tt no "
+          + ne + bs + lr)
+os.system("PYTHONPATH='/home/nsde/Documents/ddtn' python mnist_classifier.py -tt affine "
+          + ne + bs + lr)
+os.system("PYTHONPATH='/home/nsde/Documents/ddtn' python mnist_classifier.py -tt affinediffeo "
+          + ne + bs + lr)
+os.system("PYTHONPATH='/home/nsde/Documents/ddtn' python mnist_classifier.py -tt homografy "
+          + ne + bs + lr)
+os.system("PYTHONPATH='/home/nsde/Documents/ddtn' python mnist_classifier.py -tt TPS "
+          + ne + bs + lr)
+os.system("PYTHONPATH='/home/nsde/Documents/ddtn' python mnist_classifier.py -tt CPAB "
+          + ne + bs + lr)