CSC 578 HW 5: Backprop Hyperparameters

• This hyperparameter argument specifies the cost function.
• Options are 'QuadraticCost', 'CrossEntropy' or 'LogLikelihood'.
• Each one must be implemented as a Python class. The scheme for class function is explained later in this document (see Class Function notes below).

• The class should have two static functions: fn() executes the definition of the function (to compute the cost in during evaluation), and derivative() executes the function's derivative (to compute the error during learning).

  1. QuadraticCost is fully implemented already in the starter code, as shown below (and you do not need to modify it).
  2. For CrossEntropy and LogLikelihood, partial/skeleton code is written. You replace the line 'pass' and write your code.

  class QuadraticCost(object):
      @staticmethod
      def fn(a, y):
          """Return the cost associated with an output ``a`` and desired output ``y``."""
          return 0.5*np.linalg.norm(y-a)**2
  
      @staticmethod
      def derivative(a, y):
          """Return the first derivative of the function."""
          return -(y-a)
  
  class CrossEntropyCost(object):
      @staticmethod
      def fn(a, y):
          """Return the cost associated with an output ``a`` and desired output
          ``y``.  Note that np.nan_to_num is used to ensure numerical
          stability.  In particular, if both ``a`` and ``y`` have a 1.0
          in the same slot, then the expression (1-y)*np.log(1-a)
          returns nan.  The np.nan_to_num ensures that that is converted
          to the correct value (0.0)."""
          return np.sum(np.nan_to_num(-y*np.log(a)-(1-y)*np.log(1-a)))
  
      @staticmethod
      def derivative(a, y):
          """Return the first derivative of the function."""
          ###
          ### FILL IN HERE
          ###
          pass

• All cost functions receive a (activation) and y (target output), which are from one data instance and represented by column vectors.
• fn() returns a scalar, while derivative() returns a column vector (containing the cost derivative for each node in the output layer; no multiplication by the derivative of the activation function).
• NOTES on LogLikelihood:
  1. You have written the function (fn()) in HW#3.
  2. For the derivative formula, you can look at Slide 14 in Lecture note #4 (Optimizations).  Basically you return a column vector of zeros except for the node for which the target output is 1 -- its value should be -1/activation of the node.

• This parameter specifies the activation function for nodes on all hidden layers (and EXCLUDING the output layer).
• Parameter options are 'Sigmoid', 'Tanh', 'ReLU', 'LeakyReLU' and 'Softmax'.

• Each one must be implemented as a Python class (see Class Function notes below). The class should have two functions: a static method fn() executes the definition of the function (to compute the node activation value), and a class method derivative() executes the function's derivative (to compute the error during learning).

  1. Sigmoid is fully implemented already in the starter code, as shown below (and you do not need to modify it).
  2. Softmax is partially written. You must add fn()..
  3. For Tanh, ReLU and LeakyReLU, skeleton code is written. You replace the line 'pass' and write your code.
  4. For LeakyReLU, look at the formula (of the function) on Slide 8 in the lecture note #4 (Optimizations).   Use alpha = 0.3 (which you can hard-code it if you like).  Note that in the slide, the variable z is meant to be a scalar.  But in your code, z (in fn(z) also derivative())) is a vector/array.  One reference is TensorFlow tf.keras.layers.LeakyReLU.  Figure out the derivative of the function by yourself too.

  class Sigmoid(object):    
      @staticmethod
      def fn(z):
          """The sigmoid function."""
          return 1.0/(1.0+np.exp(-z))
  
      @classmethod
      def derivative(cls,z):
          """Derivative of the sigmoid function."""
          return cls.fn(z)*(1-cls.fn(z))
  
  class Softmax(object):
      @staticmethod
      # Parameter z is an array of shape (len(z), 1).
      def fn(z):
          """The softmax of vector z."""
          ###
          ### FILL IN HERE
          ###
          pass
  
      @classmethod
      def derivative(cls,z):
          """Derivative of the softmax.  
          IMPORTANT: The derivative is an N*N matrix.
          """
          a = cls.fn(z) # obtain the softmax vector
          return np.diagflat(a) - np.dot(a, a.T)
  

• This parameter specifies the activation function for nodes on the output layer.

• Parameter options are 'Sigmoid' and 'Softmax'.  But for simplicity, you don't have to check if other functions are passed in this assignment.

• This parameter specifies the regularization method.
• Parameter options are 'L2', 'L1' and None.
• If the parameter value is None, you do NOT apply regularization at all.
• The selected method is applied to all hidden layers and the output layer.

• You can implement them in any way you like, for example as function classes or inline if-else conditionals. For definitions/formulas and explanation, see Slides 24, 25, 28, 29 Lecture note #4 (Optimizations).

  IMPORTANT NOTE: The start-up code has L2 hard-coded in (unchanged from the original NNDL code for this part). You make necessary changes to the code by yourself to incorporate the two methods.

• The regularization is relevant at two places in the backprop algorithm:

  1. During training, when weights are adjusted at the end of a mini-bath – the function update_mini_batch().
  2. During evaluation, when the cost is computed – the function total_cost().

  NOTE: Both L2 and L1 functions utilize the hyperparameter lmbda, which was passed in when the network is created initially.  The parameter is stored in self.lmbda..

• This parameter specifies the percentage of dropout.
• The value is between 0 and 1. For example, 0.4 means 40% of the nodes on a layer are dropped (or made inaccessible).
• Dropout consists in randomly setting a fraction rate of units in a layer to 0 at each update during training time, which helps prevent overfitting.
• Assume the same dropout percentage is applied to all hidden layers. Dropout should NOT be applied to input or output layer.
• You can implement the parameter in any way you like. You make necessary changes to the code by yourself, wherever needed.
• Many dropout schemes have been proposed in neural networks. For this assignment, you implement the following scheme.
  1. Dropout is applied during the training phase only. No dropout is applied during the testing/evaluation phrase.
  2. Use the same dropout nodes/masks during one mini-batch. That means you have to store which nodes were used/dropped somewhere else. Think about it and implement in your way.

  3. Scale the output values of the layer. This scheme is explained at this site. In particular, the following code is very useful. The first line is generating a dropout mask (u1) and the second line is applying the mask to the activation of a hidden layer (h1) during the forward propagation phase in the backprop function.

     # Dropout training, notice the scaling of 1/p
     u1 = np.random.binomial(1, p, size=h1.shape) / p
     h1 *= u1
     

     Then during the backward propagation phase, you apply the mask to the delta of a hidden layer (dh1). This is necessary because, since a dropout mask is applied as an additional multiplier (function) after the activation function, it essentially became a constant coefficient of the activation function (i.e., c*a(z)), and shows up in the derivative of that function – Let \(f(z) = c*a(z)\), then \(f'(z) = c*a'(z)\).

     dh1 *= u1 
     

     More IMPORTANT NOTES:

     • The variable p above is the ratio of nodes to RETAIN, not to remove. So essentially, p = 1 - self.dropoutpercent.
     • To select nodes to retain/filter, you MUST use random.sample() method (in Python's standard library, not Numpy), NOT np.random.binomial(), in order to guarantee the exact p proportion of selection.
     • During forward propagation, dropout should be applied to/after activation, NOT to the z/weighted sum (e.g. sigma(0) = 0.5, which is not right or convenient here).
     • During backward propagation, dropout should be applied to the delta (the error at a given hidden layer).

Static or class functions in a class can be called through the class name. When a class name is bound to an instance variable, you can invoke a specific static/class function in the class by prefixing the instance variable. For example, here is a line in the function backprop():

a_prime = (self.act_output).derivative(zs[-1])

So whatever self.act_output is bound to, the function derivative() defined inside the class is being invoked.