Tensorflow权重初始化

Question

关于tensorflow网站上的MNIST tutorial，我运行了一个实验（gist）以查看不同权重初始化对学习的影响。我注意到，与我在流行的[Xavier, Glorot 2010] paper中读到的内容相比，无论初始化权重如何，学习都很好。

不同的曲线表示不同的值w用于初始化卷积和完全连接层的权重。请注意，w的所有值都可以正常工作，即使0.3和1.0以较低性能结束，某些值训练得更快 - 特别是0.03和0.1是最快的。尽管如此，该图显示了一个相当大的范围w这是有效的，表明“鲁棒性”w.r.t.重量初始化。

def weight_variable(shape, w=0.1): 
    initial = tf.truncated_normal(shape, stddev=w) 
    return tf.Variable(initial) 

def bias_variable(shape, w=0.1): 
    initial = tf.constant(w, shape=shape) 
    return tf.Variable(initial) 

问题：为什么这个网络不消失或爆炸梯度问题的困扰？

我建议你阅读实现细节的要点，但这里有代码供参考。在我的nvidia 960m上花了大约一个小时，尽管我想它也可以在合理的时间内在CPU上运行。

import time 
from tensorflow.examples.tutorials.mnist import input_data 
import tensorflow as tf 
from tensorflow.python.client import device_lib 

import numpy 
import matplotlib.pyplot as pyplot 

mnist = input_data.read_data_sets('MNIST_data', one_hot=True) 

# Weight initialization 

def weight_variable(shape, w=0.1): 
    initial = tf.truncated_normal(shape, stddev=w) 
    return tf.Variable(initial) 

def bias_variable(shape, w=0.1): 
    initial = tf.constant(w, shape=shape) 
    return tf.Variable(initial) 


# Network architecture 

def conv2d(x, W): 
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME') 

def max_pool_2x2(x): 
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], 
        strides=[1, 2, 2, 1], padding='SAME') 

def build_network_for_weight_initialization(w): 
    """ Builds a CNN for the MNIST-problem: 
    - 32 5x5 kernels convolutional layer with bias and ReLU activations 
    - 2x2 maxpooling 
    - 64 5x5 kernels convolutional layer with bias and ReLU activations 
    - 2x2 maxpooling 
    - Fully connected layer with 1024 nodes + bias and ReLU activations 
    - dropout 
    - Fully connected softmax layer for classification (of 10 classes) 

    Returns the x, and y placeholders for the train data, the output 
    of the network and the dropbout placeholder as a tuple of 4 elements. 
    """ 
    x = tf.placeholder(tf.float32, shape=[None, 784]) 
    y_ = tf.placeholder(tf.float32, shape=[None, 10]) 

    x_image = tf.reshape(x, [-1,28,28,1]) 
    W_conv1 = weight_variable([5, 5, 1, 32], w) 
    b_conv1 = bias_variable([32], w) 

    h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1) 
    h_pool1 = max_pool_2x2(h_conv1) 
    W_conv2 = weight_variable([5, 5, 32, 64], w) 
    b_conv2 = bias_variable([64], w) 

    h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2) 
    h_pool2 = max_pool_2x2(h_conv2) 

    W_fc1 = weight_variable([7 * 7 * 64, 1024], w) 
    b_fc1 = bias_variable([1024], w) 

    h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64]) 
    h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1) 

    keep_prob = tf.placeholder(tf.float32) 
    h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob) 

    W_fc2 = weight_variable([1024, 10], w) 
    b_fc2 = bias_variable([10], w) 

    y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2 

    return (x, y_, y_conv, keep_prob) 


# Experiment 

def evaluate_for_weight_init(w): 
    """ Returns an accuracy learning curve for a network trained on 
    10000 batches of 50 samples. The learning curve has one item 
    every 100 batches.""" 
    with tf.Session() as sess: 
    x, y_, y_conv, keep_prob = build_network_for_weight_initialization(w) 
    cross_entropy = tf.reduce_mean(
     tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv)) 
    train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy) 
    correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1)) 
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 
    sess.run(tf.global_variables_initializer()) 
    lr = [] 
    for _ in range(100): 
     for i in range(100): 
      batch = mnist.train.next_batch(50) 
      train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5}) 
     assert mnist.test.images.shape[0] == 10000 
     # This way the accuracy-evaluation fits in my 2GB laptop GPU. 
     a = sum(
      accuracy.eval(feed_dict={ 
       x: mnist.test.images[2000*i:2000*(i+1)], 
       y_: mnist.test.labels[2000*i:2000*(i+1)], 
       keep_prob: 1.0}) 
      for i in range(5))/5 
     lr.append(a) 
    return lr 


ws = [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0] 
accuracies = [ 
    [evaluate_for_weight_init(w) for w in ws] 
    for _ in range(3) 
] 


# Plotting results 

pyplot.plot(numpy.array(accuracies).mean(0).T) 
pyplot.ylim(0.9, 1) 
pyplot.xlim(0,140) 
pyplot.xlabel('batch (x 100)') 
pyplot.ylabel('test accuracy') 
pyplot.legend(ws) 

Answer 1

物流功能更容易消失的梯度，因为他们的梯度都是< 1，所以他们的越多，你反向传播，较小的梯度变（和相当快）中繁殖，而RELU有积极部分的梯度为1，所以它没有这个问题。

此外，您的网络还不足以承受这一点。

Answer 2

权重初始化策略在改进模型方面可能是一个重要且经常被忽视的步骤，而且现在这是Google的最高结果，我认为它可能需要更详细的答案。

通常，每层的激活函数梯度，传入/传出连接数（fan_in/fan_out）和权重方差的总积应该等于1。这样，当您通过网络反向传播时，输入和输出梯度之间的差异将保持一致，并且您不会遇到爆炸或渐变消失。尽管ReLU更能抵抗爆炸/消失梯度，但您仍可能遇到问题。

OP使用的tf.truncated_normal进行随机初始化，鼓励权重以不同方式更新，但而不是是否考虑了上述优化策略。在较小的网络上，这可能不成问题，但是如果您想要更深层次的网络或更快的培训时间，那么您最好尝试基于最近的研究进行权重初始化策略。

对于RELU前面的权重函数，你可以使用默认设置：

tf.contrib.layers。variance_scaling_initializer

的正切/乙状结肠激活层 “泽维尔” 可能更合适：在这两个功能和相关论文

tf.contrib.layers.xavier_initializer

更多细节被发现在： https://www.tensorflow.org/versions/r0.12/api_docs/python/contrib.layers/initializers

超越重量初始化策略，进一步优化c应探索批量标准化：https://www.tensorflow.org/api_docs/python/tf/nn/batch_normalization