当改变输入数据时，Tensorflow模型没有训练

Question

我正在尝试使用张量流来训练使用交通标志图像的神经网络（LeNet）。我想检查预处理技术对nn性能的影响。因此，我预处理了图像，并将结果（训练图像，验证图像，testimages，最终testimages）作为一个字典存储。

然后我试图遍历该字典，然后使用tensorflow的训练和验证操作如下

import tensorflow as tf 
from sklearn.utils import shuffle 


output_data = [] 
EPOCHS = 5 
BATCH_SIZE = 128 
rate = 0.0005 

for key in finalInputdata.keys(): 
    for procTypes in range(0,(len(finalInputdata[key]))): 
     if np.shape(finalInputdata[key][procTypes][0]) !=(): 
      X_train = finalInputdata[key][procTypes][0] 
      X_valid = finalInputdata[key][procTypes][1] 
      X_test = finalInputdata[key][procTypes][2] 
      X_finaltest = finalInputdata[key][procTypes][3] 


      x = tf.placeholder(tf.float32, (None, 32, 32,np.shape(X_train)[-1])) 
      y = tf.placeholder(tf.int32, (None)) 
      one_hot_y = tf.one_hot(y,43) 

      # Tensor Operations 
      logits = LeNet(x,np.shape(X_train)[-1]) 

      cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits,one_hot_y) 
      softmax_probability = tf.nn.softmax(logits) 

      loss_operation = tf.reduce_mean(cross_entropy) 
      optimizer = tf.train.AdamOptimizer(learning_rate=rate) 
      training_operation = optimizer.minimize(loss_operation) 
      correct_prediction = tf.equal(tf.argmax(logits,1), tf.argmax(one_hot_y,1)) 
      accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 


       # Pipeline for training and evaluation 

      sess = tf.InteractiveSession() 

      sess.run(tf.global_variables_initializer()) 
      num_examples = len(X_train) 

      print("Training on %s images processed as %s" %(key,dict_fornames['proctypes'][procTypes])) 
      print() 
      for i in range(EPOCHS): 
       X_train, y_train = shuffle(X_train, y_train) 
       for offset in range(0, num_examples, BATCH_SIZE): 
        end = offset + BATCH_SIZE 
        batch_x, batch_y = X_train[offset:end], y_train[offset:end] 
        sess.run(training_operation, feed_dict = {x: batch_x, y: batch_y}) 

       training_accuracy = evaluate(X_train,y_train) 

       validation_accuracy = evaluate(X_valid, y_valid) 

       testing_accuracy = evaluate(X_test, y_test) 

       final_accuracy = evaluate(X_finaltest, y_finalTest) 

       print("EPOCH {} ...".format(i+1)) 
       print("Training Accuracy = {:.3f}".format(training_accuracy)) 
       print("Validation Accuracy = {:.3f}".format(validation_accuracy)) 
       print() 
       output_data.append({'EPOCHS':EPOCHS, 'LearningRate':rate, 'ImageType': 'RGB',\ 
            'PreprocType': dict_fornames['proctypes'][0],\ 
            'TrainingAccuracy':training_accuracy, 'ValidationAccuracy':validation_accuracy, \ 
            'TestingAccuracy': testing_accuracy}) 


      sess.close() 

的评价功能如下

def evaluate(X_data, y_data): 
    num_examples = len(X_data) 
    total_accuracy = 0 
    sess = tf.get_default_session() 
    for offset in range(0,num_examples, BATCH_SIZE): 
     batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE] 
     accuracy = sess.run(accuracy_operation, feed_dict = {x:batch_x, y:batch_y}) 
     total_accuracy += (accuracy * len(batch_x)) 
    return total_accuracy/num_examples 

一旦我执行程序，它对数据集的第一次迭代有效，但从第二次迭代开始，网络不会训练并继续为所有其他迭代执行此操作。

Training on RGB images processed as Original 

EPOCH 1 ... 
Training Accuracy = 0.525 
Validation Accuracy = 0.474 

EPOCH 2 ... 
Training Accuracy = 0.763 
Validation Accuracy = 0.682 

EPOCH 3 ... 
Training Accuracy = 0.844 
Validation Accuracy = 0.723 

EPOCH 4 ... 
Training Accuracy = 0.888 
Validation Accuracy = 0.779 

EPOCH 5 ... 
Training Accuracy = 0.913 
Validation Accuracy = 0.795 

Training on RGB images processed as Mean Subtracted Data 

EPOCH 1 ... 
Training Accuracy = 0.056 
Validation Accuracy = 0.057 

EPOCH 2 ... 
Training Accuracy = 0.057 
Validation Accuracy = 0.057 

EPOCH 3 ... 
Training Accuracy = 0.057 
Validation Accuracy = 0.056 

EPOCH 4 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.056 

EPOCH 5 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.058 

Training on RGB images processed as Normalized Data 

EPOCH 1 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.054 

EPOCH 2 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.054 

EPOCH 3 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.054 

EPOCH 4 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.054 

EPOCH 5 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.054 

但是，如果我重新启动内核并使用任何数据类型（任何迭代），它都可以工作。我发现我必须清除图表或者为多个数据类型运行多个会话，但是我不清楚如何实现这一点。我尝试使用tf.reset_default_graph()，但似乎没有任何效果。有人能指出我正确的方向吗？

感谢

Answer 1

你可能想使之与被归为零均值，方差其送入网络之前的数据，例如尝试通过缩放图像到-1..1范围;那说，0..1范围大多也听起来理智。取决于网络中使用的激活，数值范围可能会造成所有差异：例如，对于低于零的输入，ReLUs将消失，当值低于-4或高于+4并且tanh激活错过半数时，S形开始饱和它们的值范围如果没有任何值低于0 - 如果值范围太大，渐变也可能爆炸，从而无法完全训练。从this paper，作者似乎减去（批）图像的平均值，而不是数值范围的平均值。

你可以尝试使用小学习速率以及（虽然就个人而言，我通常开始试验周围0.0001亚当）。

至于你的多个会话部分的问题：它是目前在你的代码中实现的方式，你基本上是混乱的默认图形。通过调用

for key in finalInputdata.keys(): 
    for procTypes in range(0,(len(finalInputdata[key]))): 
     if np.shape(finalInputdata[key][procTypes][0]) !=(): 

    # ... 

    x = tf.placeholder(tf.float32, (None, 32, 32,np.shape(X_train)[-1])) 
    y = tf.placeholder(tf.int32, (None)) 
    one_hot_y = tf.one_hot(y,43) 

    # Tensor Operations 
    logits = LeNet(x,np.shape(X_train)[-1]) 

    # ... etc ... 

要创建len(finalInputdata) * NLeNet的不同情况下，都在默认的图形。当变量在网络中被内部重用时，这可能是一个问题。

如果你想重置默认图，以尝试不同的超参数，尽量

for key in finalInputdata.keys(): 
    for procTypes in range(0,(len(finalInputdata[key]))): 

     tf.reset_default_graph() 
     # define the graph 

     sess = tf.InteractiveSession() 
     # train 

，但它可能是更好的显式地创建图形和会话像这样：

for key in finalInputdata.keys(): 
    for procTypes in range(0,(len(finalInputdata[key]))): 

     with tf.Graph().as_default() as graph: 
      # define the graph 

     with tf.Session(graph=graph) as sess: 
      # train 

您可以直接使用sess参考，而不是调用sess = tf.get_default_session()。

我还发现，在网络上迭代时，Jupyter内核和GPU启用的TensorFlow不能很好地一起玩，有时会遇到内存不足错误或彻底崩溃浏览器选项卡。