Keras LSTM预测1个时间步长在一个时间

Question

编辑补充： 我发现了什么，我认为是一个可行的解决方案：https://bleyddyn.github.io/posts/2017/10/keras-lstm/

---

我试图用一个转换次数/ LSTM网络控制机器人。我想我已经设置了一切，所以我可以开始在重播内存中对批量数据进行培训，但我无法弄清楚如何实际使用它来控制机器人。简体测试代码如下。

import numpy as np 

from keras.models import Sequential 
from keras.layers import Dense, Flatten, Input 
from keras.layers import Convolution2D 
from keras.layers.recurrent import LSTM 
from keras.layers.wrappers import TimeDistributed 
from keras.utils import to_categorical 

def make_model(num_actions, timesteps, input_dim, l2_reg=0.005): 
    input_shape=(timesteps,) + input_dim 
    model = Sequential() 
    model.add(TimeDistributed(Convolution2D(8, (3, 3), strides=(2,2), activation='relu'), input_shape=input_shape)) 
    model.add(TimeDistributed(Convolution2D(16, (3, 3), strides=(2,2), activation='relu',))) 
    model.add(TimeDistributed(Convolution2D(32, (3, 3), strides=(2,2), activation='relu',))) 
    model.add(TimeDistributed(Flatten())) 
    model.add(LSTM(512, return_sequences=True, activation='relu', unroll=True)) 
    model.add(Dense(num_actions, activation='softmax',)) 
    model.compile(loss='categorical_crossentropy', optimizer='adam') 
    return model 

batch_size = 16 
timesteps = 10 
num_actions = 6 
model = make_model(num_actions, timesteps, (84,84,3)) 
model.summary() 

# Fake training batch. Would be pulled from a replay memory 
batch = np.random.uniform(low=0, high=255, size=(batch_size,timesteps,84,84,3)) 
y = np.random.randint(0, high=5, size=(160)) 
y = to_categorical(y, num_classes=num_actions) 
y = y.reshape(batch_size, timesteps, num_actions) 
# stateful should be false here 
pred = model.train_on_batch(batch, y) 

# move trained network to robot 

# This works, but it isn't practical to not get outputs (actions) until after 10 timesteps and I don't think the LSTM internal state would be correct if I tried a rolling queue of input images. 
batch = np.random.uniform(low=0, high=255, size=(1,timesteps,84,84,3)) 
pred = model.predict(batch, batch_size=1) 

# This is what I would need to do on my robot, with the LSTM keeping state between calls to predict 
max_time = 10 # or 100000, or forever, etc. 
for i in range(max_time) : 
    image = np.random.uniform(low=0, high=255, size=(1,1,84,84,3)) # pull one image from camera 
    # stateful should be true here 
    pred = model.predict(image, batch_size=1) 
    # take action based on pred 

我上的错误“model.predict（图片...”行是：

ValueError: Error when checking : expected time_distributed_1_input to have shape (None, 10, 84, 84, 3) but got array with shape (1, 1, 84, 84, 3)

这是可以理解的，但我不能找到办法解决它 我。不知道Keras不够好，甚至不知道如果我正确使用TimeDistributed层。

那么，这甚至可能在Keras？如果是这样，怎么样？

如果没有，是否有可能在TF或Py火炬？

感谢您的任何建议！

编辑添加运行代码，虽然它不一定是正确的。仍然需要在OpenAI健身房任务中进行测试。

import numpy as np 

from keras.models import Sequential 
from keras.layers import Dense, Flatten, Input 
from keras.layers import Convolution2D 
from keras.layers.recurrent import LSTM 
from keras.layers.wrappers import TimeDistributed 
from keras.utils import to_categorical 

def make_model(num_actions, timesteps, input_dim, l2_reg=0.005): 
    input_shape=(1,None) + input_dim 
    model = Sequential() 
    model.add(TimeDistributed(Convolution2D(8, (3, 3), strides=(2,2), activation='relu'), batch_input_shape=input_shape)) 
    model.add(TimeDistributed(Convolution2D(16, (3, 3), strides=(2,2), activation='relu',))) 
    model.add(TimeDistributed(Convolution2D(32, (3, 3), strides=(2,2), activation='relu',))) 
    model.add(TimeDistributed(Flatten())) 
    model.add(LSTM(512, return_sequences=True, activation='relu', stateful=True)) 
    model.add(Dense(num_actions, activation='softmax',)) 
    model.compile(loss='categorical_crossentropy', optimizer='adam') 
    return model 

batch_size = 16 
timesteps = 10 
num_actions = 6 
model = make_model(num_actions, 1, (84,84,3)) 
model.summary() 

# Fake training batch. Would be pulled from a replay memory 
batch = np.random.uniform(low=0, high=255, size=(batch_size,timesteps,84,84,3)) 
y = np.random.randint(0, high=5, size=(160)) 
y = to_categorical(y, num_classes=num_actions) 
y = y.reshape(batch_size, timesteps, num_actions) 

# Need to find a way to prevent the optimizer from updating every b, but accumulate updates over an entire batch (batch_size). 
for b in range(batch_size): 
    pred = model.train_on_batch(np.reshape(batch[b,:], (1,timesteps,84,84,3)), np.reshape(y[b,:], (1,timesteps,num_actions))) 
    #for t in range(timesteps): 
    # pred = model.train_on_batch(np.reshape(batch[b,t,:], (1,1,84,84,3)), np.reshape(y[b,t,:], (1,1,num_actions))) 
    model.reset_states() # Don't carry internal state between batches 

# move trained network to robot 

# This works, but it isn't practical to not get outputs (actions) until after 10 timesteps 
#batch = np.random.uniform(low=0, high=255, size=(1,timesteps,84,84,3)) 
#pred = model.predict(batch, batch_size=1) 

# This is what I would need to do on my robot, with the LSTM keeping state between calls to predict 
max_time = 10 # or 100000, or forever, etc. 
for i in range(max_time) : 
    image = np.random.uniform(low=0, high=255, size=(1,1,84,84,3)) # pull one image from camera 
    # stateful should be true here 
    pred = model.predict(image, batch_size=1) 
    # take action based on pred 
    print(pred) 

Answer 1

您需要的第一件事就是了解您的数据。

这5个维度是否意味着什么？

我会试着猜测：

- 1 learning example 
- 1 time step (this is added by TimeDistributed, normal 2D convolutions don't take this) 
- 84 image side 
- 84 another image side 
- 3 channels (RGB) 

的TimeDistributed的目的是为了增加额外的timesteps尺寸，这样你就可以模拟在不应该与序列工作层的序列。

你的错误消息告诉你：

• 你input_shape参数是(None, 10, 84, 84, 3)，其中没有一个是批量大小（的样本/实例号）。
• 您的输入data，即代码中的batch是(1, 1, 84, 84, 3)。

存在不匹配情况，您应该使用包含10个时间步骤的批次（由您的input_shape定义）。 stateful=False型号可以批量打包10张图像并进行训练。

但后来，在stateful=True的情况下，您将需要该input_shape只是一个步骤。 （您可以创建一个新模型，仅用于预测并将所有权重从训练模型预测并复制到预测模型中，也可以尝试在该时间步长维度中使用None，这意味着您可以使用不同的时间步长进行训练和预测）

现在，与卷积不同，LSTM层是已经预计时间步骤。所以你应该找到一种方法来以更小的维度压缩你的数据。

该LSTM将预计(None, timeSteps, features)。时间步骤与以前相同。 10为训练，1为预测，你可以尝试在None那里去。

因此，而不是一个Flatten()内TimeDistributed，你应该简单地重塑数据，冷凝不在批量大小的尺寸或步骤：

model.add(Reshape((8,9*9*32))) #the batch size doesn't participate in this definition, and it will remain as it is. 

的9*9*32是前面的卷积及其两侧32个过滤器。 （我只是不确定双方是9，也许他们是8，你可以在当前model.summary()看到）。

最后，对于stateful=True的情况，您将不得不使用batch_shape而不是input_shape来定义模型。批次中的样品数量必须是固定数量，因为该模型会假定第二批次中的样品属于上一批样品中的新步骤。 （所有批次的样品数量需要相同）。