如何在MATLAB中加载MNIST数字和标签数据？

Question

我试图运行中的链接

https://github.com/bd622/DiscretHashing

离散哈希给出的代码是降维的方法是在近似最近邻搜索中使用。我想加载在http://yann.lecun.com/exdb/mnist/中可用的MNIST数据库上的实现。我已经从压缩的gz格式中提取文件。

问题1：

利用该解决方案来读取MNIST数据库Reading MNIST Image Database binary file in MATLAB

提供我收到以下错误：

Error using fread 
Invalid file identifier. Use fopen to generate a valid file identifier. 

Error in Reading (line 7) 
A = fread(fid, 1, 'uint32'); 

下面是代码：

clear all; 
close all; 

%//Open file 
fid = fopen('t10k-images-idx3-ubyte', 'r'); 

A = fread(fid, 1, 'uint32'); 
magicNumber = swapbytes(uint32(A)); 

%//For each image, store into an individual cell 
imageCellArray = cell(1, totalImages); 
for k = 1 : totalImages 
    %//Read in numRows*numCols pixels at a time 
    A = fread(fid, numRows*numCols, 'uint8'); 
    %//Reshape so that it becomes a matrix 
    %//We are actually reading this in column major format 
    %//so we need to transpose this at the end 
    imageCellArray{k} = reshape(uint8(A), numCols, numRows)'; 
end 

%//Close the file 
fclose(fid); 

UPDATE：问题1解决，并且修改后的代码是

clear all; 
close all; 

%//Open file 
fid = fopen('t10k-images.idx3-ubyte', 'r'); 

A = fread(fid, 1, 'uint32'); 
magicNumber = swapbytes(uint32(A)); 

%//Read in total number of images 
%//A = fread(fid, 4, 'uint8'); 
%//totalImages = sum(bitshift(A', [24 16 8 0])); 

%//OR 
A = fread(fid, 1, 'uint32'); 
totalImages = swapbytes(uint32(A)); 

%//Read in number of rows 
%//A = fread(fid, 4, 'uint8'); 
%//numRows = sum(bitshift(A', [24 16 8 0])); 

%//OR 
A = fread(fid, 1, 'uint32'); 
numRows = swapbytes(uint32(A)); 

%//Read in number of columns 
%//A = fread(fid, 4, 'uint8'); 
%//numCols = sum(bitshift(A', [24 16 8 0])); 

%// OR 
A = fread(fid, 1, 'uint32'); 
numCols = swapbytes(uint32(A)); 

for k = 1 : totalImages 
    %//Read in numRows*numCols pixels at a time 
    A = fread(fid, numRows*numCols, 'uint8'); 
    %//Reshape so that it becomes a matrix 
    %//We are actually reading this in column major format 
    %//so we need to transpose this at the end 
    imageCellArray{k} = reshape(uint8(A), numCols, numRows)'; 
end 

%//Close the file 
fclose(fid); 

问题2：

我无法理解如何将4个文件的MNIST应用中的代码。代码包含变量

traindata = double(traindata); 
testdata = double(testdata); 

如何准备MNIST数据库以便我可以应用于实施？

UPDATE：我实现了解决方案，但我不断收到此错误

Error using fread 
Invalid file identifier. Use fopen to generate a valid file identifier. 

Error in mnist_parse (line 11) 
A = fread(fid1, 1, 'uint32'); 

这些文件

demo.m％，这是调用该函数在MNIST数据读取主文件

clear all 
clc 
[Trainimages, Trainlabels] = mnist_parse('C:\Users\Desktop\MNIST\train-images-idx3-ubyte', 'C:\Users\Desktop\MNIST\train-labels-idx1-ubyte'); 

[Testimages, Testlabels] = mnist_parse('t10k-images-idx3-ubyte', 't10k-labels-idx1-ubyte'); 

k=5; 
digit = images(:,:,k); 
lbl = label(k); 

---

function [images, labels] = mnist_parse(path_to_digits, path_to_labels) 

% Open files 
fid1 = fopen(path_to_digits, 'r'); 

% The labels file 
fid2 = fopen(path_to_labels, 'r'); 

% Read in magic numbers for both files 
A = fread(fid1, 1, 'uint32'); 
magicNumber1 = swapbytes(uint32(A)); % Should be 2051 
fprintf('Magic Number - Images: %d\n', magicNumber1); 

A = fread(fid2, 1, 'uint32'); 
magicNumber2 = swapbytes(uint32(A)); % Should be 2049 
fprintf('Magic Number - Labels: %d\n', magicNumber2); 

% Read in total number of images 
% Ensure that this number matches with the labels file 
A = fread(fid1, 1, 'uint32'); 
totalImages = swapbytes(uint32(A)); 
A = fread(fid2, 1, 'uint32'); 
if totalImages ~= swapbytes(uint32(A)) 
    error('Total number of images read from images and labels files are not the same'); 
end 
fprintf('Total number of images: %d\n', totalImages); 

% Read in number of rows 
A = fread(fid1, 1, 'uint32'); 
numRows = swapbytes(uint32(A)); 

% Read in number of columns 
A = fread(fid1, 1, 'uint32'); 
numCols = swapbytes(uint32(A)); 

fprintf('Dimensions of each digit: %d x %d\n', numRows, numCols); 

% For each image, store into an individual slice 
images = zeros(numRows, numCols, totalImages, 'uint8'); 
for k = 1 : totalImages 
    % Read in numRows*numCols pixels at a time 
    A = fread(fid1, numRows*numCols, 'uint8'); 

    % Reshape so that it becomes a matrix 
    % We are actually reading this in column major format 
    % so we need to transpose this at the end 
    images(:,:,k) = reshape(uint8(A), numCols, numRows).'; 
end 

% Read in the labels 
labels = fread(fid2, totalImages, 'uint8'); 

% Close the files 
fclose(fid1); 
fclose(fid2); 

end 

Answer 1

我是方法＃1的原始作者，您提到过。读取培训数据和测试标签的过程非常简单。就读取图像而言，上面显示的代码完美地读取文件，并且处于单元格阵列格式。但是，您在中缺少读取图像数量，行数和列数为的文件。请注意，该文件的MNIST格式采用以下方式。左边一列是你相对于参考的开始字节偏移量：

[offset] [type]   [value]   [description] 
0000  32 bit integer 0x00000803(2051) magic number 
0004  32 bit integer 60000   number of images 
0008  32 bit integer 28    number of rows 
0012  32 bit integer 28    number of columns 
0016  unsigned byte ??    pixel 
0017  unsigned byte ??    pixel 
........ 
xxxx  unsigned byte ??    pixel 

前四个字节是一个神奇的数字：2051，以确保您的文件中正确读取。接下来的四个字节表示图像的总数，接下来的四个字节是行，最后四个字节是列。应该有2800×28×28的大小的图像。在此之后，像素以行主格式交织，因此您必须循环显示一系列28 x 28像素并存储它们。在这种情况下，我将它们存储在单元格数组中，并且此单元格数组中的每个元素都是一个数字。测试数据的格式也相同，但是有10000个图像。

至于实际的标签，这是大致相同的格式，但也有一些细微的差别：

[offset] [type]   [value]   [description] 
0000  32 bit integer 0x00000801(2049) magic number (MSB first) 
0004  32 bit integer 60000   number of items 
0008  unsigned byte ??    label 
0009  unsigned byte ??    label 
........ 
xxxx  unsigned byte ??    label 

前四个字节是一个神奇的数字：2049，然后是第二组的四个字节告诉您如何那里有很多标签，最后数据集中每个对应的数字都有1个字节。测试数据也是相同的格式，但有10000个标签。因此，一旦读入标签集中的必要数据，您只需要一个fread调用，并确保数据是无符号的8位整数，以读取其余标签。

现在你为什么要使用swapbytes的原因是因为MATLAB将在小端格式的数据读取，这意味着从一组字节至少显著字节在第一个被读取。完成后，您可以使用swapbytes重新排列此顺序。

因此，我已经修改了这段代码，以便它是一个实际的函数，它接受一组两个字符串：数字图像文件的完整路径和数字的完整路径。我还更改了代码，以便图像是3D数字矩阵，而不是单元阵列，以便进行更快的处理。请注意，当您开始读取实际图像数据时，每个像素都是无符号的8位整数，因此不需要进行任何字节交换。

function [images, labels] = mnist_parse(path_to_digits, path_to_labels) 

% Open files 
fid1 = fopen(path_to_digits, 'r'); 

% The labels file 
fid2 = fopen(path_to_labels, 'r'); 

% Read in magic numbers for both files 
A = fread(fid1, 1, 'uint32'); 
magicNumber1 = swapbytes(uint32(A)); % Should be 2051 
fprintf('Magic Number - Images: %d\n', magicNumber1); 

A = fread(fid2, 1, 'uint32'); 
magicNumber2 = swapbytes(uint32(A)); % Should be 2049 
fprintf('Magic Number - Labels: %d\n', magicNumber2); 

% Read in total number of images 
% Ensure that this number matches with the labels file 
A = fread(fid1, 1, 'uint32'); 
totalImages = swapbytes(uint32(A)); 
A = fread(fid2, 1, 'uint32'); 
if totalImages ~= swapbytes(uint32(A)) 
    error('Total number of images read from images and labels files are not the same'); 
end 
fprintf('Total number of images: %d\n', totalImages); 

% Read in number of rows 
A = fread(fid1, 1, 'uint32'); 
numRows = swapbytes(uint32(A)); 

% Read in number of columns 
A = fread(fid1, 1, 'uint32'); 
numCols = swapbytes(uint32(A)); 

fprintf('Dimensions of each digit: %d x %d\n', numRows, numCols); 

% For each image, store into an individual slice 
images = zeros(numRows, numCols, totalImages, 'uint8'); 
for k = 1 : totalImages 
    % Read in numRows*numCols pixels at a time 
    A = fread(fid1, numRows*numCols, 'uint8'); 

    % Reshape so that it becomes a matrix 
    % We are actually reading this in column major format 
    % so we need to transpose this at the end 
    images(:,:,k) = reshape(uint8(A), numCols, numRows).'; 
end 

% Read in the labels 
labels = fread(fid2, totalImages, 'uint8'); 

% Close the files 
fclose(fid1); 
fclose(fid2); 

end 

调用这个函数，只需指定两个图像文件和标签文件的路径：这多个字节读一个fread电话时，只要求。假设你正在运行在该文件所在的同一目录下这个文件，你会做的训练图像执行下列操作之一：

[images, labels] = mnist_parse('train-images-idx3-ubyte', 'train-labels-idx1-ubyte'); 

而且，你会做以下的测试图像：

[images, labels] = mnist_parse('t10k-images-idx3-ubyte', 't10k-labels-idx1-ubyte'); 

要访问k个数字，你会简单地做：

digit = images(:,:,k); 

为k个位数对应的标签是：

lbl = label(k); 

---

终于得到这个数据到一个格式，是，我已经看到在Github上的代码可以接受的，他们认为该行对应于训练样本和列对应的功能。如果你想要这种格式，只需重塑数据，以便图像像素分布在列上。

因此，只是这样做：

[trainingdata, traingnd] = mnist_parse('train-images-idx3-ubyte', 'train-labels-idx1-ubyte'); 
trainingdata = double(reshape(trainingdata, size(trainingdata,1)*size(trainingdata,2), []).'); 
traingnd = double(traingnd); 

[testdata, testgnd] = mnist_parse('t10k-images-idx3-ubyte', 't10k-labels-idx1-ubyte'); 
testdata = double(reshape(testdata, size(testdata,1)*size(testdata_data,2), []).'); 
testgnd = double(testgnd); 

以上使用相同的变量在脚本，所以你应该能够在堵塞这一点，它应该工作。第二行重塑矩阵，以便每个数字位于列中，但我们需要对此进行转置，以便每个数字位于列中。我们还需要将其转换为double，因为这是Github代码的功能。相同的逻辑适用于测试数据。另外请注意，我已明确将培训和测试标签投放到double，以确保您决定使用此数据的任何算法具有最大的兼容性。

---

快乐的数字黑客！