5
我在使用Java中的零拷贝将大文件从文件传输到套接字时看到一些奇怪的行为。我的环境：FileChannel zero-copy transferTo无法将字节复制到SocketChannel
- Windows 7 64位JDK 1.6.0_45和1.7.0_79。
- 的Centos 6.6 64位JDK 1.6.0_35
该程序的功能：客户端拷贝输入文件插入插座，和服务器副本插座使用零拷贝的方法输出文件：transferFrom和的transferTo。如果文件大小相对较大，并非所有字节都到达服务器，在Windows中为100Mb +，在Centos中为2GB +。客户端和服务器驻留在同一台机器上，本地主机地址用于传输数据。
行为因操作系统而异。在Windows上，客户端成功完成transferTo方法。传输的字节数等于输入文件大小。
long bytesTransferred = fileChannel.transferTo(0, inputFile.length(), socketChannel);
在另一方面的服务器，报告较低数量的接收到的字节。
long transferFromByteCount = fileChannel.transferFrom(socketChannel, 0, inputFile.length());
在Linux bytesTransferred客户端为2GB，即使输入文件的大小为4GB。两种配置都有足够的空间。
在Windows上，我能够使用以下解决方法之一传输130Mb文件：1）增加服务器上的接收缓冲区大小，2）在客户端中添加线程休眠方法。这导致我认为当所有字节被发送到套接字发送缓冲区而不是服务器时，客户端上的transferTo方法完成。无法保证这些字节是否能够传输到服务器，这会对我的使用情况造成问题。
在Linux能够使用单个transferTo调用的最大文件大小是2Gb，但是至少客户端报告发送到服务器的字节数是正确的。
我的问题：什么是最好的方式为客户确保文件到服务器的交付，跨平台？在Windows上用什么机制来模拟sendfile（）？
下面的代码：
客户端 - ZeroCopyClient.java：
import org.apache.commons.io.FileUtils;
import java.io.*;
import java.net.*;
import java.nio.channels.*;
public class ZeroCopyClient {
public static void main(String[] args) throws IOException, InterruptedException {
final File inputFile = new File(args[0]);
FileInputStream fileInputStream = new FileInputStream(inputFile);
FileChannel fileChannel = fileInputStream.getChannel();
SocketAddress socketAddress = new InetSocketAddress("localhost", 8083);
SocketChannel socketChannel = SocketChannel.open();
socketChannel.connect(socketAddress);
System.out.println("sending " + inputFile.length() + " bytes to " + socketChannel);
long startTime = System.currentTimeMillis();
long totalBytesTransferred = 0;
while (totalBytesTransferred < inputFile.length()) {
long st = System.currentTimeMillis();
long bytesTransferred = fileChannel.transferTo(totalBytesTransferred, inputFile.length()-totalBytesTransferred, socketChannel);
totalBytesTransferred += bytesTransferred;
long et = System.currentTimeMillis();
System.out.println("sent " + bytesTransferred + " out of " + inputFile.length() + " in " + (et-st) + " millis");
}
socketChannel.finishConnect();
long endTime = System.currentTimeMillis();
System.out.println("sent: totalBytesTransferred= " + totalBytesTransferred + "/" + inputFile.length() + " in " + (endTime-startTime) + " millis");
final File outputFile = new File(inputFile.getAbsolutePath() + ".out");
boolean copyEqual = FileUtils.contentEquals(inputFile, outputFile);
System.out.println("copyEqual= " + copyEqual);
if (args.length > 1) {
System.out.println("sleep: " + args[1] + " millis");
Thread.sleep(Long.parseLong(args[1]));
}
}
}
服务器 - ZeroCopyServer.java：
import java.io.*;
import java.net.*;
import java.nio.channels.*;
import org.apache.commons.io.FileUtils;
public class ZeroCopyServer {
public static void main(String[] args) throws IOException {
final File inputFile = new File(args[0]);
inputFile.delete();
final File outputFile = new File(inputFile.getAbsolutePath() + ".out");
outputFile.delete();
createTempFile(inputFile, Long.parseLong(args[1])*1024L*1024L);
System.out.println("input file length: " + inputFile.length() + " : output file.exists= " + outputFile.exists());
ServerSocketChannel serverSocketChannel = ServerSocketChannel.open();
serverSocketChannel.socket().setReceiveBufferSize(8*1024*1024);
System.out.println("server receive buffer size: " + serverSocketChannel.socket().getReceiveBufferSize());
serverSocketChannel.socket().bind(new InetSocketAddress("localhost", 8083));
System.out.println("waiting for connection");
SocketChannel socketChannel = serverSocketChannel.accept();
System.out.println("connected. client channel: " + socketChannel);
FileOutputStream fileOutputStream = new FileOutputStream(outputFile);
FileChannel fileChannel = fileOutputStream.getChannel();
long startTime = System.currentTimeMillis();
long transferFromByteCount = fileChannel.transferFrom(socketChannel, 0, inputFile.length());
long endTime = System.currentTimeMillis();
System.out.println("received: transferFromByteCount= " + transferFromByteCount + " : outputFile= " + outputFile.length() + " : inputFile= " + inputFile.length() + " bytes in " + (endTime-startTime) + " millis");
boolean copyEqual = FileUtils.contentEquals(inputFile, outputFile);
System.out.println("copyEqual= " + copyEqual);
serverSocketChannel.close();
}
private static void createTempFile(File file, long size) throws IOException{
RandomAccessFile f = new RandomAccessFile(file.getAbsolutePath(), "rw");
f.setLength(size);
f.writeDouble(Math.random());
f.close();
}
}
更新1：的Linux代码固定循环。
更新2：我正在考虑的一种可能的解决方法需要客户端 - 服务器合作。在传输结束时，服务器将接收到的数据的长度写回到客户端，客户端在阻塞模式下读取它。
服务器响应：
ByteBuffer response = ByteBuffer.allocate(8);
response.putLong(transferFromByteCount);
response.flip();
socketChannel.write(response);
serverSocketChannel.close();
客户端块与读：
ByteBuffer response = ByteBuffer.allocate(8);
socketChannel.read(response);
response.flip();
long totalBytesReceived = response.getLong();
其结果是，客户端等待字节通过发送和接收套接字缓冲区，而事实上等待将字节存储在输出文件中。无需实施带外确认，并且在文件内容可变的情况下，客户端也不需要等待第II.A https://linuxnetworkstack.files.wordpress.com/2013/03/paper.pdf中的建议。
UPDATE 3 “重写文件的相同 部分之前的时间量‘
’等待” 适当：
阿通过@EJP和@ the8472变形例掺入修复，具有长度和文件校验和验证，无需输出跟踪。请注意，计算大文件的CRC32校验和可能需要几秒钟才能完成。
客户：
import java.io.*; import java.net.*; import java.nio.*; import java.nio.channels.*; import org.apache.commons.io.FileUtils; public class ZeroCopyClient { public static void main(String[] args) throws IOException { final File inputFile = new File(args[0]); FileInputStream fileInputStream = new FileInputStream(inputFile); FileChannel fileChannel = fileInputStream.getChannel(); SocketAddress socketAddress = new InetSocketAddress("localhost", 8083); SocketChannel socketChannel = SocketChannel.open(); socketChannel.connect(socketAddress); //send input file length and CRC32 checksum to server long checksumCRC32 = FileUtils.checksumCRC32(inputFile); ByteBuffer request = ByteBuffer.allocate(16); request.putLong(inputFile.length()); request.putLong(checksumCRC32); request.flip(); socketChannel.write(request); long totalBytesTransferred = 0; while (totalBytesTransferred < inputFile.length()) { long bytesTransferred = fileChannel.transferTo(totalBytesTransferred, inputFile.length()-totalBytesTransferred, socketChannel); totalBytesTransferred += bytesTransferred; } //receive output file length and CRC32 checksum from server ByteBuffer response = ByteBuffer.allocate(16); socketChannel.read(response); response.flip(); long totalBytesReceived = response.getLong(); long outChecksumCRC32 = response.getLong(); socketChannel.finishConnect(); System.out.println("CRC32 equal= " + (checksumCRC32 == outChecksumCRC32)); } }服务器：
import java.io.*; import java.net.*; import java.nio.*; import java.nio.channels.*; import org.apache.commons.io.FileUtils; public class ZeroCopyServer { public static void main(String[] args) throws IOException { final File outputFile = new File(args[0]); ServerSocketChannel serverSocketChannel = ServerSocketChannel.open(); serverSocketChannel.socket().bind(new InetSocketAddress(8083)); SocketChannel socketChannel = serverSocketChannel.accept(); //read input file length and CRC32 checksum sent by client ByteBuffer request = ByteBuffer.allocate(16); socketChannel.read(request); request.flip(); long length = request.getLong(); long checksumCRC32 = request.getLong(); FileOutputStream fileOutputStream = new FileOutputStream(outputFile); FileChannel fileChannel = fileOutputStream.getChannel(); long totalBytesTransferFrom = 0; while (totalBytesTransferFrom < length) { long transferFromByteCount = fileChannel.transferFrom(socketChannel, totalBytesTransferFrom, length-totalBytesTransferFrom); if (transferFromByteCount <= 0){ break; } totalBytesTransferFrom += transferFromByteCount; } long outChecksumCRC32 = FileUtils.checksumCRC32(outputFile); //write output file length and CRC32 checksum back to client ByteBuffer response = ByteBuffer.allocate(16); response.putLong(totalBytesTransferFrom); response.putLong(outChecksumCRC32); response.flip(); socketChannel.write(response); serverSocketChannel.close(); System.out.println("CRC32 equal= " + (checksumCRC32 == outChecksumCRC32)); } }
'的transferTo（）'不指定在一次通话中完成整个转账。你必须循环。 – EJP
在客户端上传输的@EJP bytes在Windows上返回与inputFile长度相同的字节数，因此没有剩余字节。我认为我看到的是bytesTransferred在将字节放入发送缓冲区时返回，而不是在将它们发送到服务器时返回。 –
添加了@EJP建议的客户端循环 –