我花了相当多的时间尝试优化文件哈希算法以消除每一次可能的性能下降。Java NIO和非NIO性能
见我以前的SO线程:
Get File Hash Performance/Optimization
FileChannel ByteBuffer and Hashing Files
Determining Appropriate Buffer Size
有人recommened多次使用Java NIO
获取本机的性能提高(通过保持缓冲区的系统而不是将它们带入JVM)。但是,我的NIO代码运行速度相当慢,基准(反复散列相同的文件与每个算法,否定任何操作系统/驱动器“魔术”,可能会造成扭曲结果。
我现在有两个方法可以做同样的事情:
This one runs faster almost every time:
/**
* Gets Hash of file.
*
* @param file String path + filename of file to get hash.
* @param hashAlgo Hash algorithm to use. <br/>
* Supported algorithms are: <br/>
* MD2, MD5 <br/>
* SHA-1 <br/>
* SHA-256, SHA-384, SHA-512
* @param BUFFER Buffer size in bytes. Recommended to stay in<br/>
* multiples of 2 such as 1024, 2048, <br/>
* 4096, 8192, 16384, 32768, 65536, etc.
* @return String value of hash. (Variable length dependent on hash algorithm used)
* @throws IOException If file is invalid.
* @throws HashTypeException If no supported or valid hash algorithm was found.
*/
public String getHash(String file, String hashAlgo, int BUFFER) throws IOException, HasherException {
StringBuffer hexString = null;
try {
MessageDigest md = MessageDigest.getInstance(validateHashType(hashAlgo));
FileInputStream fis = new FileInputStream(file);
byte[] dataBytes = new byte[BUFFER];
int nread = 0;
while ((nread = fis.read(dataBytes)) != -1) {
md.update(dataBytes, 0, nread);
}
fis.close();
byte[] mdbytes = md.digest();
hexString = new StringBuffer();
for (int i = 0; i < mdbytes.length; i++) {
hexString.append(Integer.toHexString((0xFF & mdbytes[i])));
}
return hexString.toString();
} catch (NoSuchAlgorithmException | HasherException e) {
throw new HasherException("Unsuppored Hash Algorithm.", e);
}
}
My Java NIO method that runs considerably slower most of the time:
/**
* Gets Hash of file using java.nio File Channels and ByteBuffer
* <br/>for native system calls where possible. This may improve <br/>
* performance in some circumstances.
*
* @param fileStr String path + filename of file to get hash.
* @param hashAlgo Hash algorithm to use. <br/>
* Supported algorithms are: <br/>
* MD2, MD5 <br/>
* SHA-1 <br/>
* SHA-256, SHA-384, SHA-512
* @param BUFFER Buffer size in bytes. Recommended to stay in<br/>
* multiples of 2 such as 1024, 2048, <br/>
* 4096, 8192, 16384, 32768, 65536, etc.
* @return String value of hash. (Variable length dependent on hash algorithm used)
* @throws IOException If file is invalid.
* @throws HashTypeException If no supported or valid hash algorithm was found.
*/
public String getHashNIO(String fileStr, String hashAlgo, int BUFFER) throws IOException, HasherException {
File file = new File(fileStr);
MessageDigest md = null;
FileInputStream fis = null;
FileChannel fc = null;
ByteBuffer bbf = null;
StringBuilder hexString = null;
try {
md = MessageDigest.getInstance(hashAlgo);
fis = new FileInputStream(file);
fc = fis.getChannel();
bbf = ByteBuffer.allocateDirect(BUFFER); // allocation in bytes - 1024, 2048, 4096, 8192
int b;
b = fc.read(bbf);
while ((b != -1) && (b != 0)) {
bbf.flip();
byte[] bytes = new byte[b];
bbf.get(bytes);
md.update(bytes, 0, b);
bbf.clear();
b = fc.read(bbf);
}
fis.close();
byte[] mdbytes = md.digest();
hexString = new StringBuilder();
for (int i = 0; i < mdbytes.length; i++) {
hexString.append(Integer.toHexString((0xFF & mdbytes[i])));
}
return hexString.toString();
} catch (NoSuchAlgorithmException e) {
throw new HasherException("Unsupported Hash Algorithm.", e);
}
}
我的想法是,Java NIO
尝试使用本地系统调用等来保持系统中的处理和存储(缓冲区)并且不在JVM之中 - 这可以防止(理论上)程序不得不在JVM和系统之间来回拖动事物。从理论上讲,这应该会更快......但是也许我的MessageDigest
强制JVM带入缓冲区,否定本地缓冲区/系统调用带来的任何性能改进?我在这个逻辑中纠正了,还是我失去了方向?
Please help me understand why Java NIO is not better in this scenario.
NIO擅长并发。如果你没有并发执行,它只会在代码和处理上花费很多。我不知道什么是性能差异,但可能是这样。 – akostadinov 2013-05-01 15:39:41
从通道读入“ByteBuffer”,然后再拷贝到byte []中可能会损害nio方法的性能。如果JVM可以避免将系统空间(操作系统,磁盘,...)中的数据传输到用户空间,那么NIO魔术(除了非阻塞部分之外)主要进行操作。由于哈希算法需要读取文件的每个字节,显然没有可用的快捷方式。如果Old-IO方法的性能不足,请考虑使用分析器或测试库实现(例如[guava](http://code.google.com/p/guava-libraries/wiki/HashingExplained)以获得更好的性能) – Pyranja 2013-05-01 15:58:16
@Pranranja嗯,有趣的信息。我检出了Guava库,不幸的是我不预期它会产生比我上面的方法更高的性能提升,因为它们都依赖于默认的'java.security.MessageDigest'实现来实际执行哈希...并且如果文件是太大而无法在缓冲区中合理地修复(例如,10GB文件),那么它必须通过缓冲区进行流式处理,并且我们需要做很多I/O操作才能通过缓冲区进行流式处理,直到我们已经散列所有的位......嗯.. – SnakeDoc 2013-05-01 16:29:46