有没有办法在大文件的nodejs中生成SHA256或类似的文件?

rekjcdws  于 2023-04-11  发布在  Node.js
关注(0)|答案(2)|浏览(193)

我试图生成一个巨大的缓冲区(2.5G)的sha256,不幸的是hash.update已经抛出错误(ERR_OUT_OF_RANGE)

RangeError: data is too long
    at Hash.update (node:internal/crypto/hash:113:22)

据我所知,我不能在createHash('sha256 ')上进行多次更新。https://nodejs.org/api/crypto.html#hashupdatedata-inputencoding
除了将它们写入磁盘,然后调用ie sha256sum并处理输出之外,还有其他方法来处理大缓冲区吗?
复制的简单示例:

const {createHash} = require("crypto")

const hash = createHash('sha256')

const data = new Buffer.alloc(1024 * 1024 * 1024 * 3, 1)

hash.update(data)

console.log(hash.digest('hex'))
kmb7vmvb

kmb7vmvb1#

thx @derpirscher!
可以根据需要频繁调用hash.update,散列本身将以digest()调用结束
现在要处理更大的缓冲区,最简单的方法是将缓冲区分成更小的块并将它们传递到哈希中。

const {createHash} = require("crypto")

const hash = createHash('sha256')
const data = new Buffer.alloc(1024 * 1024 * 1024 * 3, 1)

const chunkSize = 1024 * 1024 * 1024
const chunks = Math.ceil(data.length / chunkSize)

for (let i = 0; i < chunks; i++) {
  hash.update(data.subarray(i * chunkSize, (i+1) * chunkSize))
}

console.log(hash.digest('hex'))
bnl4lu3b

bnl4lu3b2#

Hash类扩展了stream.Transform,文档展示了如何使用Hash和管道流从文件中读取数据,通过Hash转换器传递数据,并将结果(哈希)写入另一个文件:

import { createReadStream } from 'node:fs';
import { stdout } from 'node:process';
const { createHash } = await import('node:crypto');

const hash = createHash('sha256');

const input = createReadStream('test.js');
input.pipe(hash).setEncoding('hex').pipe(stdout);

此示例显示了Node.js流的强大功能,可用于生成满足您需求的函数:

const fs = require('fs');
const crypto = require('crypto');
const stream = require('stream/promises');

async function computeHash(filepath) {
  const input = fs.createReadStream(filepath);
  const hash = crypto.createHash('sha256');
  
  // Connect the output of the `input` stream to the input of `hash`
  // and let Node.js do the streaming
  await stream.pipeline(input, hash);

  return hash.digest('hex');
}

使用方法:

const hash = await computeHash('/path-to-file');

相关问题