erlang 有效读取输入文件

bqf10yzr  于 2022-12-08  发布在  Erlang
关注(0)|答案(2)|浏览(165)

Currently for a task, I am working with input files which give Matrix related test cases (Matrix Multiplication) i.e., example of an input file ->

N M
1 3 5 ... 6 (M columns)
....
5 4 2 ... 1 (N rows)

I was using simple read() to access them till now, but this is not efficient for large files of size > 10^2. So I wanted to know is there some way to use processes to do this in parallel.
Also I was thinking of using multiple IO readers based on line, so then each process could read different segments of the file but couldn't find any helpful resources.
Thank you.
PS: Current code is using this:

io:fread(IoDev, "", "~d")
i34xakig

i34xakig1#

Did you consider to use re module? I did not make a performance test, but it may be efficient. In the following example I do not use the first "M N" line. So I did not put it in the matrix.txt file.
matrix file:
I made the conversion in the shell

1> {ok,B} = file:read_file("matrix.txt"). % read the complete file and store it in a binary
{ok,<<"1 2 3 4 5 6 7 8 9\r\n11 12 13 14 15 16 17 18 19\r\n21 22 23 24 25 26 27 28 29\r\n31 32 33 34 35 36 37 38 39">>}
2> {ok,ML} = re:compile("[\r\n]+"). % to split the complete binary in a list a binary, one for each line
{ok,{re_pattern,0,0,0,
                <<69,82,67,80,105,0,0,0,0,0,0,0,1,8,0,0,255,255,255,255,
                  255,255,...>>}}
3> {ok,MN} = re:compile("[ ]+"). % to split the line into binaries one for each integer
{ok,{re_pattern,0,0,0,
                <<69,82,67,80,73,0,0,0,0,0,0,0,17,0,0,0,255,255,255,255,
                  255,255,...>>}}
4> % a function to split a line and convert each chunk into integer
4> F = fun(Line) -> Nums = re:split(Line,MN), [binary_to_integer(N) || N <- Nums] end.
#Fun<erl_eval.7.126501267>
5> Lines = re:split(B,ML). % split the file into lines
[<<"1 2 3 4 5 6 7 8 9">>,<<"11 12 13 14 15 16 17 18 19">>,
 <<"21 22 23 24 25 26 27 28 29">>,
 <<"31 32 33 34 35 36 37 38 39">>]
6> lists:map(F,Lines). % map the function to each lines
[[1,2,3,4,5,6,7,8,9],
 [11,12,13,14,15,16,17,18,19],
 [21,22,23,24,25,26,27,28,29],
 [31,32,33,34,35,36,37,38,39]]
7>

if you want to check the matrix size, you can replace the last line with:

[[NbRows,NbCols]|Matrix] = lists:map(F,Lines),
case (length(Matrix) == NbRows) andalso 
      lists:foldl(fun(X,Acc) -> Acc andalso (length(X) == NbCols) end,true,Matrix) of
    true -> {ok,Matrix};
    _ -> {error_size,Matrix}
end.
7eumitmz

7eumitmz2#

是否有某种方法可以使用进程来并行执行此操作。
当然可以了。
我也在考虑使用基于行的多个IO读取器,这样每个进程就可以读取文件的不同部分,但却找不到任何有用的资源。
您不是按行查找文件中的位置,而是按字节查找位置。虽然文件看起来像一堆行,但实际上文件只是一长串字符。因此,您需要确定要在文件中查找的字节位置。
检查文件:位置,文件:扩展。

相关问题