通过四处搜索，大多数解决方案可以归结为：
将文件分成更小的文件，在您的情况下，每个文件2gb；
对每个文件进行排序；
合并和排序
找到并打印副本。

下面的算法性能不太好。它只能说明我将要做什么。

Create an empty second file e.g. `./cache.data`.
Load the first line (l1) from the origin file. Initialize a counter with 0.
Read all lines (ln) from the origin file one by one (loop) 
  compare ln with the l1.
  If they differ 
    write the line ln into the created cache file 
    continue the loop
  if they do not differ
    counter++
    continue the loop
if counter > 0
  write out the l1 line and the counter (only the dups) 
        or counter + 1 (dups + l1) 
Replace the origin file with the cache file and create a new empty cache file
start from the beginning as long as there are lines in the source file

您可以通过一次比较多行来改进此算法。然后你还需要更多的计数器，每行一个进行比较。您必须确保比较行之间没有重复的行。