awk与读线的比较线条

Question

我有两个文件，一个带17k线，另一个带4k线。我想将位置115与第二个文件中每行的位置125进行比较，如果匹配，则将第一个文件中的整行写入新文件。我想出了一个解决方案，我使用'cat $ filename |'来读取文件同时阅读LINE'。但需要大约8分钟才能完成。还有没有其他方式像使用'awk'来减少这个过程时间。

我的代码

cat $filename | while read LINE 
do 
    #read 115 to 125 and then remove trailing spaces and leading zeroes 
    vid=`echo "$LINE" | cut -c 115-125 | sed 's,^ *,,; s, *$,,' | sed 's/^[0]*//'` 
    exist=0 
    #match vid with entire line in id.txt 
    exist=`grep -x "$vid" $file_dir/id.txt | wc -l` 
    if [[ $exist -gt 0 ]]; then 
    echo "$LINE" >> $dest_dir/id.txt 
    fi 
done 

Answer 1

这是怎么回事：

FNR==NR {      # FNR == NR is only true in the first file 

    s = substr($0,115,10)  # Store the section of the line interested in 
    sub(/^\s*/,"",s)   # Remove any leading whitespace 
    sub(/\s*$/,"",s)   # Remove any trailing whitespace 

    lines[s]=$0    # Create array of lines 
    next      # Get next line in first file 
} 
{        # Now in second file 
    for(i in lines)   # For each line in the array 
     if (i~$0) {   # If matches the current line in second file 
      print lines[i]  # Print the matching line from file1 
      next    # Get next line in second file 
     } 
} 

保存到一个脚本script.awk和像运行：

$ awk -f script.awk "$filename" "${file_dir}/id.txt" > "${dest_dir}/id.txt" 

这仍将b因为对于第二个文件中的每一行，您需要查看第一个中约50％的独特行（假设大多数行确实匹配）。如果您可以确认第二个文件中的行与子字符串完全匹配，则可以显着提高此功能。

---

对于全行一致，这应该是更快：

FNR==NR {      # FNR == NR is only true in the first file 

    s = substr($0,115,10)  # Store the section of the line interested in 
    sub(/^\s*/,"",s)   # Remove any leading whitespace 
    sub(/\s*$/,"",s)   # Remove any trailing whitespace 

    lines[s]=$0    # Create array of lines 
    next      # Get next line in first file 
} 
($0 in lines) {     # Now in second file 
    print lines[$0]  # Print the matching line from file1 
}