如何从Python中的文本数据文件中提取数据子集

Question

我有一个文本文件，其中相关数据（行x列）仅出现在“开始”和“结束”关键字之间。见下文。我想编写一个可以提取这些数据子集的代码。如果一行以“开始”开始，然后是数据，但没有后续的“结束”关键字，那么我想忽略这些数据。在我的例子中，data1和data3是相关的，但data2并不是因为它没有被“开始”和“结束”关键字所包围。

start 
data1 (matrix of data) /relevant because data1 is enclosed by "start" and "end" 
end 
start 
data2 (matrix of data) /not relevant because there is no "end" 
. 
start 
data3 (matrix of data) /relevant for same reason as for data1 
end 
. 
. 
and so on 

我以为我可以这样开始：

with open(file_path,'r') as file: 

    text = file.readlines() 
    start_indexes = [] 
    end_indexes = [] 

    for i, line in enumerate(text): 
     if line.startswith('start'): 
      start_indexes.append(i) 
     elif line.startswith('end'): 
      end_indexes.append(i) 

    for i in range(len(start_indexes)): 
     for j in range(len(end_indexes)): 
      if (start_indexes[i] < end_indexes[j] < start_indexes[i+1]): 
       print(start_indexes[i],end_indexes[j]) 

上面的代码给了我两个起始行号和结束行号在有相关数据。这是我有点卡住的地方。我现在如何提取相关数据？在我下面的例子中，它将是data1，data3。我是否以“正确”的方式处理问题？我应该使用熊猫吗？有没有更高效和直接的方法？

Answer 1

嵌套循环？

您正在经历开始和结束范围的每种组合。你只需要那些对应于相同数据的那些数据。

像这样的东西替换你的for循环：

for start, end in zip(start_indexes, end_indexes): 
    print(text[start + 1:end]) 

zip(a, b, ...)返回与a, b, ...列，根本上给[(a[0], b[0], ...), (a[1], b[1], ...), ...]一个新的列表。通过start_indexes, end_indexes的每一列，给出相应的开始和结束值，然后使用列表片段访问来获取这些行的数据。

Answer 2

我会通过只读顺序读取文件来使用另一种方式（假设“开始” - “结束” - 数据块中的数据不会太大）。我将创建一个data变量来收集当前块的数据（无论是否相关）以及具有状态转换的变量。

一些伪的Python：

if state == OUTSIDE_BLOCK and line.startswith('start'): 
    state = INSIDE_BLOCK 
elif state == INSIDE_BLOCK and <line contains data>: 
    <Add to data variable> 
elif state == INSIDE_BLOCK and line.startswith('end'): 
    state = OUTSIDE_BLOCK 
    <Process collected data> 
elif state == INSIDE_BLOCK and line.startswith('start'): 
    <Throw away possibly collected data because it was irrelevant> 

Answer 3

我个人觉得正则表达式是处理这种情况的最好办法：

import re woof0='''start data1 (matrix of data) /relevant because data1 is enclosed by "start" and "end" end start data2 (matrix of data) /not relevant because there is no "end" . start data3 (matrix of data) /relevant for same reason as for data1 end . . and so on ''' re.findall(r'start(\sdata.*|\Sdata.*)\nend',woof0)

输出：

['\ndata1 (matrix of data) /relevant because data1 is enclosed by "start" and "end"', '\ndata3 (matrix of data) /relevant for same reason as for data1']

Answer 4

设置：

s = '''start 
data1 (matrix of data) /relevant because data1 is enclosed by "start" and "end" 
end 
start 
data2 (matrix of data) /not relevant because there is no "end" 
start 
data3 (matrix of data) /relevant for same reason as for data1 
end 
start 
data4 blah 
''' 
import io 
f = io.StringIO(s) 

对文件进行迭代一次，测试每行的开头;找出将有效数据块放入子列表并将它们附加到结果列表中所需的逻辑...

result = [] 
sub = [] 

for line in f: 
    if line.startswith('start'): 
     # possible new data block 
     if sub: 
      # if it isn't empty it must contain 
      # a start --> data block with no end 
      result.append(sub) 
      sub = [] 
     sub = [line] 
    elif line.startswith('end'): 
     # start over 
     sub = [] 
    elif line.startswith('data'): 
     sub.append(line) 
    else: 
     # for lines that don't startwith data, start or end - if any 
     sub.append(line) 

if sub: 
    # start --> data --> EOF or end of string 
    result.append(sub)