Erxin, Shang(Edwin, 尚尔鑫)'s Home Page » Algorithm http://www.shangerxin.com Knowledge should be free. Tue, 24 Aug 2021 02:27:23 +0000 en-US hourly 1 http://wordpress.org/?v=3.8.13 Get required lasts lines from a huge text file in python http://www.shangerxin.com/?p=94 http://www.shangerxin.com/?p=94#comments Fri, 19 Dec 2014 13:28:30 +0000 http://www.shangerxin.com/?p=94 During my work, I need to parse the result from some huge log files. The result is always save at the end of the file. So I need find a way to get the required last lines to accelerate the parse speed. Here is the function I have written to achieve this requirement in python.

  1. def last_lines(filename, lines=1):
  2.     '''
  3.     return the last lines from the specify files
  4.     '''
  5.     with open(filename, 'rb') as fh:
  6.         fh.seek(0, os.SEEK_END)
  7.         if fh.tell() == 0:
  8.             return []
  9.         fh.seek(-2, os.SEEK_END)
  10.         #while not BOF
  11.         while fh.tell() > 0:
  12.             cur_char = fh.read(1).decode()
  13.             if cur_char == '\n':
  14.                 lines -= 1
  15.             if lines > 0:
  16.                 fh.seek(-2, os.SEEK_CUR)
  17.             else:
  18.                 break
  19.         return [line.decode() for line in fh.readlines()]

 

]]> http://www.shangerxin.com/?feed=rss2&p=94 0