In this post, we present a simple log file monitor agent implemented in python.
- Log files are in append mode.
- The log files are rotated in the same folder.
- The names of log files share the same pattern.
- Each line in the log file represents an event.
- The delimiter is the new line character
A log file monitor agent would need to have the following functionality:
- Identify the latest log file in the folder.
- Detect new lines of the log file that is currently being monitored.
To identify the latest log file, we could first get all the files that match the given pattern in the folder and then select the file with the latest creation time. The tool we use is
To detect new lines of the log file that is being monitored, we need to track the offset of the log file so that when we read the file next time, we don't go from the very beginning. The tools used for this purpose are
There are two other considerations:
- When we read a log file, some applications are still writing to it and the line that is being written to the file may not be a complete line (e.g. The line may not contain the new line character).
- When a new log file is generated due to the log rotation, some parts of the previous log file may not be processed yet.
Here is the implementation:
import os import time import glob def selectLatestFile(pattern): files = glob.glob(pattern) maxCreationTime = 0 result = None for f in files: s = os.stat(f) if s.st_ctime > maxCreationTime: result = f maxCreationTime = s.st_ctime return result def readLines(file, offset, callback): with open(file, 'r') as f: f.seek(offset) line = f.readline() newOffset = offset while line and line[-1] == '\n': callback(line.strip()) newOffset = f.tell() line = f.readline() return newOffset def isEmpty(file): s = os.stat(file) return s.st_size == 0 def printLine(l): print("> " + l) if __name__ == '__main__': pattern = "<pattern-of-log-file>" waitingTimeInSec = 1 currentOffset = 0 currentFile = None while True: file = selectLatestFile(pattern) if file != currentFile: if currentFile is not None: currentOffset = readLines(currentFile, currentOffset, printLine) currentFile = file currentOffset = 0 if not isEmpty(file): currentOffset = readLines(file, currentOffset, printLine) time.sleep(waitingTimeInSec)
----- END -----
©2019 - 2022 all rights reserved