Python - Simple Log File Monitor Agent

Subscribe Send me a message

#python #log #log file #monitor #agent

In this post, we present a simple log file monitor agent implemented in python.

Assumptions:

Log files are in append mode.
The log files are rotated in the same folder.
The names of log files share the same pattern.
Each line in the log file represents an event.
The delimiter is the new line character \n.

A log file monitor agent would need to have the following functionality:

Identify the latest log file in the folder.
Detect new lines of the log file that is currently being monitored.

To identify the latest log file, we could first get all the files that match the given pattern in the folder and then select the file with the latest creation time. The tool we use is glob and os.stat().

To detect new lines of the log file that is being monitored, we need to track the offset of the log file so that when we read the file next time, we don't go from the very beginning. The tools used for this purpose are f.seek(offset) and f.tell().

There are two other considerations:

When we read a log file, some applications are still writing to it and the line that is being written to the file may not be a complete line (e.g. The line may not contain the new line character).
When a new log file is generated due to the log rotation, some parts of the previous log file may not be processed yet.

Here is the implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

import os
import time
import glob

def selectLatestFile(pattern):
    files = glob.glob(pattern)
    maxCreationTime = 0
    result = None

    for f in files:
        s = os.stat(f)

        if s.st_ctime > maxCreationTime:
            result = f
            maxCreationTime = s.st_ctime
    return result

def readLines(file, offset, callback):
    with open(file, 'r') as f:
        f.seek(offset)
        line = f.readline()
        newOffset = offset

        while line and line[-1] == '\n':
            callback(line.strip())
            newOffset = f.tell()
            line = f.readline()

        return newOffset

def isEmpty(file):
    s = os.stat(file)
    return s.st_size == 0

def printLine(l):
    print("> " + l)

if __name__ == '__main__':
    pattern = "<pattern-of-log-file>"
    waitingTimeInSec = 1
    currentOffset = 0
    currentFile = None

    while True:
        file = selectLatestFile(pattern)
        if file != currentFile:
            if currentFile is not None:
                currentOffset = readLines(currentFile, currentOffset, printLine)
            currentFile = file
            currentOffset = 0

        if not isEmpty(file):
            currentOffset = readLines(file, currentOffset, printLine)
        time.sleep(waitingTimeInSec)

----- END -----