Python - Simple Log File Monitor Agent

Subscribe Send me a message home page tags


In this post, we present a simple log file monitor agent implemented in python.

Assumptions:

A log file monitor agent would need to have the following functionality:

To identify the latest log file, we could first get all the files that match the given pattern in the folder and then select the file with the latest creation time. The tool we use is glob and os.stat().

To detect new lines of the log file that is being monitored, we need to track the offset of the log file so that when we read the file next time, we don't go from the very beginning. The tools used for this purpose are f.seek(offset) and f.tell().

There are two other considerations:

Here is the implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
import os
import time
import glob

def selectLatestFile(pattern):
    files = glob.glob(pattern)
    maxCreationTime = 0
    result = None

    for f in files:
        s = os.stat(f)

        if s.st_ctime > maxCreationTime:
            result = f
            maxCreationTime = s.st_ctime
    return result

def readLines(file, offset, callback):
    with open(file, 'r') as f:
        f.seek(offset)
        line = f.readline()
        newOffset = offset

        while line and line[-1] == '\n':
            callback(line.strip())
            newOffset = f.tell()
            line = f.readline()

        return newOffset

def isEmpty(file):
    s = os.stat(file)
    return s.st_size == 0

def printLine(l):
    print("> " + l)

if __name__ == '__main__':
    pattern = "<pattern-of-log-file>"
    waitingTimeInSec = 1
    currentOffset = 0
    currentFile = None

    while True:
        file = selectLatestFile(pattern)
        if file != currentFile:
            if currentFile is not None:
                currentOffset = readLines(currentFile, currentOffset, printLine)
            currentFile = file
            currentOffset = 0

        if not isEmpty(file):
            currentOffset = readLines(file, currentOffset, printLine)
        time.sleep(waitingTimeInSec)

----- END -----

Welcome to join reddit self-learning community.
Send me a message Subscribe to blog updates

Want some fun stuff?

/static/shopping_demo.png