Files and directories

In the previous section we learned how to build a path. In this section we'll do something with those paths: list folders, inspect files, and move, delete and compress them.

We'll do this by building a log maintenance script step by step. Every server produces log files, and without maintenance they keep growing until the disk is full. By the end of this section, the script will:

1. find every log file in a folder and its subfolders;
2. move log files older than 7 days to an archive folder;
3. delete log files older than 90 days;

The test folder

The create_test_logs.py script creates a folder with log files of different sizes and ages, so we can test the maintenance script without the risk of deleting real files.

logs
├── README.txt                   (90 days old)
├── backup.log
├── app
│   ├── app.log
│   ├── app.log.1                (20 days old)
│   └── app.log.2                (60 days old)
├── nginx
│   ├── access.log
│   ├── access.log.1             (3 days old)
│   ├── access.log.2             (10 days old)
│   ├── access.log.3             (45 days old)
│   ├── error.log
│   └── error.log.1              (12 days old)
└── old
    └── legacy.log.1             (120 days old)

Files such as access.log.1 are rotated log files: when a log file becomes too large, the service renames it and starts a new, empty access.log. Run create_test_logs.py again whenever you want to start over.

Listing a folder

Concept: Listing files and directories

There are two ways to list files, per folder or recursively.

The iterdir method from the pathlib library can be used to list contents of a single directory. It returns an iterator of Path objects, which can be filtered to only include files or directories.

from pathlib import Path

for path in Path('/some/dir').iterdir():
    # Do something with the Path object.

If the contents of the children must also be listed, the walk method can be used. It returns a list of tuples, each containing a Path object for the directory, a list of subdirectories and a list of files. The subdirectories and files are strings, not Path objects. They can be converted to Path objects by joining them with the directory path.

from pathlib import Path

for root, dirs, files in Path('/some/dir').walk():
    # The dirs and files lists contain strings, not Path objects.
    dir_paths = [root / d for d in dirs]
    file_paths = [root / f for f in files]

06_log_maintenance_v1.py

from pathlib import Path

LOG_DIR = Path("logs")

def find_log_files():
    log_files = []

    for root, dirs, files in LOG_DIR.walk():
        # Completed in the next section.
        
    return log_files

Note

The LOG_DIR variable is defined in uppercase, which is a convention used for constants (variables that will not change).

Since walk() returns every file in the folder and its subfolders, we must filter the results to only include log files. There are two types of log files: the current log file, which ends in .log, and rotated log files, which end in a number and have a stem that ends in .log.

Concept: Stem and suffix

A path object that links to a file has a stem and a suffix. The stem is the name of the file without the suffix, and the suffix is the extension of the file.

from pathlib import Path

log_file = Path("/var/log/nginx/access.log")
print(log_file.stem)   # Output: access
print(log_file.suffix) # Output: .log

These two properties can be combined into a function that returns True for log files and False for everything else. Which can in turn be used to filter the results of walk(). Note that we need to remove the dot from the suffix before checking if it's numeric, otherwise .2 would return False.

06_log_maintenance_v1.py

from pathlib import Path

LOG_DIR = Path("logs")

def is_log_file(path):
    if path.suffix == ".log":
        return True

    return path.suffix[1:].isnumeric() and path.stem.endswith(".log")

def find_log_files():
    log_files = []

    for root, dirs, files in LOG_DIR.walk():
        for file in files:
            file_path = root / file
            if is_log_file(file_path):
                log_files.append(file_path)

    return log_files
    
log_files = find_log_files()
print(log_files)

This code prints the paths of all log files in the logs folder and its subfolders (formatted for readability):

[
    PosixPath('logs/backup.log'), PosixPath('logs/nginx/error.log'),
    PosixPath('logs/nginx/access.log.3'), PosixPath('logs/nginx/access.log.2'),
    PosixPath('logs/nginx/access.log'), PosixPath('logs/nginx/error.log.1'),
    PosixPath('logs/nginx/access.log.1'), PosixPath('logs/old/legacy.log.1'),
    PosixPath('logs/app/app.log.2'), PosixPath('logs/app/app.log'),
    PosixPath('logs/app/app.log.1')
]

README.txt is not part of the list while access.log.1 and access.log.2 are, so the filtering works as expected.

Size and age

To decide what to do with a file, the script needs to know its size and age.

Concept: File stats

A path object allows us to retrieve information about the file it points to, such as its size and last modification time, using the stat method.

from pathlib import Path

log_file = Path("/var/log/nginx/access.log")
stats = log_file.stat()

The stat method returns a os.stat_result object, which has a lot of attributes. The most useful ones are listed below:

Attributes

Attribute	Description
st_mode	File type and permissions
st_ino	The inode number of Unix and the file index on Windows
st_dev	The device the file resides on
st_nlink	The number of hard links to the file
st_uid	The user ID of the file owner
st_gid	The group ID of the file owner
st_size	The size of the file in bytes

Timestamps

Attribute	Description
st_atime	The last access time of the file in seconds since epoch time
st_mtime	The last modification time of the file in seconds since epoch time
st_ctime	The most recent metadata change expressed in seconds
st_birthtime	The creation time of the file in seconds since epoch time

Since the timestamp returned by stat() is expressed in the number of seconds since epoch time, we can calculate the age of a file in days by subtracting the last modification time from the current time and dividing by the number of seconds in a day. To get the current time, we can use the time module, which is part of the standard library.

07_log_maintenance_v2.py

from pathlib import Path
import time

LOG_DIR = Path("logs")
SECONDS_IN_DAY = 24 * 60 * 60

def is_log_file(path):
    if path.suffix == ".log":
        return True

    return path.suffix[1:].isnumeric() and path.stem.endswith(".log")

def get_age_in_days(path):
    return (time.time() - path.stat().st_mtime) / SECONDS_IN_DAY

def find_log_files():
    log_files = []

    for root, dirs, files in LOG_DIR.walk():
        for file in files:
            file_path = root / file
            if is_log_file(file_path):
                log_files.append({
                    "path": file_path,
                    "size": file_path.stat().st_size,
                    "age": get_age_in_days(file_path),
                })

    return log_files

log_files = find_log_files()
print(log_files)

The script now produces the following output, which shows the path, size in bytes and age in days of every log file (formatted for readability):

[
    {'path': PosixPath('logs/backup.log'), 'size': 12288, 'age': 0.12611763194479325},
    {'path': PosixPath('logs/nginx/error.log'), 'size': 40960, 'age': 0.12611758223010433},
    {'path': PosixPath('logs/nginx/access.log.3'), 'size': 1003520, 'age': 45.12611758309106},
    ...
    {'path': PosixPath('logs/app/app.log.1'), 'size': 532480, 'age': 20.126117579454075}]

Warning

A Path can be used in an f-string, but not with a format such as :<28. f"{path:<28}" raises a TypeError, so convert the path to a string first: f"{str(path):<28}".

Moving and deleting

The script can now act on the files. Log files older than 7 days move to an archive folder, in the same subfolder they came from: logs/nginx/access.log.2 becomes archive/nginx/access.log.2. Log files older than 90 days are deleted.

We'll use the following methods to move and delete files:

• path.relative_to(LOG_DIR) returns the part of the path inside logs (nginx/access.log.2), which we add to the archive folder with the / operator.
• destination.parent.mkdir(parents=True, exist_ok=True) creates the destination folder, like mkdir -p. parents=True also creates missing parent folders, exist_ok=True prevents an error when the folder already exists.
• shutil.move() moves the file, like mv.
• path.unlink() deletes the file, like rm.

Moving or deleting can fail, for example when the script isn't allowed to change a file. Rather than checking the permissions first, we simply try and catch the PermissionError, so that one locked file doesn't stop the maintenance of all the others.

08_log_maintenance_v3.py

from pathlib import Path
import time
import shutil
import sys

LOG_DIR = Path("logs")
ARCHIVE_DIR = Path("archive")
SECONDS_IN_DAY = 24 * 60 * 60
ARCHIVE_AFTER_DAYS = 7
DELETE_AFTER_DAYS = 90

# ... is_log_file(), get_age_in_days() and find_log_files() are unchanged ...

def archive_log_file(path):
    destination = ARCHIVE_DIR / path.relative_to(LOG_DIR)
    destination.parent.mkdir(parents=True, exist_ok=True)
    shutil.move(path, destination)

    return destination

for log_file in find_log_files():
    path = log_file["path"]

    try:
        if log_file["age"] >= DELETE_AFTER_DAYS:
            path.unlink()
            print(f"DELETE {path}")
        elif log_file["age"] >= ARCHIVE_AFTER_DAYS:
            destination = archive_log_file(path)
            print(f"Archived {path} --> {destination}")
    except PermissionError:
        print(f"No permission to change {path}, skipped.", file=sys.stderr)

Archived logs/nginx/access.log.3 --> archive/nginx/access.log.3
Archived logs/nginx/access.log.2 --> archive/nginx/access.log.2
Archived logs/nginx/error.log.1 --> archive/nginx/error.log.1
DELETE logs/old/legacy.log.1
Archived logs/app/app.log.2 --> archive/app/app.log.2
Archived logs/app/app.log.1 --> archive/app/app.log.1