Subprocesses

Python can do a lot on its own, but every operating system already comes with hundreds of tools that administrators use daily: ping, ipconfig, systemctl, netstat, nslookup, ... Instead of rewriting those tools in Python, a script can simply start them and read their result. The subprocess module makes this possible.

Tips

Only use subprocess when Python can't do the job itself. To list a folder, use iterdir() rather than starting ls: it works on every system, it's faster, and it returns Path values instead of text you'd have to take apart.

In chapter 3, we used the following example to explain continue:

for i in range(1, 255):
    if not host_is_reachable(f"192.168.1.{i}"):
        continue

    open_ports = get_open_ports_for_host(f"192.168.1.{i}")

At the time, host_is_reachable() didn't exist, and we promised to come back to it. Since Python doesn't include a function to check whether a host is reachable, we'll write it ourselves using the ping command.

We'll test the function on a few public DNS servers, which are always online and answer pings, and on a private address that doesn't exist on our network:

Host	Address
Cloudflare DNS	1.1.1.1
Google DNS	8.8.8.8
Quad9 DNS	9.9.9.9
A non-existent host	192.168.255.254

Note

Some networks, such as public Wi-Fi or certain company networks, block pings to the internet. If none of the DNS servers is reachable, try again from another network.

Starting a command

subprocess.run() starts a command and waits until it has finished. The command is passed as a list: the first element is the name of the program, every following element is one argument, exactly as you would type them separated by spaces in the terminal.

ping -c 1 1.1.1.1 (send one ping to 1.1.1.1) therefore becomes:

import subprocess

subprocess.run(["ping", "-c", "1", "1.1.1.1"])

Caution

You'll find examples online that pass the command as a single string, with shell=True:

subprocess.run(f"ping -c 1 {host}", shell=True)

Avoid this. The string is passed to the shell, which also interprets characters such as ; and |. If host comes from user input, a user who enters 8.8.8.8; rm -rf ~ runs a second command of their own choosing. With a list, the whole value is one argument to ping, which simply reports that the host doesn't exist.

While this works on Linux and macOS, Windows uses different options for the same thing. Using the sys.platform function from the previous section, we can detect the OS and build the command accordingly.

09_ping_v1.py

import subprocess
import sys

IS_WINDOWS = sys.platform == "win32"

def build_command(host):
    if IS_WINDOWS:
        return ["ping", "-n", "1", host]
    return ["ping", "-c", "1", host]

result = subprocess.run(build_command("1.1.1.1"))

The output of ping appears in the terminal, just as if we'd typed the command ourselves. subprocess.run() returns a result, and its returncode property contains the exit code of the command.

PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.4 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 11.4/11.4/11.4/0.000 ms
0

In chapter 3 we used sys.exit(1) to tell the system that our script failed. Every command does the same, and ping uses its exit code to tell us whether the host answered:

Exit code	Meaning (Linux ping)
0	The host replied
1	No reply
2	Another error, e.g. a host name that doesn't exist

We don't need to read the output at all: the exit code is all we need.

A first version of host_is_reachable()

Putting the command in a function gives us a first version of host_is_reachable(). The function returns True when the exit code is 0.

10_ping_v2.py

import subprocess
import sys

IS_WINDOWS = sys.platform == "win32"

def build_command(host):
    if IS_WINDOWS:
        return ["ping", "-n", "1", host]
    return ["ping", "-c", "1", host]

def host_is_reachable(host):
    result = subprocess.run(build_command(host))
    return result.returncode == 0

for host in ["1.1.1.1", "192.168.255.254"]:
    print(f"{host}: {host_is_reachable(host)}")

The function works, but it has two problems:

1. The output of ping clutters the output of our own script.
2. Run the script and you'll notice that the private address takes several seconds. When there is no reply, ping keeps waiting for one. For a single host that's annoying, for 254 hosts it's unacceptable.

PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=19.2 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 19.228/19.228/19.228/0.000 ms
1.1.1.1: True
PING 192.168.255.254 (192.168.255.254) 56(84) bytes of data.

--- 192.168.255.254 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

192.168.255.254: False

Capturing the output

With capture_output=True, the output of the command no longer appears in the terminal. Instead, it is stored in the result: result.stdout contains the normal output and result.stderr the error messages, the same two outputs we used in chapter 3. text=True returns that output as a string; without it, Python returns raw bytes.

The second problem is solved by ping itself: -w 1 tells it to wait at most one second for a reply. On Windows the option is -w 1000, which waits 1000 milliseconds (1 second).

11_ping_v3.py

import subprocess
import sys

IS_WINDOWS = sys.platform == "win32"

def build_command(host):
    if IS_WINDOWS:
        return ["ping", "-n", "1", "-w", "1000", host]
    return ["ping", "-c", "1", "-w", "1", host]

def host_is_reachable(host):
    result = subprocess.run(build_command(host), capture_output=True, text=True)
    return result.returncode == 0

for host in ["1.1.1.1", "192.168.255.254"]:
    print(f"{host}: {host_is_reachable(host)}")

1.1.1.1: True
192.168.255.254: False

host_is_reachable() doesn't need the captured output, but other scripts do. The output is an ordinary string, so everything you know about strings applies to it:

result = subprocess.run(["ping", "-c", "1", "-w", "1", "dns.invalid"], capture_output=True, text=True)

print(result.returncode)   # Output: 2
print(result.stderr)       # Output: ping: dns.invalid: Name or service not known

The final version

The final version which pings all four hosts and prints the result doesn't include any new concepts, and is therefore not included it. It can, however, be found on GitPub