Running Shell Commands in Python Using subprocess

Posted by Afsal on 06-Feb-2025

Hi Pythonistas!

In the previous post we have discussed how to run shell commands using OS module.The os.system() method works for basic tasks but lacks flexibility. If we need to:
✅ Capture command output
✅ Handle errors properly
✅ Run commands securely

Then Python’s subprocess module is the way to go!

Using subprocess.run() for Better Control

Let’s start by executing a simple command and capturing its output.

code

In [35]: import subprocess

In [36]: result = subprocess.run(['ls', '-la'], capture_output=True, text=True)

In [37]: result
Out[37]: CompletedProcess(args=['ls', '-la'], returncode=0, stdout='total 35876\ndrwxrwxr-x 7 afsal afsal     4096 Feb  6 04:11 .\ndrwxrwxr-x 4 afsal afsal     4096 Jan 14 10:20 ..\ndrwxrwxr-x 6 afsal afsal     4096 Feb  6 04:10 app\n-rw-rw-r-- 1 afsal afsal      951 Jan 20 03:58 constants.py\n-rw-rw-r-- 1 afsal afsal     1917 Jan 13 03:09 fyers_data_socket.py\n-rw-rw-r-- 1 afsal afsal     2158 Jan 13 03:09 fyers_order_socket.py\n-rw-rw-r-- 1 afsal afsal     5446 Jan 13 03:09 fyers_utils.py\n-rw-rw-r-- 1 afsal afsal 36647434 Jan 16 09:11 instruments.json\n-rw-rw-r-- 1 afsal afsal     3058 Feb  6 10:58 kite_utils.py\n-rw-rw-r-- 1 afsal afsal      108 Jan 13 03:09 long_running.py\n-rwxrwxr-x 1 afsal afsal      672 Jan 13 03:09 manage.py\ndrwxrwxr-x 2 afsal afsal     4096 Feb  6 10:58 __pycache__\n-rw-rw-r-- 1 afsal afsal      944 Jan 16 09:11 requirements.txt\ndrwxrwxr-x 4 afsal afsal     4096 Jan 20 03:58 static\n-rw-rw-r-- 1 afsal afsal    13473 Feb  6 09:20 stratergies.py\ndrwxrwxr-x 3 afsal afsal     4096 Jan 23 03:51 trading_platform\n-rw-rw-r-- 1 afsal afsal      536 Feb  5 12:31 utilities.py\ndrwxrwxr-x 6 afsal afsal     4096 Jan 17 03:33 venv\n', stderr='')

In [38]: result.returncode
Out[38]: 0

In [39]: result.stdout
Out[39]: 'total 35876\ndrwxrwxr-x 7 afsal afsal     4096 Feb  6 04:11 .\ndrwxrwxr-x 4 afsal afsal     4096 Jan 14 10:20 ..\ndrwxrwxr-x 6 afsal afsal     4096 Feb  6 04:10 app\n-rw-rw-r-- 1 afsal afsal      951 Jan 20 03:58 constants.py\n-rw-rw-r-- 1 afsal afsal     1917 Jan 13 03:09 fyers_data_socket.py\n-rw-rw-r-- 1 afsal afsal     2158 Jan 13 03:09 fyers_order_socket.py\n-rw-rw-r-- 1 afsal afsal     5446 Jan 13 03:09 fyers_utils.py\n-rw-rw-r-- 1 afsal afsal 36647434 Jan 16 09:11 instruments.json\n-rw-rw-r-- 1 afsal afsal     3058 Feb  6 10:58 kite_utils.py\n-rw-rw-r-- 1 afsal afsal      108 Jan 13 03:09 long_running.py\n-rwxrwxr-x 1 afsal afsal      672 Jan 13 03:09 manage.py\ndrwxrwxr-x 2 afsal afsal     4096 Feb  6 10:58 __pycache__\n-rw-rw-r-- 1 afsal afsal      944 Jan 16 09:11 requirements.txt\ndrwxrwxr-x 4 afsal afsal     4096 Jan 20 03:58 static\n-rw-rw-r-- 1 afsal afsal    13473 Feb  6 09:20 stratergies.py\ndrwxrwxr-x 3 afsal afsal     4096 Jan 23 03:51 trading_platform\n-rw-rw-r-- 1 afsal afsal      536 Feb  5 12:31 utilities.py\ndrwxrwxr-x 6 afsal afsal     4096 Jan 17 03:33 venv\n'

In [40]:

capture_output=True : Captures command output instead of printing it.
text=True : Ensures output is returned as a string instead of bytes

result.returncode == 0 means the process completed succesfully let us check a failure case

Handling Errors Gracefully
Let’s see what happens when the command fails:

code

In [40]: result = subprocess.run(['ls', '/abcd'], capture_output=True, text=True)

In [41]: result.returncode
Out[41]: 2

In [42]: result.stderr
Out[42]: "ls: cannot access '/abcd': No such file or directory\n"

In [43]:

Real-Time Command Execution with Popen
If we need to process output as it’s generated (e.g., for long-running commands like ping), we use subprocess.Popen().

code

In [49]: process = subprocess.Popen(['ping', '-c', '10','google.com'], stdout=subprocess.PIPE, text=True)

In [50]: for line in process.stdout:
    ...:     print(line)
    ...: 
PING google.com (172.217.166.110) 56(84) bytes of data.

64 bytes from maa05s09-in-f14.1e100.net (172.217.166.110): icmp_seq=1 ttl=118 time=13.4 ms

64 bytes from maa05s09-in-f14.1e100.net (172.217.166.110): icmp_seq=2 ttl=118 time=13.4 ms

64 bytes from maa05s09-in-f14.1e100.net (172.217.166.110): icmp_seq=3 ttl=118 time=13.2 ms

64 bytes from maa05s09-in-f14.1e100.net (172.217.166.110): icmp_seq=4 ttl=118 time=13.5 ms

64 bytes from maa05s09-in-f14.1e100.net (172.217.166.110): icmp_seq=5 ttl=118 time=13.0 ms

64 bytes from maa05s09-in-f14.1e100.net (172.217.166.110): icmp_seq=6 ttl=118 time=13.1 ms

64 bytes from maa05s09-in-f14.1e100.net (172.217.166.110): icmp_seq=7 ttl=118 time=13.2 ms

64 bytes from maa05s09-in-f14.1e100.net (172.217.166.110): icmp_seq=8 ttl=118 time=13.1 ms

64 bytes from maa05s09-in-f14.1e100.net (172.217.166.110): icmp_seq=9 ttl=118 time=13.4 ms

64 bytes from maa05s09-in-f14.1e100.net (172.217.166.110): icmp_seq=10 ttl=118 time=13.3 ms



--- google.com ping statistics ---

10 packets transmitted, 10 received, 0% packet loss, time 9015ms

rtt min/avg/max/mdev = 13.048/13.265/13.525/0.155 ms

Why use Popen?

  • Useful for long-running commands.
  • Reads output line by line without waiting for the command to finish.

Choosing Between os.system() and subprocess

Feature os.system() subprocess.run() subprocess.Popen()
Captures Output ❌ No ✅ Yes ✅ Yes
Error Handling ❌ Limited ✅ Better ✅ Better
Security ❌ Risky ✅ Safer ✅ Safer
Real-time Output ❌ No ❌ No ✅ Yes

The subprocess module gives us more power and flexibility when running shell commands in Python. While os.system() is simple, subprocess is the recommended approach for most cases.