News

Welcome to End Point’s blog

Ongoing observations by End Point people

Getting realtime output using Python Subprocess

The Problem

When I launch a long running unix process within a python script, it waits until the process is finished, and only then do I get the complete output of my program. This is annoying if I'm running a process that takes a while to finish. And I want to capture the output and display it in the nice manner with clear formatting.

Using the subprocess and shlex library

Python has a “batteries included” philosophy. I have used 2 standard libraries to solve this problem.
import subprocess 
import shlex 
  • subprocess - Works with additional processes
  • shlex - Lexical analysis of shell-style syntaxes

subprocess.popen

To run a process and read all of its output, set the stdout value to PIPE and call communicate().
import subprocess
process = subprocess.Popen(['echo', '"Hello stdout"'], stdout=subprocess.PIPE)
stdout = process.communicate()[0]
print 'STDOUT:{}'.format(stdout)
The above script will wait for the process to complete and then it will display the output. So now we are going to read the stdout line by line and display it in the console untill it completes the process.
output = process.stdout.readline()
This will read a line from the stdout.
process.poll()
The poll() method will return
  • the exit code if the process is completed.
  • None if the process is still running.
while True:
        output = process.stdout.readline()
        if output == '' and process.poll() is not None:
            break
        if output:
            print output.strip()
    rc = process.poll()
The above will loop and keep on reading the stdout and check for the return code and displays the output in real time.
I had one more problem in parsing the shell commands to pass it to popen when I set the shell=False. Below is an example command:
rsync -avzXH --delete --exclude=*.swp --exclude=**/drivers.ini /media/lgisos/lg.iso root@42-a:/isodevice
To split the string using shell-like syntax I have used shlex library's split method.

Here is the final code looks like

def run_command(command):
    process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE)
    while True:
        output = process.stdout.readline()
        if output == '' and process.poll() is not None:
            break
        if output:
            print output.strip()
    rc = process.poll()
    return rc

10 comments:

Dan O said...

Why are you using self.logger? It doesn't look like you're running this from an instance method. Is this just a typo or am I missing something?

Kannan Ponnusamy said...

@Dan,
You are right, this is a instance method. I have updated the code snippet. Thanks.

Adrien Desprez said...

Thanks a lot, very useful post, it works great !

William Isaac said...

There is still multiple "self." variables in this example.

stuaxo said...

Grabbing stderr at the same time as stdout in a nice way (and keeping the correct order) is still somewhat challenging.

Kannan Ponnusamy said...

Thanks @William Isaac, Updated the post.

shakedunay said...
This comment has been removed by the author.
ramtha Z28 said...

hi there
i tried you method but not working i am using pythin 2.7.1 and iam tring to read from c++ exe to the python as input so far when i put the while loop with readline in just hold and then crash

Kannan Ponnusamy said...

@ramtha Have you tried by setting `shell=True`. This tells subprocess to use the OS shell to open and run your script.

tfurrows said...

How do you handle if the subprocess goes into cooked mode? In my case, I have the .C source of the subprocess, and I can see when it goes into cbreak. When it does, I get no output on the screen when running as a subprocess... then at the end of my subprocess, all of the output type comes at once.

Any hints to a solution for that would be awesome.