How to get the max memory usage of a program using psutil in Python

Question

I am using the following code to get the max memory usage of the program.

    import os, subprocess , psutil
    def mem(cmd):
        try:
            with open('in.txt','r') as infile, open('out.txt', 'w') as outfile:
                p=psutil.Popen("./"+cmd,shell=False,stdin=infile,stdout = outfile)
            print p.memory_info()
        except Exception:
             print "Error"
    cmd=raw_input()
    mem(cmd)

The problem is sometimes for initial runs of the program the memory usage output is (0,0) but subsequently it displays the correct output. I dont why this happening. For some programs such as the hello world program in c++ the output is pmem(rss=4096, vms=315392) which is about 0.3M( I think the output is in bytes ) but running the hello world program in ideone.com gives the output as ~3M. Why is there this disperency?

cmd is the name of the executable.

The output of command print subprocess.check_output(['ps', 'v', '-p', str(p.pid)])

PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 16150 pts/16 Z+ 0:00 0 0 0 0 0.0 [a.out] <defunct>

One of my sample C++ program:

`int a[1000000];
int main()
{
    return 0;
}`

returns pmem(rss=4096, vms=4313088) sometimes and pmem(rss=0,vms=0) sometimes

Answer 1

The problem here is that psutils takes a quick snapshot from the /proc filesystem, as you can see in the source.

When you run your hello world example, in some cases it finishes before python gets a chance to read the values from /proc.

Once the process is finished, it effectively no longer consumes any ram. You can confirm this with an strace.

open("/proc/13420/statm", O_RDONLY)     = 3
read(3, "0 0 0 0 0 0 0\n", 1024)        = 14

If you modify your example to use something like sleep, you'll notice that psutils consistently gives memory usage back.

#include <iostream>
#include <unistd.h>

int main()
{
  std::cout << "Hello World.. sleeping!";
  sleep(3);
}

Output of your python script...

a.out
meminfo(rss=286720, vms=12931072)

One simple way to accomplish what you are trying to do, is by using the /usr/bin/time command, which on most platforms will give you the average total memory usage of the process you launch OR use valgrind as J.F Sebastian suggests... whom posted as I was researching and testing my answer ;)

Hello World.. sleeping!0.00user 0.00system 0:03.00elapsed 0%CPU     
(0avgtext+0avgdata 1144maxresident)k
0inputs+0outputs (0major+348minor)pagefaults 0swaps