Measuring the CPU and Memory Usage for Python subprocess

The article talks about a method for measuring the resources used by a process invoked by subprocess.run or subprocess.Popen.

We can use psutil to get realtime usage data of a process. Using psutil may not provide accurate resource measurements for short-lived processes, as it samples usage at intervals and can miss brief spikes. We want a method to get the final resource usage of a process after it finishes.

The method leverages multiprocessing.Process to wrap the calling of subprocess.run or subprocess.Popen, and get the resource usage by calling resource.getrusage(resource.RUSAGE_CHILDREN) of the wrapper process. For example,

 1import resource
 2import subprocess
 3from multiprocessing import Process, Manager
 4
 5def run(cmd):
 6	'''
 7	Run a command, for example
 8	run('stress --timeout 5 -c 2 --vm-bytes 128M')
 9	'''
10    def _run(cmd, result):
11        p = subprocess.Popen(cmd,
12	                         shell=True,
13	                         stdout=subprocess.PIPE,
14	                         stderr=subprocess.PIPE)
15        stdout, stderr = p.communicate()
16        result['returncode'] = p.returncode
17        result['stdout'] = stdout
18        result['stderr'] = stderr
19        rusage = resource.getrusage(resource.RUSAGE_CHILDREN)
20        result['rusage'] = rusage
21    with Manager() as manager:
22        result = manager.dict()
23        process = Process(target=_run, args=(cmd, result))
24        process.start()
25        process.join()
26        return dict(result)

The result and usage data can be transferred back to main process by multiprocessing.Manager.

The multiprocessing.Process creates an isolated child process that becomes the direct parent of the subprocess launched via subprocess.Popen. This intermediate layer ensures that the wrapper process exclusively captures resource statistics for the subprocess it spawned. This isolation prevents interference from unrelated processes and guarantees the measured usage corresponds solely to the target command.

The method is like time command, but without affecting stdout/stderr output of the target subprocess.