Sunday, June 22, 2014

python_autoprofile - A simple way to profile all python scripts on you run.

I've just created this package over at github.

It really only useful for debian users, but other may be able to do something similar , but I don't know
what features rpm has which replicate dpkg-divert.

What this package does is allow you to add extra command line switches as 'default' to the invocation of an python script on your system. It takes these extra parameters from the environment, so unless you set any
of the it special Envvars it does nothing.

It has two special environment variables the first XPYADD_ARGS is literally just prepend to the the python command line every time python is run. I use this to add the '-3' switch in my development environment so I can worry about python3 compatibly during development . By  placing it in an environment variable is one of the simplest places to make production and development different without added processes etc, which might get forgotten. But not having '-3' in production it also means unnecessary warnings are in our production logs.

But I also use this package in production for the it's other feature the one it is named for. If the PROFILING environment variable is set, it may add "-m cProfile -o $UNIQUE_FILE" to the python command line. The will cause the python execution to be profiled and a stats file saved to UNIQUE_FILE.

I say may, because I don't want to slow production down to much by running this on every python invocation, so it uses a configuration file' /etc/python-autoprofile/PROFILING_FREQ' to decide randomly on each run whether to do so or not. The idea is that you can configure what proportion of scripts run slightly slower -as they have profiling enabled - so as not to disturb you users perceived performance but still get a broad brushstroke view of your python performance.