Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I got excited but then read that it's some kind of cloud-based web application thing.

Is there something like this (show memory use and call times for a Python process) that just runs on my computer to help me profile a long-running Python process?



psutil (https://pythonhosted.org/psutil/) is awesome for collecting valuable monitoring information.

pyrasite (http://pyrasite.com/) will let you inject code into a process. This can be used to add monitoring of private internal state etc (if you have no other options).

If you want to have locally hosted graphs then grafana and influx are my current tools of choice.

It is going to be more work than swiping a credit card, but not a crazy amount.


pyrasite can make the child process slow down markedly; this came up for me when injecting profilers into cinnamon-screensaver to try to root cause a memory leak (https://bugs.launchpad.net/linuxmint/+bug/1652489)

cinnamon-screensaver would take multiple seconds to lock the screen even after I'd stopped profiling and exited the interpreter I'd injected, and I wound up restarting it so I could lock my screen quickly again.

I don't know why this happened, but it's enough to make me think twice, and I'm definitely going to double-check my process is still performing as I hope after injecting it with pyrasite in the future.


Sure, there are plenty of deterministic and statistical profilers for python. Two examples:

https://docs.python.org/3/library/profile.html

https://vmprof.readthedocs.io/en/latest/


Yes! It's called profiling and there are many ways to do that. Python has built-in profiling tools (profile, cProfile), there are also whole-system profiling solutions like DTrace (sadly, that is not available on Linux). No fancy GUIs AFAIK, you'd have to RTFM a bit.


> DTrace (sadly, that is not available on Linux)

Sysdig is available though. It even got userspace tracers recently (including Python). Alternatively there's "perf" if you want just the kernel side.


As nobody yet has mentioned it: lptrace -> strace for Python programs @ https://github.com/khamidou/lptrace

For Python3: https://github.com/rixx/lptrace/tree/python3


I also needed to profile a running process without stopping it, and ended up using pyflame (https://github.com/uber/pyflame) together with the famous flamegraph script. It is not perfect, as it only works on Linux and only does CPU profiling, but it worked well enough for my purposes.


Yappi is a really useful one, especially for multithreaded apps because it can started and stopped at any time regardless of control flow. For example, you can setup a rest API just to handle profiling and then query it at any point during app lifetime.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: