Memory Profiling in Python
CPU, Memory, and other issues are always present when profiling applications. Python apps, on the other hand, are prone to memory management issues. This is primarily due to the fact that Python is used in Data Science and Machine Learning applications and works with massive amounts of data. Furthermore, Python relies on its Memory Management system by default, rather than leaving it up to the user. However, this trust can be risky in some cases.
To do that, we will concern a special module named ‘memory-profiler’. This library is implemented by Fabian Pedregosa and the contributors. Basically, it is a pure module that depends on the psutil module from the standard libraries inside Python but it is important to remember that there are tones of different methods to profile memory usage such as cProfile, tracemalloc, etc.
The reason we picked this special library that it is easy to use and understandable structure to see line by line in our python file. Also, it provides to log our memory usage easily.
Firstly, we need to install this package with pip below:
pip install –U memory_profiler
Then, we can start to check the sample implementation below:
We can now see the memory usage for this file, line by line, in the screenshot of our log output below:
It should also be noted that these processes are done with Python’s version 3.9.2.
References:
