Advanced Memory Profiling with tracemalloc

Hi Pythonistas!

In Part 1, we learned how to track memory usage with tracemalloc. Now, let’s take it further by identifying top memory-consuming lines, comparing memory snapshots

Finding Where Memory is Used

To identify which lines of code consume the most memory, use tracemalloc.take_snapshot():

code

import tracemalloc

tracemalloc.start()

a = [0] * 10**6  
b = [0] * 5 * 10**6
c = a + b

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics("lineno")
print("\nTop 5 memory-consuming lines:\n")

for stat in top_stats[:5]:  # Show top 5 lines
    print(stat)

tracemalloc.stop()

Output

Top 5 memory-consuming lines:

/home/afsal/Desktop/experiments/trace_malloc_post/top_5_lines.py:8: size=45.8 MiB, count=1, average=45.8 MiB
/home/afsal/Desktop/experiments/trace_malloc_post/top_5_lines.py:7: size=38.1 MiB, count=1, average=38.1 MiB
/home/afsal/Desktop/experiments/trace_malloc_post/top_5_lines.py:6: size=7813 KiB, count=2, average=3907 KiB

breaking it down

/home/afsal/Desktop/experiments/trace_malloc_post/top_5_lines.py:9
→ The memory allocation happened at line 9 in the file top_5_lines.py.

size=45.8 MiB
→ The total memory allocated at this line is 45.8 MiB (Megabytes).

count=1
→ One object was allocated at this line.

average=45.8 MiB
→ Since only one object was created, its size is 45.8 MiB.

Now we know exactly where memory is used the most!

Comparing Memory Usage Over Time

To track how memory usage changes before and after block of code, compare snapshots:

code

import tracemalloc

tracemalloc.start()

snapshot1 = tracemalloc.take_snapshot()

a = [0] * 10**6  
b = [0] * 5 * 10**6  
c = a + b

snapshot2 = tracemalloc.take_snapshot()
stats = snapshot2.compare_to(snapshot1, "lineno")
for stat in stats[:3]:
    print(stat)


del a,b, c

snapshot3 = tracemalloc.take_snapshot()

stats = snapshot3.compare_to(snapshot2, "lineno")

for stat in stats[:3]:
    print(stat)

Ouput

/home/afsal/Desktop/experiments/trace_malloc_post/comparing_two.py:16: size=45.8 MiB (+45.8 MiB), count=1 (+1), average=45.8 MiB
/home/afsal/Desktop/experiments/trace_malloc_post/comparing_two.py:15: size=38.1 MiB (+38.1 MiB), count=1 (+1), average=38.1 MiB
/home/afsal/Desktop/experiments/trace_malloc_post/comparing_two.py:14: size=7812 KiB (+7812 KiB), count=1 (+1), average=7812 KiB
/home/afsal/Desktop/experiments/trace_malloc_post/comparing_two.py:16: size=0 B (-45.8 MiB), count=0 (-1)
/home/afsal/Desktop/experiments/trace_malloc_post/comparing_two.py:15: size=0 B (-38.1 MiB), count=0 (-1)
/home/afsal/Desktop/experiments/trace_malloc_post/comparing_two.py:14: size=0 B (-7812 KiB), count=0 (-1)

breaking it down

/home/afsal/Desktop/experiments/trace_malloc_post/comparing_two.py:16
→ The memory allocation happened at line 16 in the file comparing_two.py.

size=45.8 MiB
→ The total memory allocated at this line is 45.8 MiB (Megabytes).

+45.8 MiB
→ Increment in memory by 45.8 MiB.

count=1
→ One object was allocated at this line.

average=45.8 MiB
→ Since only one object was created, its size is 45.8 MiB.

Now we can see how much memory was allocated between snapshots.

Today we have learned how to check top memory consuming lines and memory difference between block of line
in the upcoming post we will learn about how to find the memory leaks. Please share your valuable
suggestion with afsal@parseltongue.co.in