Understanding Generators in Python

Hi Pythonistas!

Today we will learn a very interesting advanced concept in Python which is generators.

Python generators are powerful tools for working with large datasets or iterating over sequences without loading everything into memory at once. They allow you to generate values on-the-fly, reducing memory usage and improving performance. We use a yield statement instead of a return in the generator.  In this post, we'll explore the concept of generators and demonstrate their memory efficiency through a simple example.

Code

def generate_even_numbers_list(limit):
    numbers = []
    for i in range(limit):
        if i % 2 == 0:
            numbers.append(i)
    return numbers

def generate_even_numbers_generator(limit):
    for i in range(limit):
        if i % 2 == 0:
            yield i

Profile memory usage

from memory_profiler import memory_usage
mem_usage_list = memory_usage((generate_even_numbers_list, (1000_000_0,)))
mem_usage_generator = memory_usage((generate_even_numbers_generator, (1000_000_0,)))
print("memory usage of using list overtime: ", mem_usage_list)
print("memory usage of using generator overtime: ", mem_usage_generator)

Output

memory usage of using list overtime:  [19.60546875, 19.7421875, 51.2109375, 86.84375, 122.76171875, 158.4765625, 193.703125, 211.359375]

memory usage of using generator overtime:  [22.33203125, 22.33203125, 22.33203125]

Note: output of the memory_usage functions is memory usage  of a function on different point of time

We can see that the function uses list grows overtime. But the function using generator remains same throughout its execution. Which means generator generate only when  asked.

Congratulation you have learned an advanced concept in python.Hope you have learned if you need more about this please connect with afsal@parseltongue.co.in