Python Generators

Python generators are very powerful for handling operations which require large amount of memory.

Let us start with simple example. Below function prints infinite sequence of numbers.

and so on...

Python Yield

Ok let us revisit our function 'generator_example1()'. What is happening in the below code?

Inside while loop, we have 'yield' statement. Yield breakes out of loop and gives back control to whomever called function generator_exampe1(). In statement 'g = generator_example1()', g is now a geneator as shown below.

Once you have a generator function, you can iterate through it using next() function. Since we have a infinite 'while' loop in the genereator_example() function, we can call iterator as many times as we want it. Each time, we use next(), generator starts the execution from previous position and prints a new value.

Python Generator Expression

Python generators can be used outside the function without the 'yield'. Check out the below example.

(x for x in range(10)) is a Python generator object. The syntax is quite similar to Python list comprehension except that instead of square brackets, generators are defined using round brackets. As usual, once we have generator object, we can call iterator next() on it to print the values as shown below.

Python Generator stop Iteration

Python generators will throw 'StopIteration' exception, if there is no value to return for the iterator.

Let us look at following example.

To avoid above error, we can catch exception like this and stop the iteration.

Python Generator send()

We can pass value to Python Generators using send() function.

Python Recursive Generator

Python generators can be used recursively. Check out the below code. In below function, "yield from generator_factorial(n - 1)" is recursive call to function generator_factorial().

Python Generator throw() Error

Continuing with above example, let us say we want generator to throw error for the factorial of number greater than 100. We can add generator.throw() exception such as shown below.

Python Generators Memory Efficient

Python generators take very less memory. Let us look at following two examples. In the examples below, note the difference between byte size of memory used by 'Python list' vs 'Python generator'.

Python Generator Performance

One thing to notice here is that, Python generators are slower than Python list comprehension if the memory is large engough to compute. Let us look at below two examples from the performance perspective.

Check the number of function calls and time the 'Python generator' took to compute the sum compare to Python 'list comprehension'.

Data Pipeline with Python Generator

Let us wrap up this tutorial with Data Pipelines. Python generators are great for building the pipelines.

Let us open a CSV file and iterate through it using Python generator.

Let us say, we want to replace the commas in the CSV for each line with spaces, we can build a pipeline for this.

Wrap Up:

It takes a little practice to get hold on Python generators but once mastered, Python generators are very useful for not only building data pipelines but also handling large data operations such as reading a large file.