BETTER PROGRAMMING

How to Use Generator and yield in Python

Work with large datasets or files using Python generators

Image for post
Image for post

What are generators in Python?

Some use cases of generators

Reading large files

csv_gen = csv_reader("some_file.txt")
row_count = 0

for row in csv_gen:
row_count += 1

print(f"Row count is {row_count}")
def csv_reader(file_name):
file = open(file_name)
result = file.read().split("\n")
return result
Traceback (most recent call last):
File "ex1_naive.py", line 22, in <module>
main()
File "ex1_naive.py", line 13, in main
csv_gen = csv_reader("file.txt")
File "ex1_naive.py", line 6, in csv_reader
result = file.read().split("\n")
MemoryError
def csv_reader(file_name):
for row in open(file_name, "r"):
yield row
Row count is 65123455
csv_gen = (row for row in open(file_name))

Generating an infinite sequence

a = range(5)
print(list(a))
[0, 1, 2, 3, 4]
def infinite_sequence():
num = 0
while True:
yield num
num += 1
for i in infinite_sequence():
print(i, end=" ")
>> gen = infinite_sequence()
>>> next(gen)
0
>>> next(gen)
1
>>> next(gen)
2
....

More on yielding

>>> def multiple_yield():
... value = "I'm here for the first time"
... yield value
... value = "My Second time here"
... yield value
...
>>> multi_gen = multiple_yield()
>>> print(next(multi_gen))
I'm here for the first time
>>> print(next(multi_gen))
My Second time here
>>> print(next(multi_gen))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

Advance generator methods

def isPrime(n):
if n < 2 or n % 1 > 0:
return False
elif n == 2 or n == 3:
return True
for x in range(2, int(n**0.5) + 1):
if n % x == 0:
return False
return True

def getPrimes():
value = 0
while True:
if isPrime(value):
yield value
value += 1

How to use .send()

prime_gen = getPrimes()
print(next(prime_gen))
print(prime_gen.send(1000))
print(next(prime_gen))
2
3
5
def getPrimes():
value = 0
while True:
if isPrime(value):
i = yield value
if i is not None:
value = i
value += 1
prime_gen = getPrimes()
print(next(prime_gen))
print(prime_gen.send(1000))
print(next(prime_gen))
2
1009
1013

How to use .throw()

prime_gen = getPrimes()

for x in prime_gen:
if x > 10:
prime_gen.throw(ValueError, "I think it was enough!")
print(x)
2
3
5
7
Traceback (most recent call last):
File "test.py", line 25, in <module>
prime_gen.throw(ValueError, "I think it was enough!")
File "test.py", line 15, in getPrimes
i = yield value
ValueError: I think it was enough!

How to use .close()

prime_gen = getPrimes()

for x in prime_gen:
if x > 10:
prime_gen.close()
print(x)
2
3
5
7
11

Conclusion

Written by

I’m an entrepreneur, developer, author, speaker, and doer of things. I write about JavaScript, Python, AI, and programming in general.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store