Iterators in Python Land.

AI & Data Science 604

6 min read

·

Jul 10, 2026

--

Listen

Share

Handy functions that will allow you to effectively work with iterators.

Introduction.

In this blog, we’ll learn all about iterators and iterables, which you’ve already worked with when writing for loops. We’ll learn some handy functions that will allow you to effectively work with iterators. And we’ll work with a use case that is pertinent to the world of data science and dealing with large amounts of data — in this case, data from Twitter that you will load in chunks using iterators.

Press enter or click to view image in full size

Iterating with a for loop

There’s no reason to be scared of iterators because you have actually been working with them for some time. When you use a for loop to print out each element of a list, you’re iterating over the list.

employees = ['Nick', 'Lore', 'Hugo']
for employee in employees:
   print(employee)
#Output
"""
Nice
Lore
Hugo
"""

You can also use a for loop to iterate over characters in a string such as you see here.

for letter in 'Hello':
    print(letter)

"""
#output
H
e
l
l
o
""

You can also use a for loop to iterate over a sequence of numbers produced by a special range object.

The reason that we can loop over such objects is that they are special objects called iterables: lists, strings dictionaries, and range objects are all iterables.

The actual definition of an iterable is an object that has an associated iter() method. Once this iter method is applied to an iterable, an iterator object is created.
Under the hood, this is actually what a for loop is doing: it takes an iterable, creates the associated iterator object, and iterates over it!

Iterating over iterables: next()

An iterator is defined as an object that has an associated next() method that produces the consecutive values.

word = 'Da'
it = iter(word)

next(it)

# Output > 'D'

next(it)
# output >'a'

next(it)
#output > StopIteration Traceback (most recent call last)

To create an iterator from an iterable, all we need to do is use the function iter() and pass it the iterable. Once we’ve the iterator defined, we pass it to the function next() and this returns the 1st value. Calling next again on the iterator returns the next value, until there’re no values left to return and then it throws us a StopIteration error.

Iterating at once with *

You can also print all values of an iterator in one fell swoop using the star operator, referred to as the splat operator in some circles.

word = 'Data'
it = iter(word)
print(*it)

"""
#Output
D a t a
"""

This star operator unpacks all elements of an iterator or an iterable. Be warned, however, once you do so, you cannot do it again as there are no more values to iterate through! We would have to redefine our iterator to do so.

Iterating over dictionaries

We mentioned before that dictionaries and file connections are iterables as well. To iterate over the key-value pairs of a Python dictionary, we need to unpack them by applying the items method to the dictionary as you can see here.

pythonistas = {'hugo': 'bowne-anderson', 'francis': 'castro'}
for key, value in pythonistas.items():
    print(key, value)

Iterating over file connections

With respect to file connections, here you can see how to use the iter and next methods to return the lines from a file, file.txt. This has been your crash course in the fundamentals of iterables and iterators.

file = open('file.txt')
it = iter(file)
print(next(it))

# output > The 1st line.

print(next(it))
# output > The 2nd line

Playing with iterators

We’re now going to dive a bit deeper into the world of iterables and iterators by checking out some very cool, very useful functions.

enumerate()

The 1st function, enumerate, will allow us to add a counter to any iterabl. The enumerate is a function that takes any iterable as argument, e.g., list, and returns a special enumerate object, which consists of pairs containing the elements of the original iterable, along with their index within the iterable.

avengers = ['hawkeye', 'iron man', 'thor', 'quicksilver']
e = enumerate(avengers)
print(type(e))

# Output: <class 'enumerate'>

e_list = list(e)
print(e_list)
# Output: [(0, 'hawkeye'), (1, 'iron man'), (2, 'thor'), (3, 'quicksilver')]

We can use the function list to turn this enumerate object into a list of tuples and print it to see what it contains.

The enumerate object itself is also an iterable and we can loop over it while unpacking its elements using the clause for index, value in enumerate(avengers). It is the default behavior of enumerate to begin indexing at 0. However, you can alter this with a 2nd argument, start, which you can see here.

avengers = ['hawkeye', 'iron man', 'thor', 'quicksilver']

for index, value in enumerate(avengers):
    print(index, value)

"""
Output:
0 hawkeye1 iron man
2 thor
3 quicksilver
"""
for index, value in enumerate(avengers, start=10):
    print(index, value)

"""
Output
10 hawkeye11 iron man
12 thor
13 quicksilver
"""

zip()

The 2nd function, zip, will allow us to stitch together an arbitrary number of iterables. It accepts an arbitrary number of iterables and returns an iterator of tuples. Here we’ve 2 lists, one of the avengers, the other of their names.

avengers = ['hawkeye', 'iron man', 'thor', 'quicksilver']
names = ['barton', 'stark', 'odinson', 'maximoff']

z = zip(avengers, names)
print(type(z))

# Output: <class 'zip'>

z_list = list(z)
print(z_list)

"""
# Output: [('hawkeye', 'barton'), ('iron man', 'stark'), 
    ('thor', 'odinson'), ('quicksilver', 'maximoff')]
"""

Zipping them together creates a zip object which is an iterator of tuples. We can turn this zip object into a list and print the list. The first element is a tuple containing the first elements of each list that was zipped. The 2nd element is a tuple containing the 2nd elements of each list that was zipped and so on.

avengers = ['hawkeye', 'iron man', 'thor', 'quicksilver']
names = ['barton', 'stark', 'odinson', 'maximoff']
for z1, z2 in zip(avengers, names):
    print(z1, z2)

"""
Output: 
hawkeye barton
iron man stark
thor odinson
quicksilver maximoff
"""

Alternatively, we could use a for loop to iterate over the zip object and print the tuples.

We could also have used the splat operator to print all the elements! Now it’s your turn to play with these useful functions.

avengers = ['hawkeye', 'iron man', 'thor', 'quicksilver']
names = ['barton', 'stark', 'odinson', 'maximoff']

z = zip(avengers, names)
print(*z)

"""

('hawkeye', 'barton') ('iron man', 'stark') 
('thor', 'odinson') ('quicksilver', 'maximoff')
"""

Loading data in chunks

Now that you’re more comfortable with iterables, iterators and how they work, we’re going to check out a particular use case: dealing with large amounts of data.

Let’s say that you are pulling data from a file, database or API and there’s so much of it, just so much data, that you can’t hold it in memory.

One solution is to load the data in chunks, perform the desired operation or operations on each chuck, store the result, discard the chunk and then load the next chunk; this sounds like a place where an iterator could be useful!

To surmount this challenge, we are going to use the pandas function read_csv, which provides a wonderful option whereby you can load data in chunks and iterate over them.

All we need to do is to specify the chunk using the argument chunksize. As with much of what we do in Data Science, this is best illustrated by an example.

import pandas as pd
result = []
for chunk in pd.read_csv('data.csv', chunksize=1000):    
    result.append(sum(chunk['x']))

total = sum(result)
print(total)4252532

Let’s say that we have a csv with a column called ‘x’ of numbers and I want to compute the sum of all the numbers in that column. However, the file is too large to store in memory.

We first import pandas and then initialize an empty list result to hold the result of each iteration. We then use the read_csv function, utilizing the argument chunksize, setting it to the size of the chunks I want to read in. In this example, we use a chunk size of 1,000. You can play around with it.

The object created by the read_csv call is an iterable so I can can iterate over it, using a for loop, in which each chunk will be a DataFrame. Within the for loop, that is, on each iteration, we compute the sum of the column of interest, and we append it to the list result. Once this is executed, we can take the sum of the list result, and this gives us our total sum of the column of interest. Iterators to the rescue!

Also note that we need not have used a list to store each result — we could have initialized total to zero before iterating over the file and added each sum during the iteration procedure.

Next Blog about list comprehension and genertor.

https://medium.com/@Mustafa77/list-comprehensions-generators-a277c2c28e15?sharedUserId=Mustafa77