Python MemoryError: What is it?

In this article, let us learn about Python’s MemoryError exception. In particular, we will try to answer the following 2 questions

what is MemoryError?
What causes MemoryError?

Without further ado, let’s begin!

Understanding MemoryError

MemoryError is simply an exception raised when you run out of RAM to allocate to your program. This is when the space allocated for your code in your computer’s memory is exhausted and your computer cannot carry on with the execution of your code. In other words, your code is trying to use more memory than your system can provide.

For example, the following program triggers the MemoryError by consuming a lot of memory. We start with an empty list and then use nested for loops to add strings to it:

DISCLAIMER: The demonstration code below is for educational purposes only, We are not responsible for any data losses that might occur if you run the example code below yourself.

CAUTION: If you decide to run the below code yourself, and your code is stuck, just be patient for a 5 to 10 minutes and let the code finish executing. If the code is stuck for 30 minutes or more, try using Ctrl+C to end the program. If Ctrl+C doesn’t do it for you, try closing the Python interpreter. If that doesn’t do it either then, force shutting down the laptop might be the only option (pressing the power button for 10 seconds usually works in most brands of laptops, but you will probably lose any unsaved work, beware!)

my_list = []
for a in range(1000):
   for b in range(1000):
       for c in range(1000):
           my_list.append("Hi")

By the end of the program, the list is supposed to have 1,000,000,000 (1 Billion) copies of the string ‘hi’!

If you run the above code yourself you will see that the code is executing for a long time (cursor is just blinking away)

Assuming each instance of the string “Hi” occupies 3 bytes of space, you have just exhausted 3 billion bytes (approximately 3GB) of space in your RAM!

If you look at the resources consumed by the program, the RAM usage will keep on growing as shown below.

As you can see the simple 4 lines of code were consuming a whopping 53% of memory in my case, and it was still growing! I decided to kill the code using a KeyboardInterrupt (Ctrl + C) to prevent it from slowing my entire system!

If you let the code run, you will end up with an exception as shown below.

Traceback (most recent call last):
  File "/home/main.py", line 1, in <module>
    my_list = [0] * 10**9 
MemoryError

What causes it?

MemoryError happens mainly because of 2 reasons.

You haven’t set up your Python environment properly or
your code is simply occupying too much memory like in the example above!

Reason#1: Environment Issues

It can be that you have installed a 32-bit version of Python. 32-bit applications are limited to a very low memory amount of user-mode address space. (2^32 is 4GB)

If your computer has enough RAM, installing a 64-bit version of Python will fix this problem.

Reason#2: Your code needs more space than your system can provide

Your program could also be storing large datasets that in turn require vast amounts of memory allocation. As we have already seen, a simple list with a huge number of items is enough to trigger a MemoryError

Furthermore loading a huge dataset into your RAM from your secondary memory (SSD or HDD) and running computations on it can quickly eat up your memory.

It is good practice to write your program in such a way that it uses the system’s memory efficiently. Writing code that misuses memory space such as unnecessary copying of data or retaining references to objects that are no longer needed uses up your memory over time.

my_list = [1, 2, 3, 4]
reference_to_list = my_list
#Now  my_list is not eligible for garbage collection even if it's no longer needed

This also includes not closing files after you’re done working with them:

f = open('big_file.txt', 'r')
my_file = f.read()

In this example, we are opening a large file and do not close it after we’re done working with it. In this case, as long as the program is executing, the entire content of the file is being held in memory!

By keeping these causes in mind, you can avoid the pesky MemoryError from crashing your program!

If you still haven’t solved the problem, I recommend you read our other article on how to fix MemoryError.

Python MemoryError: How to Fix it? A Step by Step Guide!

Also it is worth noting that MemoryError is not one of the most common exceptions you will run into in your daily life as a Python developer. Instead, you need to focus on 7 other errors mentioned in the article below if you wish to gain mastery in Python!

7 Most Common In-Built Exceptions in Python!

If you are a visual learner here is a YouTube video that we made on that same topic!