Python MemoryError: How to Fix it? A Step by Step Guide!

In this article, we will learn how to fix the MemoryError exception in Python. We will look through each of its causes along with its remedy.

MemoryError is usually caused by 2 reasons,

improper setup of the Python environment or
because your code is simply occupying too much memory.

Let’s tackle them both!

DISCLAIMER: The demonstration code examples below are for educational purposes only, We are not responsible for any data losses that might occur if you run the code examples below on your own computers.

CAUTION: If you decide to run the below code yourself, and your code is stuck, just be patient for a 5 to 10 minutes and let the code finish executing. If the code is stuck for 30 minutes or more, try using Ctrl+C to end the program. If Ctrl+C doesn’t do it for you, try closing the Python interpreter. If that doesn’t do it either then, force shutting down the laptop might be the only option (pressing the power button for 10 seconds usually works in most brands of laptops, but you will probably lose any unsaved work, beware!)

Improper Setup

One reason to get MemoryError is that you have installed a 32-bit version of Python. 32-bit applications are limited to a meager memory amount of 4GB of user-mode address space. Installing a 64-bit version of Python will fix this problem.

In other cases, using Conda to install Python is the solution to MemoryError. Conda installs better and improved memory management packages which might just be what you need in your case.

Inefficient Memory Usage

Usage of Huge Datasets

Another reason is that you may be inefficiently using your memory by storing a large amount of data in your RAM:

my_list = [0] * 10**9  #creating a list with too many items

Traceback (most recent call last):
  File "/home/main.py", line 1, in <module>
    my_list = [0] * 10**9 
MemoryError

Unfortunately, there is no straight way to fix it other than tailoring your code to fit the memory requirements.

In these types of situations, batching always saves the day. Batching is simply splitting the work into batches allowing the memory to be freed in between calls.

For example, instead of dealing with 1 billion bytes of data at a time, you can split that into 1 million byte chunks and work with them.

Remember to use generator functions as they prove to be handy in these situations. Several Python libraries such as TensorFlow or Keras include dedicated generator methods and classes when you install them.

Freeing memory using del Keyword

If you are using huge datasets in your code, try freeing the memory from time to time manually using the del keyword

my_list = [1, 2, 3, 4]

# do whatever is needed 

del my_list

Closing files in file handling

This one is pretty straightforward, always close your files after you’re done with them!

Have a look at the following program:

f = open('big_file.txt', 'r')
my_file = f.read()

We opened a big file and we didn’t close it. During this time, the entire content of the file is being held in memory!

Simply close the file after you have used it:

f.close()

Or use the with statement provided by Python to ensure that the file gets automatically closed once you are done using it.

with open('big_file.txt', 'r') as f:
	my_content = f.read()
 
print(my_content)

Another good programming practice to follow is to stream the data whenever you can instead of holding the entire thing in memory. I leave it to you to figure out how to stream stuff!

MemoryError is not one of the most common errors you will run into in your daily life as a Python developer. Instead, you need to focus on 7 other errors mentioned in the article below if you wish to gain mastery in Python!

7 Most Common In-Built Exceptions in Python!

If you are a visual learner here is a YouTube video that we made on that same topic!