In this article, we will learn how to fix the MemoryError exception in Python. We will look through each of its causes along with its remedy.
MemoryError is usually caused by 2 reasons,
- improper setup of the Python environment or
- because your code is simply occupying too much memory.
Let’s tackle them both!
One reason to get MemoryError is that you have installed a 32-bit version of Python. 32-bit applications are limited to a meager memory amount of 4GB of user-mode address space. Installing a 64-bit version of Python will fix this problem.
In other cases, using Conda to install Python is the solution to MemoryError. Conda installs better and improved memory management packages which might just be what you need in your case.
Inefficient Memory Usage
Usage of Huge Datasets
Another reason is that you may be inefficiently using your memory by storing a large amount of data in your RAM:
my_list =  * 10**9 #creating a list with too many items
Traceback (most recent call last):
File "/home/main.py", line 1, in <module>
my_list =  * 10**9
Unfortunately, there is no straight way to fix it other than tailoring your code to fit the memory requirements.
In these types of situations, batching always saves the day. Batching is simply splitting the work into batches allowing the memory to be freed in between calls.
For example, instead of dealing with 1 billion bytes of data at a time, you can split that into 1 million byte chunks and work with them.
Remember to use generator functions as they prove to be handy in these situations. Several Python libraries such as TensorFlow or Keras include dedicated generator methods and classes when you install them.
Freeing memory using del Keyword
If you are using huge datasets in your code, try freeing the memory from time to time manually using the del keyword
my_list = [1, 2, 3, 4]
# do whatever is needed
Closing files in file handling
This one is pretty straightforward, always close your files after you’re done with them!
Have a look at the following program:
f = open('big_file.txt', 'r')
my_file = f.read()
We opened a big file and we didn’t close it. During this time, the entire content of the file is being held in memory!
Simply close the file after you have used it:
Or use the with statement provided by Python to ensure that the file gets automatically closed once you are done using it.
with open('big_file.txt', 'r') as f:
my_content = f.read()
Another good programming practice to follow is to stream the data whenever you can instead of holding the entire thing in memory. I leave it to you to figure out how to stream stuff!
MemoryError is not one of the most common errors you will run into in your daily life as a Python developer. Instead, you need to focus on 7 other errors mentioned in the article below if you wish to gain mastery in Python!
If you are a visual learner here is a YouTube video that we made on that same topic!
And with that being said, let’s end the article!
Congratulations on making it to the end of the article, not many have the perseverance to do so!
I hope you enjoyed reading this article and found it useful!
Feel free to share it with your friends and colleagues!
If your thirst for knowledge has not been quenched yet, here are some related articles that might spark your interest!
Thanks to Namazi Jamal for his contributions in writing this article!