Python Memory Management and Tips Transcripts
Chapter: Course conclusion and review
Lecture: Memory and functions
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
When we looked at functions, we saw two basic patterns that would really help us
0:04
aim the memory usage. We have this one here.
0:07
It looks really nice. It's gonna load some data and create this data pipeline. It's going
0:11
to then scale it, pass it along,
0:12
getting the next step back, and then it's gonna filter it with that middle step and
0:16
then get the final step, the filtered data.
0:18
While this was good, and as sort of a code style,
0:21
it's really easy to read, we saw that if we just did something silly,
0:26
like reuse the variable name, like "here's the data
0:28
at this step", we were able to get dramatically better memory performance.
0:32
So here you can see we're up over 100 megabytes on this one on the left.
0:36
The one on the right is 71.
0:39
It's 34 megabytes less just by using a different variable name.
0:43
That's kind of insane. You can do this and have a lot better memory usage.
0:48
The reason is original, scaled, those two stick around for the entire length of the
0:54
function, even though after the second step,
0:56
original is no longer needed, so it would allow us to like free that memory
1:00
up and then reuse it without getting more from the operating system.
1:03
So this is really cool. And then what blew this out of the water,
1:06
even, this technique, was to use generators.
1:08
Now generators limit what you can do with the data you get back.
1:12
But in this pipeline scenario, generators are so perfect.
1:15
So we saw that, even though going from 105 to 71 megabytes
1:19
is pretty amazing, with generators,
1:21
we went all the way down to 9 megabytes.
1:24
It was off the charts good.
1:25
So a little bit, tiny bit slower,
1:28
CPU wise, computational wise, but in terms of memory,
1:31
it was the clear winner. We have this pattern of not letting the intermediate variables
1:37
hang around too long, which is pretty awesome in a general thing,
1:40
and then if it works, generators are really good for memory.