Python Memory Management and Tips Transcripts
Chapter: Course conclusion and review
Lecture: Memory and functions
0:00 When we looked at functions, we saw two basic patterns that would really help us
0:04 aim the memory usage. We have this one here.
0:07 It looks really nice. It's gonna load some data and create this data pipeline. It's going
0:11 to then scale it, pass it along,
0:12 getting the next step back, and then it's gonna filter it with that middle step and
0:16 then get the final step, the filtered data.
0:18 While this was good, and as sort of a code style,
0:21 it's really easy to read, we saw that if we just did something silly,
0:26 like reuse the variable name, like "here's the data
0:28 at this step", we were able to get dramatically better memory performance.
0:32 So here you can see we're up over 100 megabytes on this one on the left.
0:36 The one on the right is 71.
0:39 It's 34 megabytes less just by using a different variable name.
0:43 That's kind of insane. You can do this and have a lot better memory usage.
0:48 The reason is original, scaled, those two stick around for the entire length of the
0:54 function, even though after the second step,
0:56 original is no longer needed, so it would allow us to like free that memory
1:00 up and then reuse it without getting more from the operating system.
1:03 So this is really cool. And then what blew this out of the water,
1:06 even, this technique, was to use generators.
1:08 Now generators limit what you can do with the data you get back.
1:12 But in this pipeline scenario, generators are so perfect.
1:15 So we saw that, even though going from 105 to 71 megabytes
1:19 is pretty amazing, with generators,
1:21 we went all the way down to 9 megabytes.
1:24 It was off the charts good.
1:25 So a little bit, tiny bit slower,
1:28 CPU wise, computational wise, but in terms of memory,
1:31 it was the clear winner. We have this pattern of not letting the intermediate variables
1:37 hang around too long, which is pretty awesome in a general thing,
1:40 and then if it works, generators are really good for memory.