Python Memory Management and Tips Transcripts
Chapter: Memory and functions
Lecture: Concept: Dropping intermediate data
0:00 We saw with two very similar patterns,
0:02 we got quite different memory usage. Here
0:04 we're saying we're gonna have three steps and we've named the data from them: original,
0:09 scaled, and filtered. And we could change this
0:12 to just say "we're going to call
0:14 the current data were working with, data".
0:16 Now I don't love this, right.
0:17 It's not as clear, it's not as obvious what's happening.
0:21 But at the same time, if it dramatically lowers the memory usage and lets you
0:26 do more data analysis or run a lot cheaper because you can use smaller servers when
0:31 you pay for your cloud computing, all that, well,
0:33 it's probably worth a little bit of less clear code.
0:37 So I actually graphed this in a tool that we're going to see a little bit
0:41 later. And if you look at the clean code version,
0:45 that one actually had 105 megabytes.
0:47 It's slightly larger because we scaled and then filtered,
0:50 which means, when you're working with floats,
0:53 more steps. But nonetheless,
0:55 it's basically the same idea. So we got 105 MB over there,
0:59 and this simple change dropped it down by 34 megabytes. Just those three lines made such
1:06 a big difference. Very similar patterns,
1:09 but actually forcing Python to drop the intermediate data as soon as it can.
1:14 There's of course other ways. You could set the original to none,
1:18 and you could set scaled to none
1:19 after you're done working with them.
1:20 That would be fine, too.
1:22 It's just gonna be more lines of code. So you can work with it however you
1:24 like, but I think this is a pretty interesting pattern to think about to force
1:28 Python to clean up the intermediate data sooner.