Python Memory Management and Tips Transcripts
Chapter: Memory and functions
Lecture: Concept: Dropping intermediate data

Login or purchase this course to watch this video and the rest of the course contents.
0:00 We saw with two very similar patterns, we got quite different memory usage. Here
0:05 we're saying we're gonna have three steps and we've named the data from them: original, scaled, and filtered. And we could change this
0:13 to just say "we're going to call the current data were working with, data". Now I don't love this, right.
0:18 It's not as clear, it's not as obvious what's happening. But at the same time, if it dramatically lowers the memory usage and lets you
0:27 do more data analysis or run a lot cheaper because you can use smaller servers when you pay for your cloud computing, all that, well,
0:34 it's probably worth a little bit of less clear code. So I actually graphed this in a tool that we're going to see a little bit
0:42 later. And if you look at the clean code version, that one actually had 105 megabytes. It's slightly larger because we scaled and then filtered,
0:51 which means, when you're working with floats, more steps. But nonetheless, it's basically the same idea. So we got 105 MB over there,
1:00 and this simple change dropped it down by 34 megabytes. Just those three lines made such a big difference. Very similar patterns,
1:10 but actually forcing Python to drop the intermediate data as soon as it can. There's of course other ways. You could set the original to none,
1:19 and you could set scaled to none after you're done working with them. That would be fine, too.
1:23 It's just gonna be more lines of code. So you can work with it however you
1:25 like, but I think this is a pretty interesting pattern to think about to force Python to clean up the intermediate data sooner.

Talk Python's Mastodon Michael Kennedy's Mastodon