Python Memory Management and Tips Transcripts
Chapter: Memory and functions
Lecture: Concept: Dropping intermediate data
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
We saw with two very similar patterns, we got quite different memory usage. Here
0:05
we're saying we're gonna have three steps and we've named the data from them: original, scaled, and filtered. And we could change this
0:13
to just say "we're going to call the current data were working with, data". Now I don't love this, right.
0:18
It's not as clear, it's not as obvious what's happening. But at the same time, if it dramatically lowers the memory usage and lets you
0:27
do more data analysis or run a lot cheaper because you can use smaller servers when you pay for your cloud computing, all that, well,
0:34
it's probably worth a little bit of less clear code. So I actually graphed this in a tool that we're going to see a little bit
0:42
later. And if you look at the clean code version, that one actually had 105 megabytes. It's slightly larger because we scaled and then filtered,
0:51
which means, when you're working with floats, more steps. But nonetheless, it's basically the same idea. So we got 105 MB over there,
1:00
and this simple change dropped it down by 34 megabytes. Just those three lines made such a big difference. Very similar patterns,
1:10
but actually forcing Python to drop the intermediate data as soon as it can. There's of course other ways. You could set the original to none,
1:19
and you could set scaled to none after you're done working with them. That would be fine, too.
1:23
It's just gonna be more lines of code. So you can work with it however you
1:25
like, but I think this is a pretty interesting pattern to think about to force Python to clean up the intermediate data sooner.