Python Memory Management and Tips Transcripts
Chapter: Memory and functions
Lecture: Tracking memory usage
0:00 Our code works, actually it works really well.
0:02 I see no problem with it.
0:03 It's gonna go through and process this data in this pipeline that we talked about.
0:07 But let's see if we can understand it's memory usage and then we can explore to
0:10 see if we can make it better.
0:12 Now I want to change one thing really quick here. Just because this is going to
0:18 uncover some of the challenges that we might see. Let's say,
0:21 "head" just like this and this will be the "tail" right? Kind of beginning
0:28 and the end of this data.
0:29 And let's just put five of them in there and let's go from -5 to
0:35 the end. Why is this gonna be interesting?
0:37 Because some of the advancements that we'll be able to do later will have a harder
0:41 time looking at the end. So if you reasonably want to go through the data
0:45 and, like, ask questions like "what
0:46 are the last five?" you'll see both the benefits and the drawbacks.
0:50 So this will give us kind of a realistic constraint as we go.
0:53 Alright, so what do we want to do?
0:54 Well, first of what we want to do is we want to go and bring
0:59 in our little size utility again.
1:01 So the size utility, this is what we've used so far,
1:03 but it also has this thing called "report process memory",
1:06 which will return the number of megabytes used,
1:09 as well do a little print statement.
1:11 So we're gonna use that and in order to use it without PyCharm,
1:15 right? Again, we can Just "import size_
1:18 util" which is fine here, but without PyCharm,
1:21 we've got to make sure that it's getting found in the path here. Sort of
1:26 annoying, but so it is.
1:28 So this is going to make sure you can always run it,
1:30 even if you just go and run it from the terminal or command prompt right in
1:35 that one folder. Let's go and do, to find a little function here, that's called
1:39 "report", and it's going to take a "step_name",
1:42 which is a string. Alright,
1:46 so it's just gonna go to the size_util and say "report process
1:49 memory" but we also want to just show a little bit of information about that
1:53 so that you can see at this step it's doing such and such rather than just
1:57 having the numbers come out three times.
1:59 So we'll do a little print with an f-string here.
2:01 We'll say "step is this".
2:10 And then we also want to set the end to just be a little space.
2:14 Actually got two spaces there. So we're gonna say step name
2:18 like "loading data" or "filtering" or whatever,
2:21 and then we'll see the print statement from before,
2:23 okay? So let's go ahead and use it here.
2:26 Now I'm gonna write some funky code.
2:27 Ah.. I'll write it
2:29 Like this. Let's say "report original
2:35 loaded". I'll just say "loaded".
2:39 Over here, "report starting" right?
2:42 So we have the baseline of what we got and "filtered" and "scaled".
2:54 Then we do "done", I guess.
2:59 Alright, let's run it and see how it works.
3:02 We started with 9 megabytes used then we loaded that one million numbers that went from 9
3:07 to 48 million. And then when we filtered it,
3:10 it went up some more, then we scaled
3:13 it went up some more. So at the end,
3:15 we were at 92.48 So how much did our little bit of code take to run?
3:19 It took 83 megabytes. As we go and improve it over this
3:28 what I'm gonna call "naive mode" or just the most straightforward mode,
3:31 you'll see that maybe we could do better in this.
3:33 That's the number that we gotta beat if it's going to be better.
3:36 Now, I did say I was gonna write a little bit of funky code. And what
3:38 I wanted to write, Maybe
3:40 I'll actually put it here,
3:41 is this, because, you don't normally see semi-colons in Python,
3:45 but check this out. So what I didn't want to do is I kind of
3:49 wanted to just keep the same little pattern,
3:51 but make it report at the end of each step.
3:52 So I'm gonna go and leave it like this,
3:55 right? The report is just here for us to kind of know what's going on.
4:00 I don't really want to change this pattern.
4:01 I just want to look really simple and clear
4:02 and just these three lines, these three lines are actually the problem
4:05 and where the solution lies.