Python Memory Management and Tips Transcripts
Chapter: Investigating memory usage
Lecture: Profiling with Fil
0:00 Alright, well let's use Fil. Again,
0:01 we're gonna have to have it installed and put it up,
0:05 well I'll leave it at the bottom.
0:06 So the way we do it is we say,
0:08 "filprofiler" here. Hit install. Wait for a moment.
0:13 Alright, that's all good.
0:14 And let's go ahead and tell PyCharm
0:16 it is not misspelled because you know what, it's not.
0:19 And let's cd, I'll show you a better way to get into that folder.
0:22 We can right click, say "open in terminal",
0:26 there we go. See, right there is our stuff.
0:29 So what we can do is we can say,
0:31 "fil-profile", like this,
0:35 the app that we want to run,
0:37 That's it. Let's run it and see what we get.
0:40 Oh, it looks like I forgot run.
0:41 That's right, "fil-profile run", okay,
0:45 now it's doing its magic and it says it's gonna open the browser when it's done.
0:48 Also, I commented out the "@profile" decorator that I'd put on from Memory
0:53 Profiler. We don't want those two things to fight or one to measure the other.
0:57 It's done, generating report. Now look,
1:02 it opens my Web browser, and what do we get? Cool,
1:05 it says "Fil Memory profiler". The command we ran was this,
1:08 so it would record, like, command line arguments and stuff like that.
1:12 And here's the main star of the show, let me make it full screen.
1:16 So what we get is what's called a "flame graph".
1:19 The width tells us how much memory is allocated as well as of the color,
1:25 and you can dive into it. So over here,
1:27 we're doing our main, and there's other stuff happening.
1:30 Let's dive into the main, and notice as I'm moving the mouse around,
1:35 you can see how many bytes were used,
1:37 how much of the total memory, the peak memory, was used.
1:40 Remember, this is all about measuring peak memory.
1:43 Go down here. This one 50%, on this line.
1:48 See, we've got our original load_data and you can see what, you know, what this function is
1:53 doing, right? It's loading data then it's creating the randoms which is going to do this
1:57 list comprehension which goes down into the guts of Python.
2:00 We could go back and we could dive into,
2:04 like, over here, this scaled part where we're actually doing the data scaling and
2:09 the filtering and you can see that took 26 megabytes,
2:12 roughly or 36% of the data.
2:14 Okay, so this is a super cool way
2:16 to go and explore. You can click back here.
2:19 You can, you know, if you're down here,
2:20 just hit reset, zoom, and it pulls it all back out.
2:23 So this, like there's more stuff
2:24 we could go down and see, but,
2:26 Not a lot happening,
2:27 just extra space. So this is a pretty interesting way to measure the memory, or explore the
2:33 memory usage of our application. And, like I said,
2:37 its goal is to be focused on data science,
2:39 script work flows, which a lot of times,
2:42 you know, those use a ton of data because you want to load up a
2:44 whole bunch of things and process it.