Python Memory Management and Tips Transcripts
Chapter: Investigating memory usage
Lecture: A data science-focused profiler
0:00 I think memory profiler does a pretty good job,
0:03 but sometimes you're using certain libraries or other things that maybe it's not going to do
0:08 a great job of capturing information about, and so we can go and use another one
0:13 that's really built for data science use cases.
0:16 What do I mean by that?
0:17 Its job is to find precise places where memory hits its peak.
0:24 And if you're doing data science,
0:25 you want to run a script and you're gonna say,
0:27 Well, we're gonna need however much memory,
0:28 the top of that graph basically is going to require, and notice when
0:32 we were doing the graph it was doing sampling rather than actually hooking into the allocation mechanisms being
0:38 used. We're gonna use this thing called "Fil", and Fil was meant to basically do
0:44 memory profiling for these scripts that run a few things,
0:48 have a few peak memory usage moments and then go away.
0:52 Also, it tracks allocation from many of the places where data science libraries might allocate
0:59 data. So normal python code, that's cool,
1:01 but also C code with malloc, as well a C++ with new, and Fortran
1:07 90. It'll also keep track of that.
1:11 So a lot of these base libraries used in data science are based on these other
1:15 languages, and you're just controlling or orchestrating them with Python.
1:18 So here's a cool way to get some analysis on that.
1:21 We're going to do basically the same analysis we did,
1:24 but with this tool instead and see what we get.