Python Memory Management and Tips Transcripts
Chapter: Investigating memory usage
Lecture: A data science-focused profiler

Login or purchase this course to watch this video and the rest of the course contents.
0:00 I think memory profiler does a pretty good job, but sometimes you're using certain libraries or other things that maybe it's not going to do
0:09 a great job of capturing information about, and so we can go and use another one that's really built for data science use cases.
0:17 What do I mean by that? Its job is to find precise places where memory hits its peak. And if you're doing data science,
0:26 you want to run a script and you're gonna say, Well, we're gonna need however much memory,
0:29 the top of that graph basically is going to require, and notice when
0:33 we were doing the graph it was doing sampling rather than actually hooking into the allocation mechanisms being
0:39 used. We're gonna use this thing called "Fil", and Fil was meant to basically do memory profiling for these scripts that run a few things,
0:49 have a few peak memory usage moments and then go away. Also, it tracks allocation from many of the places where data science libraries might allocate
1:00 data. So normal Python code, that's cool, but also C code with malloc, as well a C++ with new, and Fortran 90. It'll also keep track of that.
1:12 So a lot of these base libraries used in data science are based on these other
1:16 languages, and you're just controlling or orchestrating them with Python. So here's a cool way to get some analysis on that.
1:22 We're going to do basically the same analysis we did, but with this tool instead and see what we get.

Talk Python's Mastodon Michael Kennedy's Mastodon