Python Memory Management and Tips Transcripts
Chapter: Investigating memory usage
Lecture: A data science-focused profiler
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
I think memory profiler does a pretty good job,
0:03
but sometimes you're using certain libraries or other things that maybe it's not going to do
0:08
a great job of capturing information about, and so we can go and use another one
0:13
that's really built for data science use cases.
0:16
What do I mean by that?
0:17
Its job is to find precise places where memory hits its peak.
0:24
And if you're doing data science,
0:25
you want to run a script and you're gonna say,
0:27
Well, we're gonna need however much memory,
0:28
the top of that graph basically is going to require, and notice when
0:32
we were doing the graph it was doing sampling rather than actually hooking into the allocation mechanisms being
0:38
used. We're gonna use this thing called "Fil", and Fil was meant to basically do
0:44
memory profiling for these scripts that run a few things,
0:48
have a few peak memory usage moments and then go away.
0:52
Also, it tracks allocation from many of the places where data science libraries might allocate
0:59
data. So normal python code, that's cool,
1:01
but also C code with malloc, as well a C++ with new, and Fortran
1:07
90. It'll also keep track of that.
1:11
So a lot of these base libraries used in data science are based on these other
1:15
languages, and you're just controlling or orchestrating them with Python.
1:18
So here's a cool way to get some analysis on that.
1:21
We're going to do basically the same analysis we did,
1:24
but with this tool instead and see what we get.