Python Memory Management and Tips Transcripts
Chapter: Allocating memory in Python
Lecture: Allocator stats
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
Well, with all this abstract talk about blocks and pools and arenas can be a little
0:05
bit hard to understand. So, let's just actually go and see what Python will tell us about all of this stuff and the Small Object Allocator.
0:13
So, go here and create an "app_something", Remember? That means you can run it
0:19
and I'll just call it "stats". Real simple, we'll just do our fmain magic once again, we're gonna need to use sys, so there's our sys, Perfect.
0:31
And let's you put a little message to see what this is about. Great, so over here, we can just run "sys.", It's not great, kind of this semi-hidden,
0:42
but you can still get it.. "debugmallocstats()", like so. Let's go ahead and run this, see what we get here.
0:53
Notice it didn't come up in our auto-complete, Right? "_debug", no, not there. But, we tell PyCharm not to obscure it and not to tell
1:05
us it's not there. Hey, We can see that it looks like it's okay. Alright, so let's look and see what we got out of this.
1:10
Check this out. Run it once from scratch. If we get to the top, what do we got? We have this "small block threshold".
1:18
Remember, things that are managed by the Small Object Allocator, they are how big? 512 bytes or less. And there are 32 different size classes.
1:30
So, 16, 32, 48, 64, 80 and so on, is apparently what we got right now.
1:36
There's how many pools of each size of those are allocated and how many blocks are in use within those pools? so like 5 pools,
1:46
we've got 558 blocks used within those 5 pools, and we don't need to go allocate more until we've used up 72 more of them.
1:57
And of course, all of that is only for 32 byte elements. All right, so let's scroll through this.
2:02
This just gives you all the information about the blocks, their sizes, how many pools go with them and whatnot.
2:09
And then here's our arenas. How many arenas have we allocated? 12. So that would be 12*256, Should be 3MB areas reclaimed. 3, The max we've had is 9.
2:23
Currently, we have 9. And then it tells you that this much memory is being used right now. You can go through and you can see a lot about it
2:32
here, Right? Then we come down here and it actually tells you what is allocated. So we have "PyCFunctionObjects",
2:39
we have 56 bytes each. We have 5 of those, so that's cool and so on. Right. So You can see all the different objects: tuples,
2:50
stack frames, floats, dictionaries and whatnot being used here. So you get a lot of operations.
2:57
Let's go and make it do something. So I'll just come down here and say "make it do memory things". So I'm just gonna copy some code.
3:07
What we're gonna do is we're gonna create a bunch of larger numbers,
3:10
so that's going to make sure they get outside the flyweight pattern and they're gonna actually
3:14
get allocated every time. And then we're gonna create a list, then we're gonna put a list which has the string repeated 100 times of whatever this
3:26
large number is that we get here. So we're creating a bunch of objects. Basically, we're doing a bunch of things here.
3:44
Alright, let's try this one more time and see if it did anything different this time. Scroll, scroll, scroll, scroll, where is my separator?
3:53
It didn't flush quick enough, did it? Alright, so we gotta go through the first one, right? The first time we see 512,
4:01
there we go. That's the second one here. So you would see that there's potentially more of these in use. Not a whole lot more. You can see,
4:11
I think more frame objects were in use here and so on. Let's look one more time. Yeah, the data is still there. Make it a lot bigger.
4:23
Here we go, it's slow. It's doing stuff. Okay, great. Now you can see more arenas were allocated. We had 62, the highwater mark was 34 and we're still
4:34
there, were using 8.9MB of RAM and so on. Then we can see our list object and all these different things.
4:41
Okay, we have more of these blocks in use than we had before. So the 64 one's, 1000, 700, and before we had only 138.
4:50
So you can see as we do more work, It's consuming more memory, it's getting allocated into the different spaces,
4:56
and so on. Not sure how much meaningful information you're going to get from, like, actionable stuff you can do here.
5:03
But I definitely think it helps understand some of these basic ideas that we've been talking
5:07
about and gives you a look at the current state of the system, and that's pretty cool.