Python Memory Management and Tips Transcripts
Chapter: Allocating memory in Python
Lecture: Allocator stats
0:00 Well, with all this abstract talk about blocks and pools and arenas can be a little
0:04 bit hard to understand. So, let's just actually go and see what Python will tell
0:09 us about all of this stuff and the Small Object Allocator.
0:12 So, go here and create an "app_something",
0:16 Remember? That means you can run it
0:18 and I'll just call it "stats". Real simple, we'll just do our fmain magic
0:23 once again, we're gonna need to use sys,
0:28 so there's our sys, Perfect.
0:30 And let's you put a little message to see what this is about.
0:35 Great, so over here, we can just run "sys.",
0:38 It's not great, kind of this semi-hidden,
0:41 but you can still get it..
0:44 "debugmallocstats()", like so. Let's go ahead
0:48 and run this, see what we get here.
0:52 Notice it didn't come up in our auto-complete,
0:54 Right? "_debug", no,
0:56 not there. But, we tell PyCharm not to obscure it and not to tell
1:04 us it's not there. Hey,
1:05 We can see that it
1:06 looks like it's okay. Alright,
1:08 so let's look and see what we got out of this.
1:09 Check this out. Run it once from scratch.
1:13 If we get to the top,
1:14 what do we got? We have this "small block threshold".
1:17 Remember, things that are managed by the Small Object
1:21 Allocator, they are how big? 512 bytes or less.
1:25 And there are 32 different size classes.
1:29 So, 16, 32, 48, 64, 80
1:32 and so on, is apparently what we got right now.
1:35 There's how many pools of each size of those are allocated and how many blocks are
1:41 in use within those pools? so like 5 pools,
1:45 we've got 558 blocks used within those 5 pools,
1:50 and we don't need to go allocate more
1:53 until we've used up 72 more of them.
1:56 And of course, all of that is only for 32 byte elements.
2:00 All right, so let's scroll through this.
2:01 This just gives you all the information about the blocks,
2:04 their sizes, how many pools go with them and whatnot.
2:08 And then here's our arenas. How many arenas have we allocated? 12. So that would
2:13 be 12*256, Should be 3MB areas reclaimed.
2:19 3, The max we've had is 9.
2:22 Currently, we have 9. And then it tells you that this much memory is
2:27 being used right now. You can go through and you can see a lot about it
2:31 here, Right? Then we come down here and it actually tells you what is allocated.
2:35 So we have "PyCFunctionObjects",
2:38 we have 56 bytes each. We have 5 of those,
2:42 so that's cool and so on.
2:45 Right. So You can see all the different objects: tuples,
2:49 stack frames, floats, dictionaries and whatnot being used here.
2:53 So you get a lot of operations.
2:56 Let's go and make it do something. So I'll just come down here and say "make
3:00 it do memory things". So I'm just gonna copy some code.
3:06 What we're gonna do is we're gonna create a bunch of larger numbers,
3:09 so that's going to make sure they get outside the flyweight pattern and they're gonna actually
3:13 get allocated every time. And then we're gonna create a list,
3:19 then we're gonna put a list which has the string repeated 100 times of whatever this
3:25 large number is that we get here.
3:26 So we're creating a bunch of objects.
3:28 Basically, we're doing a bunch of things here.
3:43 Alright, let's try this one more time and see if it did anything different
3:47 this time. Scroll, scroll,
3:49 scroll, scroll, where is my separator?
3:52 It didn't flush quick enough, did it?
3:55 Alright, so we gotta go through the first one,
3:58 right? The first time we see 512,
4:00 there we go. That's the second one here.
4:03 So you would see that there's potentially more of these in use.
4:08 Not a whole lot more. You can see,
4:10 I think more frame objects were in use here and so on.
4:14 Let's look one more time. Yeah,
4:16 the data is still there. Make it a lot bigger.
4:22 Here we go, it's slow. It's doing stuff.
4:24 Okay, great. Now you can see more
4:28 arenas were allocated. We had 62, the highwater mark was 34 and we're still
4:33 there, were using 8.9MB of RAM and so on.
4:37 Then we can see our list object and all these different things.
4:40 Okay, we have more of these blocks in use than we had before.
4:44 So the 64 one's, 1000, 700, and before we had only 138.
4:49 So you can see as we do more work,
4:51 It's consuming more memory, it's getting allocated into the different spaces,
4:55 and so on. Not sure how much meaningful information you're going to get from,
4:59 like, actionable stuff you can do here.
5:02 But I definitely think it helps understand some of these basic ideas that we've been talking
5:06 about and gives you a look at the current state of the system, and that's pretty cool.