Python Memory Management and Tips Transcripts
Chapter: Allocating memory in Python
Lecture: Small object allocation introduction
0:00 Let's look at one of the red pills in CPython around object allocation. So this is "obmalloc.c",
0:09 and if you look here, you can see here's the ASCII art part that I was
0:12 telling you about before. This is when the second take that we did on what allocation looks like. We have physical RAM,
0:19 we have virtual memory, then we have malloc, then we have PyMem, API, the allocator,
0:25 but there's something really interesting at the bottom that we didn't talk about then and check
0:29 that out. It says "a fast special purpose memory allocator for small blocks to be used on top of a general purpose malloc heavily based on prior art".
0:39 And if you want to go check this bit of the source code, this is literally straight out of GitHub, just go to bit.ly/pyobjectallocators,
0:47 and it will take you right to this line, and you can look through it. So what's the deal? To reduce the overhead for small objects,
0:54 that is, objects that are less than 512 bytes in the "sys.getobjectsize", not the whole traversal,
1:02 but the small bit, Python sub-allocates larger blocks of memory, and then, as you need it, will free up, or give, that memory to different objects,
1:13 which potentially could be reused once that object is cleaned up. Larger objects are just routed to the standard malloc.
1:21 But for these smaller ones, which are most of them, the Small Object Allocator uses these three levels of abstraction.
1:29 We've got arenas and pools and blocks. At the lowest level we've got this thing called a block, and then pools manage blocks and arenas manage pools.
1:38 So we're gonna go through all of those. But there's this trifecta of ideas or algorithms that we're going to use to manage,
1:46 remember, the small objects. And this little quote right here comes from an article about Python memory management by Artem Golubin,
1:55 and he's done some fantastic research and writing around it. So I recommend that you check out his blog. There'll be a couple of articles that
2:02 I think I refer to. Definitely, I've read as researching all the stuff for this course,
2:07 so check out his article here. It has a lot of interesting analysis on what's happening and why it's being done.