Python Memory Management and Tips Transcripts
Chapter: Memory and classes
Lecture: Where do classes store memory?
0:00 Let's do another little exploration here. To understand why classes take as much memory as they
0:06 do, we need to figure out where they store their data. and it's also setting
0:11 the stage for a very big improvement that we're gonna make in a minute,
0:15 but you need to understand this first.
0:16 So let's go and say "app_storage". Set that that to run. Do our fmain magic, and
0:26 up here, I'm actually gonna paste some code
0:28 that's not worth watching me type out.
0:30 But, we're gonna do a little trick so
0:31 we can import size utility. I'm gonna define a class called "thing".
0:35 Now, thing contains thing1 and thing2,
0:38 and it has a way to print out what things
0:41 it has. So a thing object with the things in here, and also like this.
0:46 So we're just going to use that. There's not a whole lot to it,
0:49 just think of this as a standard class that has two fields,
0:53 and it just knows how to talk about itself,
0:54 Okay? That's all there is that's happening here.
0:57 So let's go and create some things. We'll have a thing, which is a hat, and a
1:05 mat, you know, thing1 and
1:07 thing2, we could do this Dr.Seuss style,
1:09 I suppose. Hat and mat, dog and cat,
1:15 bat and bird, car and bike.
1:21 And let's just print out the things, we'll make a new line. There
1:26 we have thing object at some address with hat and
1:29 mat. Notice 20, and 4, different location, with dog and cat and so on.
1:34 These are our things. Pretty cool,
1:36 right? Nothing special there. Standard class stuff.
1:40 If you look at any of these things,
1:43 any class at all, it will have a "__dict__"
1:46 and it's, this is where Python fields,
1:50 when you say "self.something" here,
1:53 what that really does is it creates an entry.
1:55 This is equivalent to saying
1:57 "self.dictionary of thing1 equals t1", right?
2:03 Those, the lines above these two are the same. Put that down here so I can
2:11 keep it. Those two things mean the same thing.
2:15 Alright, so whenever you have an instance of a class,
2:18 you have the instance of a dictionary on top of all the other stuff that might
2:22 have to be tracked and whatnot.
2:24 So the cost in terms of memory of a class is a dictionary.
2:30 Now you might think like many languages,
2:32 the cost is that string and that string,
2:36 but there's actually two sides to that coin.
2:38 The cost is this string right here, thing, and that. The entry that goes there,
2:45 these are actually reused. So there's only one,
2:49 I guess there's not too much,
2:50 but there is a pointer, got a 64 bit system, so that's 8 bytes plus this thing
2:54 for every field. So that's pretty interesting.
2:58 And down here, this is gonna be one of those.
3:02 So let's actually look at all of them.
3:06 So we'll say "obj.__dict__, for obj in things".
3:13 And then let's just print out the dictionaries. And look at that. Here we have a
3:20 thing with hat and mat and here we have a dictionary, thing1
3:23 is hat, thing2 is mat. thing1
3:26 is dog, thing2 is cat, right?
3:28 There it is. Those are the fields
3:31 self.thing1 self.thing2.
3:35 So you might wonder, well how much size does this use? So we could print
3:39 out the size, we'll just get the first one of the dictionary.
3:45 How much size does that use?
3:48 And let's also do class, things zero. Class itself. Perfect. So we run those.
4:01 You can see the size of the dictionary.
4:04 Ah, we're using the wrong one.
4:06 Let's go and use our size
4:09 util, get full size of the object.
4:11 That's right. It's not traversing,
4:13 is it? There we go.
4:15 That makes more sense.
4:16 So the dictionary is 318 and the class is 366.
4:20 So it's another 48 Bytes added on here to
4:24 have a class, but really the understanding that the dictionary is the thing that really
4:29 holds most of the stuff to do with the class.
4:32 That's pretty interesting, right? So this is where it's stored. Now,
4:36 question is, are these different dictionaries?
4:38 Are they copies? What's the story?
4:41 So what we can actually do is we can go and look at the locations of
4:45 the dictionaries. We could say "this is going to be the
4:47 id of d, for d in dicts".
4:51 And we could just print those, print the locations.
4:55 And If we look at that, what this is telling us is these are the memory
4:58 locations of the dictionaries of the classes. And so the same class,
5:03 over and over and over again,
5:04 right? Those are different. So what that's telling you is there's every time you say
5:08 a new class, you get a new dictionary and that allocates some bits that are
5:12 required to be tracked and managed in memory for every single one of these. In particular,
5:18 as I said at the beginning,
5:20 that you've got to keep track of the entries,
5:22 right? You can see thing1, thing1, thing1, now the value of that string
5:28 is actually reused, but, the fact that it's appearing over and over,
5:32 you still have to keep track of that in the dictionary,
5:34 right? And that takes a lot of space, relatively. Over here,
5:37 we've got new dictionaries. You might wonder,
5:40 why is that required? So let's go over here and say things, go to the last
5:47 thing, I don't know which one it is,
5:49 but we're going to say that,
5:50 um, it was thing1 and thing2,
5:53 those are its two fields. Let's say we want to give it a
5:55 thing3. Python is a dynamic language,
5:58 and will let you just dynamically apply these things. Now,
6:00 PyCharm says "you're asking for a hurt here",
6:04 but you can tell PyCharm
6:05 "don't bother me, I want to do this" and then we can go and we
6:09 can print out all the things again.
6:11 Let's do, I'll just print the dictionaries.
6:18 So check this out. We have different ones, and notice thing1
6:21 thing2, thing1
6:24 thing2, and the next one,
6:25 thing1 thing2, but the third
6:27 one, thing1 thing2
6:29 thing3, right? Now there's a key
6:31 3, thing3 in the third one because we did this line 59 here,
6:36 but the others they were unmodified.
6:38 Because of this dynamic nature, you can just go to an object and go "bam,
6:41 You have more stuff than you knew about".
6:43 That means you have to have a dictionary that's dedicated to each class that tracks all
6:47 the things that not just came with it,
6:49 but also were dynamically added. That's really flexible,
6:52 but it also adds a lot of overhead.
6:54 Keep that in mind. We're going to see if we could trade off this flexibility
6:59 for much better performance. Hint, we can. The take away from
7:03 this is whenever you have a standard class,
7:05 the fields actually get stored like this into "__dict__" field name equals field value,
7:12 and when you create a class,
7:14 each one of them gets a separate,
7:16 dedicated dictionary that is its own thing that was created in memory and managed and allocated
7:22 and populated and taken care of like that.
7:24 So we, come over here and we look for the locations.
7:27 They're all different. And the reason that you have a unique one for each class
7:30 is to support this dynamism here,
7:33 right? Thing3 is not really an aspect of things,
7:36 that's why PyCharm, if I don't tell it to stop, complains unresolved attribute reference
7:41 thing3 for class thing, because you're kind of not treating this right.
7:45 But it will accept it, and some programs were written this way,
7:48 so Python needs to be flexible for classes like this.