Python for the .NET developer Transcripts
Chapter: Memory management in Python
Lecture: Reference counting demo
0:00 It's fun to talk about memory management in the abstract
0:02 but let's actually write a little program that explores it.
0:07 So here I've got a new little section and I wrote
0:09 create a file called mem_explorer
0:12 mostly so I could drop in this beauty right here.
0:15 But we got this utility that will let us reach down
0:18 into Python's internals and actually let us use
0:22 the C API to actually ask how many references
0:26 does this particular object have to it at the moment.
0:30 Right, everything in Python
0:33 down in the C level is one of these.
0:34 It's a PyObject pointer.
0:37 But what we're doing is we're just getting it back
0:38 and it has a reference count on it.
0:39 We can just say, hey, how many things are referring to you?
0:42 Remember, if that equals 0
0:44 boom, this object is cleaned up.
0:47 But we're going to use this little memutil thing
0:49 to work on playing around with some ideas.
0:52 And we also got another class over here that
0:54 it gives us more diagnostic information.
0:56 But I'm going to start out with some primitive types
0:58 then we'll bring this other thing in.
1:00 So we'll start with our standard layout here.
1:03 We're going to have two versions.
1:04 So I'm going to have a ref counting version
1:06 and we're going to have a GC version.
1:09 I'll just call it gcing, I guess and let's just
1:12 define that to do nothing for the minute.
1:16 And then we're going to work on this one.
1:17 We'll come back later and work on the GC story.
1:20 Remember, that's only for cycles in this world.
1:22 So let's go over here and create a variable
1:24 and it's going to be
1:28 the super fun little string like this.
1:31 And then we can get information about it.
1:33 I can get the id from our memutil.
1:36 Why do I want to get the id from it?
1:39 Because if I actually hold on to a reference to it
1:43 that will keep it alive, and what I want to demonstrate
1:45 to you is under what circumstances will it
1:48 automatically clean itself up deterministically.
1:52 But, obviously, if I hang on to it
1:54 I'm going to have a problem.
1:55 So I'm going to get, basically, the memory address of it
1:57 and then I can ask, hey, the thing that used to be
2:00 at that memory address, is it alive or is it not alive?
2:03 Okay, so we're going to create this;
2:06 and then I want to just print out, we'll say step one.
2:10 Now we'll go to the F string; I'll say ref out out
2:16 not going to put the variable there, again we'll hang on to it.
2:18 So I'm going to say, we can go over to our memutil
2:20 and we can get refs to v1_id.
2:24 Alright, well, let's just go and run that
2:26 and while we're here, tell PyCharm that's not a problem.
2:29 Oh, whoops, what did I put, not there; what did I put here?
2:34 Oh, I want to, I called that a little bit earlier.
2:36 I'm going to, there's a id built in that will give us the id
2:38 and then we're going to pass it here, okay.
2:40 So we're going to get a hold of this id
2:41 and we're going to see if it's alive.
2:43 There we go; right now there are three
2:46 reference counts to this object.
2:49 Let's go over here and say v2 is equal to v1.
2:53 This should increase the reference count
2:56 but won't accept you. Alright, we run it again.
2:59 So, step one the ref counts is three;
3:01 and then now, it's four. Why?
3:03 Because we have a new variable, v2
3:06 pointing over to this in memory.
3:09 Remember we got that, and we now got this
3:12 pointing up to it as well.
3:14 Now let's go over here and see if we can
3:16 start to roll this back a little bit.
3:20 I guess we don't really need the id to it, right?
3:22 That should be the same.
3:24 Actually, it would be exactly the same.
3:25 So, in step three down here, what I want to do is
3:28 I want to basically make that point to somewhere else.
3:31 It could go to other, or it could go to none.
3:35 Alright, I'll just leave it as other
3:36 it doesn't really matter.
3:38 But now, the reference count for v1 should go down a bit.
3:42 There we go, we have one fewer there.
3:46 And let's do this one other time.
3:48 Let's go to v1, put that to None, this will be step four.
3:55 Alright, we'll see what's referring to it now.
3:58 To, okay, well maybe a string is not really the best object.
4:01 Let's put, I think it's interning.
4:04 Let's put some, some data structure like this.
4:07 Here we go that's a little bit better.
4:09 So, we come in, we have some object we're allocating
4:12 on the heap is one, have two of them over here, right
4:16 this one; even though PyCharm is telling us, hey
4:18 this doesn't do anything, it is doing something
4:21 it's just not doing anything conceptually
4:23 productive in a programming language.
4:25 But it, this new variable now points at this list
4:28 and v1 also points at the list, that's two.
4:31 Down here we're making v2, maybe we could just
4:34 make it a list of one, like, another list
4:38 just so it's more consistent.
4:39 So, v2's no longer pointing at the original list
4:41 it's pointing at this list, but it should still be the same.
4:45 Now that makes one.
4:46 And then once we tell v1 to stop pointing as well
4:49 well, there's only two.
4:50 There's v1 and v2 pointing at it, and now it's garbage.
4:53 It should be gone, okay.
4:56 Notice, let's, well, you can't really
4:58 tell if it's finished, right?
5:00 Let's print end of method, and let's go flush equals true
5:04 just to make sure that there's nothing going on here.
5:08 All right, so end of method
5:09 how do we know whether this was deleted?
5:11 Well, take Python's word; it's reference count of 0
5:14 so it's probably deleted.
5:15 But it would be nicer to know, yes
5:17 this thing actually got deleted.
5:19 So let's go over here and change this.
5:21 I'm going to go import, let's say from doomed import doomed.
5:27 And doomed is a class.
5:28 We can go down here and we can allocate a doomed.
5:31 The cool thing about doomed, I'll have to go look at it
5:35 is that doomed allows you to pass
5:38 other objects it holds onto but, most importantly
5:41 it tells you, hey, I was created.
5:44 Hey, I was the, the delete thing was called to me.
5:47 Very basically, it's like it's finalized, right.
5:49 The garbage collector is cleaning me up now
5:53 and then it also has some nice string representation.
5:56 So let's use this version and run it one more time, okay?
6:00 So we'll have a doomed, and then we'll just
6:03 yeah, leave it like that, as None; okay?
6:06 So we're going to run it; it's going to be the same thing
6:08 but the doomed object will describe its life cycle to us.
6:12 Alright, so run this.
6:14 Also, before we could assign it to variable one
6:17 it created itself and then we ask
6:19 hey, how many things point to you.
6:21 One, v1, points to it.
6:23 Later down here, we added v2 to point at v1
6:27 then we said, now how many things point at you; two.
6:30 Then we set v2 to nothing, so now
6:32 that rolled it back to just v1.
6:35 Ask here how many things pointed to you.
6:37 On line 22 right here, when we said, well
6:41 we don't want anything point at anymore, reference count
6:43 hits zero before we could even print the statement, right.
6:47 Before line 23, when that went to zero, what happened?
6:50 Boom, finalizer effectively was called.
6:54 That the object was deleted
6:55 removed from memory, boom, it's gone.
6:58 And then we ask, oh, by the way, now that it's gone
7:00 how many things point at it? 0, right?
7:02 We just couldn't report at the same time
7:04 that it's being deleted; but like the very next thing
7:07 once we realize it's zero, like, this makes it zero
7:10 boom, it's gone and then, yeah
7:11 we confirmed that right there.
7:13 And that's the end of the method.
7:14 But there's no garbage collection happening at all.
7:17 In fact, we can even just like .net, we could import gc
7:23 and we can come over here and say gc.disable.
7:26 If you thought maybe garbage collection
7:27 had something to do with this.
7:30 Run again, yeah, no, it's exactly the same
7:33 because nothing here had anything
7:35 to do with the garbage collection.
7:36 It's just reference counting.
7:39 You add a pointer to it, it goes up.
7:41 You remove a pointer, it goes down.
7:42 That pointer hits zero, boom, it's deleted, right.
7:44 So it's very very deterministic, right.
7:46 You can see it's exactly at every step
7:49 happening just like you would expect.
7:51 You would not see that in C#.
7:53 You would have to do things like explicitly
7:55 call gc.collect each step, or like trigger the GC
7:58 to run because it's non-deterministic here
8:01 reference counting, very deterministic.
8:03 This is Python's primary memory management model.