Python for .NET Developers Transcripts
Chapter: Memory management in Python
Lecture: Reference counting demo
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
It's fun to talk about memory management in the abstract but let's actually write a little program that explores it.
0:08
So here I've got a new little section and I wrote create a file called mem_explorer mostly so I could drop in this beauty right here.
0:16
But we got this utility that will let us reach down into Python's internals and actually let us use the C API to actually ask how many references
0:27
does this particular object have to it at the moment. Right, everything in Python down in the C level is one of these. It's a PyObject pointer.
0:38
But what we're doing is we're just getting it back and it has a reference count on it. We can just say, hey, how many things are referring to you?
0:43
Remember, if that equals 0 boom, this object is cleaned up. But we're going to use this little memutil thing to work on playing around with some ideas.
0:53
And we also got another class over here that it gives us more diagnostic information. But I'm going to start out with some primitive types
0:59
then we'll bring this other thing in. So we'll start with our standard layout here. We're going to have two versions.
1:05
So I'm going to have a ref counting version and we're going to have a GC version. I'll just call it gcing, I guess and let's just
1:13
define that to do nothing for the minute. And then we're going to work on this one. We'll come back later and work on the GC story.
1:21
Remember, that's only for cycles in this world. So let's go over here and create a variable and it's going to be the super fun little string like this.
1:32
And then we can get information about it. I can get the id from our memutil. Why do I want to get the id from it?
1:40
Because if I actually hold on to a reference to it that will keep it alive, and what I want to demonstrate to you is under what circumstances will it
1:49
automatically clean itself up deterministically. But, obviously, if I hang on to it I'm going to have a problem.
1:56
So I'm going to get, basically, the memory address of it and then I can ask, hey, the thing that used to be
2:01
at that memory address, is it alive or is it not alive? Okay, so we're going to create this; and then I want to just print out, we'll say step one.
2:11
Now we'll go to the F string; I'll say ref out out not going to put the variable there, again we'll hang on to it.
2:19
So I'm going to say, we can go over to our memutil and we can get refs to v1_id. Alright, well, let's just go and run that
2:27
and while we're here, tell PyCharm that's not a problem. Oh, whoops, what did I put, not there; what did I put here?
2:35
Oh, I want to, I called that a little bit earlier. I'm going to, there's a id built in that will give us the id
2:39
and then we're going to pass it here, okay. So we're going to get a hold of this id and we're going to see if it's alive.
2:44
There we go; right now there are three reference counts to this object. Let's go over here and say v2 is equal to v1.
2:54
This should increase the reference count but won't accept you. Alright, we run it again. So, step one the ref counts is three;
3:02
and then now, it's four. Why? Because we have a new variable, v2 pointing over to this in memory. Remember we got that, and we now got this
3:13
pointing up to it as well. Now let's go over here and see if we can start to roll this back a little bit.
3:21
I guess we don't really need the id to it, right? That should be the same. Actually, it would be exactly the same.
3:26
So, in step three down here, what I want to do is I want to basically make that point to somewhere else. It could go to other, or it could go to none.
3:36
Alright, I'll just leave it as other it doesn't really matter. But now, the reference count for v1 should go down a bit.
3:43
There we go, we have one fewer there. And let's do this one other time. Let's go to v1, put that to None, this will be step four.
3:56
Alright, we'll see what's referring to it now. To, okay, well maybe a string is not really the best object. Let's put, I think it's interning.
4:05
Let's put some, some data structure like this. Here we go that's a little bit better. So, we come in, we have some object we're allocating
4:13
on the heap is one, have two of them over here, right this one; even though PyCharm is telling us, hey this doesn't do anything, it is doing something
4:22
it's just not doing anything conceptually productive in a programming language. But it, this new variable now points at this list
4:29
and v1 also points at the list, that's two. Down here we're making v2, maybe we could just make it a list of one, like, another list
4:39
just so it's more consistent. So, v2's no longer pointing at the original list it's pointing at this list, but it should still be the same.
4:46
Now that makes one. And then once we tell v1 to stop pointing as well well, there's only two. There's v1 and v2 pointing at it, and now it's garbage.
4:54
It should be gone, okay. Notice, let's, well, you can't really tell if it's finished, right? Let's print end of method, and let's go flush equals true
5:05
just to make sure that there's nothing going on here. All right, so end of method how do we know whether this was deleted?
5:12
Well, take Python's word; it's reference count of 0 so it's probably deleted. But it would be nicer to know, yes this thing actually got deleted.
5:20
So let's go over here and change this. I'm going to go import, let's say from doomed import doomed. And doomed is a class.
5:29
We can go down here and we can allocate a doomed. The cool thing about doomed, I'll have to go look at it is that doomed allows you to pass
5:39
other objects it holds onto but, most importantly it tells you, hey, I was created. Hey, I was the, the delete thing was called to me.
5:48
Very basically, it's like it's finalized, right. The garbage collector is cleaning me up now and then it also has some nice string representation.
5:57
So let's use this version and run it one more time, okay? So we'll have a doomed, and then we'll just yeah, leave it like that, as None; okay?
6:07
So we're going to run it; it's going to be the same thing but the doomed object will describe its life cycle to us. Alright, so run this.
6:15
Also, before we could assign it to variable one it created itself and then we ask hey, how many things point to you. One, v1, points to it.
6:24
Later down here, we added v2 to point at v1 then we said, now how many things point at you; two. Then we set v2 to nothing, so now
6:33
that rolled it back to just v1. Ask here how many things pointed to you. On line 22 right here, when we said, well
6:42
we don't want anything point at anymore, reference count hits zero before we could even print the statement, right.
6:48
Before line 23, when that went to zero, what happened? Boom, finalizer effectively was called. That the object was deleted
6:56
removed from memory, boom, it's gone. And then we ask, oh, by the way, now that it's gone how many things point at it? 0, right?
7:03
We just couldn't report at the same time that it's being deleted; but like the very next thing once we realize it's zero, like, this makes it zero
7:11
boom, it's gone and then, yeah we confirmed that right there. And that's the end of the method. But there's no garbage collection happening at all.
7:18
In fact, we can even just like .net, we could import gc and we can come over here and say gc.disable. If you thought maybe garbage collection
7:28
had something to do with this. Run again, yeah, no, it's exactly the same because nothing here had anything to do with the garbage collection.
7:37
It's just reference counting. You add a pointer to it, it goes up. You remove a pointer, it goes down.
7:43
That pointer hits zero, boom, it's deleted, right. So it's very very deterministic, right. You can see it's exactly at every step
7:50
happening just like you would expect. You would not see that in C#. You would have to do things like explicitly
7:56
call gc.collect each step, or like trigger the GC to run because it's non-deterministic here reference counting, very deterministic.
8:04
This is Python's primary memory management model.