MongoDB for Developers with Python Transcripts
Chapter: High-performance MongoDB
Lecture: Concept: Document design for performance
Login or
purchase this course
to watch this video and the rest of the course contents.
0:01
One of the most important things you can do for performance in your database and these document databases is think about your document design,
0:09
should you embed stuff, should you not, what embeds where, do you embed just ids, do you embed the whole thing;
0:15
all of these are really important questions and it takes a little bit of experience to know what the right thing to do is.
0:21
It also really depends on your application's use case, so something that's really obviously a thing we should consider
0:29
is this service history thing, this adds the most weight to these car objects, so we've got this embedded document list field
0:39
so how often do we need these histories? How many histories might a car have? Should those maybe be in a separate collection
0:50
where it has all the stuff that service record, the class has, plus car id, or something to that effect? So this is a really important question,
1:00
and it really depends on how we're using this car object, this car document if almost all the time we want to work with the service history,
1:08
it's probably good to go in and put it here, unless these can be really large or something to that effect,
1:14
but if you don't need them often, you'll consider putting them in their own collection, there's just a tension between complexity and separation,
1:21
safety and separation, speed of having them in separate so you don't pull them back all the time;
1:27
you can also consider using the only keyword or only operator in MongoEngine to say if I don't need it, exclude the service history,
1:35
it adds a little bit of complexity because you often know, hey is this the car that came with service history
1:41
or is it a car where that was excluded, things like that, but you could use performance profiling and tuning to figure out where you might use only.
1:49
Let's look at one more thing around document design. You want to consider the size of the document,
1:53
remember MongoDB has a limit on how large these documents can be, that's 16 MB per record, that doesn't mean you should think
2:02
oh it's only 10 MB so everything is fine for my document design, that might be terrible this is like a hard upper bound,
2:08
like the database stops working after it hits 16 MB, so you really want to think about what is the right size, so let's look at a couple examples:
2:17
we can go to any collection and say .stats and it will talk about the size of the documents and things like that,
2:22
so here we ran db.cars.stats in MongoEngine, and we see that the average object size is about 700 bytes,
2:30
there is information about how many there are, and all that kind of stuff, but really the most interesting thing for this discussion is
2:36
what is the average object size, 700 bytes that seems like a pretty good size to me, it's not huge by any means,
2:43
and this is the cars that contain those service histories, so this is probably fine for what we're doing. Let me give you a more realistic example.
2:51
Let's think about the Talk Python Training website, and the courses and chapters, we talked about them before,
2:57
so here if we run that same thing, db.courses.stats you can see that the average object size is 900 bytes for a course,
3:08
and remember the course has the description that shows on the page and that's probably most the size, it has a few other things as well,
3:14
like student testimonials and whatnot, but basically it's the description and a few hyperlinks.
3:20
So I think this is again a totally good object, average object size. Now one of the considerations was I could have taken the chapters
3:28
which themselves contain all the lectures, and embedded those within the course, would that have been a good idea—
3:35
I think I might have even had it created that way in the very beginning, and it was a lot slower than I was hoping for, so I redesigned the documents.
3:41
If we run this on this chapter section, you can see that the average object size is 2.3 KB,
3:47
this is starting to get a little bit big, on its own it's fine, but think about the fact that a course on average has like 10 to 20 chapters,
3:56
so if I embedded the chapters in the course instead of putting them to a separate document like I do,
4:03
this is how it actually runs at the time of the recording, then it would be something like these courses would be
4:08
24 up to maybe 50 KB of data per entry, think about that you go to like the courses page and it shows you a big list of all the courses
4:18
and there might be 10 or later 20 courses, we're pulling back and deserializing like megabytes of data
4:25
to render a really, really common page, that is probably not ok, so this is why I did not embed the chapters and lectures inside the course,
4:35
I just said okay, this is the breaking point I looked at the objects' size I looked at where the performance was
4:42
and I said you know what, really it's not that common that we actually want more than one chapter at a time,
4:47
but it is common we want lectures, so it's probably the right partitioning, but you build it one way, you try it, it doesn't work,
4:54
you just redesign your class structure, recreate the database and try it again, but you do want to think about the average object size
5:01
and you can do it super easy with db.colection name.stats.