MongoDB for Developers with Python Transcripts
Chapter: High-performance MongoDB
Lecture: The MongoDBs performance knobs
0:01 You've heard MongoDB is fast, really fast,
0:03 and you've gone through setting up your documents and modeling things,
0:07 you inserted, you imported your data, and you're ready to go;
0:11 and you run a query and it comes back,
0:13 so okay, I want to find all the service histories
0:15 that have a certain price, greater than such and such, how many are there—
0:18 apparently there's 989, but it took almost a second to answer that question.
0:22 So this is a new version of the database, so we are going to talk about it shortly.
0:27 Instead of having just a handful of cars and service histories
0:30 that we maybe entered in our little play-around app,
0:32 it has a quarter million cars with a million service histories, something to that effect.
0:38 And the fact that we were able to answer this query
0:41 of how many sort of nested documents had this property
0:44 in less than a second, on one hand that's kind of impressive,
0:48 but to be honest, it feels like MongoDB is just dragging,
0:51 this is not very special, this is not great.
0:55 So this is what you get out of the box, if you just follow what we've done so far
0:59 this is how MongoDB is going to perform.
1:02 However, in this chapter, we're going to make this better, a lot better.
1:06 How much— well, let's see, we're going to make it fast,
1:09 here's that same query after applying just some of the techniques of this chapter.
1:13 Notice now it runs in one millisecond, not 706 milliseconds.
1:17 So we've made our MongoDB just take off,
1:21 it's running over 700 times faster than what the default MongoDB does.
1:26 Well, how do we do it, how do we make this fast?
1:30 Let's have a look at the various knobs
1:33 that we can turn to control MongoDB performance.
1:35 Some of which we're going to cover in this course,
1:38 and some are well beyond the scope of what we're doing,
1:40 but it's still great to know about them.
1:42 The first knob are indexes, so it turns out
1:44 that there are not too many indexes added to MongoDB by default,
1:48 in fact, the only index that gets set up is on _id
1:52 which is basically an index as well as a uniqueness constraint,
1:55 but other than that, there are no indexes,
1:57 and it might be a little non intuitive at first, when you first hear about this,
2:02 but indexes and manually tuning and tweaking and understanding the indexes
2:06 in document databases is far more important
2:10 than understanding indexes in a third normal form designed relational database.
2:15 So why would that be? That seems really odd.
2:18 So think about a third normal form database,
2:21 you've broken everything up into little tiny tables that link back to each other
2:24 and they often have foreign key constraints traversing all of these relationships,
2:28 well, those foreign key constraints go back to primary keys on the main tables,
2:33 those are indexed, every time you have one of those relationships
2:35 it usually at least on one end has an index on that thing.
2:39 In document databases, because we take some of those external tables
2:43 and we embed them in documents,
2:45 those subdocuments while they kind of logically play the same role
2:49 there is no concept of an index being added to those.
2:52 So we have fewer tables, but we still have
2:55 basically the same amount of relationships
2:57 and because of the way documents work,
2:59 we actually have fewer indexes than we do in say a relational database.
3:04 So we're going to see that working with understanding
3:07 and basically exploring indexes is super, super important
3:09 and that's going to be the most important thing that we do.
3:12 In fact, the MongoDB folks, one of their things they do is
3:16 they sell like services, consulting and what not to help their customers
3:19 and you could hire them, say hey I got this big cluster and it's slow
3:24 can you help me make it faster— the single most dramatic thing that they do,
3:30 the thing that almost always is the problem is incorrect use of indexes.
3:34 So we're going to talk about how to use, discover and explore indexes for sure.
3:38 Next is document design, all that discussion about to embed or not to embed,
3:43 how should you relate documents, this is sort of the beginning of this conversation,
3:47 it turns out the document design has dramatic implications across the board
3:52 and we did talk quite a bit about this, but we'll touch on it again in this chapter.
3:56 Query style, how are you writing your queries,
4:01 is there a way that you could maybe restructure a query,
4:05 or ask the question differently and end up with
4:08 a more high performance query, maybe one example misses an index
4:12 and the other particular example uses a better index or something to this effect.
4:16 Projections and subsets are also something that we can control,
4:23 we saw that you could limit your set of returned responses
4:26 and this can be super helpful for performance;
4:29 you could write a query where it returns 5 MB of data
4:32 but if you restrict that to just the few fields that you actually care about
4:36 maybe its all K instead of 5 MB, it could be really dramatic,
4:40 depending on how large and nested your documents might be.
4:43 We're going to talk about how we can do this, especially from MongoEngine.
4:46 These are the knobs that we're going to turn in this course,
4:49 these are the things that will work even if you have a single individual database,
4:53 so you should always think about these things,
4:56 some of them happen on the database side, document design, indexes,
4:59 and the other, maybe is in your application interacting with the database, the other two,
5:04 but MongoDB being a NoSql database, allows for other types of interactions,
5:08 other configurations and network topologies and so on.
5:11 So, one of the things that it supports is something called replication,
5:14 now replication is largely responsible for redundancy and failover.
5:19 Instead of just having one server I could have three servers,
5:22 and they could work in triplicate, basically one is what's called the primary,
5:26 and you read and write from this database,
5:28 and the other two are just there ready to spring into action,
5:31 always getting themselves in sync with the primary,
5:34 and if one goes down, the other will spring in to be the primary
5:36 and they will sort of fix themselves as the what used to be the primary comes back.
5:40 There is no performance benefit from that at all.
5:43 However, there are ways to configure your connection to say
5:46 allow me to read not just from the primary one, but also from the secondary,
5:50 so you can configure a replication for a performance boost,
5:53 but mostly this is a durability thing.
5:55 The other type of network configuration you can do is what's called sharding.
5:59 This is where you take your data instead of putting all into one individual server,
6:02 you might spread this across 10 or 20 servers,
6:06 one 20th, hopefully, of evenly balanced,
6:09 across all of them, and then when you issue a query,
6:12 can either figure out where if it's based on the shard key,
6:15 which server to point that at and let that one
6:17 handle the query across the smaller set of data,
6:20 or if it's general like show me all the things with greater than this for the price,
6:23 it might need to fan that out to all 20 servers,
6:26 but it would run on parallel on 20 machines.
6:30 So sharding is all about speeding up performance,
6:32 especially write performance, but also queries as well,
6:35 so you can get tons of scalability out of sharding,
6:38 and you can even combine these like, when I said there is 20 shards,
6:41 each one of those could actually be a replica set,
6:43 so there is a lot of stuff you could do with network topology
6:46 and clustering and sharding and scaling and so on.
6:48 We're not turning those knobs in this course,
6:50 I'll show you how to make individual pieces fast,
6:52 the same idea applies to these replicas and shards,
6:54 just on a much grander scale if you want to go look at them.