MongoDB for Developers with Python Transcripts
Chapter: High-performance MongoDB
Lecture: Running the default configuration
0:01 Let's go ahead and run this code, you've seen the minor changes like the addition of this concept of an owner,
0:07 and how we generated all this data, and how you can restore it. Let's go ahead and run it, and see what's happening.
0:14 Let's look at this from two perspectives, let's begin over actually in Robomongo,
0:18 so we're going to ask the question, basically how many owners own a certain car
0:22 the idea is more or less we're going to call this function which goes right here, really what we're looking for is this query,
0:29 find me all of the owners where this car id is in their car ids collection, just generate and deserialize that.
0:38 The other one that we're going to focus on is show me the cars with the expensive service history, how many cars or what cars had some kind of service
0:47 that cost over 16800 dollars. Let's begin by looking at those in Robomongo.
0:55 Here we have this concept, we could simplify this a little bit, but it doesn't matter, cars here's the service history, let's go to the price
1:01 where that's greater than 16800, how many of them are there. If I run this, notice, it took a while to come back,
1:09 run it again, here's the speed right there, 0.724 sec, 0.731, 0.733, so it's pretty reliably taking around 700 milliseconds to answer that question.
1:20 We're going to come back to this. Here's a more interesting example, like go and randomly grab a car
1:26 somewhere deep in the list, in this case I put 61600, grab that car and then find me all the owners,
1:34 where that car id appears in their id list, and then we'll just dump that out,
1:39 by saying var it doesn't appear if you just state the name it will show up down here, so make sure to deselect it and run this,
1:46 and this is actually surprisingly fast, given all the stuff that's going on here, but it's taking still about 75, 80 milliseconds to run here,
1:54 which, I don't know, maybe in your database going across a 100 thousand records 80 milliseconds seems decent,
2:00 I can tell you in MongoDB 80 milliseconds is terrible you should really think about making something that's 80 milliseconds faster
2:07 it's not always possible you can do it, but most of the queries as we'll see are possible.
2:12 Let's take this one and just try to understand what's happening here and then we're going to go look at it in Python,
2:20 but let's just explore it here in the shell for just a moment. Why is this taking 700 milliseconds?
2:25 MongoDB has this way to basically ask how are you running this query, and the way you do that is you say explain, like so,
2:36 so I can say this query instead of giving me a result tell me how you're running it,
2:39 if I unselect it, it just runs the selected stuff if there's something there, so we can go and look at it in this mode,
2:45 so it says okay, here's what the query planner found for you, we've parsed this query, and this is something it's basically what went into the find,
2:53 it also might have something to the effect of like a sword and other things that are happening, but this is a simple query.
2:59 Look down here, see this winning plan, stage column scan, that is bad, that is really, really bad.
3:06 Also notice the rejected plan, so if there are multiple indexes and other things that could have done
3:11 it might have attempted a bunch of them and said no, no, no this is the best, let's see it doesn't seem to tell us any more about what it did there,
3:19 like sometimes it'll tell you how many records it scanned and things like this, but it's just basically reading entirely in the forward direction
3:26 over this and just doing a comparison. So that's why this was taking 700 milliseconds as it was literally reading and comparing 100 thousand entries
3:37 or actually more, remember their is 1.2 million search histories across those 250 thousand cars, so not 100 thousand,
3:44 1.2 million records it scanned over, that's bad, you don't want that. So what we can do is we can actually add an index,
3:52 now there's two ways to add an index, but before I add the index, let's go over here just explain is super, super valuable,
4:01 any time something is slow we're going to explain there's actually way to turn on profiling and say log all of the queries
4:08 that you see MongoDB that are slower than x, you providing them like say 10 milliseconds might be great,
4:15 show me all the queries that take more than 10 milliseconds and then you can drop them in here, put an explain
4:20 and then start creating indexes to make them faster. So just google mongodb profile enable slow queries
4:27 or something like this, it's pretty straightforward. Now let's run this code, we're asking a lot of questions
4:32 what we want to run is q and a, so we go over here and just right click and say run, notice some of these things are taking time,
4:43 the database might be cold, it might have not loaded that stuff, so let me run it one more time just to be fair,
4:50 there's a few things that are already really fast, and that's cool, so let's go here and review, how many owners are there—
4:59 well, I can tell you it doesn't show the answer it just sort of says this is the question I'm asking here is how long it takes.
5:05 Three milliseconds, that is solid, how many cars— half a millisecond.
5:08 That's pretty solid, I don't think we can improve the count on the entire collection but this one, find the 10 thousandth owner— not good,
5:15 so let's see how many cars are owned by that person— this is pretty fast actually, this is surprisingly fast,
5:24 how many owners this can have— 66 milliseconds that's the one we were looking at in there. I'm going to take these numbers and put them over here,
5:33 let's say, this will be Without indexes we're going to get this, we don't really care about the exit code, do we?
5:42 With indexes, and we're going to kind of iterate on this a little bit so let's begin over here, and we're going to talk about
5:50 how we can add an index in MongoDB and then for the most part do this in MongoEngine because it's really part of the way our application works,
6:01 what the indexes are, and it's better to make that part of our document then kind of do a separate database setup step;
6:15 Again we had the count, here's the almost 800 milliseconds, let's go over here and just I'll take this, I'll make a copy,
6:29 so here is what we can do, instead of doing the find operation we can say create index, and then we have the thing that we're doing the query on,
6:39 most the time this is one item but you can have composite indexes they are a little more nuance so we'll talk about them later,
6:46 but let's just do this one, we want to be able to query by service history's price Here we can put one of two things, one or minus one,
6:57 what do you want the default sort, descending or ascending? A lot of times it doesn't really matter,
7:02 it can read from the back or it can read from the front, whatever, you saw the forward direction on our column scan for example.
7:07 So over here we could say one, this creates an index, there's no count; the other thing we can do is we can give it a name
7:14 so we can come over here and say name is search by service history price, so if we go look in this little indexes, we'll see the name here,
7:28 we can also say run in the background, if I don't say that it's going to block the database until the index is generated,
7:34 if you're doing this in production, and you have tons and tons of data maybe background is the way to go.
7:39 Okay, anyway let's go ahead and run this and see what happens. Notice the pause, this is it's actually computing the index
7:45 right now the database is effectively down, now it's back, what do we get ok, we created collection automatically know it already existed
7:52 a number of indexes before was one, now we have two and everything was a ok so if I refresh,
8:00 here's that index and I can actually edit this over here in Robomongo, go for the advanced properties, here is the create index and background
8:10 whether it's sparse, how long it lives, whether it's based on text search or whatever, but here's just the basic thing.
8:19 We've added this index, remember this took 800 milliseconds ask the same question now, boom, 8 milliseconds.
8:25 Ask it one more time, 2, here we go, 2, 2, 2, 3, 2, 2, right, the screen sharing is probably put in a pretty heavy load on the server
8:33 that's also the database server, right but still, we're getting it down 350, 400 times faster by adding that.
8:40 Now if I go back and I ask that question explain now we get something way better, winning plan is index scan
8:51 index name search by service history price, that is really awesome; that means we're using our index which is so much faster.
9:03 There was no rejected plans, so it only found one index it tried to use it if found that it was awesome, it's very happy.
9:17 Go back to my account more time, boom 2 milliseconds, and that's a really good answer, let's go run our Python code and see what answers we get now,
9:28 that was already faster, let's go over here and load car name and ids with expensive prices and spark plugs,
9:39 20 milliseconds this is actually a pretty complicated query we'll get into cars with expensive service, 1.9 milliseconds.
9:48 This is exactly what we saw in Robomongo, so over here in MongoEngine, we're getting essentially the same results— how cool is that?
9:57 Very nice, we're going to go through and in Python from now on we're going to add the necessary index to start making these
10:06 almost all of these run super fast, all of them run fast some of them we can get incredibly fast, like one millisecond,
10:12 others not quite that fast, but we'll still do good on all of them.