MongoDB for Developers with Python Transcripts
Chapter: High-performance MongoDB
Lecture: Running the default configuration
Login or
purchase this course
to watch this video and the rest of the course contents.
0:01
Let's go ahead and run this code, you've seen the minor changes like the addition of this concept of an owner,
0:07
and how we generated all this data, and how you can restore it. Let's go ahead and run it, and see what's happening.
0:14
Let's look at this from two perspectives, let's begin over actually in Robomongo,
0:18
so we're going to ask the question, basically how many owners own a certain car
0:22
the idea is more or less we're going to call this function which goes right here, really what we're looking for is this query,
0:29
find me all of the owners where this car id is in their car ids collection, just generate and deserialize that.
0:38
The other one that we're going to focus on is show me the cars with the expensive service history, how many cars or what cars had some kind of service
0:47
that cost over 16800 dollars. Let's begin by looking at those in Robomongo.
0:55
Here we have this concept, we could simplify this a little bit, but it doesn't matter, cars here's the service history, let's go to the price
1:01
where that's greater than 16800, how many of them are there. If I run this, notice, it took a while to come back,
1:09
run it again, here's the speed right there, 0.724 sec, 0.731, 0.733, so it's pretty reliably taking around 700 milliseconds to answer that question.
1:20
We're going to come back to this. Here's a more interesting example, like go and randomly grab a car
1:26
somewhere deep in the list, in this case I put 61600, grab that car and then find me all the owners,
1:34
where that car id appears in their id list, and then we'll just dump that out,
1:39
by saying var it doesn't appear if you just state the name it will show up down here, so make sure to deselect it and run this,
1:46
and this is actually surprisingly fast, given all the stuff that's going on here, but it's taking still about 75, 80 milliseconds to run here,
1:54
which, I don't know, maybe in your database going across a 100 thousand records 80 milliseconds seems decent,
2:00
I can tell you in MongoDB 80 milliseconds is terrible you should really think about making something that's 80 milliseconds faster
2:07
it's not always possible you can do it, but most of the queries as we'll see are possible.
2:12
Let's take this one and just try to understand what's happening here and then we're going to go look at it in Python,
2:20
but let's just explore it here in the shell for just a moment. Why is this taking 700 milliseconds?
2:25
MongoDB has this way to basically ask how are you running this query, and the way you do that is you say explain, like so,
2:36
so I can say this query instead of giving me a result tell me how you're running it,
2:39
if I unselect it, it just runs the selected stuff if there's something there, so we can go and look at it in this mode,
2:45
so it says okay, here's what the query planner found for you, we've parsed this query, and this is something it's basically what went into the find,
2:53
it also might have something to the effect of like a sword and other things that are happening, but this is a simple query.
2:59
Look down here, see this winning plan, stage column scan, that is bad, that is really, really bad.
3:06
Also notice the rejected plan, so if there are multiple indexes and other things that could have done
3:11
it might have attempted a bunch of them and said no, no, no this is the best, let's see it doesn't seem to tell us any more about what it did there,
3:19
like sometimes it'll tell you how many records it scanned and things like this, but it's just basically reading entirely in the forward direction
3:26
over this and just doing a comparison. So that's why this was taking 700 milliseconds as it was literally reading and comparing 100 thousand entries
3:37
or actually more, remember their is 1.2 million search histories across those 250 thousand cars, so not 100 thousand,
3:44
1.2 million records it scanned over, that's bad, you don't want that. So what we can do is we can actually add an index,
3:52
now there's two ways to add an index, but before I add the index, let's go over here just explain is super, super valuable,
4:01
any time something is slow we're going to explain there's actually way to turn on profiling and say log all of the queries
4:08
that you see MongoDB that are slower than x, you providing them like say 10 milliseconds might be great,
4:15
show me all the queries that take more than 10 milliseconds and then you can drop them in here, put an explain
4:20
and then start creating indexes to make them faster. So just google mongodb profile enable slow queries
4:27
or something like this, it's pretty straightforward. Now let's run this code, we're asking a lot of questions
4:32
what we want to run is q and a, so we go over here and just right click and say run, notice some of these things are taking time,
4:43
the database might be cold, it might have not loaded that stuff, so let me run it one more time just to be fair,
4:50
there's a few things that are already really fast, and that's cool, so let's go here and review, how many owners are there—
4:59
well, I can tell you it doesn't show the answer it just sort of says this is the question I'm asking here is how long it takes.
5:05
Three milliseconds, that is solid, how many cars— half a millisecond.
5:08
That's pretty solid, I don't think we can improve the count on the entire collection but this one, find the 10 thousandth owner— not good,
5:15
so let's see how many cars are owned by that person— this is pretty fast actually, this is surprisingly fast,
5:24
how many owners this can have— 66 milliseconds that's the one we were looking at in there. I'm going to take these numbers and put them over here,
5:33
let's say, this will be Without indexes we're going to get this, we don't really care about the exit code, do we?
5:42
With indexes, and we're going to kind of iterate on this a little bit so let's begin over here, and we're going to talk about
5:50
how we can add an index in MongoDB and then for the most part do this in MongoEngine because it's really part of the way our application works,
6:01
what the indexes are, and it's better to make that part of our document then kind of do a separate database setup step;
6:08
we could create a script in Javascript and run it, it will do these things and that may be fine, but let's go over here and work on this.
6:15
Again we had the count, here's the almost 800 milliseconds, let's go over here and just I'll take this, I'll make a copy,
6:29
so here is what we can do, instead of doing the find operation we can say create index, and then we have the thing that we're doing the query on,
6:39
most the time this is one item but you can have composite indexes they are a little more nuance so we'll talk about them later,
6:46
but let's just do this one, we want to be able to query by service history's price Here we can put one of two things, one or minus one,
6:57
what do you want the default sort, descending or ascending? A lot of times it doesn't really matter,
7:02
it can read from the back or it can read from the front, whatever, you saw the forward direction on our column scan for example.
7:07
So over here we could say one, this creates an index, there's no count; the other thing we can do is we can give it a name
7:14
so we can come over here and say name is search by service history price, so if we go look in this little indexes, we'll see the name here,
7:28
we can also say run in the background, if I don't say that it's going to block the database until the index is generated,
7:34
if you're doing this in production, and you have tons and tons of data maybe background is the way to go.
7:39
Okay, anyway let's go ahead and run this and see what happens. Notice the pause, this is it's actually computing the index
7:45
right now the database is effectively down, now it's back, what do we get ok, we created collection automatically know it already existed
7:52
a number of indexes before was one, now we have two and everything was a ok so if I refresh,
8:00
here's that index and I can actually edit this over here in Robomongo, go for the advanced properties, here is the create index and background
8:10
whether it's sparse, how long it lives, whether it's based on text search or whatever, but here's just the basic thing.
8:19
We've added this index, remember this took 800 milliseconds ask the same question now, boom, 8 milliseconds.
8:25
Ask it one more time, 2, here we go, 2, 2, 2, 3, 2, 2, right, the screen sharing is probably put in a pretty heavy load on the server
8:33
that's also the database server, right but still, we're getting it down 350, 400 times faster by adding that.
8:40
Now if I go back and I ask that question explain now we get something way better, winning plan is index scan
8:51
index name search by service history price, that is really awesome; that means we're using our index which is so much faster.
9:03
There was no rejected plans, so it only found one index it tried to use it if found that it was awesome, it's very happy.
9:17
Go back to my account more time, boom 2 milliseconds, and that's a really good answer, let's go run our Python code and see what answers we get now,
9:28
that was already faster, let's go over here and load car name and ids with expensive prices and spark plugs,
9:39
20 milliseconds this is actually a pretty complicated query we'll get into cars with expensive service, 1.9 milliseconds.
9:48
This is exactly what we saw in Robomongo, so over here in MongoEngine, we're getting essentially the same results— how cool is that?
9:57
Very nice, we're going to go through and in Python from now on we're going to add the necessary index to start making these
10:06
almost all of these run super fast, all of them run fast some of them we can get incredibly fast, like one millisecond,
10:12
others not quite that fast, but we'll still do good on all of them.