Async Techniques and Examples in Python Transcripts
Chapter: asyncio-based web frameworks
Lecture: A note about rate limiting with external services
0:00 What we're talking about
0:01 with this wrk benchmarking tool
0:03 is otherwise what's known as load testing.
0:05 Let's hit the server with as much traffic
0:09 and as many concurrent users as we can.
0:11 Just keep ramping that up until it fails.
0:14 Well, it turns out the way we've implemented this is
0:17 we're not talking to a local database
0:19 and then just sending responses.
0:21 We're basically orchestrating external services.
0:25 So by load testing our code
0:27 we're going to end up load testing those services.
0:30 Alright, the harder we hit our service
0:31 the harder we're going to concurrently hit those services.
0:36 Now either because those services can't take it
0:39 or because those service just have built in
0:42 rate limiting to not let you do that much processing
0:46 that many concurrent requests, they're going to fail.
0:49 And that is going to make it seem like
0:50 our app is failing but it's only because
0:52 we're hammering the external services
0:54 they either can't take it or don't like it
0:56 and then they fail on us. So that's a problem.
1:00 How do we do load testing with these external services
1:03 without getting into trouble by hitting them too hard?
1:06 Well, you may have noticed this measured latency
1:09 in seconds and if used cache data.
1:12 That by the way is set right here.
1:15 I'm not going to turn it on it doesn't matter right now
1:17 but you can turn it on when we
1:20 do the load testing if you want.
1:21 You're going to need to turn it on
1:23 or you're not going to get the right numbers more or less.
1:25 So how did I come up with these numbers
1:26 and what is this doing? So I ran
1:28 some simple repetitive tests against this
1:31 over and over and over
1:32 and figured out how long does that operation
1:35 right there take?
1:37 One time it took this long and one time it took that long
1:39 and you can see all of these numbers
1:41 that's how long it took for this to be done.
1:45 And I said okay well that's pretty representative
1:46 at least from the West coast of the US.
1:48 So lets just record those numbers
1:51 and for load testing purposes we'll be able to go
1:54 and actually just simulate those.
1:56 So if we say use cache data we're going to simulate
1:59 calling that thing.
2:01 We're going to randomly pick one of the real measured latencies
2:04 and then we're going to return some fake data
2:06 just something I grabbed somewhere near Portland.
2:09 I'm not sure exactly where that is.
2:10 So if you're doing any sort of load testing
2:13 you have to go into the config file you're using
2:16 and set use cache data true
2:18 or it's just going to freak out and you're going to end up
2:20 load testing datasciencetoolkit.org
2:23 which is not really what we want to do
2:25 please don't do that to them.
2:26 And it turns out it fails anyways so it
2:28 wrecks your test, not great.
2:30 If they're not explicitly rate limiting you
2:32 maybe what they should be doing is
2:33 also using asyncio and things like that.
2:36 But anyway, you'll see this in each one of the services.
2:40 So the lat long service has it
2:42 the sun service has just some hard coded sun data.
2:47 Exact same thing, here's the measured latencies.
2:49 If you turn on this it just uses the asyncio
2:52 friendly sleep and then to simulate the time
2:55 that it takes to return the fake data.
2:57 I want to make sure you're aware
2:58 that because we're depending on these external services
3:01 for load testing, you don't want to load test them
3:03 you want to simulate what it would be like in a real system.
3:07 Now granted you may not want to actually depend on them
3:09 if they can't take your load testing
3:11 in a real one but this is a toy course demo, right?
3:15 No harm here.