Building Data-Driven Web Apps with Pyramid and SQLAlchemy Transcripts
Chapter: Testing web apps
Lecture: Sitemaps and testing
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
You've heard of the Pareto principle in one form or another. It's often referred to as the 80/20 rule the idea that most of the value can be gotten
0:09
by doing a little bit of the work... 80% value, 20% work. That's good right? We can actually accomplish a lot of that benefit
0:17
by having a site map. Well, what's a site map? So, a site map is actually a listing of all of the URLs and a little bit of detail about them
0:27
targeted for search engines. Google, Search bots, Bing, things like that. Here's an example that might make sense for our...
0:36
And what we can do is we could define a Chameleon template that will generate this. So we can have one for the site, one for account/login
0:44
one for account/register and then every package on the site we could use data-driven behaviors to pull out all of that.
0:52
See that down there at the bottom tal:repeat p in packages, we're going to spit that out. You want to have one of these types of things anyway.
0:59
It's massively helpful for search engines. So we could have this template and our view model would be really simple.
1:05
It's just going to go and say "Give me all the packages." We may want to set a limit, probably not, more likely
1:10
you want to just cache this and refresh it periodically. Anyway, we want to generate this somehow and here's our controller, super easy.
1:19
We register sitemap.xml as the URL and than we just generate this stuff using that template. If we have this in place
1:27
and I'm going to give you one for this website we're not going to write it from scratch. I'll give it to you, you can adapt it for your own.
1:32
Then we can use this for testing. We can go through here and just request every single page in the sitemap and make sure that none of them 500
1:40
you know server crash or 404 or things like that. That's actually really helpful. This is the 80/20 rule.
1:47
We can just do a little bit of work to grab the site map and request everything. It won't test everything we're looking for
1:52
but like I said most of the time when these pages fail they fail hard and just fully crash 500, 404, something like that.
2:00
They don't just send back wrong HTML. That does happen, but they fully crash most of the time which is actually kind of good for testing.
2:10
So, let's see this just in the concept here and then we'll go look at it in code for just a moment. We're going to write a little bit of code
2:16
to get the sitemap text, so what we need to do is actually use our functional test behavior to request /sitemap.xml.
2:26
It's going to return some value, response. We can go to the text and we can say so the XPath queries are simpler let's get rid of the namespaces
2:35
replace this namespace that has to be there with nothing that means we don't have to worry about it in what we're about to do.
2:41
We can write a test that'll say I want to go and test every URL there, so get us that text. I want to create an XML DOM element, an etree.from_string
2:52
and in there we're going to do an XPath query for URL/location. This will pull through, pull it out and generate every single URL on our site.
3:03
Here where we are doing the replace localhost:6552 to say, I mean we would put that in our sitemap but we just want to do a relative request here
3:12
like /account and so on. And then, we can just now we have just the URLs as a flat string.
3:20
Well, at that point let's just go ahead with a bunch of requests. So, we'll go through each one of these and, we'll just do a request for them.
3:28
Self.app.get URL status should be 200. I know that I threw a "if the project is in here" in the URL. We've already tested it, lets not do it again.
3:40
In the real PyPI, imagine, this was the real PyPI data we'd have 140,000 different packages. It may be valuable to request every one
3:51
but, you'll probably get most of the value by just requesting an example one. So, we're just, if we've requested one
3:59
so okay we've tested that part of our site we don't need to it do it over and over and over again. So, it's an optimization, it gives up some tests
4:06
makes it much faster, put it in here if you like. And this is it. We can go and write this simple test against this simple site map you want anyway
4:15
and get a lot of verification right there.