Building data-driven web apps with Pyramid and SQLAlchemy Transcripts
Chapter: Testing web apps
Lecture: Sitemaps and testing

Learn more about this course
Login or purchase this course to watch this video and the rest of the course contents.
0:00 You've heard of the Pareto principle
0:01 in one form or another.
0:03 It's often referred to as the 80/20 rule
0:05 the idea that most of the value can be gotten
0:08 by doing a little bit of the work...
0:10 80% value, 20% work.
0:13 That's good right? We can actually accomplish a lot of that benefit
0:16 by having a site map. Well, what's a site map?
0:20 So, a site map is actually a listing of all of the URLs
0:24 and a little bit of detail about them
0:26 targeted for search engines.
0:28 Google, Search bots, Bing, things like that.
0:32 Here's an example that might make sense for our...
0:35 And what we can do is we could define a Chameleon template
0:38 that will generate this.
0:40 So we can have one for the site, one for account/login
0:43 one for account/register and then every package on the site
0:47 we could use data-driven behaviors to pull out all of that.
0:51 See that down there at the bottom
0:53 tal:repeat p in packages, we're going to spit that out.
0:56 You want to have one of these types of things anyway.
0:58 It's massively helpful for search engines.
1:01 So we could have this template
1:02 and our view model would be really simple.
1:04 It's just going to go and say "Give me all the packages."
1:07 We may want to set a limit, probably not, more likely
1:09 you want to just cache this and refresh it periodically.
1:12 Anyway, we want to generate this somehow
1:15 and here's our controller, super easy.
1:18 We register sitemap.xml as the URL
1:22 and than we just generate this stuff using that template.
1:25 If we have this in place
1:26 and I'm going to give you one for this website
1:27 we're not going to write it from scratch.
1:29 I'll give it to you, you can adapt it for your own.
1:31 Then we can use this for testing.
1:33 We can go through here and just request
1:35 every single page in the sitemap
1:37 and make sure that none of them 500
1:39 you know server crash or 404 or things like that.
1:42 That's actually really helpful. This is the 80/20 rule.
1:46 We can just do a little bit of work to grab the site map
1:48 and request everything.
1:50 It won't test everything we're looking for
1:51 but like I said most of the time when these pages fail
1:54 they fail hard and just fully crash
1:56 500, 404, something like that.
1:59 They don't just send back wrong HTML.
2:01 That does happen, but they fully crash most of the time
2:06 which is actually kind of good for testing.
2:09 So, let's see this just in the concept here
2:11 and then we'll go look at it in code for just a moment.
2:13 We're going to write a little bit of code
2:15 to get the sitemap text, so what we need to do
2:18 is actually use our functional test behavior
2:22 to request /sitemap.xml.
2:25 It's going to return some value, response.
2:28 We can go to the text and we can say
2:30 so the XPath queries are simpler
2:32 let's get rid of the namespaces
2:34 replace this namespace that has to be there with nothing
2:37 that means we don't have to worry about it
2:38 in what we're about to do.
2:40 We can write a test that'll say
2:41 I want to go and test every URL there, so get us that text.
2:46 I want to create an XML DOM element, an etree.from_string
2:51 and in there we're going to do an XPath query for URL/location.
2:55 This will pull through, pull it out
2:58 and generate every single URL on our site.
3:02 Here where we are doing the replace localhost:6552
3:05 to say, I mean we would put that in our sitemap
3:08 but we just want to do a relative request here
3:11 like /account and so on. And then, we can just
3:15 now we have just the URLs as a flat string.
3:19 Well, at that point let's just go ahead with a bunch of requests.
3:22 So, we'll go through each one of these
3:23 and, we'll just do a request for them.
3:27 Self.app.get URL status should be 200.
3:31 I know that I threw a "if the project is in here"
3:34 in the URL.
3:36 We've already tested it, lets not do it again.
3:39 In the real PyPI, imagine, this was the real PyPI data
3:43 we'd have 140,000 different packages.
3:47 It may be valuable to request every one
3:50 but, you'll probably get most of the value
3:52 by just requesting an example one.
3:55 So, we're just, if we've requested one
3:58 so okay we've tested that part of our site
4:00 we don't need to it do it over and over and over again.
4:03 So, it's an optimization, it gives up some tests
4:05 makes it much faster, put it in here if you like.
4:09 And this is it. We can go and write this simple test
4:12 against this simple site map you want anyway
4:14 and get a lot of verification right there.