Python-powered Chat Apps with Twilio and SendGrid Transcripts
Chapter: Appendix: Using SQLAlchemy
Lecture: Inserting real PyPI data

Login or purchase this course to watch this video and the rest of the course contents.
0:00 Now you've seen how to insert data with sequel Commie. We're going to insert the actual real data and it turns out that this data I
0:09 got is the actual Pipi I data I got from, ah, couple of AP eyes I put together and I have the top 100 packages
0:17 in all their details in a bunch of Jason files. So what we're gonna do is load those Jason files,
0:22 pull them apart, do some type conversion and things like that and insert them all
0:27 into a database that is super into D gritty and it's really not worth going into
0:31 So let's just skim quickly across that first off to run the program we were
0:35 going to use to new requirements progress bar to so we can have a cool progress as we're doing our import, which is really,
0:42 really nice. And then Python Dash date util, which is a really, really nice way to parse dates much better than the built
0:49 in stuff. So I've already pip installed these, so they're just in the requirements now. So over here at the top of your repository,
0:56 I have the pie p I top 100 each one of these is just a Jason file. For example, let's look at click circa a little while ago written my
1:06 r Monroe knicker originally at least now managed by David Lord and the Pallets Project. But for the day that we got,
1:13 this is what it says And it has the licenses BST but notice it doesn't just say licenses BST it has this, like, sort of funky name space style,
1:21 if you will. It talks about the languages Python and being Colin Colin three. We're gonna parts that apart. Yeah,
1:26 the license BST. It works on Python to and Python three and so on so you can scroll through and see,
1:31 like, here's all of our releases All the details about the releases and the dates and Wolf has a lot of stuff,
1:37 right? So we're gonna go and parse that apart and insert it into the database and that's gonna happen over here pretty straightforward.
1:43 What we're gonna do is we're going to just go and ask really quickly like, Hey, is there any data in this database and the way of checking is are
1:51 there any users? It could look at all the tables and you just ask. Are there any users? If so, Hey, we've probably already done this,
1:58 so don't reinsert duplicates. Just don't do anything. Do a little summary there at the end. But if it happens to be empty,
2:05 go load up those files, all of them. All the Jason files skin across all of them, find the distinct users, import them,
2:12 do all the packages and their releases and so on. Pull out the languages licenses like that colon, colon BSD license thing we just saw.
2:21 And then finally do a little summary. So we're just gonna run this through its Let's just look at the import languages and
2:28 goes and uses our progress bar, which is pretty, pretty sweet.
2:31 And it rates over them and pulls out the language classification that the the interesting data base part is it says that we're going Teoh,
2:40 just create session creative programming language, set the details of it added to the session and call commit and then update our
2:46 progress bar Super straight for right. This is what we did before. It's just all this gu of juggling the Jason Files.
2:52 All right, so let's go and run this and because it uses the Progress bar It looks better outside apply charm. It will run in here. No problem.
2:59 But let's just make it as nice as possible. So I want to figure out where to activate my virtual environment.
3:04 That's a long enough directory, don't you think? I will say that. Slash Activate. And then I want to runs a Python.
3:14 The name of this script here where that one is gonna do the import. One other thing we also need to add system dup half upend toe our path.
3:25 This folder right here because we're importing pipeline, not or in pyjamas is gonna totally work smooth,
3:32 because guess what? Pie charm does that forest right there. But if we try to run this outside without setting this up,
3:40 it's a package or something like this is not gonna work so great. Now her ready to run our code here till gonna run Python out of our virtual
3:47 environment Pointed at a low data Here goes. So hearing see, it's loading up all of the users.
3:53 All the projects it found 96 packages said there were 100. I think for some reason, some couldn't be downloaded. So let's go with 96 Top 96.
4:01 Out of there, it went through, and it found the users found the packages and releases the languages. So in the end we found 84 users.
4:08 96 packages, 5400 releases embedded within those documents, 10 maintainers, 25 languages and 30 different licenses. All right, well, that's it.
4:19 We should now have a whole lot more data over here. And if we go local quick, let's just go to the packages and jump to the console.
4:26 Say, select star from packages. We run it like that. We had a whole bunch of them. Here's am Q P. Actors are pars,
4:34 Flake eight and so on. We running for releases, you see a whole bunch of stuff and there related to their various packages over here
4:41 on the right. Pretty awesome, huh? So now when we run our app, forgive over and run the actual app itself click on it,
4:50 you can see Well, we're not quite using the data yet, but we're going to be able to start using all that data we've just loaded up
4:56 and dropping it into these locations. So that's gonna be really awesome. We have true, accurate, realistic or even a snapshot in time.
5:05 RealD data from Pipi I toe work with to finish building out in testing your app

Talk Python's Mastodon Michael Kennedy's Mastodon