Building Data-Driven Web Apps with Flask and SQLAlchemy Transcripts
Chapter: Using SQLAlchemy
Lecture: Inserting real PyPI data
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
Now you've seen how to insert data with sequel Commie. We're going to insert the actual real data and it turns out that this data I
0:09
got is the actual Pipi I data I got from, ah, couple of AP eyes I put together and I have the top 100 packages
0:17
in all their details in a bunch of Jason files. So what we're gonna do is load those Jason files,
0:22
pull them apart, do some type conversion and things like that and insert them all
0:27
into a database that is super into D gritty and it's really not worth going into
0:31
So let's just skim quickly across that first off to run the program we were
0:35
going to use to new requirements progress bar to so we can have a cool progress as we're doing our import, which is really,
0:42
really nice. And then Python Dash date util, which is a really, really nice way to parse dates much better than the built
0:49
in stuff. So I've already pip installed these, so they're just in the requirements now. So over here at the top of your repository,
0:56
I have the pie p I top 100 each one of these is just a Jason file. For example, let's look at click circa a little while ago written my
1:06
r Monroe knicker originally at least now managed by David Lord and the Pallets Project. But for the day that we got,
1:13
this is what it says And it has the licenses BST but notice it doesn't just say licenses BST it has this, like, sort of funky name space style,
1:21
if you will. It talks about the languages Python and being Colin Colin three. We're gonna parts that apart. Yeah,
1:26
the license BST. It works on Python to and Python three and so on so you can scroll through and see,
1:31
like, here's all of our releases All the details about the releases and the dates and Wolf has a lot of stuff,
1:37
right? So we're gonna go and parse that apart and insert it into the database and that's gonna happen over here pretty straightforward.
1:43
What we're gonna do is we're going to just go and ask really quickly like, Hey, is there any data in this database and the way of checking is are
1:51
there any users? It could look at all the tables and you just ask. Are there any users? If so, Hey, we've probably already done this,
1:58
so don't reinsert duplicates. Just don't do anything. Do a little summary there at the end. But if it happens to be empty,
2:05
go load up those files, all of them. All the Jason files skin across all of them, find the distinct users, import them,
2:12
do all the packages and their releases and so on. Pull out the languages licenses like that colon, colon BSD license thing we just saw.
2:21
And then finally do a little summary. So we're just gonna run this through its Let's just look at the import languages and
2:28
goes and uses our progress bar, which is pretty, pretty sweet.
2:31
And it rates over them and pulls out the language classification that the the interesting data base part is it says that we're going Teoh,
2:40
just create session creative programming language, set the details of it added to the session and call commit and then update our
2:46
progress bar Super straight for right. This is what we did before. It's just all this gu of juggling the Jason Files.
2:52
All right, so let's go and run this and because it uses the Progress bar It looks better outside apply charm. It will run in here. No problem.
2:59
But let's just make it as nice as possible. So I want to figure out where to activate my virtual environment.
3:04
That's a long enough directory, don't you think? I will say that. Slash Activate. And then I want to runs a Python.
3:14
The name of this script here where that one is gonna do the import. One other thing we also need to add system dup half upend toe our path.
3:25
This folder right here because we're importing pipeline, not or in pyjamas is gonna totally work smooth,
3:32
because guess what? Pie charm does that forest right there. But if we try to run this outside without setting this up,
3:40
it's a package or something like this is not gonna work so great. Now her ready to run our code here till gonna run Python out of our virtual
3:47
environment Pointed at a low data Here goes. So hearing see, it's loading up all of the users.
3:53
All the projects it found 96 packages said there were 100. I think for some reason, some couldn't be downloaded. So let's go with 96 Top 96.
4:01
Out of there, it went through, and it found the users found the packages and releases the languages. So in the end we found 84 users.
4:08
96 packages, 5400 releases embedded within those documents, 10 maintainers, 25 languages and 30 different licenses. All right, well, that's it.
4:19
We should now have a whole lot more data over here. And if we go local quick, let's just go to the packages and jump to the console.
4:26
Say, select star from packages. We run it like that. We had a whole bunch of them. Here's am Q P. Actors are pars,
4:34
Flake eight and so on. We running for releases, you see a whole bunch of stuff and there related to their various packages over here
4:41
on the right. Pretty awesome, huh? So now when we run our app, forgive over and run the actual app itself click on it,
4:50
you can see Well, we're not quite using the data yet, but we're going to be able to start using all that data we've just loaded up
4:56
and dropping it into these locations. So that's gonna be really awesome. We have true, accurate, realistic or even a snapshot in time.
5:05
RealD data from Pipi I toe work with to finish building out in testing your app