Data Science Jumpstart with 10 Projects Transcripts
Chapter: Project 4: Understanding Grouping and Aggregation Retail Data
Lecture: Using Feather and Pyarrow to Speed up loading Retail Data in Pandas
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
In this section I want to show a way to speed up loading your data. So I'm going to write this data to a format called feather and I just want to
0:09
time how long it takes to read the feather data compared to how long it takes to read the Excel data. I'll sit here and let that Excel data load again.
0:20
So this time it took 26.7 seconds here, so I'm going to change this to
0:29
26.7 and this is going to be 20 milliseconds for reading the feather data set. So
0:34
quite a huge improvement there. Again, if you want to have your data stored in other formats such as
0:42
feather or parquet, you're going to save a large amount of time over reading it from Excel.