Using Excel with big data:

Excel is one of the most common spreadsheet programs available today, thanks to a sleek UI and  powerful reporting tools, it’s a go to for most companies trying to make data visualization easier.

Although not the best with huge data sets, Excel can be a good way to build nice reporting and functionality into a set if you are dealing with a manageable amount of data.

While the max file size used by Excel of 2GB hampers my ability to put all of my crypto coin pricing data into a single Excel calculator (my latest project), I can still run reports on ~ 196 days worth of crypto currency pricing data from my program without much issue (86,400 X 196 = ~ 16,856,000 records). But regardless of this and my comment on huge data sets, it is still used a very powerful reporting tool.

Take for example a single day of Crypto pricing data from my collector, a solid 86,400 records (10.2mb) (Link below). We can use the reporting features of excel to break down and understand this data on a human level. To make an otherwise dense, unreadable data set into sensible reports.

Using a Pivot Table we can break down a large data set into a more readable format. We can separate each individual token into a single record and view more generalized reporting data. Using this method, I have turned my data from 6-17-18 (85,681 records) into a manageable 63 records, not bad!

I have attached a link to the actual data set, functions included, which shows an example of the power of the pivot table. Refer to the reports sheet to view general market trends over the last 24 hours of actual real time CC data.

We start by taking our data set in one sheet, and applying the pivot table in another. We select a column from raw data to be used as rows in our pivot table, after which we can build general reports based on the data in each corresponding field based on the matching rows. Using this technique, a lot of data can be visualized easily to the human eye in a concise fashion.

So what can we visualize from this example data set using the crypto pricing data from all of 6/17/18 broken down with a pivot table? Notice columns (E:F): 7 day percentage change, is mostly in the red. But looking at columns (G:H): 1 day percentage change, we see less red, and more green. From there, looking forward to columns (I:J): 1 hour percentage change: you can see more green and yellow than previous columns. Based on this data it is easy to see the market bottoming out and the resurgence of funds into the market visually, based on a giant CSV data set. Visualizing data is the key to understanding it, which is why tools like Excel are essential at times.
Visualizing the data is one of the first steps in understanding it!

Let me know if you have any questions or concerns!
Constructive criticism is the best kind of criticism!
Thanks for reading!

Data Set Example Link:
http://www.tpwebdevelopment.com/blog/tutorials/Reporting_Test.xlsx

Pivot Table Tutorial Link:
https://support.office.com/en-us/article/create-a-pivottable-to-analyze-worksheet-data-a9a84538-bfe9-40a9-a8e9-f99134456576