Building Big Data


I’ve recently been obsessed with getting my hands on CryptoCurrency value information in order to try and optimize trading as well as learn about dealing with the data as a whole. With enough data an AI could be taught to find correlations in price changes among various coins in order to make accurate predictions on price forecasting.

The Issue is finding this data. Anywhere I’ve looked, they only give very generalized version of the data (daily reports, incomplete reports, etc). In order to train an AI, I would need very detailed information, so I went straight to the source and figured who better to collect the data than myself?

So I started with a plan;
– Pull data from a web server on a number of coins at a time every minute.
– Parse each of those coins data into fields of their own individual coin objects.
– Build an object out of an array of coin objects and a date time field for acquisition data.
– Use the date time data to hash the array containing class to a hash map.
– Write the hash map to a binary file for storage.

And then suddenly BAM! ~1200 lines of handwritten, organic, gluten free Java code appeared from the grass! And now we’ve got a nifty little data collector prototype!

This gives me the ability to save and traverse all of the data perfectly without taking up as much space as it would otherwise thanks to the hashing and binary storage features. From here the GUI will implement a CSV report output function, low and high values over a given time span, as well as a graph to display pricing data, Along with the already functioning coin selection, percent changes, and current values.

Wish me luck on the final touches!