We are looking for a big data expert with experience in hadoop, spark, presto, hive, mysql. You should be super experienced in setting up the hadoop architecture on any cloud server.
On the high level, this is what the application needs to be doing.
1) We need to fetch a lot of data from several different apis which have different formats of responses and different data representations.
2) We need to clean this data and put them into a sql database.
3) This data will then need to be retrieved by an application that requires a data source which will be the datatable to reference - this is the front end that will pull the data from our database: http://www.tableau.com/
4) Fetching data from these data endpoints / apis needs to be done in the most efficient manner possible and also retrieving data from our database needs to be efficient.
We are looking for someone highly experienced in:
- Big data and has experience in working with large / huge data sets of millions...