I need to spider all products and product details, pricing, from these sites.
I want all the items to be matched for price comparison purposes.
i need it to run fast so that i can rerun it often (on a daily basis), i also want to be able to refresh specific items on demand,
you will need to use multiple proxies so that we dont get blocked,
i need an interface to:
1. Schedule, the scraper,
2. search / filter data
2. so that i can refresh specific items, on demand
3. access data
4. set alerts based search / filter data. we want to be able to set email alerts if new data meets certain criteria, so if when scraping the data the systems finds results that meet specified filters, it will send me an email.
5. I will be adding many more sites approx 300 sites in total. so once this project is complete and works well we have plenty of more work to do together.
Data should be stored in a mysql database.
should be in the lamp environment.
For backend use responsive Metronic theme (http://keenthemes.com/preview/metronic/). We will send source of theme.