Php Crawler modifications

Web, Mobile & Software Dev Web Development Posted 2 years ago

Fixed Price

Delivery by January 4, 2013




I already have a php crawler which runs from my pc using WAMP. It places the information into a mysl database and largely does the job it is supposed to.

What I want to do is:

- Ensure that the crawler obeys the robots.txt if I want it to (it used to do this but somehow the code is broken so I have to comment out this line now)

- Makes sure it crawls every URL - it used to show me the URLs one by one in the CMD window but now looks quite messy for some reason

- Queue up sites so it can do more than one site at a time

- I want to be able to specify URLs from different sites to be crawled - but only those URLS, I might one 10 URLs crawled from site 1, 5 URls crawled from site 2

If you are a php master, please get in touch.

Skills Required:

Client Activity on this Job

Last Viewed: 2 years ago

Proposals: 3

Hired: 1

About the Client

(4.97) 49 reviews

United Kingdom
Salford 09:44 AM

140 Jobs Posted
42% Hire Rate, 1 Open Job

$8,925 Total Spent
61 Hires, 1 Active

$11.96/hr Avg Hourly Rate Paid
565 Hours

Member Since Sep 15, 2010