Php Crawler modifications

Web, Mobile & Software Dev Web Development Posted 2 years ago

Fixed Price

Delivery by January 4, 2013

$50.00

Budget

Details

I already have a php crawler which runs from my pc using WAMP. It places the information into a mysl database and largely does the job it is supposed to.

What I want to do is:

- Ensure that the crawler obeys the robots.txt if I want it to (it used to do this but somehow the code is broken so I have to comment out this line now)

- Makes sure it crawls every URL - it used to show me the URLs one by one in the CMD window but now looks quite messy for some reason

- Queue up sites so it can do more than one site at a time

- I want to be able to specify URLs from different sites to be crawled - but only those URLS, I might one 10 URLs crawled from site 1, 5 URls crawled from site 2

If you are a php master, please get in touch.

Skills Required:

Client Activity on this Job

Last Viewed: 2 years ago

Applicants: 3

Hired: 1


About the Client

(4.97) 48 reviews

United Kingdom
Salford 02:52 PM

140 Jobs Posted
42% Hire Rate, 1 Open Job

$8,655 Total Spent
61 Hires, 2 Active

$11.76/hr Avg Hourly Rate Paid
552 Hours

Member Since Sep 15, 2010