Php Crawler modifications

Web, Mobile & Software Dev Web Development Posted 3 years ago

Fixed Price

Delivery by January 4, 2013




I already have a php crawler which runs from my pc using WAMP. It places the information into a mysl database and largely does the job it is supposed to.

What I want to do is:

- Ensure that the crawler obeys the robots.txt if I want it to (it used to do this but somehow the code is broken so I have to comment out this line now)

- Makes sure it crawls every URL - it used to show me the URLs one by one in the CMD window but now looks quite messy for some reason

- Queue up sites so it can do more than one site at a time

- I want to be able to specify URLs from different sites to be crawled - but only those URLS, I might one 10 URLs crawled from site 1, 5 URls crawled from site 2

If you are a php master, please get in touch.

  • Other Skills:

Activity on this Job

Last Viewed by Client: 3 years ago

Hired: 1

Other open jobs by this client ({{ jobsCount }})

About the Client

(4.98) 50 reviews

United Kingdom
Salford 02:52 AM

148 Jobs Posted
40% Hire Rate, 1 Open Job

Over $10,000 Total Spent
61 Hires, 0 Active

$12.97/hr Avg Hourly Rate Paid
647 Hours

Member Since Sep 15, 2010