Web Crawler Jobs

75 were found based on your criteria {{ paging.total|number:0 }} were found based on your criteria

show all
  • Hourly ({{ jobTypeController.getFacetCount("0")|number:0}})
  • Fixed Price ({{ jobTypeController.getFacetCount("1")|number:0}})
Hourly - Intermediate ($$) - Est. Time: 1 to 3 months, 10-30 hrs/week - Posted
Lesson Lace is a new and growing company that builds lesson plans databases from existing out-dated lesson plans. I need someone who is a good communicator, knows Common Core Lesson Planning, can scrape websites, and clean data. This is a big opportunity to grow with the company for the right person. We are looking to grow our existing lesson plan database, clean it, and optimize it. If this sounds fun and interesting to you please apply. Do not apply if you know nothing about lesson planning!
  • Number of freelancers needed: 2
Skills: Web Crawler Data mining Data scraping Scrapy
Fixed-Price - Entry Level ($) - Est. Budget: $20 - Posted
I am looking for a knowledgable custom android developer whom is very skilled with java & android development & is able to think outside the box on projects - I have some specifications & ideas & am looking for the right android developer to help me execute them.. If you feel you can accomplish any task & are willing to learn & grow please contact me for more details! I am looking for a custom script to block Google when viewing applications upon submitting to the play store - the script would automatically make your app close/ crash when it picks up googles internet service provider upon reviewing my app What are your thoughts & are you capable? looking forward to finding the right android devs
Skills: Web Crawler Android Android App Development Android SDK
Fixed-Price - Intermediate ($$) - Est. Budget: $1,000 - Posted
What we need: a web based tool to find expired domains we can register with a minimum criteria of majesticseo metrics (Fresh TF and CF above 10, minimum referring IPs, etc). The process of the tool, would be something along these lines. 1. Custom web crawler like Google bot/xenu link sleuth/etc so that scrape sites for links constantly. (The crawler will be deployed into the AWS - But should be designed to scale to work faster.) 2. Add only unique root domains to a database, not full links (example just want www.site.com and not www.site.com/certain-page) 3. Check to see which dead and available to buy (Link with Godaddy, or Namecheap API). 4. Check metrics with Majestic api and MOZ Api and filter based on have above criteria of metrics (if good add to database) 5. Download excel and or view online option. This product will compete directly with http://sourcerevive.net/, so once the functionality needs to be built easily into a GUI. Please let me know three things: 1 - time you expect this would take to finish 2 - how much you would charge to do it 3 - why you are the best person for this position Please read here for more information: https://docs.google.com/document/d/189DIRzs3PkStTQHzi0TvuGVz6o-_K0ojwL0v94B-m0A/edit# And the diagram: (which will explain exactly what is needed) https://docs.google.com/drawings/d/1qL7J7UTZ_yMy9XQbNphuNBh7JQjg3JDbQg4ER_NswBg/edit The link below includes a script that has been put together on GitHub. https://github.com/christophebe/crawler.ninja Also we don't want a desktop client due to these reasons: 1. will not have central database (will be only on that pc) 2. is limited to the resources of that pc (limited to the number of the internet connection, processor, ram, etc). If it's hosted on a cloud service for example it can use as many resources as it needs, it can crawl up to tens of thousands of domains per minute, or more, there is no real limit on this... 3. you have to keep the pc open, it's not scalable, multiple users can't use it effectively and it will make the pc it's working on kinda unusable if it will work on full power I have a large credit with AWS so you will need to deploy the software into an AWS Instance. IMPORTANT: Must crawl sites not google searches or registrars The steps for this project will be: 1. Scope the project out - Determine how useful existing scripts are and how they will work as a part of the project. 2. Let me know how much it will be to get the software up and running with a basic GUI 3. Let me know how much it will cost to complete a compelling GUI such as http://sourcerevive.net/.
Skills: Web Crawler PHP Python SQL
Fixed-Price - Entry Level ($) - Est. Budget: $20 - Posted
I am looking for a knowledgable custom android developer whom is very skilled with java & android development & is able to think outside the box on projects - I have some specifications & ideas & am looking for the right android developer to help me execute them.. If you feel you can accomplish any task & are willing to learn & grow please contact me for more details! looking forward to finding the right android devs
Skills: Web Crawler Android Android App Development Android SDK
Fixed-Price - Entry Level ($) - Est. Budget: $50 - Posted
Hello, I am looking for an expert in online advertising. I want to launch my spa. The spa website is almost completed,therefore am searching for someone to advertise my company by making my website visible to by many.
  • Number of freelancers needed: 10
Skills: Web Crawler
Hourly - Entry Level ($) - Est. Time: Less than 1 week, 10-30 hrs/week - Posted
I need a few lists built out of the Angellist (angel . co) website Key skill needed is: ability to build extractors, crawlers, connectors in import.io Should be very easy for the right person who already knows how to use the tool. Please include past experience with import.io This could result in follow-on work which requires scraping for lead generation.
Skills: Web Crawler Web Crawling Data mining Data scraping
Fixed Price Budget - Expert ($$$) - $100 to $500 - Posted
We are looking to develop a price comparison website. Please apply with some sample work and the best cost. Preferred languages Website - Ruby on rails Crawler- Scrapy (python) However we are open to good suggestions.
Skills: Web Crawler PHP Ruby on Rails Scrapy
Fixed Price Budget - Intermediate ($$) - $100 to $300 - Posted
I estimate the total work here is 4-5 hours. I have a small web crawler software that I need some work done for. I hired a programmer to make this a while ago and now he is no longer responsive. PLEASE VIEW A FULL VIDEO about the script and everything I need done here: https://www.youtube.com/watch?v=6KG5xXbfXq4 (I tried my best to explain everything below here in the post but you will understand a lot better if you just watch the short video) PART 1: Advise And Help Put Software On A Public Server I need to take this software(which is currently just a script running on my personal server in cPanel), and put it on a server for public use with an account system. I need to know what type of server to get, and there needs to be a system installed that will allow users to sign up/create a free account and have their own account of the software. PART 2: Fix A Bug In The Script There is a bug in the script that is explained in this video. Basically it is a PHP bug that is causing the stored data to not refresh on each scan. It causes the software to just keep accumulating data as time goes on- even when the data no longer exists on the site that is crawled. Please include *READ* as the first word of your proposal/response or I will not be able to consider your bid. I need somebody who can follow directions.
Skills: Web Crawler PHP