Seeking experienced developer with stong Artificial Intelligence and Machine Learning skills
to build a sophisticated crawler. The crawler needs to have a very complex computation to
meet our needs for fetching specific data while disregarding other data.
Suppose that we have a database with URLs from the internet.
Dataset : url, crawled, validated, rating
0. If dataset is empty, exit. Else, fetch an entry from db.
1. If entry has attribute "validated:yes" goto 0, else continue.
2. Fetch the site using URL.
3. Process the site's content using pattern recognition(Explained below)
4. If successful, mark "validated:yes, confidence:<<output from ANN>>, else mark "validated:error".
5. Goto 0.
Whenever the parser finds word "free", "trial", "free trial", "basic", "introductory", "$0"
or similar terms on the page, that page should be given a good confidence rating.
Choice of programming language is left open.
Please submit your proposal as well as any thoughts or questions you may have.. We are
developers and there is nothing we can't answer.