Thanks for your time for our project. We need to develop an application from the scratch.
We need certain data. The required data need to have a uniform structured format. We’ve identified the data are scattered through different public URLs. So we have prepared an expanding list of URLs of the relevant data sources. The URLs are mainly pointing to a browse permitted publicly accessible web directory.
We’ve several local servers for storing files. We need to create a local mirror of each of the URL, basically the mirror will contains every file whatever there in the given URL, following by successive unlimited depth with exactly same directory structure. We'll denote this mirrors as 'Browsed mirrors'. Yes, there are free applications to do this job (likewise, HTTrack).
We’ve also some URL’s not directly browse permitted as such, but exposes tabular data interface as per query with user selected options. We also need to make mirror of the search options and results for those types of URLs. We'll often calling this mirror as 'scrapped mirror's.
Now, the most important part of the application, we need keep these mirrors as a manner so it stays updated as per the source URL. If any directory, files, search option been added, updated in the web directory of the source URL, it will also been added, updated with same location structure in our mirror. It might not be instant, but a periodic check and mirror update will have to happen once a week, for scraped mirror also.
Yes, this is the prime requirement description. Now, we have some specification diagrams, as per this specs this total application will have to develop. There will have a certain number of small independent functional modules in the entire application. The modules are needed to develop separately, and later be integrated in the main application to make the functionality chain.
Development Technology and run-time
1. Microsoft DOT NET 4 and above. VB and/or C#
2. Database MS – SQL 2014 or later technology.
3. Should not use any 3rd party application/library, but only whatever inside DOT NET framework.
4. Functionality should be objects based (not likely OOP, but to be standalone and portable).
I’ve attached a schematic diagram with this job posting, please feel free to ask any question after viewing this. Thanks in advance.
* Please write “Mirroring with DOT NET” on the top of your application.