- Hourly
- Expert
- Est. time: 1 to 3 months, Less than 30 hrs/week
(In your message to me please provide me with a project price and elapsed time you would need to complete. ) Experienced Python dev needed to build data extraction tool. Tool should utilize Selenium to handle dynamic single-page application (SPA) content and collect relevant data into well-structured CSV files or postgre db. Type of data will be financial related. General coin info, metrics, transactions, comment threads. Handle pagination to scrape up to x pages. Implement retry logic for network issues. Co-pilot code gen is accepted. Plus if you have experience with blockchain and block explorer tools (Solana, ETH, Cosmos, or Bitcoin). Details: - SPA website has pretty clear html. does not look difficult. - 6 tables of data will be involved. (coin info, market metrics, transactions, comments, holder distribution, etc.). - Implement retry logic with a configurable retry limit. - Implement good error handling and a verbose logging system to show errors and each scraping action. - Ensure thread-safe data writes to CSVs or db. - Provide modular utility functions for URL parsing, CSV/db management, and error handling, logging. - Create a configuration file (config.json) for customizable scraping parameters. - README.md with setup and usage instructions with useful inline comments throughout the script. Bonus: setup with postgre db Bonus: setup with Django + celery or similar to manage processes. Qualifications: - Experienced with web scraping using Python (Selenium, BeautifulSoup, or other libraries). - Familiarity with SPA scraping, dynamic content, and error handling. - Can write clear, maintainable, and modular code. Let me know your ball park project price. Or ask me questions. Thanks. NDA required.