You will get Web & API Scraping, Save data to excel or database using Python Scripts
Project details
I have extensive knowledge of scripting with Python & scraping HTML elements, integrations with API, and profound experience working with databases to store data indestructible, designing schema & data structure for optimum usage.
Fetch or scrape data from any Websites or API, process the data and store them either in CSV, Excel, Google Sheets, or Databases or pass data to any other API.
Python scripts are very modular and accept a wide range of Data Sources and Output Formats.
Customizing the configuration is very easy, as the main scraping script is different, and there will be a configuration file to control the scraping script.
Users can modify the text configuration file, to define sources, Data type, and Output Formats.
The script is very resilient and can handle many types of Errors and issues, with auto max retries, use of proxy, IP rotations etc.
Fetch or scrape data from any Websites or API, process the data and store them either in CSV, Excel, Google Sheets, or Databases or pass data to any other API.
Python scripts are very modular and accept a wide range of Data Sources and Output Formats.
Customizing the configuration is very easy, as the main scraping script is different, and there will be a configuration file to control the scraping script.
Users can modify the text configuration file, to define sources, Data type, and Output Formats.
The script is very resilient and can handle many types of Errors and issues, with auto max retries, use of proxy, IP rotations etc.
Programming Languages
HTML & CSS, PythonCoding Expertise
Cross Browser & Device Compatibility, PSD to HTML, DesignWhat's included $20
These options are included with the project scope.
$20
- Delivery Time 3 days
- Number of Revisions 3
- Number of Pages Mined/Scraped 10
- Number of Sources Mined/Scraped 2
- Install Script
- Test Script
- Task Automation
Optional add-ons
You can add these on the next page.
Fast 2 Days Delivery
+$10About Suraj
Full Stack Developer | Python, Django, Database, Jira, Linux, Cloud
Bengaluru, India - 1:12 pm local time
- Python, Bash Shell & PowerShell.
- Django Web development & REST API development
- Automation Scripting & Web-hooks
- JIRA Development, Business Automation & Administration.
- SQL, Database (DBA) & Linux Administration.
- DevOps: Docker Containers, Jenkins, GitHub-Actions, AWS Lambda.
- AWS, Google Cloud & Azure operations.
• Django
- Developing full stack Web applications using Django, with support for Authentication & Authorization.
- Django boiler plate templates to start and finish most projects within weeks time using DRY modular codes & Object oriented classes.
- Developing Responsive mobile friendly and desktop website & application using end HTML, CSS & Bootstrap, htmx library.
- Single page application (SPA) functionality, async functions to update pages without refresh using HTMX library.
- Developing custom REST API applications.
- Dashboards & Graphs & Charts in Django Web applications.
- Backend Job integrations & scheduling with Celery, redis & other applications.
• Python, Automation, DevOps, Cloud
- Working extensively developing & integrating applications via Python & REST API.
- Building and deploying full fledged web applications either on premise or on Cloud.
- Developing REST API applications using AWS gateway, Lambda & Databases (AWS Mysql, Postgres, Dynamo DB) etc.
- Automation scripting for IT infrastructre (servers, databases, networks, cloud).
- CI/CD using GitHub-Actions, Jenkins, custom scripts.
- Cloud Operations & Automation (AWS, GCP, Azure, EC2, S3, AWS gateway, AWS Lambda, Google/Azure Functions etc).
- Infrastructure & Data Center automation using Jenkins, Ansible & Python/Bash/PowerShell scripts.
- Experience integrating Apache & nginx webservers for load balancing & reverse proxy.
- Data processing using SQL, pandas, NumPy, OpenPyXL, Databases, CSVs, Excel.
• Projects:
- Opensource infrastructure monitoring tool
- Web scraping. Data processing from and into Web pages, API, CSV, Excel,
- Technology Blog (Launch own blog, Software as a Service).
- E-commerce product price tracking alert.
- E commerce platform to buy/rent any commodities
- SolarWinds & Jira integration for datacenter infrastructure monitoring and ticket/incident generation, using SQL, python & REST API. (Read more)
- Automated User HR Onboarding & Asset management application using Jira and custom workflow using Python, REST API & Jenkins.
• Jira Development
- Jira Development, Automation & Administration.
- Developing full stack application with front end as Jira Forms & backend development with Jira Workflows, Automation, WebHooks, REST API, Python, AWS lambda functions, GitHub-Actions.
- Building & designing conditional & dynamic Jira Forms.
- External data source (databases/REST-API) integration with custom fields & Front end forms.
- Designing complex workflows with conditions, validators & post-functions.
- Automated & Dynamic workflows based on business logic, inputs & thresholds.
- Custom HTML Email notifications & templates.
- Advance automation using Webhooks, REST-API, Python script, CI/CD, Jenkins, AWS Lamda, Github-Actions etc.
- External & Internal customers security & management.
- Expertise on Jira Cloud.
- Jira Software & Data-Center installation, with multiple nodes.
- Nginx load balancer setup.
- User authentication with Active-Directory, Single Sign-on using Okta.
• Database
- Database Administration on Oracle, MySQL, MS SQL Server, PostgreSQL, AWS RDS.
- SQL & PL/SQL Programming.
- Database Backup & Restore, Automated scripts.
- Database High availability, replication, installation, upgrades, patching.
- Automated database housekeeping and tablespace storage management, automated table partitioning.
- Automated database refresh & cloning across data centers.
Steps for completing your project
After purchasing the project, send requirements so Suraj can start the project.
Delivery time starts when Suraj receives requirements from you.
Suraj works on your project following the steps below.
Revisions may occur after the delivery date.
Discuss Requirements
Describe the requirements of your Source Website or API, what kind of Data you want to scrape, and Output Format, and where to store like Database, CSV, Excel, Google Sheets, Email, SharePoint etc.