You will get Complex Data Extraction , Transformation , Data Quality related projects
Rising Talent
Rising Talent
Project details
>> Relational Databases (RDBMS).
Oracle, MySQL, PostgreSQL, MS SQL Server. Etc.
>> NoSQL Databases.
MongoDB, Redis, Cassandra.
>> Web Sources. Web APIs:
Determine the endpoint URL, request method (GET, POST), headers, and authentication if needed.
>> Web Scraping:
Identify the web page URL, elements to scrape, and any dynamic content-loading mechanisms.
>> Files. File Types:
CSV, Excel, JSON, XML,EdiFact
>> Cloud Platforms.
AWS S3, Google Cloud Storage, Azure Blob Storage.
>> File types in Hadoop Ecosystem
Avro, Parquet, and ORC
>> Data Streams.
Apache Kafka, RabbitMQ, real-time IoT devices.
>> Hybrid and Complex Sources
Combination of the above sources, needing a more complex extraction logic.
Designing end-to-end data transformation and data validation requirements to ensure data integrity, accuracy, and consistency throughout the process. Here’s a structured approach, that we are following during data transformation
Oracle, MySQL, PostgreSQL, MS SQL Server. Etc.
>> NoSQL Databases.
MongoDB, Redis, Cassandra.
>> Web Sources. Web APIs:
Determine the endpoint URL, request method (GET, POST), headers, and authentication if needed.
>> Web Scraping:
Identify the web page URL, elements to scrape, and any dynamic content-loading mechanisms.
>> Files. File Types:
CSV, Excel, JSON, XML,EdiFact
>> Cloud Platforms.
AWS S3, Google Cloud Storage, Azure Blob Storage.
>> File types in Hadoop Ecosystem
Avro, Parquet, and ORC
>> Data Streams.
Apache Kafka, RabbitMQ, real-time IoT devices.
>> Hybrid and Complex Sources
Combination of the above sources, needing a more complex extraction logic.
Designing end-to-end data transformation and data validation requirements to ensure data integrity, accuracy, and consistency throughout the process. Here’s a structured approach, that we are following during data transformation
Data Tool
PythonWhat's included
| Service Tiers |
Starter
$100
|
Standard
$200
|
Advanced
$400
|
|---|---|---|---|
| Delivery Time | 5 days | 10 days | 20 days |
Number of Revisions | 2 | 3 | 4 |
Number of Scenarios | 5 | 10 | 20 |
Model Documentation | - | - | - |
Data Source Connectivity | - | - | - |
Model Validation/Testing | - | - | - |
About Salinda
Data Engineering /Data Analyst | Informatica / Oracle / PLSQL
Guruthalawa, Sri Lanka - 9:19 pm local time
🛠️ Specializing in the Telco and Airline domains, I bring comprehensive expertise in System Analysis, ETL using PLSQL, and CRM functionalities encompassing Campaign and Loyalty Management.
📈 Proficient in Data Engineering, I leverage Python for ETL processes, crafting scalable pipelines to efficiently extract, transform, and load data. My proficiency extends to Python Django for building robust backend systems, ensuring seamless data flow and integrity.
🔍 With a keen eye for detail and a deep understanding of data dynamics, I excel in designing and implementing data solutions that drive business insights and decision-making processes.
💼 Past projects underscore my ability to architect and implement data pipelines, optimize data workflows, and develop custom solutions tailored to unique business needs, resulting in tangible improvements in operational efficiency and business outcomes.
🌟 Let's collaborate to unlock the full potential of your data assets. Whether it's optimizing data infrastructure, implementing advanced analytics, or building custom data solutions, I'm here to deliver excellence every step of the way.
Steps for completing your project
After purchasing the project, send requirements so Salinda can start the project.
Delivery time starts when Salinda receives requirements from you.
Salinda works on your project following the steps below.
Revisions may occur after the delivery date.
1. Identification of Data Sources and respective Data structure.
During this phase, we will analyze data source and the data source could be a database, a web API, a CSV file, an Excel, or other sources. The source determines the methods and tools that we need to need to use during the transformations phase.
Step 2: Design Data Transformation Process
3. Data Profiling 4. **Data Mapping:** 5. **Data Transformation Logic:** - Include steps for data validation, error handling, and logging during transformation. - Optimize transformation processes for performance and scalability.