First of all, I would like to state that I worked in Istanbul Metropolitan Municipality and I have experience in big data such as public transportation, traffic, wi-fi, and solution center that Istanbul has. I also provided consultancy to companies operating in Turkey that publish data bulletins on end-to-end data management (database administration, data engineering, data analysis, data scientist, data visualization). I will have to write a few paragraphs to explain my knowledge. The first paragraph will be about my experiences with database management, the second paragraph will be about data engineering, the third paragraph will be about data science, and the fifth paragraph will be about data visualization.
I made the installation, setup and management processes of the PostgreSQL database that I am responsible for. In addition, I carried out the PostgreSQL installation, package installations and administration processes for the companies I consulted.
Establishing schema structures on the relational databases I have worked with (Vertica, Oracle, MS SQL Server, Hadoop, PostgreSQL, Citus), structuring the relational tables in an optimized way. Establishment of data pipelines. ETL writing of data coming via API via Apache Kafka. Integrating the tables obtained through web-scrapping, analyzed and regularly imported with csv, into the database with Apache Airflow. In addition, writing the scrap codes of the data needed on the websites and directly integrate them into the database. (Note: There is no website where I can't do web-scrapping ":)" )
I will explain my experience in data science over a few cases. I created a future forecast by analyzing the daily fruit-vegetable prices within the scope of my work with the Istanbul vegetable-fruit market directorate. These forecasts were updated daily, weekly, monthly and yearly. If a price was realized above the forecast range, an e-mail was sent to the managers. Another project was the analysis of vehicle data collected by sensors on main roads. sensors are perishable devices due to weather conditions. For this reason, when the data was examined, it was seen that some sensors were missing or no data was received. In order to complete the missing or missing data, the model was built on python and the missing data was completed, and the prediction model was created and the number of vehicles that would pass through the analyzed sensors in the future was calculated instantly.
I have used Tableau, Powerbi, etc BI tools throughout my business life. While I was working in Istanbul metropolitan municipality, Tableau was used, but the license could not be renewed due to high cost and there was no data visualization program in the municipality. That's why I discovered the apache superset. I did a lot of research about the program and finally got it into the municipality. I made the development of the tool that I installed on the ubuntu server. it is currently running smoothly up to eight thousand requests at the moment. I also gave training to municipal employees on how to use it. I made data visualizations using this superset in all of the works I mentioned above. I also provide support to the companies I serve for every stage of the superset (installation, development, data visualization).
Program-Tool Experience Knowledge
Python 4 Years pandas, matplotlib, statsmodels, sklearn, numpy, pickle, requests, json, psycopg2, vertica_python, time, selenium, beautifulsoup, smtplib, csv
Apache Kafka 1.5 Years Installing, Developing, ETL
ApacheAirflow 1 Years Installing, Developing, ETL
Postgresql 3 Years Installing, Tuning, Administration, Data Pipelines, Data Analysis, Reporting,
Vertica, Oracle, Hadoop, Mssql 3 Years Data Analysis, Reporting
Apache Superset 2 Years Installing, Developing, Data Visualization, Reporting
Tableau, PowerBI 3 Years Data Visualization, Reporting
SPSS 9 Years Data Analysis
Microsoft Excel 9 Years Reporting, Data Analysis
E-Views 3 Years Data Analysis
Stata 3 Years Data Analysis
ArcGIS 2 Years Spatial Data Analysis, Mapping
Microsoft Power BI Data Visualization