Big Data Developer job description template

An effective description can help you hire the best fit for your job. Check out our tips to provide details that skilled professionals are looking for.

Trusted by


Tips for Writing a Big Data Engineer Job Description

A big data engineer is a professional who is responsible for the management of data sets that are too big for traditional database systems to handle. They create, design, and implement data processing jobs in order to transform the data into a more usable format. They also ensure that the data is secure and complies with industry standards to protect the company’s information. 

Below, we will cover a sample job description, exploring the daily responsibilities and necessary qualifications for a big data engineer. 

The Job Overview

We are seeking a big data engineer to join our data analytics team. The successful candidate will be responsible for overseeing the creation and maintenance of our database infrastructure, including collecting and maintaining data, ensuring the integrity of our data, and creating and training data models.

Responsibilities

Below are some of the responsibilities of a big data engineer:

  • Design the architecture of our big data platform
  • Perform and oversee tasks such as writing scripts, calling APIs, web scraping, and writing SQL queries
  • Design and implement data stores that support the scalable processing and storage of our high-frequency data
  • Maintain our data pipeline
  • Customize and oversee integration tools, warehouses, databases, and analytical systems
  • Configure and provide availability for data-access tools used by all data scientists
Job Qualifications and Skill Sets

Below are the qualifications expected of a big data engineer:

  • 3 to 5 years of relevant data engineering experience
  • Bachelor’s degree or higher in computer science, data science, or a related field
  • Hands-on experience with data cleaning, visualization, and reporting
  • At least 2 years of relevant experience with real-time data stream platforms such as Kafka and Spark Streaming
  • Experience working in an agile environment
  • Familiarity with the Hadoop ecosystem
  • Experience with platforms such as MapReduce, Apache Cassandra, Hive, Presto, and HBase
  • Excellent analytical and problem-solving skills
  • Excellent communication and interpersonal skills
Big Data Developer Hiring Resources
Explore talent to hire
Learn about cost factors
ar_FreelancerAvatar_altText_292
ar_FreelancerAvatar_altText_292
ar_FreelancerAvatar_altText_292

4.8/5

Rating is 4.8 out of 5.

clients rate Big Data Developers based on 1K+ reviews

Hire Big Data Developers

Big Data Developers you can meet on Upwork

Chunyi  W.
$50/hr
Chunyi W.

Big Data Developer

5.0/5(267 jobs)
Shoreline, WA
  • Trophy Icon Big Data
  • SAS
  • R
  • Data Science
  • Linear Regression
  • Data Visualization
  • Quantitative Analysis
  • Statistics
  • Analytics
  • Logistic Regression
  • Biostatistics
  • Statistical Analysis
  • Epidemiology
  • Healthcare & Medical
  • Public Health

I obtained my Ph.D. degree in Epidemiology at the University of Michigan and I also have the SAS Programmer certification. Currently, I am a Lead Data Analyst in Medical School. I have a strong background in biostatistics/ epidemiology and have 14 years experiences on analyzing large epidemiological, clinical, genetic and National Inpatient Sample data using various software packages (SAS, SPSS, R and R studio program). I have extensive knowledge of statistical models, and have developed various analysis strategies for different studies and meta-analysis. Statistical methods that I have applied in the research projects: 1. Multilevel Logistic Regression Models, and Ordinal Logistic/Logistic Regression Models 2. Linear Mixed Models and Linear Regression Models 3. Survival Models, Cox Proportional Hazards model, Accelerated Failure Time Modeling, Kaplan-Meier Plot) 4. Poisson Regression Model 5. GEE (Generalized Estimating Equations) 6. Propensity Score Matching (PSM) 7. ROC curve, ANOVA, T-test, Nonparametric Statistics (Kruskal-Wallis test and Wilcoxon Signed Rank Test), Cohen's alpha, Pearson's Correlation Coefficients, Chi-squared test. 8. CMS-HCC Risk Adjustment Model (HCC, RxHCC, ESRD) 9. Data analysis with weighted data in the survey sample. 10. Power Analysis In addition, I have performed the statistical analysis by using the large longitudinal national data in the past: A. Health Retirement Study B. National Health and Nutrition Examination Survey (NHANES) C. National Inpatient Sample (NIS), and Healthcare Cost and Utilization Project (HCUP)) D. CMS-HCC Risk Adjustment Model (HCC, RxHCC, ESRD) E. Meta-analysis to perform the analysis on a large database (Genome-Wide Association Studies) efficiently. . As a data scientist, I am passionate about data analysis, solving complex and interesting task. Once you hire me as a freelancer, the results will be delivered to you within 1-10 days (including weekends). Small project: 1-4 hours. Results will be delivered within 1-2 days. Medium project: 4-10 hours. Results will be delivered within 2-4 days. Large project: 10-20 hours. Results will be delivered within 4-6 days. Project more than 20 hours: Results will be delivered within 5-15 days. Please feel free to contact me and I will response your message within 24 hours. Thank you.

...
Amar K.
$80/hr
Amar K.

Big Data Developer

5.0/5(26 jobs)
Bengaluru, KA
  • Trophy Icon Big Data
  • DevOps
  • Amazon Web Services
  • Google Cloud Platform
  • AWS Lambda
  • PySpark
  • MongoDB
  • Content Writing
  • Apache Kafka
  • SQL
  • Apache Airflow
  • Data Engineering
  • Docker
  • Python

Top Rated | #1 Freelancer in India for Big Data, Python, GCP, AWS etc. I have 𝟴+ 𝘆𝗲𝗮𝗿𝘀 of professional 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 and 𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝗺𝗶𝗻𝗴 experience with 𝗣𝘆𝘁𝗵𝗼𝗻 and 𝗝𝗮𝘃𝗮 with 𝗚𝗖𝗣 & 𝗔𝗪𝗦 Cloud. I am fortunate to have worked with 𝗙𝗼𝗿𝘁𝘂𝗻𝗲 𝟱𝟬𝟬, 𝘁𝗼𝗽 𝗶𝗻𝘃𝗲𝘀𝘁𝗺𝗲𝗻𝘁 𝗯𝗮𝗻𝗸𝗶𝗻𝗴 companies in the past. Moreover, I posses solid 𝗗𝗲𝘃𝗢𝗽𝘀 experience with good hands-on in Cloud Infrastructure. Currently, I am an Upwork 𝗧𝗼𝗽-𝗥𝗮𝘁𝗲𝗱 freelancer who focuses on providing premium service to my clients and quality projects with on-time delivery. Previously, I have worked full-time with top-notch product companies which includes - 𝗖𝗲𝗿𝗻𝗲𝗿 𝗯𝘆 𝗢𝗿𝗮𝗰𝗹𝗲, 𝗞𝗣𝗠𝗚, 𝗚𝗼𝗹𝗱𝗺𝗮𝗻 𝗦𝗮𝗰𝗵𝘀, 𝗠𝗼𝗿𝗴𝗮𝗻 𝗦𝘁𝗮𝗻𝗹𝗲𝘆, etc. Skills : - 𝗖𝗹𝗼𝘂𝗱 ⌥ GCP (Google Cloud Platform) , AWS (Amazon Web Services) - 𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝗺𝗶𝗻𝗴 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 ⌥ Java, Scala, Python, Ruby, Groovy - 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 ⌥ Spark, Kafka, Crunch, MapReduce, Hive, HBase - 𝗗𝗲𝘃𝗢𝗽𝘀 ⌥ GitHub, GitLab. BitBucket, CHEF, Jenkins, Bamboo, Nexus, JFrog, etc - 𝗔𝗣𝗜 ⌥ SpringBoot, Jersey, Flask

...
Muhammad Jarir K.
$40/hr
Muhammad Jarir K.

Big Data Developer

5.0/5(3 jobs)
Lilongwe, C
  • Trophy Icon Big Data
  • Python
  • Machine Learning
  • Apache Spark
  • Data Engineering
  • Database Design
  • R
  • Data Science
  • Marketing Data Analytics
  • Data Visualization

I'm a data science and analytics professional with a physics degree to boot. I'm currently pursuing a master's degree in computer science at Georgia Tech, focusing on machine learning and AI. If you have a data problem, I'm your guy. Data engineering, visualization, and analysis, I can do it all — using either Python or R. Want to build a machine learning model for your business but lack the technical expertise? I will build it for you! And I can also design a database to help you manage all that precious data or use big data tools like Spark and MapReduce to help improve your data pipeline. Some of my skills include: ✔️ Data Science ✔️ Machine Learning ✔️ Data Engineering ✔️ Big Data Systems (Hadoop, MapReduce, Spark) ✔️ High Dimensional Data Analysis ✔️ Database Design (MySQL, Postgres, BigQuery, EER models, DB normalisation, etc.) ✔️ Python, R, SQL ✔️ Flask ✔️ Python libraries: NumPy, pandas, SciPy, sklearn, matplotlib, seaborn, networkx, Tensorly, OpenCV, PyCaret, CatBoost, TensorFlow, etc. ✔️ R libraries: tidyverse, ggplot, caret, kernlab, etc.

...
Ioannis P.
$120/hr
Ioannis P.

Big Data Developer

5.0/5(8 jobs)
Athens, I
  • Trophy Icon Big Data
  • Python
  • PyTorch
  • Deep Learning
  • Machine Learning
  • Data Science
  • Cloud Computing
  • SQL

I am an experienced Machine Learning Engineer, skillful in Python, Big Data, Deep Learning and Earth Observation. As a researcher, I am very familiar with the state-of-the-art and can provide in-depth analyses. I have completed several projects including time-series forecasting, anomaly detection, computer vision, remote sensing & satellite data analysis. I have implemented and deployed several Machine Learning models that are currently used in production: - An NLP model that identifies cardiology-related scientific abstracts and categorizes them into more specific categories. - A wildfire forecasting model that uses weather and satellite data. - An anomaly detection model that produces risk scores for vulnerable servers. I am excited to solve diverse problems as a freelance Data Scientist and waiting to tackle the next challenge.

...
John M.
$34/hr
John M.

Big Data Developer

5.0/5(12 jobs)
Malvern, AR
  • Trophy Icon Big Data
  • Python
  • Machine Learning
  • API
  • Microsoft SQL Server
  • Java
  • C#
  • Database Design
  • Data Visualization
  • ASP.NET
  • Adobe Inc.
  • Web Development
  • CSS
  • JavaScript

👋 I have been working as a web developer for over 4 years. My design approach is always clean, modern, and simple. Over the past 4 years, I have developed a wide range of websites using, C#, Java, SQL Server, Postgresql, JavaScript, CSS3, and HTML5. These sites were made for startup companies, small businesses, corporations, and individuals. Has extensive experience working with the various REST APIs. Up to date on all modern web design trends and standards and can build you a site to be proud of. I enjoy developing, coding, and maintaining clean, professional, easy to navigate websites. I can help you or your business create a website from design concept to a completed, polished, and professional site. I can also help maintain and update existing websites. What makes me the best choice for your business? I'm driven to over deliver on every project project and communicate effectively with project managers. I'll work night and day until your project is how you want and how you deserve it to be. I pride myself on finding creative ways to save my clients money and automate their existing processes. Below is a list of my professional services: ✔️ eCommerce Solutions (Shopify) ✔️ HTML / CSS / jQuery / JavaScript / Liquid / Bootstrap / Ajax ✔️ Shopify Custom Development ✔️ Landing Page ✔️ Converting into functional website from Adobe XD Sketch / Figma / Illustrator /Photoshop ✔️ Custom Websites (PSD to HTML / Shopify) ✔️ Responsive Web Design ✔️ Page Speed Optimization ✔️ SEO friendly website ✔️ Website maintenance ✔️ ETL Batch Processors ✔️ API Creation Skills Profile: ✔️ C# ✔️ Python ✔️ JavaScript ✔️ Java ✔️ SQL I am also prior service military personnel.

...
Rai S.
$40/hr
Rai S.

Big Data Developer

5.0/5(4 jobs)
Lahore, PUNJAB
  • Trophy Icon Big Data
  • Python
  • Apache Hadoop
  • Tableau
  • Deep Learning
  • Machine Learning
  • Apache Spark
  • SQL
  • Data Visualization
  • Microsoft SQL Server
  • PySpark
  • BigQuery
  • Data Analysis
  • Cloudera
  • Business Intelligence

Rai Shahnawaz, a Sr. Data Scientist with comprehensive mathematical and programming background, is well-versed with and vastly experienced in big data technologies, machine learning and statistics. Rai has worked on building large scale data warehouse solutions with integration of heterogeneous data sources on top of Google big-query (PaaS), Hive & Vertica. Rai is currently working as a Senior Data Scientist at ADDO-AI, the leading AI-based IT firm in Pakistan. Prior to this, he served as a Research Associate for Data Science Lab (DSL) & Fin-tech Center at ITU where his collaborative work with the University of Washington’s DFSRG group was focused on bias aware decision making and financial fraud mitigation on top of big data technologies. Moreover, he was a Teaching Assistant for Data Science and Big Data courses offered at Information Technology University which ranks among the top IT universities in Pakistan. His research work for mitigating financial fraud and fairness in decision making for machine learning models is directly applicable to the legal, social, and economic domains particularly after strict privacy regulations like GDPR. Before that, he worked as a product support engineer for three years at i2c which is a leading stakeholder in the Payment Card Industry. At i2c, his prime responsibilities included efficient and effective analysis of customers’ product-related issues using SQL at Informix db and backend server application logs. He is profoundly experienced in working on a variety of product modules ranging from OLAP, CMS, Distributed Schedulers and Campaign Management to Web Services API and OLTP at Linux Platform. Rai graduated from Fast-NU, Lahore with Bachelor in Computer Science and thereafter, completed his Master degree in Computer Science with minor in Data Sciences from ITU, Lahore. Languages: Python; Java; SQL; Linux bash scripting, C++;. Tools and Technologies: AWS Sage maker, EMR, Spark, Tensor flow, Keras, Sklearn, numpy, pandas, Pyspark, Hadoop, Bigquery, Hive, Vertica, Talend, Google Cloud Platform (GCP), Apache Beam, Hadoop HDFS, Weka, Informix DB, MySQL, Tableau, Qlik, Pycharm, Datagrip, Spyder, MS SQL Server, SSMS, Dbeaver, MS Visual Studio, C#/C++, XAMPP, Web Service API Client, Unix/Linux machines, Eclipse, OpenCV, Java for Android, Notepad++, Swing and NetBeans IDE

...
Aser O.
$50/hr
Aser O.

Big Data Developer

5.0/5(7 jobs)
Rome, METROPOLITAN CITY OF ROME
  • Trophy Icon Big Data
  • Statistical Analysis
  • Forecasting
  • Data Analysis
  • Data Visualization
  • Artificial Intelligence
  • Data Analytics
  • Statistics
  • SQL
  • Microsoft Power BI
  • Microsoft Excel
  • Data Mining
  • Machine Learning
  • Python
  • pandas

✅ **100% Satisfaction or Full Refund** I enjoy solving chronic business problems, automating tedious tasks, finding patterns in ambiguous datasets and providing professional analysis and AI/ML models to help you optimize your earnings. My solutions have successfully been implemented at different organizations and industries, from aspiring start-ups to leading multinationals in Africa, Asia, North America and Europe. With a background in Software engineering, and extensive professional experience in data analysis and automation; I equipped myself with a wide range of tools to efficiently answer my clients' needs, including: -Python, for statistics, Machine Learning, data science, linear programming, process automation and general purpose programming. -Excel (VBA Macros, M and DAX) for office solutions and business dashboards. -Power BI, Matplotlib, Seaborn, and Plotly for charts and visualization. -Cloud Services such as AWS and Azure Feel free to reach out for a quick chat if you have any doubts, I'll be more than happy to clear them for you.

...
Stanley B.
$90/hr
Stanley B.

Big Data Developer

5.0/5(8 jobs)
Yorba Linda, CA
  • Trophy Icon Big Data
  • Data Warehousing & ETL Software
  • IBM Cloud
  • Microsoft SQL SSAS
  • Netezza
  • Snowflake
  • Marketing Data Analytics
  • Azure Blockchain Service
  • SQL
  • Amazon Redshift
  • AWS CloudFormation
  • Google Cloud Platform
  • Apache Kafka

Cloud Solution Architect with engineering experience in Cloud SQL Big Data and multi-database technologies including Data Architecture to support the Business Intelligence needs. Solution Architect in Technical Teams on Cloud Data Solutions into various Cubes, Data Marts, and ERP Systems. Developed data structures for business using various Analysis Services Cubes, BI Dashboard, and Scorecards. Implemented standards and data designs for HIPAA, SOX, regulatory, compliance, financial, reporting, and auditing. Cloud Services Database include Snowflake Data Cloud, Azure Cloud, Go/Language 9JSON and yaml) with Anaconda python programming. Loaded large Data Lakes from on-prem up to SnowFlake using SnowPipe and SnowSQL using command line scripts.

...
Hemant J.
$35/hr
Hemant J.

Big Data Developer

5.0/5(6 jobs)
Hyderabad, TELANGANA
  • Trophy Icon Big Data
  • Business Intelligence
  • Apache Spark
  • ETL
  • iReport
  • Jaspersoft Studio
  • Data Warehousing
  • Talend Data Integration
  • Java
  • Talend Open Studio
  • JasperReports

Certified Talend and AWS Developer with 8+ Years of experience with vast knowledge on Data Integrations projects. Skills include Talend Data Integration, Talend Big Data, AWS developer.

...
POLYCHRONIS A.
$45/hr
POLYCHRONIS A.

Big Data Developer

5.0/5(53 jobs)
Athens, ATTICA
  • Trophy Icon Big Data
  • Web Crawling
  • Data Scraping
  • Web Scraper
  • Python
  • Machine Learning
  • Apache Spark
  • Python Pandas
  • Scrapy
  • ETL Pipeline
  • API

I have great experience in web scraping and ETL, mainly using Python and the panda's library. I am familiar with proxies and many scraping techniques. Also, running them on the cloud is my forte, as I am familiar with many cloud services. Finally, I also have experience in Big Data and machine learning using Apache Spark (both Scala and Python). I have acquired that from my job as a freelancer, in which I applied machine learning algorithms to economic data (cryptocurrency).

...
Mohamed R.
$75/hr
Mohamed R.

Big Data Developer

4.7/5(18 jobs)
El Aiun, Morocco
  • Trophy Icon Big Data
  • pandas
  • Statistical Programming
  • R
  • Statistics
  • R Shiny
  • Data Scraping
  • Shiny
  • Stata
  • Django
  • Python
  • API Integration
  • Data Mining
  • Machine Learning
  • Marketing Analytics

***********************I shall either find a way or make it one******************** I'm an Engineer in Operational Research , I've got many skills and expertise that will allow me to achieve perfectly all the projects and missions. I have advanced knowledge's in: Vba Excel Optimization Queuing Theory Linear Programming Economics modeling, Data Mining: Factor analysis, Principal Component Analysis, Regression (Simple, Multiple, logistic, hierarchical, Poisson), Anova, Clustering, Hierarchical Clustering... I master Lingo, Ampl, Spss, Stata, R, Eviews, I'll provide you perfect reporting using MsWord. Also I have a huge experience / Knowledge with creating software for scraping/extracting data from web site with VBNET

...
Joaquin M.
$100/hr
Joaquin M.

Big Data Developer

5.0/5(9 jobs)
Wellington, FL
  • Trophy Icon Big Data
  • Artificial Intelligence
  • Machine Learning
  • Natural Language Processing
  • Analytics
  • Business Intelligence
  • Cloud Computing
  • Azure
  • Deep Learning
  • AWS Glue
  • PySpark

2019-20. Researched, analyzed, designed, coded and implemented an automated car collision detection system for US car insurance company based on a convolutional neural network using Python, Tensorflow 2.0 and keras 2.0, as well as digital processing algorithms and GoogleMaps APIs. This system provides all the information needed for an operator to contact the nearest police and hospital with the accident location within a minute of the accident occurring, as well as providing a second-by-second animation useful for accident reconstruction. 2019 Researched, analyzed, designed, coded and implemented an Asset Management Risk Management system for a major asset investment firm based on a deep learning neural network, hidden markov chain, time series and NLP algorithms to reduce the expected risk associated with asset management an average of 10%. Used Python, Tensorflow, scikit-learn, BERT, spaCy and keras. The deployment platform was AWS Cloud with EC2, Sagemaker and AWS Deep Learning Containers. 2018. Recently received, in partnership with Oracle, the 2018 Innovation Challenge Award from The Guardian Life Insurance Company for architecting a prediction analytics system to increase sales of insurance products to new customer prospects using machine learning and neural networks. 2017-18. Researched, analyzed, designed, coded, implemented and deployed complete predictive/prescriptive analytics platform and dynamic pricing/ yield management on Azure Cloud for major international parking systems/services corporation to allow parking owners to predict potential demand for parking and maximize their profits by 15-20%. 2014-17. Led digital transformation of sales and marketing groups of a couple very large international corporations using predictive/prescriptive analytics and machine learning, from generating new sales leads to creating strategies for international sales campaigns, increasing selling revenue 3- to 6-fold, and cross-sales/up-sales by $100M+. 2008. As Chief Architect, lead a team of 30 architects in the US and 120 developers/dbas in India, to design and implement Ally Bank, the first US online bank, in a record six months, working 70-90 hours per week. Was responsible for securing a TARP grant of $6.5 billion by meeting extremely tight deadline.

...
Julia S.
$75/hr
Julia S.

Big Data Developer

5.0/5(17 jobs)
Seattle, WA
  • Trophy Icon Big Data
  • Researcher
  • Internet Research
  • Project Management
  • Agriculture & Forestry
  • Data Entry
  • Market Research
  • Public Speaking
  • Project Planning
  • Writing

Experienced researcher and communicator with a demonstrated history of linking disparate pieces of information to produce new ideas and knowledge products. Quick learner who enjoys working with culturally diverse teams and stakeholders and values: openness, inquisitiveness, honesty and integrity. Background in international agricultural development, honing skills in project planning and management, securing grants and engaging in an array of projects spanning the globe.

...
Archana G.
$75/hr
Archana G.

Big Data Developer

5.0/5(5 jobs)
Milpitas, CA
  • Trophy Icon Big Data
  • Microsoft Excel
  • Tableau
  • Data Mining
  • SQL
  • Machine Learning
  • R
  • Python
  • TensorFlow
  • Data Analysis
  • Keras
  • Artificial Intelligence
  • Convolutional Neural Network
  • Deep Neural Network
  • Data Science

Work Experience Data Scientist –– Wonder Chrome (December 2021 - PRESENT) Artificial Intelligence Engineer –– Uniquify Inc (August 2021 - October 2021 ) ● Tensorflow and neural network training ● Debugging the framework for automating neural network and tensorflow scripts ● Running experiments with the tensorflow scripts produced by the framework ● Manipulating data and neural network structure and specs to optimize accuracy in the neural networks ● Analyze the results from training the neural network models to better understand and improve the models and frameworks ● developing image processing and image segmentation algorithms Data Analyst –– Centriqe Inc (February 2020 - January 2021) ● Provide insights and proposals to support business improvement using analytical and technical expertise ● Build predictive models and forecasting models using various machine learning tools ● Actively engaged in the quantitative analysis of sophisticated models to address business issues ● Identify the trends and key metrics and generate dashboards using various data visualization tools. Technical Skills Languages : Python, R, Core Java Data Analysis : Data manipulation techniques, Plotting and visualization, Exploratory data analysis, Estimation techniques, Regression model, Simulation techniques Machine Learning and Data Mining : Bayesian classifiers, PCA , Linear classifiers and regression, Classifier performance evaluation, KNN, Hidden Markov models, Ensemble learning and Decision trees, Neural networks, Natural Language Processing Deep learning and AI: TensorFlow, keras, Multilayer perceptrons, CNN, RNN Testing Tools : Selenium WebDriver, Cucumber,TestNG, SOAPUI, Maven, Postman Other : Elasticsearch, Logstash, Kibana , Plotly, Excel, Pandas, Numpy, Matlibplot, Seaborn, Sklearn,ggplot, BeautifulSoup, Spacy, Mongodb, Robo 3T, MySQL, PostgreSQL, AWS

...
Gilberto P.
$35/hr
Gilberto P.

Big Data Developer

5.0/5(5 jobs)
Bogota, BOGOTA
  • Trophy Icon Big Data
  • Robotic Process Automation
  • Automation Anywhere
  • Project Management
  • Product Roadmap
  • Project Management
  • Product Backlog
  • Agile Project Management

Systems Engineer, with more than 16 years of experience in Technology Management over processes, business solutions, and infrastructure in Cloud and OnPremise environments, in strategic, operational, and technical positions for Latin America in the Banking, Energy, and Technology sectors with multicultural teams, applying best practices, and existing methodologies.

...
Nino A.
$70/hr
Nino A.

Big Data Developer

5.0/5(2 jobs)
Skopje, GRAD SKOPJE
  • Trophy Icon Big Data
  • Business Intelligence
  • PostgreSQL
  • Database Administration
  • Amazon Redshift
  • Data Warehousing
  • Data Visualization
  • Docker
  • Linux System Administration
  • Python
  • Data Science
  • Machine Learning
  • Cluster Computing

Having a Master's Degree in Computer Science, specialized in intelligent systems, I have worked intensively with different technologies over the past six years. My focus is on systems architecture, data analysis, data processing and anything related to massive data flows. This includes machine learning (design and development of machine learning models), clustering, database administration and tuning, data warehouse and data lake design, NLP, web scraping, DevOps and many more. I have worked on various machine learning research projects at Stanford University and have worked remotely with professors from Harvard. These projects were focused on designing case-specific machine learning models, generating interpretable explanations, reinforcement learning, and so on. I have also intensively worked with optimization algorithms for different real-world problems, such as bail-out decisions, employee metric analysis and prediction, data mining, reinforcement learning algorithms, network embedding algorithms, network modeling, clustering and so on. On the other hand, I have worked with huge databases and complex data models with thousands of entities. This was a part of complete re-design of existing databases with sensitive data about students and citizens, data warehouse design, ETL pipeline development and Business Intelligence development. I love DevOps: Docker, Kubernetes, graph databases, message brokers and anything that help us develop highly scalable and elastic infrastructure to run interesting code including ML models. I love speeding up Python code, serving TensorFlow models, designing micro-services, load balancing, and, of course, in-memory databases suitable to massive transactions (especially writes). I also have experience in academic writing, project proposal writing, product development, project management and project analysis.

...
Akram A.
$50/hr
Akram A.

Big Data Developer

5.0/5(1 job)
Bulandshahr, UP
  • Trophy Icon Big Data
  • Data Analysis
  • Apache Hive
  • Sqoop
  • Informatica
  • SQL
  • Bash Programming
  • Python
  • Java
  • ETL

• 8+ years of data product development experience including 5+ years of experience in big data engineering development along with 7+ years of experience in data Engineering, data warehousing and business Intelligence. • Good Experience building systems to perform real-time data processing using spark streaming, Kafka, spark sql, pyspark and cloudera. • Worked extensively with dimensional modeling, data migration, data cleansing, data profiling, and ETL processes features for data lake and data warehouse. • Design and build ETL pipelines to automate ingestion of structured and unstructured data in batch and real time mode using Nifi, Kafka, spark sql, spark streaming, hive, Impala and different ETL tools. • Worked with multiple ETL tools like Informatica Big Data Edition 10.2.2., Alteryx, Talend, Kalido. • Good knowledge of Azure Databrick, Azure HDInsight, ADLS, ADF and Azure storage Analyzed and processed complex data sets using advanced querying, visualization, and analytics tools.

...
Anuj S.
$35/hr
Anuj S.

Big Data Developer

5.0/5(11 jobs)
Dubai, DUBAI
  • Trophy Icon Big Data
  • Apache Spark
  • Scala
  • Apache Kafka
  • Apache HBase
  • CI/CD
  • Apache Hive
  • Analytics
  • Python
  • Machine Learning
  • PySpark
  • Apache Cassandra

Technology skills - Hadoop 2.0, Real-Time streaming and batch, Python, PySpark, Scala, Java, Spark, Hive, Certified AWS, Certified ML Engineer in Python, Sqoop, Apache Kafka, Hbase, Cassandra, Unix (Shell scripting) Responsibilities includes - Design, Development and Automate Big Data solutions in various ecosystems. 12 Years IT Experience and handled more than 15 big data projects in Python, Scala and Java Data Lake Raw - Development and Quality checks Skills to transfer and check data between external systems and clusters. This includes the following: Ingest batch, real-time and near-real-time streaming data into HDFS Process batch structured data as it loaded from RDBMS(SQL, Oracle, CSV, etc.) into HDFS Process XML/Json from kafka streaming data as it is loaded onto HDFS / NoSQL (Hbase) Process Quality checks using Apache Griffin / Automation Test framework Data Lake Transform Filter raw data and store back to HDFS different file formats (parquet/avro) Process Quality checks using Apache Griffin / Automation Test framework Data Modelling - Transform, Stage, and Store Convert a set of data values in a given format stored in HDFS into new data values or a new data format and write them into HDFS. Read and write files in a variety of file formats Perform standard extract, transform, load (ETL) processes on data Data Analysis Use Spark RDD, Dataframes and Datasets to process the daily reports based on historical data Generate reports in csv/hive against loaded data. Write queries that calculate aggregate statistics Test generated report against business logic using hive/spark SQL with the help of framework written in spark scala

...
Nakiboudine M.
$42/hr
Nakiboudine M.

Big Data Developer

5.0/5(2 jobs)
Paris, France
  • Trophy Icon Big Data
  • CSS
  • HTML
  • SQL
  • Node.js
  • React
  • Vue.js
  • Ruby
  • Docker
  • DevOps

My name is Naki, and I'm a Full Stack Developer based in Paris. I worked for five years as a software developer for startup companies and law firms. My most accomplished work was automating the analysis of financial accounting books and fraud detection at Arsene Innovation, where I created the algorithm to detect fraud in financial accounting books instantly. Here are the technologies I work on : Tools & Frameworks: Liquid, Shopify, WordPress, Git, Websockets, Docker, Virtual Machines. Frontend: Javascript, React, jQuery, Bootstrap. Backend: Node.js, MySQL, Ruby, MongoDB, Redis, Apache, Nginx.

...
Anurag B.
$60/hr
Anurag B.

Big Data Developer

5.0/5(3 jobs)
Ottawa, ON
  • Trophy Icon Big Data
  • Data Visualization
  • Data Mining
  • Machine Learning
  • Agile Software Development
  • Web Scraper
  • Exploratory Data Analysis
  • Statistical Analysis
  • Natural Language Processing
  • Deep Learning

I am currently a Lead Data Scientist at RBC Ventures and have a Masters Degree in Big Data from Simon Fraser University - Vancouver. I have a great amount of experience with Machine Learning, Data Analysis, Natural Language Processing, Statistical Analysis, Data Visualization, and other Automated processes. I can provide services like data collection through web scrapping or perform data analysis with interactive visualization. You can check out my LinkedIn profile or GitHub repository to get a sense of my previous works.

...
Bekpasha D.
$35/hr
Bekpasha D.

Big Data Developer

5.0/5(4 jobs)
Almaty, ISTANBUL
  • Trophy Icon Big Data
  • JavaFX
  • Deep Learning
  • Natural Language Processing
  • Probability Theory
  • Python
  • Calculus
  • C++
  • Statistics
  • Machine Learning
  • Reinforcement Learning
  • R
  • HTML
  • CSS
  • PHP

I have more than 3 years of experience in Data Science and around 5 years in programming. I have a deep knowledge in mathematics (calculus, linear algebra, probability, statistics, etc.), great programming skills, and proficiency in ML/DL. My professional background consists of the experience in Competitive Programming in high school years, a bachelor's degree in Computer Science, study at Big Data Academy MADE of Mail.ru Group, and work experience in Big Data projects for leading companies in Russia/Turkey/Kazakhstan. Areas of expertise: - Computer Vision (Image classification, Image segmentation, and detection, Facial detection) - Natural Language Processing (Text classification, Text generation, Machine translation, Sentiment analysis, etc.) - Data processing and analysis - Unsupervised learning and clustering - Search systems - Time Series Analysis and ARIMA models - Distributed systems - Data Structures and Algorithms - C, Python, C++, C# - R programming language, Statistics and AB calculus. I have expertise and skills in the following topics and tools: - Machine Learning (Deep Learning): Scikit-learn, PyTorch, NumPy, Pandas, Matplotlib, Seaborn, Jupyter Notebook, Anaconda, NLTK, Word2Vec, Gensim - Machine Learning Algorithms: Linear Regression (+ Polynomial Regression), Logistic Regression, Naive Bayes, KNN, Random Forest, Decision Trees, Gradient Boosting, Support Vector Machines, PCA, Clustering algorithms (KMeans, DBSCAN, Agglomerative Clustering) - Time Series: ARIMA models, Correlation Analysis - Programming languages: Python, Java (Scala, Kotlin), C/C++ - Big Data: Apache Hadoop, Apache Spark, HDFS, MapReduce, Apache Phoenix, Apache Kafka - Web Frameworks: Spring Boot, Django, Flask - Database Technologies: PostgreSQL, MongoDB, Redis, MyBatis, Liquibase - DevOps Tools: Docker, Docker Compose, Gitlab Pipelines, Git - Cloud and infrastructure: Amazon Web Services, DigitalOcean - Work Management Tools: Jira, Confluence, Trello I guarantee you will be satisfied with the work done by me and both of us will gain great experience of cooperation.

...
Gunjan R.
$35/hr
Gunjan R.

Big Data Developer

5.0/5(4 jobs)
Bogota, DC
  • Trophy Icon Big Data
  • Kubernetes
  • Docker
  • Git
  • Unix
  • CI/CD
  • Amazon Web Services
  • Python
  • Java
  • AWS CloudFormation
  • Terraform
  • DevOps

I have 9 years of total experience with strong expertise in AWS, Azure, DevOps, CI/CD Pipelines, Automated Builds & Deployments, UNIX Scripting, Terraform, Docker-Kubernetes, Atlassian Tools Administration, Infrastructure As a Code, JAVA and Python Scripting. I have hands on experience on Revision Control System (SVN, Gitlab, Bitbucket and Git), and continuous integration using Jenkins for Java, Android, Big data projects. I also have experience in using DevOps tools such as Ansible, Cloud-Formation and Lambda. Training and certifications : AWS Certified Solutions Architect-Associate Level (Credential Id : K2561DXK3F14Q6K3) Programming and scripting languages : UNIX, Python, Perl, Java, C++ Frameworks, tools, and libraries : JIRA, SVN, GIT, Stash, TeamCity, HTML, YAML, Jenkins, Maven, Git, SVN, Lambda, RDS, CloudWatch, EMR, ECS, S3, VPC, ELB, Docker, Ansible, Cloud Formation, GitLab, TortoiseSVN, TortoiseGit and GitLab Servers and platforms : Apache, JBoss Devices and OS : Linux (Centos, Red Hat, Ubuntu), Windows

...
Muhammad A.
$70/hr
Muhammad A.

Big Data Developer

5.0/5(42 jobs)
Karachi, SINDH
  • Trophy Icon Big Data
  • ETL
  • Apache Hadoop
  • Amazon Redshift
  • Apache Spark
  • AWS Glue
  • Data Warehousing
  • Data Management
  • ETL Pipeline
  • Python
  • SQL
  • Data Visualization
  • Apache NiFi
  • Marketing Analytics
  • AWS Lambda
  • Solution Architecture Consultation

𝗖𝗲𝗿𝘁𝗶𝗳𝗶𝗲𝗱 𝗗𝗮𝘁𝗮 𝗣𝗿𝗼𝗳𝗲𝘀𝘀𝗶𝗼𝗻𝗮𝗹 with 𝟲+ 𝘆𝗲𝗮𝗿𝘀 of experience and hands-on expertise in Big Data, Data Engineering, Data Warehousing and Data Analytics. Looking for someone with a broad skill set, minimal oversight and ownership mentality then contact me to discuss in detail the value and strength I can bring to your company. 𝙄 𝙝𝙖𝙫𝙚 𝙚𝙭𝙥𝙚𝙧𝙞𝙚𝙣𝙘𝙚 𝙞𝙣 𝙩𝙝𝙚 𝙛𝙤𝙡𝙡𝙤𝙬𝙞𝙣𝙜 𝙖𝙧𝙚𝙖𝙨, 𝙩𝙤𝙤𝙡𝙨 𝙖𝙣𝙙 𝙩𝙚𝙘𝙝𝙣𝙤𝙡𝙤𝙜𝙞𝙚𝙨: ► BIG DATA & DATA ENGINEERING Apache Spark, Hadoop, MapReduce, YARN, Pig, Hive, Kudu, HBase, Impala, Delta Lake, Oozie, NiFi, Kafka, Airflow, Kylin, Druid, Flink, Presto, Drill, Phoenix, Ambari, Ranger, Cloudera Manager, Zookeeper, Spark-Streaming, Streamsets, Snowflake ► CLOUD AWS -- EC2, S3, RDS, EMR, Redshift, Lambda, VPC, DynamoDB, Athena, Kinesis, Glue GCP -- BigQuery, Dataflow, Pub/Sub, Dataproc, Cloud Data Fusion Azure -- Data Factory, Synapse. HDInsight ► ANALYTICS, BI & DATA VISUALIZATION Tableau, Power BI, SSAS, SSMS, Superset, Grafana, Looker ► DATABASE SQL, NoSQL, Oracle, SQL Server, MySQL, PosgreSQL, MongoDB, PL/SQL, HBase, Cassandra ► OTHER SKILLS & TOOLS Docker, Kubernetes, Ansible, Pentaho, Python, Scala, Java, C, C++, C# 𝙎𝙤𝙢𝙚 𝙤𝙛 𝙢𝙮 𝙢𝙖𝙟𝙤𝙧 𝙥𝙧𝙤𝙟𝙚𝙘𝙩𝙨 𝙞𝙣𝙘𝙡𝙪𝙙𝙚𝙙 - Designing Big Data architectures for the financial and telecom sector to power their data-driven digital transformation. - Implementing Data Lake and Data Warehousing solutions using Big Data tools. - Developing ETL workflows using Apache Spark, Apache NiFi, Streamsets, Apache Airflow, etc. - Hands-on experience with Big Data and Cloud technologies in implementation and architectural design of Data Lake and Data Warehouse. - Experienced in working with Cloudera, Hortonworks, AWS, GCP, and other Big Data and Cloud technologies. 𝙒𝙝𝙚𝙣 𝙮𝙤𝙪 𝙝𝙞𝙧𝙚 𝙢𝙚, 𝙮𝙤𝙪 𝙘𝙖𝙣 𝙚𝙭𝙥𝙚𝙘𝙩: - Outstanding results and service - High-quality output on time, every time - Strong communication - Regular & ongoing updates Your complete satisfaction is what I aim for, so the job is not complete until you are satisfied! Warm Regards, 𝗔𝗻𝗮𝘀

...
Muhammad Akmal M.
$40/hr
Muhammad Akmal M.

Big Data Developer

5.0/5(9 jobs)
Khanewal, PB
  • Trophy Icon Big Data
  • Data Mining
  • Artificial Intelligence
  • Image Processing
  • Deep Learning Modeling
  • Python
  • Model Optimization
  • Data Science
  • Natural Language Processing
  • Apache Spark
  • Machine Learning Model
  • PyTorch
  • TensorFlow
  • Keras
  • Computer Vision

I am a highly skilled Data Scientist, with master's degree in Data Science, Top Rated Plus on Upwork, with extensive 4+ year of experience in the field, offers a broad range of NLP and Computer Vision services. I have an extensive experience building models for NLP, which comprises text preprocessing, sentiment analysis, topic modeling, text classification, OCR, Visual Question Answering, text summarization, document classification, named entity recognition NER, text generation, machine translation, speech-to-text and text-to-speech capabilities for audio data. I can utilize the latest NLP tools and technologies, such as SBert, spaCy, the Hugging Face library, and SentenceTransformer, to guarantee high accuracy and efficiency in all projects. I also have a very firm grip over the Computer Vision field and can handle a wide range of video processing tasks, including action recognition, object tracking, optical flow analysis, and scene segmentation. I can provide my services in tasks, which are critical for a variety of applications, such as sports analysis, medical imaging, and security systems, where real-time video processing is essential. I can handle your image-related problems such as object tracking, image segmentation, image classification, object detection, image captioning, visual question answering, and video processing. Whether you're looking to integrate multimodal models, such as CLIP, BLIP, Donut, LayoutLM v3, or utilize diffusion techniques for art generation models, including stable diffusion and DreamBooth training. I am well-versed in using the latest computer vision and NLP tools, technologies, libraries and frameworks, such as OpenCV, Python, Pytorch, and TensorFlow, SBert, spaCy, the Hugging Face library, SentenceTransformer.

...
Aleksei S.
$60/hr
Aleksei S.

Big Data Developer

5.0/5(10 jobs)
Dubai, DU
  • Trophy Icon Big Data
  • Databricks
  • Machine Learning
  • Scala
  • Python
  • Java
  • Apache Spark
  • Apache Kafka

Hi, my name is Aleksei. I'm working as data engineer. I have experience with Java, Scala, Python and Big Data technologies.

...
Muhammad Javaid I.
$35/hr
Muhammad Javaid I.

Big Data Developer

4.7/5(38 jobs)
Lahore, PUNJAB
  • Trophy Icon Big Data
  • Data Science
  • Machine Learning
  • Python
  • Deep Learning
  • MATLAB
  • Data Mining
  • Computer Vision
  • Image Processing
  • Natural Language Processing
  • Data Analysis
  • Artificial Intelligence
  • R
  • Data Science Consultation
  • Online Research

I have 5+ years of experience in Artificial Intelligence, Data Science, Machine Learning, and Deep Learning. After completing my Masters (MS) in Computer Science from COMSATS University Pakistan. I worked as a Researcher and Developer at JTech Pvt Ltd. Detailed experience: - Building, Testing Machine Learning, and Deep Learning Models - Natural Language Processing Techniques - Speech Recognition and Speech Generation - Integrating machine learning algorithms - Digital Image Processing Applications - Computer Vision Applications - Leading projects with Machine Learning and Analytics background - Data Analysis and Prediction Systems - Time Series Analysis - Building data analysis API's - Strong math and statistics background Skills: - Languages/Technologies: Python, R, MATLAB - Frameworks: TensorFlow, Pytorch, Keras, DJango - Libraries: Numpy, Scikit-Learn, Matplotlib, SciPy, Pandas, NLTK, LightGBM, XGBoost, etc. - DB/Storing: MySQL, MongoDB, SQLite - Version Control: GIT, Mercurial, SVN - Analysis: Statistic, Calculus, Classification/Clustering, Machine Learning, Deep Learning - Methodologies: Scrum, TDD Others: - AWS experienced user - Microsoft Azure - Google Cloud Engine advanced user - Google Data Studio - Data scraping - Kaggle experienced - IBM Certified in Deep Learning and Data Science Projects GitHub: javaidiqbal11

...
Maxime V.
$60/hr
Maxime V.

Big Data Developer

5.0/5(2 jobs)
Bois-Colombes, ÎLE-DE-FRANCE
  • Trophy Icon Big Data
  • Python
  • C++
  • Adobe Photoshop
  • Adobe After Effects
  • Excel
  • Python Scikit-Learn
  • Machine Learning
  • MySQL
  • Social Video Marketing

I am a Canadian & French engineer. I have two years of work experience in investment (VC & PE) and within software & data teams (at a NYC startup, then at a leading-edge VFX studio in Vancouver). I hold a MSc from Université Paris Saclay (ranked #1 best university in France in the Shanghai Ranking). Tech skills: - Python: Data science, machine learning, computer vision (Pandas, Scikit Learn, Open CV) - C++ - SQL

...
Want to browse more talent?Sign Up

Join the world’s work marketplace

Find Talent

Post a job to interview and hire great talent.

Hire Talent
Find Work

Find work you love with like-minded clients.

Find Work