Big Data Developer job description template

An effective description can help you hire the best fit for your job. Check out our tips to provide details that skilled professionals are looking for.

Trusted by


Tips for Writing a Big Data Engineer Job Description

A big data engineer is a professional who is responsible for the management of data sets that are too big for traditional database systems to handle. They create, design, and implement data processing jobs in order to transform the data into a more usable format. They also ensure that the data is secure and complies with industry standards to protect the company’s information. 

Below, we will cover a sample job description, exploring the daily responsibilities and necessary qualifications for a big data engineer. 

The Job Overview

We are seeking a big data engineer to join our data analytics team. The successful candidate will be responsible for overseeing the creation and maintenance of our database infrastructure, including collecting and maintaining data, ensuring the integrity of our data, and creating and training data models.

Responsibilities

Below are some of the responsibilities of a big data engineer:

  • Design the architecture of our big data platform
  • Perform and oversee tasks such as writing scripts, calling APIs, web scraping, and writing SQL queries
  • Design and implement data stores that support the scalable processing and storage of our high-frequency data
  • Maintain our data pipeline
  • Customize and oversee integration tools, warehouses, databases, and analytical systems
  • Configure and provide availability for data-access tools used by all data scientists
Job Qualifications and Skill Sets

Below are the qualifications expected of a big data engineer:

  • 3 to 5 years of relevant data engineering experience
  • Bachelor’s degree or higher in computer science, data science, or a related field
  • Hands-on experience with data cleaning, visualization, and reporting
  • At least 2 years of relevant experience with real-time data stream platforms such as Kafka and Spark Streaming
  • Experience working in an agile environment
  • Familiarity with the Hadoop ecosystem
  • Experience with platforms such as MapReduce, Apache Cassandra, Hive, Presto, and HBase
  • Excellent analytical and problem-solving skills
  • Excellent communication and interpersonal skills
Big Data Developer Hiring Resources
Explore talent to hire
Learn about cost factors
ar_FreelancerAvatar_altText_292
ar_FreelancerAvatar_altText_292
ar_FreelancerAvatar_altText_292

4.8/5

Rating is 4.8 out of 5.

clients rate Big Data Developers based on 1K+ reviews

Hire Big Data Developers

Big Data Developers you can meet on Upwork

  • $90 hourly
    Amar K.
    Big Data Developer
    • 5.0
    • (27 jobs)
    Bengaluru, KA
    vsuc_fltilesrefresh_TrophyIcon Big Data
    Google App Engine
    Software Development
    Web Development
    Machine Learning
    Google Cloud Platform
    Amazon Web Services
    BigQuery
    PySpark
    Apache Airflow
    Apache Spark
    Data Engineering
    SQL
    Python
    Java
    I pride myself on achieving a 𝗽𝗲𝗿𝗳𝗲𝗰𝘁 𝗿𝗲𝗰𝗼𝗿𝗱 𝗼𝗳 𝟱-𝘀𝘁𝗮𝗿 𝗿𝗮𝘁𝗶𝗻𝗴𝘀 𝗮𝗰𝗿𝗼𝘀𝘀 𝗮𝗹𝗹 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀. My expertise in 𝗰𝗹𝗼𝘂𝗱 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 and 𝗳𝘂𝗹𝗹-𝘀𝘁𝗮𝗰𝗸 development has been honed through experience with premier institutions like 𝗚𝗼𝗹𝗱𝗺𝗮𝗻 𝗦𝗮𝗰𝗵𝘀, 𝗠𝗼𝗿𝗴𝗮𝗻 𝗦𝘁𝗮𝗻𝗹𝗲𝘆, a member of the 𝗕𝗶𝗴 𝗙𝗼𝘂𝗿 and a 𝗙𝗼𝗿𝘁𝘂𝗻𝗲 𝟱𝟬𝟬 company. With over 9 years of experience in Data Engineering and Programming, I bring a commitment to excellence and a passion for perfection in every project I undertake. My approach is centered around delivering not just functional, but 𝗵𝗶𝗴𝗵𝗹𝘆 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗮𝗻𝗱 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 code, ensuring top-quality outputs that consistently impress my clients. My expertise combined with extensive experience on both GCP and AWS Cloud platforms, allows me to provide solutions that are not only effective but also innovative and forward-thinking. I believe in going beyond the basics, striving for excellence in every aspect of my work, and delivering results that speak for themselves. 𝗖𝗵𝗼𝗼𝘀𝗲 𝗺𝗲 𝗶𝗳 𝘆𝗼𝘂 𝗽𝗿𝗶𝗼𝗿𝗶𝘁𝗶𝘇𝗲 𝘁𝗼𝗽-𝗻𝗼𝘁𝗰𝗵 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀 𝗮𝗻𝗱 𝗮𝗽𝗽𝗿𝗲𝗰𝗶𝗮𝘁𝗲 𝗮 𝗳𝗿𝗲𝗲𝗹𝗮𝗻𝗰𝗲𝗿 𝘄𝗵𝗼 𝗮𝘂𝘁𝗼𝗻𝗼𝗺𝗼𝘂𝘀𝗹𝘆 𝗺𝗮𝗸𝗲𝘀 𝗼𝗽𝘁𝗶𝗺𝗮𝗹 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻𝘀, 𝘀𝗲𝗲𝗸𝗶𝗻𝗴 𝗰𝗹𝗮𝗿𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 𝗼𝗻𝗹𝘆 𝘄𝗵𝗲𝗻 𝗮𝗯𝘀𝗼𝗹𝘂𝘁𝗲𝗹𝘆 𝗻𝗲𝗰𝗲𝘀𝘀𝗮𝗿𝘆. 𝗔𝗿𝗲𝗮𝘀 𝗼𝗳 𝗘𝘅𝗽𝗲𝗿𝘁𝗶𝘀𝗲: - 𝗖𝗹𝗼𝘂𝗱: GCP (Google Cloud Platform), AWS (Amazon Web Services) - 𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝗺𝗶𝗻𝗴 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲: Java, Scala, Python, Ruby, HTML, Javascript - 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴: Spark, Kafka, Crunch, MapReduce, Hive, HBase, AWS Glue, PySpark, BiqQuery, Snowflake, ETL, Datawarehouse, Databricks, Data Lake, Airflow, Cloudwatch 𝗖𝗹𝗼𝘂𝗱 𝗧𝗼𝗼𝗹𝘀: AWS Lambda, Cloud Functions, App Engine, Cloud Run, Datastore, EC2, S3, - 𝗗𝗲𝘃𝗢𝗽𝘀: GitHub, GitLab. BitBucket, CHEF, Docker, Kubernetes, Jenkins, Cloud Deploy, Cloud Build, - 𝗪𝗲𝗯 & 𝗔𝗣𝗜: SpringBoot, Jersey, Flask, HTML & JSP, ReactJS, Django 𝗥𝗲𝘃𝗶𝗲𝘄𝘀: "Amar is a highly intelligent and experienced individual who is exceeding expectations with his service. He has very deep knowledge across the entire field of data engineering and is a very passionate individual, so I am extremely happy to have finished my data engineering project with such a responsible fantastic guy. I was able to complete my project faster than anticipated. Many thanks...." "Amar is an exceptional programmer that is hard to find on Upwork. He combines top-notch technical skills in Python & Big Data, excellent work ethic, communication skills, and strong dedication to his projects. Amar systematically works to break down complex problems, plan an approach, and implement thought-out high-quality solutions. I would highly recommend Amar!" "Amar is a fabulous developer. He is fully committed. Is not a clock watcher. Technically very very strong. His Java and Python skills are top-notch. What I really like about him is his attitude of taking a technical challenge personally and putting in a lot of hours to solve that problem. Best yet, he does not charge the client for all those hours, He still sticks to the agreement. Very professional. It was a delight working with him. and Will reach out to him if I have a Java or Python task."
  • $50 hourly
    Umar S.
    Big Data Developer
    • 5.0
    • (29 jobs)
    Lahore, PUNJAB
    vsuc_fltilesrefresh_TrophyIcon Big Data
    Financial Planning
    ETL
    Financial Modeling
    Microsoft Power BI Data Visualization
    Microsoft Power BI Development
    Financial Analysis
    Microsoft Power BI
    Microsoft Excel
    Here at Digital Accounting Services, we have a pool of Power BI Expertise: Data Model, Visualizations, Power Query (Get & Transform), Power Pivot, M, DAX Power Automate Expertise: Business process automation Excel Expertise: Pivot Tables, Charts, Nested If, Lookup, Array Calculations, VBA Bookkeeping Expertise: XERO (Certified Xero Advisor), MYOB HS Classification Expertise: HS code Assignment, Research Work Finance Expertise: Budgeting, business evaluations, and management reporting
  • $35 hourly
    Gaurav S.
    Big Data Developer
    • 4.8
    • (3 jobs)
    Delhi, NCT
    vsuc_fltilesrefresh_TrophyIcon Big Data
    Data Warehousing
    Serverless Computing
    Deployment Automation
    AWS Systems Manager
    ETL
    Service Cloud Administration
    Golang
    JavaScript
    Python
    Node.js
    React Native
    Expert in High level and low level system design. DevOps and distributed systems are my passion. Other than software development, I also provide corporate training on new and upcoming technologies. When it comes to programming, I am a polyglot programmer with expertise in python and golang and a love hate relationship with Node and PHP
Want to browse more talent? Sign up

Join the world’s work marketplace

Find Talent

Post a job to interview and hire great talent.

Hire Talent
Find Work

Find work you love with like-minded clients.

Find Work