Data engineer: the analytics role in high demand

Pro

Image: Stockfresh

27 July 2018

Data engineers are responsible for finding trends in data sets and developing algorithms to help make raw data more useful to the enterprise. This IT role requires a significant set of technical skills, including a deep knowledge of SQL database design and multiple programming languages. But data engineers also need communication skills to work across departments to understand what business leaders want to gain from the company’s large datasets.

Data engineers are often responsible for building algorithms to help give easier access to raw data, but to do this, they need to understand company’s or client’s objectives. It’s important to have business goals in line when working with data, especially for companies that handle large and complex datasets and databases.

Data engineers also need to understand how to optimise data retrieval and how to develop dashboards, reports and other visualisations for stakeholders. Depending on the organisation, data engineers may also be responsible for communicating data trends. Larger organisations often have multiple data analysts or scientists to help understand data, while smaller companies might rely on a data engineer to work in both roles.

Data engineer role
According to Dataquest, there are three main roles that data engineers can fall into. These include:

Generalist: Generalists are typically found on small teams or in small companies. In this setting, data engineers wear many hats as one of the few “data-focused” people in the company. Generalists are often responsible for every step of the data process, from managing data to analysing it. Dataquest says this is a good role for anyone looking to transition from data science to data engineering, since smaller businesses will not need to worry as much about engineering “for scale.”

Pipeline-centric: Often found in midsize companies, pipeline-centric data engineers work alongside data scientists to help make use of the data they collect. Pipeline-centric data engineers need “in-depth knowledge of distributed systems and computer science,” according to Dataquest.

Database-centric: In larger organisations, where managing the flow of data is a full-time job, data engineers focus on analytics databases. Database-centric data engineers work with data warehouses across multiple databases and are responsible for developing table schemas.

Data engineer responsibilities
Data engineers are tasked with managing and organising data, while also keeping an eye out for trends or inconsistencies that will impact business goals. It’s a highly technical position, requiring experience and skills in areas like programming, mathematics and computer science. But data engineers also need soft skills to communicate data trends to others in the organisation and to help the business make use of the data it collects. Some of the most common responsibilities for a data engineer include:

Develop, construct, test and maintain architectures
Align architecture with business requirements
Data acquisition
Develop data set processes
Use programming language and tools
Identify ways to improve data reliability, efficiency and quality
Conduct research for industry and business questions
Use large data sets to address business issues
Deploy sophisticated analytics programs, machine learning and statistical methods
Prepare data for predictive and prescriptive modelling
Find hidden patterns using data
Use data to discover tasks that can be automated
Deliver updates to stakeholders based on analytics

Data engineer salaries
According to Glassdoor, the average salary for a data engineer is $137,776 per year, with a reported salary range of $110,000 to $155,000 depending on skills, experience and location. Senior data engineers earn an average salary of $172,603 per year, with a reported salary range of $152,000 to $194,000.

Here’s what some of the top tech companies pay their data engineers, on average, according to Glassdoor:

Company Reported range Average annual salary

Amazon $78,000 – $133,000 $103,849

Hewlett-Packard $64,000 – $105,000 $86,164

Facebook $93,000 – $171,000 $122,695

IBM $90,000 – $116,000 $99,351

Data engineer skills
The skills on your resume might impact your salary negotiations — in some cases by more than 10 or 15%, depending on the skill. According to data from PayScale, the following data engineering skills are associated with a significant boost in reported salaries:

Scala: +17%
Apache Spark: +16%
Data warehouse: +14%
Java: +13%
Data modelling: +12%
Apache Hadoop: +11%
Linux: +11%
Amazon Web Services (AWS): +10%
ETL (extra, transform, load): +7%
Big data analytics: +6%
Software development: +2%

Becoming a data engineer
Data engineers typically have a background in computer science, engineering, applied mathematics or have a degree in other related IT fields. Since the role requires heavy technical knowledge, aspiring data engineers might find a bootcamp or certification alone will not cut it against the competition. Most data engineering jobs require at least a relevant bachelor’s degree in a related discipline, according to PayScale.

You will need experience with multiple programming languages, including Python and Java, and knowledge of SQL database design. If you already have a background in IT, or in a related discipline such as mathematics or analytics, a bootcamp or certification can help tailor your resume to data engineering positions. For example, if you’ve worked in IT but haven’t held a specific data job, you could enrol in a data science bootcamp or get a data engineering certification to prove you have the skills on top of your other IT knowledge.

If you do not have a background in tech or IT, you might need to enrol in an in-depth program to demonstrate your proficiency in the field or invest in an undergraduate program if you do not have a degree. If you have an undergraduate degree, but it’s not in a relevant field, you can always look into master’s programmes in data analytics and data engineering.

Ultimately, it will depend on your situation and the types of jobs you have your eye on. Take time to browse job openings to see what companies are looking for, and that will give you a better idea of how your background can fit into that role.

Data engineer certifications
There are only a few certifications that are specific to data engineering; however, there are plenty of other data science and big data certifications for you to pick from if you want to expand beyond data engineering skills. But if you are looking to prove your merit as a data engineer, any one of these certifications will look great on your CV: