Companies are increasingly collecting and analysing vast amounts of data to improve their services and decision making. However, the automation of data management inside organisations comes with multiple challenges and security is viewed as the top one, according to a new survey.
Data management automation, also known as DataOps, involves the use of multiple technologies and processes to simplify the collection, storage, access and analysis of data to enable agile, data-driven business decisions and product development. The problem is that without a strategy in place that takes security into consideration, businesses are at risk of experiencing serious data breaches that can result in financial losses and negative impact to their reputations.
Configure and secure
Over the past several years many large organisations have experienced data breaches not because of hacking, but because they failed to properly configure and secure various data-related technologies such as Amazon S3 buckets, Elasticsearch nodes, MongoDB databases, Hadoop deployments and others. In a survey conducted by 451 Research that was released today, two-thirds of respondents cited security and compliance as the top challenge associated with data management. This was followed by concerns related to data quality, the governance of distributed data, access to internal data and data pipeline reliability.
Understanding the compliance requirements related to data privacy was also viewed as a challenge by 40% of respondents.
The survey questioned 150 representatives from US-based organisations with over 1,000 employees and a minimum of 2 petabytes of managed data, and who had a good understanding of their organisation’s data management strategy. The organisations were from various industry segments, including banking, healthcare, technology, manufacturing, retail, government, insurance and telecom. Around a fifth considered DataOps already ingrained in their culture, 37% were in the course of DataOps adoption across multiple departments and 34% were still defining a DataOps strategy.
Distribution and volume growth
“Data security is a perennial concern, of course, while addressing data quality is essential if organisations are to ensure they are making decision based on accurate data,” 451 Research said in its report. “Additionally, the fact that data is increasingly distributed across multiple data platforms (relational, nonrelational and Apache Hadoop/Spark), and multiple locations (on-premises and multiple clouds) complicates the understanding of what data organisations truly have at their disposal.”
The push of moving data into the cloud is only expected to accelerate. Nearly a third of respondents said that their organisation’s data grows at a rate of 100 gigabytes to 500 gigabytes per day. Nineteen percent reported growth rates between 500 gigabytes and 1 terabyte per day, 12% between 1 terabyte and 2 terabytes and 13% over two terabytes per day. This means that’s it is becoming harder and more expensive for companies to store and analyse data on-premises.
In the cloud
According to 451 Research, 41% of surveyed organisations currently store more than half of their data in the cloud and over 70% of them are expected to reach that milestone over the next two years. Furthermore, 60% are already using NoSQL databases and almost half have distributed data clusters based on Apache Hadoop/Spark.
The increased popularity of these technologies and the fact that some of them were designed for internal use and lack strong access controls or secure default configurations could explain why security researchers are finding tens of thousands of deployments that are publicly accessible from the internet without authentication.
The good news is that many respondents whose organisations are well on their way into DataOps adoption are citing greater security and compliance as a benefit (66%). Other benefits according to the survey results include increased business agility and faster time to market (63%), improved decision making and business insight (55%), the introduction of application-driven business capabilities (54%) and cost reduction (49%).
DataOps concept
“While the concept of DataOps is perhaps most associated with operational efficiencies, the survey results indicate that those efficiency improvements are not just related to agility, but also to security and transformational change,” 451 Research said. “Companies that are already engaged with DataOps overwhelmingly agree that it is having a positive impact on their organisation, and while improved agility and efficiency are closely associated with DataOps, the biggest driver, priority and benefit is actually related to security and compliance.”
IDG News Service
Ltd
