Sanjay Srivastava faces a fundamental choice when designing and building data management environments: go with open source data management technology or buy commercial options?
For Srivastava, who works with customers on such choices, it comes down to the role data plays for the business.
“If you were born in the cloud and data is a core value and driver of your business, go open source,” said Srivastava, chief digital officer at business transformation services firm Genpact. “But not if you’re setting something up in your own environment, wanting to run it long-term, and using data to augment and support your core business.”
Many corporate data managers and IT managers find themselves in a similar situation: forced to choose between proprietary and open source software for their data management needs.
This is a choice that will likely be more common as the space expands.
Consider already the size of the market and its expected growth. Grand View Search assessed the global enterprise data management market to $72.8 billion in 2020, with an expected compound annual growth rate of 13.8% through 2028.
The high dollar value reflects the wide range of capabilities required for an enterprise data management program, as well as the many vendors and open-source options on the market.
That, however, is the heart of the challenge – the plethora of options. And, while Srivastava has a particular way of helping him decide whether to go proprietary or open source when creating an organization’s data management program, he and other experts have stated that every business should understand the benefits and challenges of these options and recognize that the choice between the two is not always obvious.
Rather, there are a series of trade-offs that must be considered.
“Open source offers cost-effective alternatives to expensive, off-the-shelf products,” said Sandhya Balakrishnan, US region manager for intelligent enterprise solutions at Brillio, a digital transformation consultancy. “However, the most common concerns with open source data management tools include security, lack of support, and often hidden costs associated with installation and ongoing maintenance.”
On the other hand, “many commercial data management tools successfully shield the complexity of data management for users,” Balakrishnan added.
Advantages and Disadvantages of Proprietary Data Management Software
Proprietary software is software whose source code is not available to users; it is sold by commercial entities as off-the-shelf solutions that may or may not be customizable to any degree.
Although its definition may seem too rigid, IT consultants have pointed out that proprietary data management systems offer several important advantages to organizations.
“The upside is that even though it’s something you paid more for [than open source]it will work in the enterprise for production,” said William McKnight, president of McKnight Consulting Group.
Many vendors offer solutions with a full range of add-on capabilities. They include integrations so that enterprise teams can more quickly and easily develop their data management environments. And they add more automation.
“They are reliable. They can deliver high performance at scale, in security, innovation and automation,” added Noel Yuhanna, vice president and principal analyst at Forrester Research.
Organizations also benefit from vendor support when opting for proprietary data management technologies, and they typically find it easier to hire the talent needed to implement and maintain data management software. commercial data – especially the more commonly used ones – versus open source options.
These are important considerations for organizations looking to rapidly advance their use of data, the experts said.
“Usability from a development and maintenance point of view, as well as the assurance of ongoing support and improvementsgives large enterprises the ability to scale by focusing on the right aspects of enterprise architecture,” Balakrishnan said.
However, proprietary data management software can have some potential downsides, experts say.
Enterprise teams cannot innovate on proprietary code and must instead rely on vendors to keep pace with the innovations needed to succeed in a rapidly changing digital landscape.
It costs more – especially in upfront costs – than open source options.
And there’s the possibility of staying locked in with one vendor, with the cost and challenges of switching to another vendor overwhelming the benefits of switching.
Advantages and Disadvantages of Open Source Data Management Software
Unlike proprietary data management software, open source options are released under a license that allows users to deploy the code to develop their own systems and also update, modify, and modify it for their own needs. .
This flexibility allows for the creation of data management solutions that meet the unique needs of each organization, McKnight said.
“With open source, if you want, you can create your own forks in code. For some, that might matter,” he said.
Open source is also cheaper to use.
“It’s obviously good for the budget, so if money is a constraining factor, then you can go open source,” McKnight added.
Additionally, companies can usually test open source options more easily, allowing them to run a proof of concept or pilot before deploying open source more widely or even moving to paid/plus enterprise versions. expensive.
Additionally, open source allows business teams to innovate on code and build on improvements other users make to it.
“With the open source community, you have more people contributing lines of code, so you’re going to get more innovation,” Srivastava said.
These benefits are driving many enterprise IT and data managers to embrace open source, Yuhanna added.
“The [COVID-19] The pandemic seemed to heighten the outlook on open source,” he said. “What we’ve seen is that open source tools definitely help you reduce your costs. It’s one of the drivers of its adoption, but open source can also help you avoid vendor lock-in and future-proof your architecture. »
As is the case with proprietary software, open source has some possible drawbacks. Open source is generally harder to integrate than proprietary alternatives, Yuhanna said, adding that it usually takes more work to get it to work well.
“It’s work that needs to be done on the spot,” he said.
Additionally, experts said organizations need technologists with the specialized skills required to create, deploy, maintain, and improve open source code. These technologists must track all changes to the code, and they may be required by license to contribute back to the open source community. They also need to be able to do all of this work without the 24/7 customer support that usually comes with commercial software products.
The future is a mix
Enterprise IT and data managers may not need to choose between proprietary and open source – it doesn’t have to be an all-or-nothing approach.
According to experts, they may instead use proprietary software for some needs and use open source for others. This, in fact, may be the optimal approach for many organizations.
“In our experience, best-of-breed is the norm with increasing acceptance of open source,” Balakrishnan said, noting that organizations could opt for commercial vendors for data movement and storage, but use open source options, such as Apache Kafka or Apache Spark for near-real-time data processing and Apache NiFi or Apache Airflow for orchestration or workflow management.
Others said organizations might want to use open source for pilots and proofs of concept, then move to commercial choices when scaling.
IT and enterprise data leaders are also increasingly turning to solutions that essentially combine the two, said Brad Ptasienski, partner at digital consulting firm West Monroe and its market leader in engineering and data analysis.
Noting that West Monroe thinks open source “is going to be the new normal and the mainstream for large-scale data processing and storage,” Ptasienski said he sees a lot of positives in using solutions. commercial “wrappers” that have open source at their core, but with some of the benefits of proprietary software surrounding it.
“It’s almost a hybrid approach,” he added. “It works more like a platform, but it’s open source at its core.”