Irion participated in the final conference of the Big Data & Business Analitycs Observatory of the Politecnico di Milano, of which it has been a Partner for years, to constantly engage with the professional and academic data community on the evolution of these issues.
Since 2021, large organizations that have adopted Data Governance roles and responsibilities have nearly doubled, but for the most part, companies are still at the beginning of their journey. In companies of 250 employees or more, these aspects have been formalized in 41% of cases (it was 25% last year): in particular, 27% have created centralized roles to provide guidelines, while 14% have involved multiple professional figures in different lines of business, without central coordination.
Among the specific solutions adopted for data governance, a major growth has been seen in Data Catalog tools, which increased from 15% in 2022 to 27% this year. Overall, the spending of Italian companies and thus the value of the market (as far as infrastructure, software, and data management and analytics services are concerned) shows a growth of 18%, touching €2.85 billion. 83% involve large enterprises, the rest is composed of SMEs and micro businesses.
These are the main trends that emerged from the 2023 survey of the Big Data & Business Analytics Observatory of the Politecnico di Milano. The conference presenting the research also highlighted trends in the use of generative artificial intelligence, which in data management processes, for example, can be relevant in the data labeling, categorization and classification phases. “Cloud is also increasing, but not at the growth rate I would have expected: it is a symptom of a certain resistance of enterprises to migrate their activities there,” points out Carlo Vercellis, professor of Machine Learning and director of the Observatory.
Data Strategy Index: where companies are today
Above average growth in manufacturing (+25%) and telecommunications and media (+20%). According to the “Data Strategy Index” calculated by the Observatory for large companies, precisely the telecommunications sector, along with financial services (banking and insurance) is the most representative in the cluster of “Advanced” organizations, those with high maturity in both Data Science and Data Management, as well as Business Intelligence. This is one-fifth of the sample. Relating these three variables, the researchers classified the remaining companies as “Immature” (16%; same figure for those “First Steps” that have succeeded in making cultural change, but still in their infancy on the rest) followed by “Intrapreneurial” (13%; often struggle to internalize skills), “Prudent” (23%, often start with little corporate involvement, now more focused on governance) and “Focused” (12%; prefer internal developments).
“We are moving from data-driven design to the concept of data product,” explained Irene Di Deo, senior researcher at the Observatory, recalling that for DJ Patil and Simon Regan, the central aspect is that data as a “product” must be focused on a goal. Today’s research shows that only 19% of the organizations surveyed perceive high or very high data quality and usability, while 43% of the sample admits to “some difficulties” (despite the fact that the data is considered generally usable) and 30 % say they are working to introduce “new guidelines.” The most macroscopic aspect is that nearly two out of three companies (62% of responses) have not yet defined metrics to measure the effectiveness of Data Management activities. Currently, only 13% say they adopt a “data as a product” approach.
The added value of Data Product
“The difficulty in evaluating hinders industrialization and the willingness to continue experimenting,” adds Di Deo. In half of the cases, there is also no structured process for gathering feedback from internal users. It is therefore an unconsolidated activity, although it is also true that “not all Data Science projects can become Data Products, perhaps because of the brevity of time and scope.” As for the deployment of Data Mesh architecture, on the other hand, 44 % of organizations say it is a topic currently being discussed in the enterprise. “But regardless of this architecture, even in less complex and structured organizations the Data Product approach will be critical to making sure that all data activities are effective, assessable, measurable, and appreciated by internal users,” the researcher concluded.
The basics: responsive architecture and metadata
“There are no longer only people, technology, processes and capital: to become truly data-driven companies the central aspect today is data as a productive factor,” highlights Mario Vellella, Domain Advisory Leader at Irion, in his speech at the Politecnico di Milano conference. “But data acquires value only if it is shared. And to do this we need a flexible and responsive architecture. As a first step, we need to create links between those who have the information need and those who will then have to build the Data Product.”
What does it take to make these two actors in the enterprise communicate? “A standardized semantic first layer: everything around the data as a product is based on metadata, not just descriptive, but also actionable. The requirements junction must go through semantics: metadata makes it possible to use the Data Product in practice. We have to take the risk of lowering the bar a bit: there is no point in designing maximum systems and then nothing is used. Better to have a good result on a ‘slice’ of knowledge, as long as it is not too small,” the expert explained.
Metadata are therefore a driver of change and flexibility, in a responsive architecture that must achieve semi-automatic results: “They also serve to make machines talk to each other. Today, the real platform is the customer’s data: we have to try to find all the services, wherever they are, and make sure we connect them via APIs, on a metadata layer that guarantees interoperability of information within. The architecture is built and rebuilt – as Gartner says, “business composable architecture”– also based on the roles (technical or business) that ask us for timely answers.”
Data Marketplace: six steps to success
“Irion was a pioneer in talking about rule-based data management, 20 years ago,” recalls Principal Advisor Mauro Tuvo, “then it evolved on end-to-end data management, listening to the market. But how is it possible to make visible the business value of the immense underlying work beneath the tip of the iceberg? First, by addressing decision makers who need to allocate budgets, for which the Value Based Data Governance model was created, developed by Irion in collaboration with Credem and university professor Franco Francia.
For end users, the Data Marketplace is the answer to the needs of data use in organizations. In Irion’s vision, Tuvo explained at the event, the Data Journey consists of six phases: Publish (creating and publishing a Data Product), followed by the Shop experience (the ability to search and choose from available products) and Checkout, and then moving on to the Tracking steps (fourth phase) critical to knowing who is using the available data and how.
This metric is also useful for understanding the “level of data quality” that can be delivered and maintained on an ongoing basis, depending on the interest of internal users; finally, Fulfillment i.e., our ability to deliver the data, following the approval of the relevant Data Owner; and the final Monitor step, which allows us to use the KPIs produced by our marketplace to keep track of data distribution and consumption. “However, the marketplace assumes certain prerequisites,” Tuvo points out, “we need to have worked with the appropriate Data Governance standards first to make it possible.”
For further details:
