The Role of Data Architecture for Data Management Professionals
Executive Summary
‘The Role of Data Architecture for Data Management Professionals’ explores the role of data architecture and management in today's business landscape. Howard Diesel expands upon the opportunity for Businesses to create enterprise data architecture and governance plans by understanding the data value chain and context diagrams. He suggests that it is essential to involve business people in data architecture, as data plays a significant role in the success of any business. The webinar explores Data flow and lineage and how they are essential aspects of data architecture, which helps create data consistency and integration. Data modelling and architectural development are also significant aspects of enterprise data architecture. Additionally, Cataloguing business catalogues for utility sites help businesses understand the data value chain, data latency, and data lineage. Lastly, Howard touches on data privacy and ethical frameworks as critical considerations for data handling, processing, and cataloguing.
Webinar Details:
Title: The Role of Data Architecture for Data Management Professionals
Date: 29 January 2021
Presenter: Howard Diesel
Meetup Group: Data Professionals
Write-up Author: Howard Diesel
Contents
Role of Data Architecture and Data Management
Data Value Chain and Data Architecture Discussion.
Understanding Data Architecture and Context Diagrams.
Enterprise Data Architecture and Data Governance.
Involving Business People in Data Architecture.
Importance of Data in Business.
Importance of Data Flow and Data Lineage in Business.
Key Roles and Responsibilities of Data Architects and Data Modelers.
Enterprise Data Modeling and Architecture.
Cataloguing Business Catalogues for Utility Sites.
Enterprise Data Modeling and Architectural Development
Data Integration and Business Consistency.
Importance of Data Architecture Practice in Business.
Governance Challenges in Agile Development
Agile Teamwork and Structure.
Understanding the Data Value Chain.
Data Latency and Data Lineage.
Understanding Data Lineage and Data Value Chain.
Data Handling and Value in Processing.
Data Catalogs and Data Privacy.
Ethical Framework and Privacy in Business.
Discussion on Follow-Up and Contribution on LinkedIn.
Data Value Chain and Data Architecture Discussion
Howard Diesel discusses the contention in the data management industry about the right approach to building an enterprise data model or doing a data value chain. Newcomers to data management may need to ramp up quickly and understand the role of data architects, the use of the DMBOK framework, and the relevance of data policies in the data management process. Howard notes that the discussion aims to hear from others about their thoughts on data governance in architecture. Veronica provides a high-level explanation of data architecture and introduces the context diagram.
Figure 1 Data Architecture Context Diagram
Understanding Data Architecture and Context Diagrams
A context diagram is a composite of the DMBOK that provides a definition, goals, business drivers, inputs, activities, and deliverables related to data architecture. The activities in data architecture are divided into planning, tactical, and operational phases, with specific roles for participants like enterprise data architects and data modellers. Inputs for data architecture include enterprise architecture, including business architecture, data architecture, application architecture, and technology architecture, used to establish energy for data within the organisation.
Enterprise Data Architecture and Data Governance
Enterprise architecture and data governance are two crucial practices that help establish and maintain standards for naming conventions, data strategies, and the data required by businesses. The outputs from these activities become the inputs for data integration and modelling and are reviewed and approved by data governance. To establish an effective enterprise data architecture and data governance practice, a high-level checklist in the form of a context diagram is necessary, and alignment, collaboration, and communication are key in maintaining a strong relationship between the two practices.
Involving Business People in Data Architecture
The importance of involving business people in data architecture to ensure alignment with the business and increase the value of data to the organisation is discussed. Howard emphasises the need for data governance, strategy, architecture, and modelling to align with the business and for data quality to be business-focused. Using context diagrams as a reference and involving subject matter experts in the process is highlighted as crucial for success.
Importance of Data in Business
Howard notes that the focus on business processes is shifting towards data landscape and model diagrams, as data is becoming a key driver in decision-making. With the emergence of artificial intelligence and data-driven decision-making, machine learning and AI are introducing strategic decision-making beyond traditional business processes. The importance of aligning data with business goals is emphasised to show value to the business. The transition from academic and theoretical discussions to practical applications is encouraged, marking a shift from the traditional focus on business processes. Data is now seen as vital in driving business decisions and insights, and its significance is starting to be realised.
Importance of Data Flow and Data Lineage in Business
Demonstrating the value of data flow in an organisation is crucial, especially for protecting and controlling assets, such as in financial services under BCBS 239. Data lineage, value chain, and enterprise data model are important components to achieve this. However, many organizations lack data architects who can provide data flow diagrams and information on the journey of data within the organisation. Thus, understanding and demonstrating data lineage and flow is essential to avoid the risk of distributing incorrect information to external parties
Figure 2 Data Architecture Business Drivers: Goal - Structures
Key Roles and Responsibilities of Data Architects and Data Modellers
Data architects and modellers are essential in creating and managing enterprise data. While data architects are responsible for developing conceptual models of the entities and data flow at a high level, data modellers focus on creating database structures that support specific applications or business needs. Data architects ensure data integration and provide blueprints to guide data harmony and consistency across the organisation. In contrast, data modellers work on master data projects and may be involved in looking at data across the organisation. Overall, the main goal of data architects is to design structures and plans to meet the current and long-term requirements of the enterprise rather than delivering business applications.
Enterprise Data Modelling and Architecture
To ensure data and design consistency across all areas, architects are responsible for enterprise taxonomy, ontology, and business terminology. Enterprise data models, derived from industry models and customised for specific organisations, enforce business rules consistently across all applications. These models also provide blueprints and cover all required concept models. Enterprise data modelling ensures that requirements and concepts are converted into project development. Business glossary and conceptual modelling are essential for expressing business terms and relationships. Business stewards and quality checks are involved in conceptual modelling for business glossary maintenance.
Cataloguing Business Catalogues for Utility Sites
The utility industry is facing a problem procuring spares for their different sites due to inconsistent numbering and versioning of the catalogue for each site. To address this issue, the enterprise data model must be enforced consistently and operationalised through a master data application to prevent duplicate or different pricing across sites. This requires differentiation between conceptual, logical, and physical models, alignment between enterprise and master data models, and sign-off from the data architect to ensure consistent rule enforcement.
Enterprise Data Modelling and Architectural Development
The business rule on the model stipulates that only one price is allowed throughout the organisation, while the architect argues that different sites can have their prices. The architect suggests verifying if the allowance for each site to have a different price is valid per the business requirement; if wrong, the enterprise data model needs to be updated. Backfilling is harvesting application models and incorporating them into the enterprise data model in cases where there is no industry model. The logical level can be translated into various physical representations like a JSON document, an ER design, a document database, or a knowledge graph database, and the business solution remains unchanged irrespective of the physical implementation. Data modelling tools are being installed on production to investigate and construct a data model from JSON documents, ensuring conformity with the industry model and discovering new rules. The outcome and business advantage of enterprise data modelling is aligned within the organisation, making it an indispensable part of its data management strategy.
Data Integration and Business Consistency
Data integration plays an essential role in an organisation's data management strategy. It enables data sharing between departments and ensures it is easily understood and aligned with standardised rules and terminology. However, integrating data can be challenging, and it often takes dedicated subject matter experts to make sense of it and align it with business rules. The move towards business platforms and integrated systems like SAP and cloud applications drives the shift away from siloed organisational structures. Nonetheless, non-aligned fields in systems can still pose a challenge. Failure to integrate data effectively can hinder effective decision-making and cause delays of up to three months in understanding data from another department.
Importance of Data Architecture Practice in Business
Howard highlights the importance of having a data architecture practice in a business. Integrating data between different areas is essential, as different applications have their strengths and weaknesses. Howard also mentions that big applications like SAP should not be assumed to handle all data effectively and that best-of-breed applications like Salesforce may outperform SAP in certain areas. He emphasises that the misalignment of operations, lack of data integration, and different business rules could be the consequences of not having a data architecture practice in the business. Data architects should be able to articulate the benefits and value they bring to the business, addressing misaligned operations and data integration.
Governance Challenges in Agile Development
Effective governance in agile development requires a balanced focus on alignment and control while addressing business needs. However, a rigid focus on enforcing rules and standards can result in intolerance of deviations and resistance to business needs. Howard notes that flexibility in delivery and data modelling may be required to prioritise business demands over consistency to ensure success. Collaboration between data modellers and architects is essential for effective project delivery, and involving architects in story planning can help prevent friction and gaps in project implementation.
Agile Teamwork and Structure
Agile project management emphasises team cohesion and overall success rather than individual success. To achieve effective implementation, team members must be appropriately aligned and coordinated. Governance and compliance should not hinder agility but instead be integrated into an assurance policy to establish a three-line defence system for overseeing and coordinating work. To ensure smooth collaboration, rules and structure must be established upfront. Each team member's role is important; no one should overshadow anyone else. In addition, the enterprise data model and data lineage are critical aspects of agile project management that should agree for cohesive flow and alignment.
Understanding the Data Value Chain
Focusing on the source and people supplying it is critical to build trust in data and use it effectively. Data lineage and flow help us understand the origin and path of the data, while provenance emphasises the importance of trusting the data supplier. Understanding the data's journey helps determine its timeliness and latency, allowing for better management and utilisation of the data.
Data Latency and Data Lineage
Latency issues can be a major obstacle to deriving insights from data, as delays in data reaching users can impact response time. A value stream model can help measure the time it takes for data to reach users promptly. For data to be useful, it must be up-to-date. Data lineage and flow diagrams are crucial for GDPR compliance, showing the processing path and who processed the data. However, handling child data requires authorisation from regulators to avoid legal consequences. Ensuring data processing is done appropriately is crucial for avoiding risks and civil penalties.
Understanding Data Lineage and Data Value Chain
Data lineage and flow are critical aspects of data management, including the content and movement of sensitive data like card numbers, employee information, and children's details. Penalties and fines may result from unauthorised access or mishandling of data. The data value chain enhances the value and usefulness of the data to the business by understanding its relevance and target personas and leveraging it to make better decisions. However, data barriers, such as distrust in transformed data, can lead to lost value. Trust and accuracy are, therefore, crucial to maintain and enhance the value of data.
Data Handling and Value in Processing
Quality checks are essential when moving data to avoid any loss of value. Aggregating data across dimensions can lead to the loss of important attributes, reducing the value of the data set. ETL processes can potentially discard essential data, leading to a loss of value. To quantify the value of data, a data value chain can be used to identify any increases or decreases, distinguishing between assets and liabilities. Furthermore, implementing machine learning algorithms can benefit businesses by providing insight into which data generates value and optimising digital decision-making processes.
Data Catalogues and Data Privacy
Data catalogues are becoming increasingly important for businesses as they provide valuable insights into which data generates the most value and help optimise digital decision-making processes. Data lineage, including data flows and business processes, plays a crucial role within data catalogues. However, using AI to enhance personal information carries an increased risk, especially when incorporating third-party data. Privacy impact assessments play a significant role in mitigating these risks and ensuring compliance with data privacy regulations.
Figure 3 Data Catalogue - Construct in Action
Ethical Framework and Privacy in Business
Howard notes that it is crucial to maintain operational efficiency without violating ethical and privacy rules. Employees should be educated and aware of ethical and privacy considerations, as they may have insights that management may not recognise. Bringing in risk management professionals into these discussions can also help to educate employees and ensure the protection of customers. Howard shares the example of Lego's commitment to the safety of the children they serve, seen in their rejection of integrating social media into their products. Choosing a different path can lead to innovative ideas for addressing issues.
If you want to receive the recording, kindly contact Debbie (social@modelwaresystems.com)
Don’t forget to join our exciting LinkedIn and Meetup data communities not to miss out!