Data Abstraction for Data Managers
Executive Summary
The topic of abstraction and its uses and dangers in development, object-oriented programming, and data modelling is discussed by Howard. He shares his experience of learning about the hazards of abstraction from Steve Hoberman’s course and how it can create more responsibility for developers.
The concept of abstraction safety guide and its use in simplifying data models and presenting information to business users is explored. Partitioning and absorption are also discussed. The process of specifying, from business requirements to technical details, and the definitions of "specification" and "abstraction" are explained in detail.
The different levels of understanding for specifications, such as a detailed description of work or materials for a project, instructions on how to do or make something, and architect specifications for a new building, are described.
Creating a believable visual presentation to define high-level requirements and using a conceptual model to get precise definitions for eliciting requirements to create a business glossary and threading logic from top to bottom is explained.
Webinar Details
Title: DATA ABSTRACTION FOR DATA MANAGERS
Date: 25 July 2023
Presenter: Howard Diesel
Meetup Group: Data Managers
Write-up Author: Howard Diesel
Contents
Abstraction and Data Models
Clarifying the Definition of Abstract for Specification.
Understanding the Importance of Specifications in Writing.
Visual Presentation and Data Modeling in Writing.
Understanding the Definition of Specification and Abstraction for Specification.
Process for Representing Essential Aspects in System Modeling.
The Six Friends of Writing Business Definitions.
The Importance of Defining Business Terms and Building System Models.
Abstraction for Specification in Software Engineering.
Agile Development and Test-Driven Development
Importance of Differentiating Functional and Non-Functional Requirements in Writing.
Analysis Artifacts and Data Life Cycle Management
Understanding Abstraction and Implementing Processes.
Notes on Agile Development and the Zachman Framework.
Developing a System Logic and Process Model
Importance of Avoiding Unnecessary Details in Business Presentations.
Importance of defining critical data elements and controlling scope in the conceptual model of a project
Abstraction and Data Models
Howard Diesel opens the webinar by highlighting the benefits of abstraction in master data projects. This technique can help in creating conceptual models and simplifying data to its minimum. However, it's crucial to understand the problems that can arise from abstraction, especially in relational data models.
Different abstraction methods exist, such as removing characteristics, using shapes and patterns, simplifying for presentation, and considering general qualities apart from concrete realities. The choice between absorption and partitioning in data modelling depends on the reasoning and understanding of the DBAs.
Howard stresses the importance of focusing on abstraction for specification. According to Google search results, abstraction enables modular programming by hiding implementation details.
Figure 1 Dictionary Definition: Abstraction
Clarifying the Definition of Abstract for Specification
Howard clarifies that they are not discussing hiding implementation details in coding modules. Instead, they are talking about different levels of specifications, including business requirements, functional specifications, and technical specifications. To help understand the process of specifying, Howard references the V model.
Before referring to the generated definition by ChatGPT, Howard wanted to focus on the general definition of specification, which he found in the Oxford definition. It defines specification as precisely identifying something or stating a precise requirement.
In addition to this definition, specification can also refer to a detailed description of the design and materials used to create something, particularly in the context of database design and implementation.
Howard also references Miriam Webster, which defines specification as a detailed and precise presentation of a plan or proposal. He then goes on to speak on determining the specifications for a project. Howard suggests asking six key questions: purpose, functions, performers, who, what, when, where, and how.
Figure 2 Dictionary Definition: Abstraction for Specification
Understanding the Importance of Specifications in Writing
Specifications thoroughly explain the tasks to be undertaken or materials to be utilised in a project. According to Miriam Webster, specifications are essentially instructions on how to create or perform something and how the equipment will be produced.
John Zachman has categorised specifications into various levels, including planning, management, and architect specifications. It is vital to present specifications in an abstract manner to executives, using visuals, icons, and colours to aid comprehension.
Specification documents can be likened to regulations or standards documents, and it is necessary to delve into the details without losing sight of the initial specification.
Figure 3 Dictionary Definition: Specification
Visual Presentation and Data Modelling in Writing
The example of a ranch house with a porch and kitchen illustrates the importance of visual presentation in writing and is referenced in an extract by Steve Hoberman. From a data modelling perspective, a conceptual model establishes precise definitions and provides a framework for how things will work. Zachman's data architecture incorporates data and factors such as business processes and organisational structure to determine how and why things will happen. The Zachman ontology can define policies, procedures, and other aspects of business operations. To create a comprehensive business glossary, it is recommended to define it, create a data model, and establish relationships with a data dictionary.
Understanding the Definition of Specification and Abstraction for Specification
Identifying something and describing it in detail is a crucial aspect of specification. Specification aims to state the requirements by eliciting core business concepts precisely. A conceptual model represents these requirements, while a logical model represents the business solution. Abstraction is a crucial process in specification, where only essential aspects of the system are represented at different levels, excluding unnecessary details. Abstraction aims to enable modular programming and facilitate understanding of requirements for business stakeholders.
Process for Representing Essential Aspects in System Modelling
When representing or identifying the most important aspects of a system, it's crucial to present it at different levels and exclude unnecessary details, especially when communicating with executives. To achieve this, the system's objects and their relationships are defined, and a model is created to capture the essential elements. The behaviour is then specified to enhance the system model even further. This process enables developers to create a more abstract representation of the system, making communicating with business or higher-level individuals easier.
Abstract specification is necessary to understand and adapt to changes in business and implementation requirements. When explaining the benefits of this process to a business client, it's important to emphasise the cost savings and advantages of more efficient and tailored system implementation.
The Six Friends of Writing Business Definitions
The V model (there is an example below) is a helpful process that enables drilling down into various levels of detail while ensuring that requirements are appropriately interpreted for different audiences.
In "The Six Friends of Writing Business Definitions" course, the breakdown of business definitions involves creating and discussing definitions in single sentences. The process includes debating and confirming each statement with the business and considering related terms or objects.
A data model can be built using the defined terms, such as abstraction for specification, process, essential aspect, and non-essential aspect. The focus is on essential aspects which determine system behaviour and should be defined at different specification levels without unnecessary details.
The Importance of Defining Business Terms and Building System Models
Creating a business glossary and understanding the relationships between terms is crucial in defining business terms. Building a comprehensive graph relationship makes it easier to see how each term fits into the system model.
One challenge is defining abstraction, which involves reducing multiple terms into one. However, building relationships between terms brings the definition to life. It's essential to agree on the process for defining abstraction before building the data model.
The level of detail and presentation of glossary terms should be tailored to the audience, whether executives, managers, architects, or developers. Abstraction for specification should also demonstrate the system's behaviour and link to other areas in the system model.
It's important to avoid including excessive information in the model that the audience may not appreciate.
Figure 4 Abstraction for Specification Structured Definition
Abstraction for Specification in Software Engineering
In the field of software engineering, the V model is not only used for data modelling but also for extracting details and requirements from the business and moving towards implementation. In this process, each artefact, from requirements analysis to coding, requires a corresponding test level for validation and verification. This constant mapping between artefacts and corresponding tests ensures that the business requirements are delivered and signed off. To simplify and present different levels of detail to the right audience, the process of abstraction for specification involves going up the hierarchy of levels. The V model layout includes requirement design, system design, architecture design, modular design, and unit testing.
Figure 5 V Model
Agile Development and Test-Driven Development
Agile development begins with a business need and product vision and follows a V-shaped model. The development process involves defining product features, creating user stories, and breaking down components into functional tasks.
Test-driven development is an important aspect of Agile development, which involves writing tests before coding and continuously testing to ensure functionality. Additionally, regression testing ensures that new changes do not break existing functionality.
Product owner testing also ensures the product fits its intended use and purpose. Business case assessment is another crucial aspect of Agile development, which includes return on investment and benefits realisation, validating the project's success.
Different areas may have specific terminology, such as client needs, functional specifications, and system/component definition. Agile development allows for both top-down and bottom-up approaches in delivering functionality.
Detailed design must be agreed upon before moving to the next level to avoid challenges and confusion. It's important to note that test-driven development is not limited to software development and can also be used in mechanical development.
Figure 6 Agile V Model
Importance of Differentiating Functional and Non-Functional Requirements in Writing
There are two types of requirements for a product: functional and non-functional. Functional requirements explain what the product does, while non-functional requirements explain how it works.
For example, functional requirements may include displaying a customer's last name as a clickable link to their account history or allowing sorting by account open date. Non-functional requirements may include requiring strong passwords or accommodating up to 200 concurrent users.
Business requirements focus on high-level goals, such as reducing incorrectly processed orders by 50% or increasing repeat orders by 10%.
As the discussion moves from business requirements to functional requirements to system specifications, the level of detail and understanding also changes. During the requirements-gathering process, an analyst may create different artefacts such as context diagrams, use case models, and conceptual data models.
Figure 8 Specifications required for Development.
Analysis Artifacts and Data Life Cycle Management
Various analysis artefacts are used during software development to define requirements and implement business needs. Data modellers and solution architects collect and build up these artefacts, such as user interface models, business process models, prioritised requirements, business rule catalogues, and prototypes. The focus is on extracting conditions from the business rather than coding and background implementation.
The data life cycle consists of several stages: planning, designing, creating, storing, using, enhancing, and disposing of data. Different specifications are produced at each stage of the data life cycle, such as data architecture, conceptual model, logical model, physical model, and database creation. Each person involved in the process has specific specifications to follow during the software development life cycle (SDLC).
The SDLC stages include feasibility analysis, project planning, and requirements analysis. Operating within the appropriate level of detail during each stage of the SDLC is crucial.
Figure 9 Requirements Definition
Understanding Abstraction and Implementing Processes
During a conversation, Howard and Gauchet discuss the significance of abstraction in specification and the importance of tailoring communication to different audiences. They also highlight the need to justify the time and effort spent on implementing a methodology or framework by highlighting its benefits to the business. Howard emphasises the necessity of obtaining sign-off at different levels of detail. At the same time, Gauchet considers budget constraints when selecting steps for system integration and testing to balance profitability and quality.
Notes on Agile Development and the Zachman Framework
When discussing agile development, it's important to note that it's not about completely removing documentation or specifications. Instead, the focus is on the implementation's speed and level of detail.
The ultimate goal of agile development is to create a minimal viable product through smaller iterations and more precise definitions. This approach helps prevent the costly mistakes of putting a faulty product into production by thoroughly testing each step and level of abstraction.
There are two main debates within agile development. One is about reducing time and using the waterfall approach, while the other centres around ensuring the right people confirm the need and definition of what needs to be built.
Agile development places significant emphasis on drilling down into the details quicker and more efficiently. The system's functionality and features help with overall adaptability to changes. When adding disparate systems and integrating them, a catalyst is necessary.
As for the discussion itself, it should cover the Zachman framework. This framework includes different levels of detail, such as data, processes, interactions, people, networks, timing specifications, and motivation.
Figure 10 Data Architecture Framework Abstractions
Developing a System Logic and Process Model
Howard stresses the importance of simultaneously and thoroughly developing the system logic and process model during the development process. He also emphasises the need to check and verify both components. Additionally, Howard recognises the necessity for the data side to be synchronized with the process model and clarifies their relationship. He asks about any disparities within the data model and requests clarification on the specific type. Howard also highlights the significance of validating functional or characteristic disparities within the data model. Paul agrees and acknowledges the importance of mapping to the test and ensuring that all necessary business requirements are covered.
Figure 11 Relationship between Data Model & Other Models in EA
Importance of Avoiding Unnecessary Details in Business Presentations
At the top level of the V model, we have the business requirement, which forms the foundation of our system. It's essential to avoid adding unnecessary details that could confuse our audience when presenting information. Instead, we should focus on communicating the vital aspects of our system through abstraction and specification. By avoiding excessive details that may lead to arguments and confusion, we can save time and gain agreement on the core aspects of our system.
When discussing data models, starting at the subject area or data estate level is best. Relationships and cardinality should be addressed at the conceptual level to ensure a clear understanding of the system. Each component in a dimensional model contributes to understanding the business requirements and solution. Data governance should be addressed early on, even before considering data sets. By identifying critical data elements through use cases, we can contribute to the business glossary and ensure a smooth implementation process.
Importance of defining critical data elements and controlling scope in the conceptual model of a project
Agreeing on the critical data elements before setting expectations for data quality is essential. Quickly defining the scope of work helps clarify the expected deliverables. At the conceptual model stage, it is crucial to avoid abstraction, which can add entities outside the project's scope. These additions can significantly increase the implementation size. The more abstract the conceptual artefact, the harder it becomes to control the project's scope. Maintaining the project's scope is essential for a tight and effective project strategy.