Explore the next major revolution in data science and artificial intelligence: causal AI

In Causal Artificial Intelligence: The Next Step in Effective Business AI, a team of distinguished AI and analytics professionals delivers an incisive and comprehensive exploration of the models and data of causal inference and causal artificial intelligence. Authors Judith Hurwitz and John Thompson offer the technical detail-explained clearly and accessibly-necessary to understand the underlying technologies, as well as the business context that frames causal AI from a perspective of daily business operations.

You'll discover meaningful and practical insights into what causality is and how it can benefit your organization and understand the critical differences between correlation-based approaches to AI and causality-based approaches. The book also includes easy-to-understand use cases and examples that demonstrate the value of causality for solving business problems.

Perfect for data scientists, subject matter experts in a variety of fields, as well as managers, executives, and other business leaders, Causal Artificial Intelligence is a one-of-a-kind resource designed to open eyes and minds to the incredible possibilities of casual AI and its implications for businesses of all kinds.

JUDITH S. HURWITZ is the chief evangelist at Geminos Software, a causal AI platform company. For more than 35 years she has been a strategist, technology consultant to software providers, and a thought leader having authored 10 books in topics ranging from augmented intelligence, data analytics, and cloud computing.

JOHN K. THOMPSON is an international technology executive with over 37 years of experience in the fields of data, advanced analytics, and artificial intelligence (AI). John is responsible for the global AI function at EY. He has previously led the global Artificial Intelligence and Rapid Data Lab teams at CSL Behring and is the bestselling author of three books on data analytics.

Foreword xix

Preface xxiii

Introduction xxix

Chapter 1 Setting the Stage for Causal AI 1

Why Causality Is a Game Changer 2

Causal AI in Perspective with Analytics 7

Analytical Sophistication Model 8

Analytics Enablers 10

Analytics 10

Advanced Analytics 11

Scope of Services to Support Causal AI 11

The Value of the Hybrid Team 13

The Promise of AI 14

Understanding the Core Concepts of Causal AI 15

Explainability and Bias Detection 15

Explainability 17

Detecting Bias in a Model 17

Directed Acyclic Graphs 18

Structural Causal Model 19

Observed and Unobserved Variables 20

Counterfactuals 21

Confounders 21

Colliders 22

Front- Door and Backdoor Paths 23

Correlation 24

Causal Libraries and Tools 25

Propensity Score 25

Augmented Intelligence and Causal AI 26

Summary 27

Note 27

Chapter 2 Understanding the Value of Causal AI 29

Defining Causal AI 30

The Origins of Causal AI 33

Why Causality? 34

Expressing Relationships 37

The Ladder of Causation 38

Rung 1: Association, or Passive Observation 40

Rung 2: Intervention, or Taking Action 40

Rung 3: Counterfactuals, or Imagining What If 42

Why Causal AI Is the Next Generation of AI 43

Deep Learning and Neural Networks 43

Neural Networks 44

Establishing Ground Truth 45

The Business Imperative of a Causal Model 46

The Importance of Augmented Intelligence 51

The Importance of Data, Visualization, and Frameworks 52

Getting the Appropriate Data 52

Applying Data and Model Visualization 55

Applying Frameworks After Creating a Model 56

Getting Started with Causal AI 57

Summary 58

Notes 59

Chapter 3 Elements of Causal AI 61

Conceptual Models 62

Correlation vs. Causal Models 63

Correlation- Based AI 63

Causal AI 63

Understanding the Relationship Between Correlation and Causality 64

Process Models 66

Correlation- Based AI Process Model 67

Causal- Based AI Process Model 69

Collaboration Between Business and Analytics Professionals 72

The Fundamental Building Blocks of Causal AI Models 75

The Relations Between DAGs and SCMs 76

Explaining DAGs 76

Causal Notation: The Language of DAGs 78

Operationalizing a DAG with an SCM 79

The Elements of Visual Modeling 81

Nodes 83

Variables 83

Endogenous and Exogenous Variables 83

Observed and Unobserved Variables 84

Paths/Relationships 84

Weights 86

Summary 88

Notes 89

Chapter 4 Creating Practical Causal AI Models and Systems 91

Understanding Complex Models 92

Causal Modeling Process: Part 1 94

Step 1: What Are the Intended Outcomes? 95

Step 2: What Are the Proposed Interventions? 97

Step 3: What Are the Confounding Factors? 99

Step 4: What Are the Factors Creating the Effects and Changes? 102

Common/Universal Effects in a Causal Model 102

Refined Effects in a Causal Model 103

Step 5: Creating a Directed Acyclic Graph 105

Step 6: Paths and Relationships 105

Types of Paths 106

Path Connecting an Unobserved Variable 107

Front- Door Paths 108

Backdoor Paths 108

Modeling for Simplicity to Understand Complexity 109

Step 7: Data Acquisition 110

Causal- Based Approach: Part 2 112

Step 8: Data Integration 113

Step 9: Model Modification 114

Step 10: Data Transformation 115

Step 11: Preparing for Deployment in Business 118

Summary 121

Notes 122

Chapter 5 Creating a Model with a Hybrid Team 125

The Hybrid Team 126

Why a Hybrid Team? 127

The Benefits of a Hybrid Team 128

Establishing the Hybrid Team as a Center of Excellence 129

How Teams Collaborate 131

But Why? 132

Defining Roles 134

Leaders and Business Strategists 137

Subject- Matter Experts 138

Data Experts 140

Software Developers 142

Business Process Analysts 143

Information Technology Expertise 143

Project Manager(s) 144

The Basics Steps for a Hybrid Team Project 145

An Overview of Model Creation 146

It Depends on Your Destination 150

Understanding the Root Cause of a Problem 151

Understanding What Happened and Why 153

The Importance of the Iterative Process 154

Summary 155

Chapter 6 Explainability, Bias Detection, and AI Responsibility in Causal AI 157

Explainability 158

The Ramifications of the Lack of Explainability 159

What Is Explainable AI in Causal AI Models? 161

Black Boxes 162

Internal Workings of Black-Box Models 162

Deep Learning at the Heart of Black Boxes 163

Is Code Understandable? 163

The Value of White-Box Models 166

Understanding Causal AI Code 167

Techniques for Achieving Explainability 169

Challenges of Complex Causal Models 169

Methods for Understanding and Explaining Complex Causal AI Models 171

The Importance of the SHAP Explainability Method 172

Detecting Bias and Ensuring Responsible AI 175

Bias in Causal AI Systems 176

Responsible AI: Trust and Fairness 178

How Causal AI Addresses Bias Detection 180

Tools for Assessing Fairness and Bias 182

The Human Factor in Bias Detection and Responsible AI 183

Summary 184

Note 184

Chapter 7 Tools, Practices, and Techniques to Enable Causal AI 185

The Causal AI Pipeline 187

Define Business Objectives 190

Model Development 193

Data Identification and Collection 195

Data Privacy, Governance, and Security 197

Synthetic Data 198

Model Validation 199

Deployment/Production 201

Monitor and Evaluate 203

Update and Iterate 205

Continuous Learning 208

The Importance of Synthetic Data 210

Why Create Synthetic Data? 210

Overcoming Data Limitations 211

Enhancing Data Privacy and Security 211

Model Validation and Testing 211

Expanding the Range of Possible Scenarios 212

Reducing the Cost of Data Collection 212

Improving Data Imbalance 213

Encouraging Collaboration and Openness 213

Streamlining Data Preprocessing 213

Supporting Counterfactual Analysis 213

Fostering Innovation and Experimentation 214

Creating Synthetic Data 214

Generative Models 214

Agent-Based Modeling 215

Data Augmentation 215

Data Synthesis Tools and Platforms 215

Conditional Synthetic Data Generation 216

Synthetic Data from Text 216

The Limitations of Synthetic Data 217

Current State of Tools and Software in Causal AI 218

The Role of Open Source in Causal AI 218

Commercial Causal AI Software 221

CausaLens 221

Geminos Software 223

Summary 223

Chapter 8 Causal AI in Action 225

Enterprise Marketing in a Business- to- Consumer Scenario 226

DDCo Marketing Causal Model: Annual Pricing Review and Update Cycle 228

Incorporating Internal and External Factors in the Model and DAG 230

Easily Enabling Iterating 231

End-User-Driven Exploration 232

Bench Testing 234

DDCo Marketing Causal Model: Semiannual Product Planning Cycle 236

Always Consider Model Reuse 237

Give and Take in Building a New Model 239

Typical Model and Process Operation: Iterating 239

Keeping the Process/Model Scope Manageable and Understandable 240

Moving from Strategy to Building and Implementing Causal AI Solutions 241

Agriculture: Enhancing Crop Yield 242

Key Causal Variables 244

Creating the DAG 246

Moving from the DAG to Implementing the Causal AI Model 247

Commercial Real Estate: Valuing Warehouse Space 250

Key Causal Variables 251

Implementing the Causal AI Model 253

Video Streaming: Enhancing Content Recommendations 254

Key Causal Variables 255

Implementing the Causal AI Model 256

Healthcare: Reducing Infant Mortality 258

Key Causal Variables 259

Implementing the Causal AI Model 261

Retail: Providing Executives Actionable Information 263

Key Causal Variables 264

Implementing the Causal Model 265

Summary 267

Chapter 9 The Future of Causal AI 271

Where We Stand Today 271

Foundations of Causal AI 273

The Causal AI Journey 274

Causal AI Today 274

What's Next for Causal AI 276

Integrating Causal AI and Traditional AI 278

The Imperative for Managing Data 279

Ensembles of Data 279

Generative AI Is Emerging as a Game Changer for Causal AI 281

The Future of Causal Discovery 282

The Emergence of Causal AI Reinforcement Learning Will Accelerate Model Training 284

Causal AI as a Common Language Between Business Leaders and Data Scientists 284

The Emergence of Probabilistic Programming Languages 286

The Predictable Model Evolution Cycle 286

The Emergence of the Digital Twin 287

Improving the Ability to Understand Ground Truth 289

The Development of More Sophisticated DAGs 289

Visualizing Complex Relationships in the DAGs 290

The Merging of Causal and Traditional AI Models 291

The Future of Explainability 291

The Evolution of Responsible AI 292

Advances in Data Security and Privacy 293

Integration Will Be Between Models and Business Applications 294

Summary 295

Glossary 299

Appendix 313

Selected Resources 329

Acknowledgments 331

About the authors 335

About the contributor 339

Index 341