Building a Data Integration Team: Skills, Requirements, and Solutions for Designing Integrations

Table of Contents
About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Chapter 1: Integration Background
	The Current Landscape: Two Distinct Models
	Waterfall Vs. Agile Models
		The Waterfall Model
		The Agile Model
	Integration Process Vs. Integration Product Ownership
	Integration Approaches
		One-Time Migration
		Nightly Integration
			Scheduled Service
			Web Service
		Hybrid Approach
Chapter 2: Key Terms
	The Simplicity of Migration
	Source System
		Connection Type
		Source Endpoints
		Metadata Model
		Data Profiling
			Structure Discovery
			Content Discovery
			Relationship Discovery
	ETL Tool
		Source-to-Target Mapping
		Design Code
		Business Rules
		Exception Handling
		Version Control
		Scheduling
		Logging
	Target
		Target Endpoint
		Data Matching
Chapter 3: Team Qualifications
	Team Personality
	Types of Teams
		Basic Team Flow
		Functional
		Cross-Functional
		Virtual Team
		Self-Directed Team
	Team Sizes
	Waterfall Vs. Agile Team Expectations
		Waterfall
		Agile
		Best Approaches
		The T-Shaped Person
	Integration Team Roles
		Project Sponsor
		Stakeholders
		Subject Matter Expert (SME)
		Product Owner
		Project Manager (PM)
		Technical Lead
		Data Integration Architect
		Software Developers
		Software Testers
		User Acceptance Testers
		Business Analyst
		Release Manager
	Recruiting Your Team
		Determining the Roles
		The Job Description
	Finding Candidates
		Internal Candidates
		External Candidates
			Referrals
			Outsourcing
			Posting to Job Boards
			Hiring a Recruiter
		Networking
		Conducting Interviews
	Summary
Chapter 4: Finding Your Purpose: Project Deliverables
	Introduction
	Project Management Phases
		Initiation
		Definition
		Design Phase
		Development
		Implementation
		Maintenance
	Integration Milestones
		Initiation
		Definition
			Data Migration Plan
				The Discovery Process: Tips and Tricks
					Learn the Architecture
					Keep Your Session Small
					Get System Access Early
			Design
				Data Profile Analysis
				Issues Log
				Source-to-Target Mapping
					Purpose
					Design
					Usage
			Data Migration Process Flow Design
				Order of Operations
				Mapping Logic
				Deployment Steps
				Log File Delivery
				Migration Results
				Final Signoff
			ETL Scripts
			Migration Results Report
			Go-Live/Deployment Plan
	Post-deployment: Maintenance and Updates
	Summary
Chapter 5: Choosing an ETL Tool
	Introduction
	ETL Vs. ELT
	ETL As Middleware
	The Components of ETL: Choosing the Right Tool
		Extraction
		Establishing Connectors
		Storing Temporary Data
		Validating Inbound Data
	Transformation
		Modifying Data
		Analyzing Data Integrity
		Validating Result Data
	Loading
		Establishing Connectors
		Determining Load Types
		Rolling Back Errors
	Tools on the Market
		Jitterbit
		Talend
		Dell Boomi
		Pentaho Data Integration (PDI)
		Microsoft SQL Server Integration Services (SSIS)
		MuleSoft
		Informatica
		Scripting Languages
	Summary
Chapter 6: A Sample ETL Project
	Introduction
	Project Management Phases Revisited
		Initiation
		Definition
		Design
			Data Profiling
			Source-to-Target Mapping
			Data Migration Process Flow Design
		Development and Implementation
			Getting Ready
				Clone the GitHub Directory
				Download and Install PDI (Community Edition)
			Opening the “create_target_table_buyer” Script
			Opening the “Prototype” Script
	Summary
Chapter 7: Platform Automation
	Introduction
	All About Environments
		Why It Matters
		Types of Environments
			Development
			Staging
			Production
		Environments for One-Time Migration
		Environments for Repeatable Integrations
		DevOps Practices: A Sample Deployment
			Our Architecture
			Preparing for Automation
			Proposed Migration and Integration Approach
			Demonstration
			The Output Step
	Rollback
	Testing the Automation
		Testing
	Job Scheduling
	Summary
Chapter 8: Monitoring Results
	Introduction
	Day One: Post-deployment
		Level of Support
		Common Integration Complaints and Solutions
		The Daily Data Log
		Updating Production
	Week One: Post-deployment
		Planning Your Next Steps
		Continuous Improvement
	Month One: Post-deployment
	The Data Management Maturity (DMM) Model
	Year One: Post-deployment
	MDM
	Summary
Chapter 9: Marketing Your Team
	Introduction
	Addressing Leadership
	Educating Others
		Internal Education
			Formal Seminars
			Workshops
			Online Training
			Lunch and Learns
		External Education
			Meetups
			Networking Event
			Conferences
	The Data Integration Roadmap
	Data Governance and the Data Governance Office
	Summary
Appendix A:
Data Migration Strategy Plan
	Overview
		Introduction
		Purpose
		References
	Assumptions, Constraints, Risks, and Gaps
		Assumptions
		Constraints
		Risks
		Gaps
	Architecture
		Current State
			Current State Data Sources
			Current State Record Counts
			Current State Data Model
			Current State Integration Points
		Future State
			Future State Data Sources
			Future State Data Model
			Future State Integration Points
	Development Tools
		Environment Usage
		Data Migration Approach
			Scope
			Approach
			Team Roles and Responsibilities
		Migration Process
			Pre-migration Activities
			Migration Activities
			Post-Migration Activities
	Contingency Plan
	Testing/Validation
		Testing Methods
		Migration
		Integration
Appendix B:
Data Profiling Template
Appendix C:
Issues Log Template
Appendix D:
Source-to-Target Mapping Template
Appendix E:
Data Migration Process Flow Design Template
	Order of Operation
	Mapping Logic
	Deployment Steps
		Description of Prerequisites
		Contact
	Account
		Project Signoff/Document Delivery
	Log File Delivery
		Individual Delivery
		Portal Storage
	Migration Results
		Contact
			Inserts
			Updates
		Account
	Final Signoff
Appendix F:
Migration Results Report Template
Index