Published by Addison-Wesley (March 21, 2012) © 2012
James Whittaker | Jason Arbon | Jeff Carollo2012 Jolt Award finalist!
Pioneering the Future of Software Test
Do you need to get it right, too? Then, learn from Google. Legendary testing expert James Whittaker, until recently a Google testing leader, and two top Google experts reveal exactly how Google tests software, offering brand-new best practices you can use even if you’re not quite Google’s size…yet!
Breakthrough Techniques You Can Actually Use
Discover 100% practical, amazingly scalable techniques for analyzing risk and planning tests…thinking like real users…implementing exploratory, black box, white box, and acceptance testing…getting usable feedback…tracking issues…choosing and creating tools…testing “Docs & Mocks,” interfaces, classes, modules, libraries, binaries, services, and infrastructure…reviewing code and refactoring…using test hooks, presubmit scripts, queues, continuous builds, and more. With these techniques, you can transform testing from a bottleneck into an accelerator–and make your whole organization more productive!
Foreword by Alberto Savoia xiii
Foreword by Patrick Copeland xvii
Preface xxiii
Chapter 1: Introduction to Google Software Testing 1
Quality?Test 5
Roles 6
Organizational Structure 8
Crawl, Walk, Run 10
Types of Tests 12
Chapter 2: The Software Engineer in Test 15
The Life of an SET 17
Development and Test Workflow 17
Who Are These SETs Anyway? 22
The Early Phase of a Project 22
Team Structure 24
Design Docs 25
Interfaces and Protocols 27
Automation Planning 28
Testability 29
SET Workflow: An Example 32
Test Execution 40
Test Size Definitions 41
Use of Test Sizes in Shared Infrastructure 44
Benefits of Test Sizes 46
Test Runtime Requirements 48
Case 1: Change in Common Library 52
Test Certified 54
An Interview with the Founders of the Test
Certified Program 57
Interviewing SETs 62
An Interview with Tool Developer Ted Mao 68
An Interview with Web Driver Creator Simon Stewart 70
Chapter 3: The Test Engineer 75
A User-Facing Test Role 75
The Life of a TE 76
Test Planning 79
Risk 97
Life of a Test Case 108
Life of a Bug 113
Recruiting TEs 127
Test Leadership at Google 134
Maintenance Mode Testing 137
Quality Bots Experiment 141
BITE Experiment 153
Google Test Analytics 163
Free Testing Workflow 169
External Vendors 173
An Interview with Google Docs TE Lindsay Webster 175
An Interview with YouTube TE Apple Chow 181
Chapter 4: The Test Engineering Manager 187
The Life of a TEM 187
Getting Projects and People 189
Impact 191
An Interview with Gmail TEM Ankit Mehta 193
An Interview with Android TEM Hung Dang 198
An Interview with Chrome TEM Joel Hynoski 202
The Test Engineering Director 206
An Interview with Search and Geo Test Director
Shelton Mar 207
An Interview with Engineering Tools Director
Ashish Kumar 211
An Interview with Google India Test Director Sujay Sahni 214
An Interview with Engineering Manager Brad Green 219
An Interview with James Whittaker 222
Chapter 5: Improving How Google Tests Software 229
Fatal Flaws in Google’s Process 229
The Future of the SET 231
The Future of the TE 233
The Future of the Test Director and Manager 234
The Future of Test Infrastructure 234
In Conclusion 235
Appendix A: Chrome OS Test Plan 237
Overview of Themes 237
Risk Analysis 238
Per-Build Baseline Testing 239
Per-LKG Day Testing 239
Per-Release Testing 239
Manual Versus Automation 240
Dev Versus Test Quality Focus 240
Release Channels 240
User Input 241
Test Case Repositories 241
Test Dashboarding 241
Virtualization 241
Performance 242
Stress, Long-Running, and Stability 242
Test Execution Framework (Autotest) 242
OEMs 242
Hardware Lab 242
E2E Farm Automation 243
Testing the Browser AppManager 243
Browser Testability 243
Hardware 244
Timeline 244
Primary Test Drivers 246
Relevant Documents 246
Appendix B: Test Tours for Chrome 247
The Shopping Tour 247
The Student Tour 248
Suggested Areas to Test 248
The International Calling Tour 249
Suggested Areas to Test 249
The Landmark Tour 249
Suggested Landmarks in Chrome 249
The All Nighter Tour 250
Suggested Areas to Test 250
The Artisan’s Tour 251
Tools in Chrome 251
The Bad Neighborhood Tour 251
Bad Neighborhoods in Chrome OS 251
The Personalization Tour 252
Ways to Customize Chrome 252
Appendix C: Blog Posts on Tools and Code 253
Take a BITE out of Bugs and Redundant Labor 253
Unleash the QualityBots 255
RPF: Google’s Record Playback Framework 257
Google Test Analytics—Now in Open Source 260
Comprehensive 260
Quick 260
Actionable 260
Sustained Value 260
Index 265
Foreword by Patrick Copeland xvii
Preface xxiii
Chapter 1: Introduction to Google Software Testing 1
Quality?Test 5
Roles 6
Organizational Structure 8
Crawl, Walk, Run 10
Types of Tests 12
Chapter 2: The Software Engineer in Test 15
The Life of an SET 17
Development and Test Workflow 17
Who Are These SETs Anyway? 22
The Early Phase of a Project 22
Team Structure 24
Design Docs 25
Interfaces and Protocols 27
Automation Planning 28
Testability 29
SET Workflow: An Example 32
Test Execution 40
Test Size Definitions 41
Use of Test Sizes in Shared Infrastructure 44
Benefits of Test Sizes 46
Test Runtime Requirements 48
Case 1: Change in Common Library 52
Test Certified 54
An Interview with the Founders of the Test
Certified Program 57
Interviewing SETs 62
An Interview with Tool Developer Ted Mao 68
An Interview with Web Driver Creator Simon Stewart 70
Chapter 3: The Test Engineer 75
A User-Facing Test Role 75
The Life of a TE 76
Test Planning 79
Risk 97
Life of a Test Case 108
Life of a Bug 113
Recruiting TEs 127
Test Leadership at Google 134
Maintenance Mode Testing 137
Quality Bots Experiment 141
BITE Experiment 153
Google Test Analytics 163
Free Testing Workflow 169
External Vendors 173
An Interview with Google Docs TE Lindsay Webster 175
An Interview with YouTube TE Apple Chow 181
Chapter 4: The Test Engineering Manager 187
The Life of a TEM 187
Getting Projects and People 189
Impact 191
An Interview with Gmail TEM Ankit Mehta 193
An Interview with Android TEM Hung Dang 198
An Interview with Chrome TEM Joel Hynoski 202
The Test Engineering Director 206
An Interview with Search and Geo Test Director
Shelton Mar 207
An Interview with Engineering Tools Director
Ashish Kumar 211
An Interview with Google India Test Director Sujay Sahni 214
An Interview with Engineering Manager Brad Green 219
An Interview with James Whittaker 222
Chapter 5: Improving How Google Tests Software 229
Fatal Flaws in Google’s Process 229
The Future of the SET 231
The Future of the TE 233
The Future of the Test Director and Manager 234
The Future of Test Infrastructure 234
In Conclusion 235
Appendix A: Chrome OS Test Plan 237
Overview of Themes 237
Risk Analysis 238
Per-Build Baseline Testing 239
Per-LKG Day Testing 239
Per-Release Testing 239
Manual Versus Automation 240
Dev Versus Test Quality Focus 240
Release Channels 240
User Input 241
Test Case Repositories 241
Test Dashboarding 241
Virtualization 241
Performance 242
Stress, Long-Running, and Stability 242
Test Execution Framework (Autotest) 242
OEMs 242
Hardware Lab 242
E2E Farm Automation 243
Testing the Browser AppManager 243
Browser Testability 243
Hardware 244
Timeline 244
Primary Test Drivers 246
Relevant Documents 246
Appendix B: Test Tours for Chrome 247
The Shopping Tour 247
The Student Tour 248
Suggested Areas to Test 248
The International Calling Tour 249
Suggested Areas to Test 249
The Landmark Tour 249
Suggested Landmarks in Chrome 249
The All Nighter Tour 250
Suggested Areas to Test 250
The Artisan’s Tour 251
Tools in Chrome 251
The Bad Neighborhood Tour 251
Bad Neighborhoods in Chrome OS 251
The Personalization Tour 252
Ways to Customize Chrome 252
Appendix C: Blog Posts on Tools and Code 253
Take a BITE out of Bugs and Redundant Labor 253
Unleash the QualityBots 255
RPF: Google’s Record Playback Framework 257
Google Test Analytics—Now in Open Source 260
Comprehensive 260
Quick 260
Actionable 260
Sustained Value 260
Index 265