|
|
很好的入门书籍。文件太大传不了，我发contents上来，如果有需要，请联系我：）
1 Introduction to data 1
1.1 Case study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Data basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.1 Observations, variables, and data matrices . . . . . . . . . . . . . . . 3
1.2.2 Types of variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.3 Relationships between variables . . . . . . . . . . . . . . . . . . . . . 7
1.3 Overview of data collection principles . . . . . . . . . . . . . . . . . . . . . 9
1.3.1 Populations and samples . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3.2 Anecdotal evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3.3 Sampling from a population . . . . . . . . . . . . . . . . . . . . . . . 10
1.3.4 Explanatory and response variables . . . . . . . . . . . . . . . . . . . 12
1.3.5 Introducing observational studies and experiments . . . . . . . . . . 13
1.4 Observational studies and sampling strategies . . . . . . . . . . . . . . . . . 13
1.4.1 Observational studies . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.4.2 Three sampling methods (special topic) . . . . . . . . . . . . . . . . 14
1.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.5.1 Principles of experimental design . . . . . . . . . . . . . . . . . . . . 17
1.5.2 Reducing bias in human experiments . . . . . . . . . . . . . . . . . . 17
1.6 Examining numerical data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.6.1 Scatterplots for paired data . . . . . . . . . . . . . . . . . . . . . . . 20
1.6.2 Dot plots and the mean . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.6.3 Histograms and shape . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.6.4 Variance and standard deviation . . . . . . . . . . . . . . . . . . . . 25
1.6.5 Box plots, quartiles, and the median . . . . . . . . . . . . . . . . . . 28
1.6.6 Robust statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
1.6.7 Transforming data (special topic) . . . . . . . . . . . . . . . . . . . . 31
1.6.8 Mapping data (special topic) . . . . . . . . . . . . . . . . . . . . . . 32
1.7 Considering categorical data . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
1.7.1 Contingency tables and bar plots . . . . . . . . . . . . . . . . . . . . 35
1.7.2 Row and column proportions . . . . . . . . . . . . . . . . . . . . . . 36
1.7.3 Segmented bar and mosaic plots . . . . . . . . . . . . . . . . . . . . 38
1.7.4 The only pie chart you will see in this book . . . . . . . . . . . . . . 40
1.7.5 Comparing numerical data across groups . . . . . . . . . . . . . . . 40
1.8 Case study: gender discrimination (special topic) . . . . . . . . . . . . . . . 42
1.8.1 Variability within data . . . . . . . . . . . . . . . . . . . . . . . . . . 42
1.8.2 Simulating the study . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
1.8.3 Checking for independence . . . . . . . . . . . . . . . . . . . . . . . 45
iiiiv CONTENTS
1.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
1.9.1 Case study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
1.9.2 Data basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
1.9.3 Overview of data collection principles . . . . . . . . . . . . . . . . . 49
1.9.4 Observational studies and sampling strategies . . . . . . . . . . . . . 50
1.9.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
1.9.6 Examining numerical data . . . . . . . . . . . . . . . . . . . . . . . . 54
1.9.7 Considering categorical data . . . . . . . . . . . . . . . . . . . . . . 62
1.9.8 Case study: gender discrimination . . . . . . . . . . . . . . . . . . . 64
2 Probability (special topic) 68
2.1 Defining probability (special topic) . . . . . . . . . . . . . . . . . . . . . . . 68
2.1.1 Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
2.1.2 Disjoint or mutually exclusive outcomes . . . . . . . . . . . . . . . . 70
2.1.3 Probabilities when events are not disjoint . . . . . . . . . . . . . . . 72
2.1.4 Probability distributions . . . . . . . . . . . . . . . . . . . . . . . . . 74
2.1.5 Complement of an event . . . . . . . . . . . . . . . . . . . . . . . . . 76
2.1.6 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
2.2 Conditional probability (special topic) . . . . . . . . . . . . . . . . . . . . . 79
2.2.1 Marginal and joint probabilities . . . . . . . . . . . . . . . . . . . . . 80
2.2.2 Defining conditional probability . . . . . . . . . . . . . . . . . . . . . 81
2.2.3 Smallpox in Boston, 1721 . . . . . . . . . . . . . . . . . . . . . . . . 83
2.2.4 General multiplication rule . . . . . . . . . . . . . . . . . . . . . . . 84
2.2.5 Independence considerations in conditional probability . . . . . . . . 86
2.2.6 Tree diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
2.2.7 Bayes’ Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
2.3 Sampling from a small population (special topic) . . . . . . . . . . . . . . . 93
2.4 Random variables (special topic) . . . . . . . . . . . . . . . . . . . . . . . . 95
2.4.1 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
2.4.2 Variability in random variables . . . . . . . . . . . . . . . . . . . . . 98
2.4.3 Linear combinations of random variables . . . . . . . . . . . . . . . . 99
2.4.4 Variability in linear combinations of random variables . . . . . . . . 101
2.5 Continuous distributions (special topic) . . . . . . . . . . . . . . . . . . . . 104
2.5.1 From histograms to continuous distributions . . . . . . . . . . . . . . 105
2.5.2 Probabilities from continuous distributions . . . . . . . . . . . . . . 106
2.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
2.6.1 Defining probability . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
2.6.2 Conditional probability . . . . . . . . . . . . . . . . . . . . . . . . . 110
2.6.3 Sampling from a small population . . . . . . . . . . . . . . . . . . . 113
2.6.4 Random variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
2.6.5 Continuous distributions . . . . . . . . . . . . . . . . . . . . . . . . . 116
3 Distributions of random variables 118
3.1 Normal distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
3.1.1 Normal distribution model . . . . . . . . . . . . . . . . . . . . . . . 119
3.1.2 Standardizing with Z scores . . . . . . . . . . . . . . . . . . . . . . . 120
3.1.3 Normal probability table . . . . . . . . . . . . . . . . . . . . . . . . . 121
3.1.4 Normal probability examples . . . . . . . . . . . . . . . . . . . . . . 122
3.1.5 68-95-99.7 rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
3.2 Evaluating the normal approximation . . . . . . . . . . . . . . . . . . . . . 127CONTENTS v
3.2.1 Normal probability plot . . . . . . . . . . . . . . . . . . . . . . . . . 128
3.2.2 Constructing a normal probability plot (special topic) . . . . . . . . 132
3.3 Geometric distribution (special topic) . . . . . . . . . . . . . . . . . . . . . 133
3.3.1 Bernoulli distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 133
3.3.2 Geometric distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 134
3.4 Binomial distribution (special topic) . . . . . . . . . . . . . . . . . . . . . . 137
3.4.1 The binomial distribution . . . . . . . . . . . . . . . . . . . . . . . . 137
3.4.2 Normal approximation to the binomial distribution . . . . . . . . . . 141
3.4.3 The normal approximation breaks down on small intervals . . . . . . 143
3.5 More discrete distributions (special topic) . . . . . . . . . . . . . . . . . . . 144
3.5.1 Negative binomial distribution . . . . . . . . . . . . . . . . . . . . . 144
3.5.2 Poisson distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
3.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
3.6.1 Normal distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
3.6.2 Evaluating the Normal approximation . . . . . . . . . . . . . . . . . 152
3.6.3 Geometric distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 153
3.6.4 Binomial distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 154
3.6.5 More discrete distributions . . . . . . . . . . . . . . . . . . . . . . . 157
4 Foundations for inference 159
4.1 Variability in estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
4.1.1 Point estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
4.1.2 Point estimates are not exact . . . . . . . . . . . . . . . . . . . . . . 161
4.1.3 Standard error of the mean . . . . . . . . . . . . . . . . . . . . . . . 162
4.1.4 Basic properties of point estimates . . . . . . . . . . . . . . . . . . . 164
4.2 Confidence intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
4.2.1 Capturing the population parameter . . . . . . . . . . . . . . . . . . 165
4.2.2 An approximate 95% confidence interval . . . . . . . . . . . . . . . . 165
4.2.3 A sampling distribution for the mean . . . . . . . . . . . . . . . . . 167
4.2.4 Changing the confidence level . . . . . . . . . . . . . . . . . . . . . . 168
4.2.5 Interpreting confidence intervals . . . . . . . . . . . . . . . . . . . . 170
4.2.6 Nearly normal population with known SD (special topic) . . . . . . 170
4.3 Hypothesis testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
4.3.1 Hypothesis testing framework . . . . . . . . . . . . . . . . . . . . . . 172
4.3.2 Testing hypotheses using confidence intervals . . . . . . . . . . . . . 173
4.3.3 Decision errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
4.3.4 Formal testing using p-values . . . . . . . . . . . . . . . . . . . . . . 177
4.3.5 Two-sided hypothesis testing with p-values . . . . . . . . . . . . . . 182
4.3.6 Choosing a significance level . . . . . . . . . . . . . . . . . . . . . . . 184
4.4 Examining the Central Limit Theorem . . . . . . . . . . . . . . . . . . . . . 185
4.5 Inference for other estimators . . . . . . . . . . . . . . . . . . . . . . . . . . 188
4.5.1 Confidence intervals for nearly normal point estimates . . . . . . . . 189
4.5.2 Hypothesis testing for nearly normal point estimates . . . . . . . . . 190
4.5.3 Non-normal point estimates . . . . . . . . . . . . . . . . . . . . . . . 192
4.5.4 When to retreat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
4.6 Sample size and power (special topic) . . . . . . . . . . . . . . . . . . . . . 193
4.6.1 Finding a sample size for a certain margin of error . . . . . . . . . . 193
4.6.2 Power and the Type 2 Error rate . . . . . . . . . . . . . . . . . . . . 194
4.6.3 Statistical significance versus practical significance . . . . . . . . . . 196
4.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197vi CONTENTS
4.7.1 Variability in estimates . . . . . . . . . . . . . . . . . . . . . . . . . 197
4.7.2 Confidence intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
4.7.3 Hypothesis testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
4.7.4 Examining the Central Limit Theorem . . . . . . . . . . . . . . . . . 206
4.7.5 Inference for other estimators . . . . . . . . . . . . . . . . . . . . . . 210
4.7.6 Sample size and power . . . . . . . . . . . . . . . . . . . . . . . . . . 211
5 Inference for numerical data 212
5.1 Paired data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
5.1.1 Paired observations and samples . . . . . . . . . . . . . . . . . . . . 213
5.1.2 Inference for paired data . . . . . . . . . . . . . . . . . . . . . . . . . 213
5.2 Difference of two means . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
5.2.1 Point estimates and standard errors for differences of means . . . . . 215
5.2.2 Confidence interval for the difference . . . . . . . . . . . . . . . . . . 217
5.2.3 Hypothesis tests based on a difference in means . . . . . . . . . . . . 217
5.2.4 Summary for inference of the difference of two means . . . . . . . . 220
5.2.5 Examining the standard error formula . . . . . . . . . . . . . . . . . 221
5.3 One-sample means with the t distribution . . . . . . . . . . . . . . . . . . . 221
5.3.1 The normality condition . . . . . . . . . . . . . . . . . . . . . . . . . 222
5.3.2 Introducing the t distribution . . . . . . . . . . . . . . . . . . . . . . 222
5.3.3 The t distribution as a solution to the standard error problem . . . . 225
5.3.4 One sample t confidence intervals . . . . . . . . . . . . . . . . . . . . 226
5.3.5 One sample t tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
5.4 The t distribution for the difference of two means . . . . . . . . . . . . . . . 230
5.4.1 Sampling distributions for the difference in two means . . . . . . . . 230
5.4.2 Two sample t test . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
5.4.3 Two sample t confidence interval . . . . . . . . . . . . . . . . . . . . 234
5.4.4 Pooled standard deviation estimate (special topic) . . . . . . . . . . 235
5.5 Comparing many means with ANOVA (special topic) . . . . . . . . . . . . . 236
5.5.1 Is batting performance related to player position in MLB? . . . . . . 237
5.5.2 Analysis of variance (ANOVA) and the F test . . . . . . . . . . . . . 239
5.5.3 Reading an ANOVA table from software . . . . . . . . . . . . . . . . 242
5.5.4 Graphical diagnostics for an ANOVA analysis . . . . . . . . . . . . . 242
5.5.5 Multiple comparisons and controlling Type 1 Error rate . . . . . . . 243
5.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
5.6.1 Paired data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
5.6.2 Difference of two means . . . . . . . . . . . . . . . . . . . . . . . . . 249
5.6.3 One-sample means with the t distribution . . . . . . . . . . . . . . . 251
5.6.4 The t distribution for the difference of two means . . . . . . . . . . . 253
5.6.5 Comparing many means with ANOVA . . . . . . . . . . . . . . . . . 258
6 Inference for categorical data 263
6.1 Inference for a single proportion . . . . . . . . . . . . . . . . . . . . . . . . . 263
6.1.1 Identifying when the sample proportion is nearly normal . . . . . . . 263
6.1.2 Confidence intervals for a proportion . . . . . . . . . . . . . . . . . . 264
6.1.3 Hypothesis testing for a proportion . . . . . . . . . . . . . . . . . . . 265
6.1.4 Choosing a sample size when estimating a proportion . . . . . . . . 266
6.2 Difference of two proportions . . . . . . . . . . . . . . . . . . . . . . . . . . 268
6.2.1 Sample distribution of the difference of two proportions . . . . . . . 268
6.2.2 Intervals and tests for p1 − p2 . . . . . . . . . . . . . . . . . . . . . . 269CONTENTS vii
6.2.3 Hypothesis testing when H0 : p1 = p2 . . . . . . . . . . . . . . . . . 271
6.3 Testing for goodness of fit using chi-square (special topic) . . . . . . . . . . 273
6.3.1 Creating a test statistic for one-way tables . . . . . . . . . . . . . . . 274
6.3.2 The chi-square test statistic . . . . . . . . . . . . . . . . . . . . . . . 274
6.3.3 The chi-square distribution and finding areas . . . . . . . . . . . . . 276
6.3.4 Finding a p-value for a chi-square distribution . . . . . . . . . . . . 279
6.3.5 Evaluating goodness of fit for a distribution . . . . . . . . . . . . . . 280
6.4 Testing for independence in two-way tables (special topic) . . . . . . . . . . 283
6.4.1 Expected counts in two-way tables . . . . . . . . . . . . . . . . . . . 284
6.4.2 The chi-square test for two-way tables . . . . . . . . . . . . . . . . . 286
6.5 Small sample hypothesis testing for a proportion (special topic) . . . . . . . 288
6.5.1 When the success-failure condition is not met . . . . . . . . . . . . . 288
6.5.2 Generating the null distribution and p-value by simulation . . . . . . 289
6.5.3 Generating the exact null distribution and p-value . . . . . . . . . . 291
6.5.4 Using simulation for goodness of fit tests . . . . . . . . . . . . . . . . 292
6.6 Hypothesis testing for two proportions (special topic) . . . . . . . . . . . . 293
6.6.1 Large sample framework for a difference in two proportions . . . . . 294
6.6.2 Simulating a difference under the null distribution . . . . . . . . . . 295
6.6.3 Null distribution for the difference in two proportions . . . . . . . . 296
6.6.4 Randomization for two-way tables and chi-square . . . . . . . . . . . 297
6.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
6.7.1 Inference for a single proportion . . . . . . . . . . . . . . . . . . . . 298
6.7.2 Difference of two proportions . . . . . . . . . . . . . . . . . . . . . . 303
6.7.3 Testing for goodness of fit using chi-square . . . . . . . . . . . . . . 307
6.7.4 Testing for independence in two-way tables . . . . . . . . . . . . . . 308
6.7.5 Small sample hypothesis testing for a proportion . . . . . . . . . . . 310
6.7.6 Hypothesis testing for two proportions . . . . . . . . . . . . . . . . . 313
7 Introduction to linear regression 315
7.1 Line fitting, residuals, and correlation . . . . . . . . . . . . . . . . . . . . . 316
7.1.1 Beginning with straight lines . . . . . . . . . . . . . . . . . . . . . . 317
7.1.2 Fitting a line by eye . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
7.1.3 Residuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
7.1.4 Describing linear relationships with correlation . . . . . . . . . . . . 322
7.2 Fitting a line by least squares regression . . . . . . . . . . . . . . . . . . . . 324
7.2.1 An objective measure for finding the best line . . . . . . . . . . . . . 324
7.2.2 Conditions for the least squares line . . . . . . . . . . . . . . . . . . 325
7.2.3 Finding the least squares line . . . . . . . . . . . . . . . . . . . . . . 325
7.2.4 Interpreting regression line parameter estimates . . . . . . . . . . . . 328
7.2.5 Extrapolation is treacherous . . . . . . . . . . . . . . . . . . . . . . . 329
7.2.6 Using R2
to describe the strength of a fit . . . . . . . . . . . . . . . 329
7.2.7 Categorical predictors with two levels . . . . . . . . . . . . . . . . . 330
7.3 Types of outliers in linear regression . . . . . . . . . . . . . . . . . . . . . . 332
7.4 Inference for linear regression . . . . . . . . . . . . . . . . . . . . . . . . . . 334
7.4.1 Midterm elections and unemployment . . . . . . . . . . . . . . . . . 334
7.4.2 Understanding regression output from software . . . . . . . . . . . . 336
7.4.3 An alternative test statistic . . . . . . . . . . . . . . . . . . . . . . . 338
7.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
7.5.1 Line fitting, residuals, and correlation . . . . . . . . . . . . . . . . . 339
7.5.2 Fitting a line by least squares regression . . . . . . . . . . . . . . . . 345viii CONTENTS
7.5.3 Types of outliers in linear regression . . . . . . . . . . . . . . . . . . 348
7.5.4 Inference for linear regression . . . . . . . . . . . . . . . . . . . . . . 349
8 Multiple and logistic regression 354
8.1 Introduction to multiple regression . . . . . . . . . . . . . . . . . . . . . . . 354
8.1.1 A single-variable model for the Mario Kart data . . . . . . . . . . . 355
8.1.2 Including and assessing many variables in a model . . . . . . . . . . 356
8.1.3 Adjusted R2 as a better estimate of explained variance . . . . . . . . 358
8.2 Model selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
8.2.1 Identifying variables in the model that may not be helpful . . . . . . 359
8.2.2 Two model selection strategies . . . . . . . . . . . . . . . . . . . . . 360
8.3 Checking model assumptions using graphs . . . . . . . . . . . . . . . . . . . 363
8.4 Logistic regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
8.4.1 Email data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
8.4.2 Modeling the probability of an event . . . . . . . . . . . . . . . . . . 368
8.4.3 Practical decisions in the email application . . . . . . . . . . . . . . 372
8.4.4 Diagnostics for the email classifier . . . . . . . . . . . . . . . . . . . 373
8.4.5 Improving the set of variables for a spam filter . . . . . . . . . . . . 375
8.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
8.5.1 Introduction to multiple regression . . . . . . . . . . . . . . . . . . . 377
8.5.2 Model selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
8.5.3 Checking model assumptions using graphs . . . . . . . . . . . . . . . 382
8.5.4 Logistic regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
A End of chapter exercise solutions 387
B Distribution tables 407
B.1 Normal Probability Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
B.2 t Distribution Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
B.3 Chi-Square Probability Table . . . . . . . . . . . . . . . . . . . . . . . . . . 412 |
|
发表于 2014-9-16 01:39:58