Probability Theory and

Mathematical

Statistics

A WILEY PUBLICATION

IN MATHEMATICAL

STATISTICS

Probability Theory and

Mathematical Statistics
MAREK

FISZ

Late Professor of Mathematics

New York University

THIRD EDITION

J ohn

Wiley & Sons, Ine.

New York London

Sydney

Published in colloboration with P WN, Polish Scientific Publishers

AUTHORIZED TRANSLATION FROM THE POLlSH.

TRANSLATED BY R. BARTOSZYNSKI'

THE FIRST AND SECOND EDlTIONS OF THIS BOOK

WERE PUBLlSHED AND COPYRIGHTED IN POLAND UNDER THE TITLE
RACHUNEK

PRAWDOPODOBIENSTWA

STATYSTYKA

MATEMATYCZNA

COPYRIGHT

1963

BY
JOHN WILEY & SONS, INC.

Al! Rights Reserved

This book or any part thereof mst not
be reproduced in any [orm without the
written permission 01 (he publisher,
TRANSLATIONS INTO OTHER LANGUAGES ARE PROHIBITED EXCEPT BY
PERMISSION OF P'!NN, POLlSH SCIENTIFIC PUBLlSHERS
THIRD PRINTING, JANUARY,

1967

LIBRARY Of CONGRESS CATALOG CARD NUMBER:

63-7554

PRINTED IN THE UNITED STATES OF AMERICA

To OIga and Aleksander

Preface to the English Edition

The opening sentence of this book is "Probability theory is a part of

mathematics which is useful in discovering and investigating the regular
features of random events." However, even in the not very remote past
this sentence would not have been found acceptable (or by any means
evident) either by mathematicians or by researchers applying probability
theory. It was not until the twenties and thirties of our century that the
very nature of probability theory as a branch of mathematics and the
relation between the concept of "probability" and that of "frequency" of
random events was thoroughly clarified. The reader will find in Section
1.3 an extensive (although, certainly, not exhaustive) list of names of
researchers whose contributions to the field of basic concepts of probability theory are important. However, the foremost importance of
Laplace's Thorie Analytique des Probabilits, von Mises' Wahrscheinlichkeit, Statistik und Wahrheit, and Kolmogorov's Grundbegriffe der Wahrscheinlichkeitsrechnung should be stressed. With each of these three works,
a new period in the history of probability theory was begun. In addition,
the work of Steinhaus' school of independent functions contributed greatly
to the clarification of fundamental concepts of probability theory.
The progress in foundations of probability theory, along with the
introduction of the theory of characteristic functions, stimulated the
exceedinglyfast development of modern probability theory. In the field of
limit theorems for sums of random variables, a fairly general theory was
developed (Khintchin, Lvy, Kolmogorov, Feller, Gnedenko, and others)
for independent random variables, whereas for dependent random
variables sorne important particular results were obtained (Bernstein,
Markov, Doeblin, Kolmogorov, and others). Furthermore, the theory
of stochastic processes became a mathematically rigorous branch of
vii

viii

PREFACE TO THE ENGLISH EDITION

probability theory (Markov, Khintchin, Lvy, Wiener, Kolmogorov,

Feller, Cramr, Doob, and others).
Sorne ideas on application of probability theory that are now used in
mathematical statistics are due to Bayes (estimation theory), Laplace
(quality control of drugs), and to Gauss (theory of errors). However, it
was not until this century that mathematical statistics grew into a selfcontained scientific subject. In order to restrict myself to the names of
just a few of the principal persons responsible for this growth, 1 mention
only K. Pearson, R. A. Fisher, J. Neyman, and A. Wald, whose ideas and
systematic research contributed so much to the high status of modern
mathematical statistics.
At present, the development of the theory of probability and mathematical statistics is going on with extreme intensity. On the one hand,
problems in classical probability theory unsolved, as of now, are attracting
much attention, whereas, on the other hand, much work is being done in
an attempt to obtain highly advanced generalizations of old concepts,
particularIy by considering probability theory in spaces more general than
the finite dimensional Euclidean spaces usually treated. That probability
theory is now closely connected with other parts of mathematics is evidenced by the fact that almost immediately after the formulation of the
distribution theories by L. Schwartz and Mikusinski their probabilistic
counterparts were thoroughly discussed (Gelfand, 1t6, Urbanik). Moreover, probability theory and mathematical statistics are no longer simply
"customers" of other parts of mathematics. On the contrary, mutual
influence and interchange of ideas between probability theory and mathematical statistics and the other areas of mathematics are constantly going
on. One example is the relation between analytic number theory and
probability theory (Borel, Khintchin, Linnik, Erds, Kac, Rnyi, and
others), and another example is the use of the theory of games in mathematical statistics and its stimulating effect on the development of the theory
of games itself (v. Neumann, Morgenstern, Wald, Blackwell, Karlin, and
others).
.
. Areas of application of probability theory and mathematical statistics
are increasing more and more. Statistical methods are now widely used in
physics, biology, medicine, economics, industry, agriculture, fisheries,
meteorology, and in communications. Statistical methodology has become
an important ,;:omponent of scientific reasoning, as well as an integral
part of well-organized business and social work.
The development of probability theory and mathematical statistics and
their applications is marked by a constantly increasing flood of scientific
papers that are published in many journals in many countries.

* * *

PREFACE TO THE ENGLISH EDlTION

IX

Having described briefly the position of modero probability theory and

mathematical statistics, 1 now state the main purpose of this book:
1. To give a systematic introduction to modero probability theory and
mathematical statistics.
2. To present an outline of many of the possible applications of these
theories, accompanied by descriptive concrete examples.
3. To provide extensive (however, not exhaustive) references of other
books and papers, mostly with brief indications as to their contents,
thereby giving the reader the opportunity to complete his knowledge of the
subjects considered.

Although great care has been taken to make this book mathematicalIy
rigorous, the intuitive approach as weIl as the applicability of the concepts
and theorems presented are heavily stressed.
For the most part, the theorems are given with complete proofs. Sorne
proofs, which are either too lengthy or require mathematical knowledge
far beyond the scope of this book, were omitted.
The entire text of .the book may be read by students with sorne background in calculus and algebra. However, no advanced knowledge in
these fields or a knowledge in measure and integration theory is required.
Sorne necessary advanced concepts (for instance, that of the Stieltjes
integral) are presented in the text. Furthermore, this book is provided
with a Supplement, in which sorne basic concepts and theorems of modero
measure and integration theory are presented.
Every chapter is folIowed by "Problerns and Complements." A large
part ofthese problems are relatively easy and are to be solved by the reader,
with the remaining ones given for information and stimulation.
This book may be used for systematic one-year courses either in
probability theory or in mathematical statistics, either for senior undergraduate or graduate students. 1 have presented parts of the material,
covered by this book, in courses at the University of Warsaw (Poland)
for nine academic years, from 1951/1952 to 1959/1960, at the Peking
University (China) in the Spring term of 1957, and in this country at the
University of Washington and at Stanford, Columbia and New York
Universities for the last several years.
This book is also suitable for nonmathematicians, as far as concepts,
theorems, and methods of application are concemed.

* * *

1 started to write this book at the end of 1950. Its first edition (374
pages) was published in Polish' in 1954. AH copies were sold within a
few months. 1 then prepared the second, revised and extended, Polish

PREFACE TO THE ENGLISH EDITION

edition, which was published in 1958, simultaneously with its German

translation .. Indications about changes and extensions introduced into
the second Polish edition are given in its preface. In comparison with
the second Polish edition, the present English one contains many extensions and changes, the most important of which are the following:
Entirely new are the Problems and Complements, the Supplement, and
the Sections: 2.7,C, 2.8,C, 3.2,C, 3.6,G, 4.6,B, 6.4,B, 6.4,C, 6.12,D,
6.12,F, 6.15, 8.11, 9.4,B, 9.6,E, 9.9,B, 1O.10,B, 1O.1l,E, 12.6,B, 13.5,E,
13.7,D, 14.2,E, 14.4,D, 15.l,C, 15.3,C, 16.3,D, 16.6, and 17.10,A. Section 8.10 (Stationary processes) is almost entirely new.
Considerably changed or complemented are Sections: 2.5,C, 3.5,
3.6,C, 4.1, 4.2, 5.6,B, 5.7, 5.I3,B, 6.2, 6.4,A, 6.5, 6.l2,E, 6.l2,G, 7.5,B,
8.4,D, 8.8,B, 8.I2, 9.1, 9.7, 10.12, 10.13, 12,4,C, 12.4,D, 13.3, 16.2,C.
These changes and extensions have all been made to fulfill more completely the main purpose of this book, as stated previously.

* * *

J. Lukaszewicz, A. M. Rusiecki, and W. Sadowski read the manuscript

of the first edition and suggested many improvements. The criticism of
E. Marczewski and the reviews by Z. W. Birnbaum and S. Zubrzycki ofthe
first edition were useful in preparing the second edition; also useful were
valuable remarks of K. Urbanik, who read the manuscript of the second
edition. Numerous remarks and corrections were suggested by J. Wojtyniak, and R. Zasepa (first edition), and by L. Kubik, R. Sulanke, and J.
Wloka (second edition). R. Bartoszynski, with the substantial collaboration of Mrs H. Infeld, translated the book from the Polish. J. Karush
made valuable comments about the language. Miss D. Garbose did the
editorial work. B. Eisenberg assisted me in the reading of the proofs. My
sincere thanks go to all these people.
New York
October, 1962

MAREK

FISZ

Contents

CHAPTER

PAGE
PART 1

PROBABILlTY THEORY

RANDOM EVENTS

1.1 Preliminary remarks

1.2 Random events and operations performed on them
1.3 The system ofaxioms of the theory of probability
1.4 Application of combinatorial formulas for computing probabilities
1.5 Conditional probability
1.6 Bayes theorem .
1.7 Independent events
Problems and Complements
2

RANDOM VARIABLES

The concept of a random variable

2.2 The distribution function .
2.3 Random variables of the discrete type and the continuous type
2.4 Functions of random variables .
2.5 Multidimensional random variables
2.6 Marginal distributions
2.7 Conditional distributions .
2.8 Independent random variables
2.9 Functions of multidimensional random variables
2.1

xi

1
3
5

11
16
18
22
24
25

29
29

31
33
36
40
46
48

52
56

xii

CONTENTS

62

2.10 Additional remarks

Problems and Complements
3

PARAMETERS OF THE DISTRIBUTION OF A RANDOM VARIABLE

3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8

Expected values
Moments.
The Chebyshev inequality
Absolute moments
Order parameters
Moments of random vectors
Regression of the first type
Regression of the second type
Problems and Complements

64
64
67
74
76
77

79
91
96
101

CHARACTERISTIC FUNCTIONS

4.1
4.2
4.3
4.4

Properties of characteristic functions

The characteristic function and moments
Semi-invariants
The characteristic function of the sum of independent random
variables .
4.5 Determination of the distribution function by the characteristic
function .
4.6 The characteristic function of multidimensionaI random vectors
4.7 Probability-generating functions
Problems and Complements

62

SOME PROBABILITY DISTRlBUTIONS

105
107
110
112
115
121
125
126

129

5.1
5.2
5.3
5.4
5.5
5.6

One-point and two-point distributions

The Bernoulli scheme. The binomial distribution .
The Poisson scheme. The generalized binomial distribution
The Plya and hypergeometric distributions
The Poisson distribution
The uniform distribution

129
130
134
135
140
145

5.7
5.8
5.9
5.10
5.11

The
The
The
The
The

147
151
154
156
158

normal distribution
gamma distribution
beta distribution
Cauchy and Laplace distributions
multidimensional normal distribution

CONTENTS

5.12 The multinomial distribution

5.13 Compound distributions
Problems and Complements

Preliminary remarks
Stoehastie eonvergenee
Bernoulli's law of large numbers
The eonvergenee of a sequenee of distribution funetions
The Riemann-Stieltjes integral
The Lvy-Cramr theorem .
6.7 The de Moivre-Laplaee theorem .
6.8 The Lindeberg-Lvy theorem
6.9 The Lapunov theorem
6.10 The Gnedenko theorem
6.11 Poisson's, Chebyshev's, and Khintehin's laws of large numbers
6.12 The strong law of large numbers
6.13 Multidimensionallimit distributions
6.14 Limit theorems for rational funetions ofsome random variables
6.15 Final remarks
Problems and Complements

MARKOV CHAINS

7.1
7.2
7.3
7.4
7.5

170

175
176
179
180
184
188
192
196
202
211
216
220
232
236
239
239
250

Preliminary remarks .
Homogeneous Markov ehains
The transition matrix
The ergodie theorem .
Random variables forming a homogeneous Markov ehain

250
250
252
255
263

Problems and Complements

267

STOCHASTIC PROCESSES

8.1
8.2
8.3
8.4
8.5

163
164

175

LlMIT THEOREMS

6.1
6.2
6.3
6.4
6.5
6.6

Xlll

The notion of a stoehastie proeess

Markov proeesses and proeesses with independent inerements
The Poisson proeess .
.'
The Furry- Yute proeess
Birth and death proeess
8.6 The Plya proeess
8.7 Kolmogorovequations

271
271
272
276

281
287
298

301

CONTENTS

XIV

8.8
8.9
8.10
8.11
8.12

Purely discontinuous and purely continuous processes

The Wiener process
Stationary processes
Martingales
Additional remarks

304
309
314
323
325

Problems and Complements

327

PART 2
9

MATHEMATICAL STATISTICS

SAMPLE MOMENTS ANO THEIR FUNCTIONS

9.1 The notion ofa sample

9.2 The notion of a statistic
9.3 The distribution ofthe arithmetic mean ofindependent normally
distributed random variables
9.4 The 'i distribution
9.5 The distribution of the statistic (X, S)
9.6 Student's t-distribution
9.7 Fisher's Z-distribution
9.8 The distribution of X for sorne non-normal populations
9.9 The distribution of sample moments and sample correlation
coefficients of a two-dimensional normal population
9.10 The distribution of regression coefficients
9.11 Limit distributions of sample moments
Problerns and Complements
10

335
335
337
337
339
343
348
354
357
358
363
366
368.

ORDER STATlSTlCS

372

10.1 Preliminary remarks

10.2 The notion of an order statistic
10.3 The empirical distribution function
10.4 Stochastic convergence of sample quantiles
10.5 Limit distributions of sample quantiles
10.6 The limit distributions of successive sample elements
10.7 The joint distribution of a group of quantiles
10.8 The distribution of the sample range
10.9 Tolerance limits
10.10 Glivenko theorem
10.11 The theorems of Kolmogorov and Smirnov
'.
10.12 Rnyi's theorem
10.13 The problem of k sarnples

372
372
374
377
379
384
387
388
388
390
394
405
407

Problems and Complements

410

CONTENTS

11

AN OUTLINE OF THE THEORY OF RUNS

11.1
11.2
11.3
11.4

12

13

415
415
416
421

Problems and Complements

423

TESTS

425

The concept of a statistical test

Parametric tests for small samples
Parametric tests for large samples
The .%2 test
Tests of the Kolmogorov and Smirnov type .
The Wald-Wolfovitz and Wilcoxon-Mann-Whitney tests
Independence tests by contingency tables

425
427
433
436
445
449
456

Problems and Complements

459

THE THEORY OF ESTIMATION

461

Preliminary notions
Consistent estimates
Unbiased estimates
The sufficiency of an estimate
The efficiency of an estimate
Asymptotically most efficient estimates
Methods of finding estimates
Confidence intervals .
Bayes theorem and estimation

461
461
462
465
467
479
484
490
494

Problems and Complements

499

METHOOS ANO SCHEMES OF SAMPLING

503

13.1
13.2
13.3
13.4
13.5
13.6
13.7
13.8
13.9

14

415

Preliminary remarks . "

The notion of a run
The probability distribution of the number of runs
The expccted value and the variance of the number of runs

SIGNIFICANCE

12.1
12.2
12.3
12.4
12.5
12.6
12.7

XV

14.1
14.2
14.3
"14.4
14.5

Preliminary remarks .
Methods of random sampling
Schemes of independent and dependent random sampling
Schemes of unrestricted and stratified random sampling
Random errors of measurements

503
504
509
512
520

Problems and Complements

522

CONTENTS

XVi

15

16

AN OUTLlNE OF ANALYSlS OF YARIANCE

524

15.1
15.2
15.3

One-wa y c1assification
Multiple c1assification
A modified regression problem

524
531
535

Problems and Complements

540

THEORY OF HYPOTHESES TESTlNG

541

16.1
16.2
16.3
16.4
16.5
lq.6
16.7

541
541
552
558
560
566
578

Preliminary remarks .
The power function and the OC function
Most powerfuI tests
Uniformly most powerfuI test
Unbiased tests .
The powcr and consistency of nonparametric
AdditionaI remarks
.

tests

Problems and Complement

17

578

ELEMENTS OF SEQUENTIAL ANALYSIS

584

17.1
17.2
17.3
17.4
17.5
17.6
17.7
17.8

584
585
587
591
592

Preliminary rernarks .
The sequential probability ratio test
Auxiliary theorems
The fundamental Identity .
The OC function of the sequential probability ratio test
The expected value E(n)
The determination of A and B
Testing a hypothesis concerning the parameter p of a zero-one
distribution
17.9 Testing a hypothesis concerning the expected value m of a
normal population
17.10 Additional remarks
.
Problems and Complements

REFERENCES
T ABLES

AUTHORINDEX
SUBJECT

597
604
608
610
612
621

SUPPLEMENT
ST ATISTICAL

595
597

INDEX

ERRATA FOLLOWS INDEX

658
665
671

PA R T

Probability
Theory

CHAPTER

Random Events

1.1

PRELIMINAR Y REMARKS

A
Probability theory is a part of mathematics which is useful in discovering and investigating the regular features of random events. The
following examples show what is ordinarily understood by the term
random event.
Example 1.1.1. Let us toss a symmetric coin. The result may be either a head
or a tail. For any one throw, we cannot predict the result, although it is obvious
that it is determined by definite causes. Among them are the initial velocity of
the coin, the initial angle of throw, and the smoothness of the table on which the
coin falIs. However, since we cannot control alI these parameters, we cannot
predetermine the result of any particular toss. Thus the result of a coin tossing,
head or tail, is a random event.
Example 1.1.2. Suppose that we observe the average monthly temperature
at a definite place and for a definite month, for instance, for January in Warsaw.!
This average depends on many causes such as the humidity and the direction and
strength of the wind. The effect of these causes changes year by year. Hence
Warsaw's average temperature in January is not always the same. Here we can
determine the causes for a given average tempera ture, but often we cannot determine the reasons for the causes themselves. As a result, we are not able to
predict with a sufficient degree of accuracy what the average temperature for a
certain January wilI be. Thus we refer to it as a random evento
B
It might seem that there is no regularity in the examples given.
But if the number of observations is large, that is, if we deal with a mass
phenomenon, sorne regularity appears.
Let us return to exarnple 1.1.1. We cannot predict the result of any
particular toss, but if we perform a long series of tossings, we notice
that the number of times heads occur is approxirnately equal to the
number of times tails appear. Let n denote the nurnber of all our tosses
and m the number of times heads appear. The fraction m/n is called the
1

See example 12.5.1.

3

PROBABILITY THEORY

frequency of appearance of heads. The frequency of appearance of tails

is given by the fraction (n - m}/n. Experience shows that if n is
sufficiently large, thus if the tossings may be considered as a mass
phenomenon, the fractions m/n and (n - m}/n differ little ; hence each of
them is approximately l. This regularity has been noticed by many
investigators who have performed a long series of coi n tossings. Buffon
TABLE 1.1.1
FREQUENCY
OF BIRTHS OF Bovs ANOGIRLS

Year
of Birth

Number of Births
Boys
m

Girls

1927
1928
1929
1930
1931
1932

496,544
513,654
514,765
528,072
496,986
482,431

462,189
477,339
479,336
494,739
467,587
452,232

Total

3,032,452

2,833,422

Total
Number
of Births

Frequency of Births
Boys
PI

Girls

958,733
990,993
994,101
1,022,811
964,573
934,663

0.518
0.518
0.518
0.516
0.515
0.516

0.482
0.482
0.482
0.484
0.485
0.484

5,865,874

0.517

0.483

+I

P2

tossed a coin 4040 times, and obtained heads 2048 times; hence the ratio
of heads was m/n = 0.50693. In 24,000 tosses, K. Pearson obtained a
frequency of heads equal to 0.5005. We can see quite clearly that the
observed frequencies oscillate about the number 0.5.
As a result of long observation, we can also notice certain regularities
in example 1.1.2. We investigate this more closely in example 12.5.1.
Example 1.1.3. We cannot predict the sex of a newborn baby in any particular case. We treat this phenornenon as a random event. But if we observe a
large number of births, that is, if we deal with a mass phenomenon, we are able to
predict with considerable accuracy what will be the percentages of boys and girls
.among all newborn babies. Let us consider the number of births of boys and
girIs in Poland in the years 1927 to 1932. The data are presented in Table 1.1.1.
In. this table m andf denote respectively the number of births of boys and girls
in particular years. Denote the frequencies of births by PI and P2' respectively;
then
m
PI = m + f'
One can see that the values of PI oscilIate about the number 0.517, and the
values of P2 oscilIate about the number 0.483.

RANDOM

EVENTS

Example 1.1.4. We throw a die. As a result of a throw one of the faces

1, ... , 6 appears. The appearance of any particular face is a random evento If,
however, we perform a long series of throws, observing all those which give face
one as a result, we will notice that the frequency of this event will oscillate about
the number i. The same is true for any other face of the die.

This observed regularity, that the frequency of appearance of any

random event oscilIates about sorne fixed number when the number of
experiments is large, is the basis of the notion of probability.
Concluding these preliminary remarks, let us stress the fact that the
theory of probability is applicable only to events whose frequency of
appearance can (under certain conditions) be either directly or indirectly
observed or deduced by logical analysis.
1.2 RANDOM EVENTS AND OPERATIONS
PERFORMED ON THEM
A
We now construct the mathematical definition of a random event,
the colloquial meaning of which was discussed in the preceding section.
The primitive notion of the axiomatic theory of probability is that of
the set 01 elementary events. This set is denoted by E.
For every particularproblem wemust decidewhat iscalled the elementary
event; this determines the set E.
Example 1.2.1. Suppose that when throwing a die we observe the frequency
of the event, an even face. Then, the appearance of any particular face i, where
i = 1, ... , 6, is an elementary event, and is denoted by ei. Thus the whole set of
elementary events contains 6 elements.
In our example we are investigating the randorn event A that an even face will
appear, that is, the event consisting of the elementary events, face 2, face 4, and
face 6. We denote such an event by the symbol (e2, e4, es). The random event
(e2' e4, es) occurs if and only if the result of a throw is either fa ce 2 or face 4 or
fa ce 6.
If we wish to observe the appearance of an arbtrary face which is not face 1,
we will have a random event consisting of five elements (e2' ea, e4, es, es).
Let us form the set Z of random events which in this example is the set of all
subsets of E.
We include in Z all the single elements of E: (el)' (e2), (ea), (e4);(eS)' (e6), where
for instance, the random event (e4) is simply the appearance of the elementary
event, face 4.
Besides the 6 one-element random events (el), ... , (es), there also belong to
the set Z 15 two-element subsets (el' e2), .. , (es, es), 20 three-element subsets
(el> e2, ea), .. ,(e4' es, e6), 15 four-element subsets (el' e2, e3, e4), ... , (ea, e4, es,
e6), and 6 five-element subsets (el' e2, ea, e4, es)' ... , (e2, ea' e4, es, es). But these
are not all.
Now consider the whole set E as an event. It is obvious that as a result of a
throw we shall certainly obtain one of the faces 1, ... ,6, that is, we are sure that
one of the elementary events of the set E will occur. Usually, if the occurrence

PROBABILlTY THEOR y

of an event is sure, we do not consider it a random event; nevertheless we shall

consider asure event as a random event and include it in the set Z of random
events.
Finally, in throwing a die, consider the event of a face with more than 6 dots
appearing. This event includes no element of E; hence as a subset of E, it is
an empty set. Such an event is, of course , irnpossible, and usually is not considered as a random evento However, we shall consider it as a random event and
we shall incIude it in the set Z of random events, denoting it by the symbol (O).
JncIuding the impossible and sure events, the set Z of random events in our
example has 64 elements.
Generally, ir the set E contains n elernents, then the set Z of random events
contains 2n elements, namely,
impossible event (empty set),

( nI)

one-element events,

(n2)

two-element events,

( n -n I)

(n - l)-element events,

sure event (the whole set of elementary events E).

B
In example 1.2.1., the set E of elementary events was finite; in the
theory of probability we also consider situations where the set E is
denumerable or is of power continuum. In the latter case the set Z of
random events des not contain alI events, that is, it does not contain
all subsets of the set E. We shall restrict our considerations to a set Z
which is a Borel field of subsets of E. The definition of such a set Z is
given at the end of this section since this book is to be available to the
readers who do not know the set operations which are involved in the
definition of a Borel field.
We now give the definition of a random event. The notion of the set Z
appears in this definition. But since it has not been given precisely, we
return to the notion of a random event once more (see definition 1.2.10).
Definition 1.2.1. Every element of the Borel field Z of subsets of the set
E of elementary events is called a random evento
Definition 1.2.2. The event containing all the elements of the set E of
elementary events is called the sure evento
Definition 1.2.3. The event which contains no elements of the set E of
.elementary events is called the impossible evento
The impossible event is denoted by.(O).
Definition 1.2.4. We say that event A is contained in event B if every
elementary event belonging to A belongs to B.
We write

and read: A is contained in B.

RANDOM

EVENTS

We illustrate this notion by Fig. 1.2.1, where square E represents the

set of elementary events and circles A and B denote subsets of E. We see
that A is contained in B.
Definition 1.2.4'. Two events A and B are called equal if A is contained
in B and B is contained in A.
We write
E

A=B.
We now postulate the folIowing properties of Z.
Property 1.2.1. The set Z 01 random events
contains as an element the whole set E.
Property 1.2.2. The set Z 01 random eoents
contains as an element the empty set (O).
These two properties state that the set Z of
random events contains as elements the sure
Fig.1.2.1
and the impossible events.
Definition 1.2.5. We say that two events A and B are exclusive if they
do not have any common element of the set E.
Example 1.2.2. Consider the random event A that two persons from the
group of n persons born in Warsaw in 1950 will stilI be alive in the year 2000 and
the event B that two or more persons from the group considered will still be alive
in the year 2000. Events A and B are not exclusive.
If, however, we consider the event B' that onlv one person will still be alive in
the year 2000, events A and B' will be exclusive.
Let us analyze this example more closely. In the group of n elements being
considered it may happen that 1, or 2, or 3 ... up to n persons will still be alive
in the year 2000, and it may happen that none of them will be alive at that time.
Then the set E consists of n + 1 elementary events eo' el' ... , en, where the ndices O, 1, ... , n denote the number of persons from the group being considered
who will still be alive in the year 2000. The random event A in this example contains only one element, namely, the elementary event e2. The random event B
contains n - 1 elementary events, namely ,e2' e3' ... ,en- The common element
of the two events A and Bis the elementary event e2, and hence these two events .
are not exclusive. However, event B' contains only one element, namely, the
elementary event el. Thus events A and B' have no common element, and are
exclusive.

e We now come to a discussion of operations on events. Let

A, A2, be a finite or denumerable sequence of random events.
Definition 1.2.6. The event A which contains those and only those
elementary events which belong to at least one of the events Al' A2, is
called the alternatioe (or sum or union) of the events Al' A2,
We write
A

Al

and read:

A2

U ...

Al or A2 or ....

or

Al

A2

+ ... ,

PROBABILlTY THEORY
E

Fig. 1.2.2

Fig.1.2.3

Let us illustrate the alternative of events by Fig. 1.2.2.

On this figure, square E represents the set of elementary events and
circles Al' A2, A3 denote three events; the shaded area represents the
alternative Al + A2 + A3'
In our definition the alternative of random events corresponds to the
set-theoretical sum of the subsets Al' A2, , of the set of elementary
events.
The alternative of the events Al' A2, occurs if and only if at least
one of these events occurs.
The essential question which arises here is whether the alternative of
an arbitrary (finite or denumerable) number of random events belongs to
Z and hence is a random event. A positive answer to this question
results from the following postulated property of the set Z of random
events.
Property 1.2.3. If a finite or denumerable number 01 events Al' A2,
belong lo Z, then their alternatioe also belongs lo Z.
It is easy to verify that for every event A the following equalities are
true:
A U E= E,
A U (O) = A.
A U A = A,
For example, we prove that
A uA

= A.

In fact, every elementary event belonging to A U A belongs to A; hence

(A U A) e A. SimilarIy, A e (A U A); thus A U A = A.
Definition 1.2.7. The random event A containing those and only those
elementary events which belong to Al but do not belong to A2 is called
the difference of the events Al and A2
We write
A = Al - A2
The difference of events is illustrated by Fig. 1.2.3, where square E
represents the set of all elementary events and circ1es Al and A2 represent
two events; the shaded area represents the difference Al - A2

RANDOM

EVENTS

The difference Al - A2 occurs if and only if event

occurs.
If events Al and A2 are exclusive, the difference Al
the event Al'
As before, wepostulate the following property of
events.
Property 1.2.4. If events Al and A2 belong to Z,
also belongs to Z.

Al but not event A2

- A2 coincides with
the set Z of random
then their difference

Example 1.2.3. Suppose that we investigate the number of children in a

group of families. Consider the event A that a family chosen at randorn! has
only one child and the event B that the family has at least one child. The alternative A + B is the event that the family has at least one child.
If it is known that in the group under investigation there are no families having
more than n children, the set of elementary events consists of n + 1 elements
which, as in example 1.2.2. is denoted by eo' el' ... ,en' Event A contains only
one elementary event el' and event B contains n elementary events el' ... en' The
difference A - Bis, of course, an impossible event since there is no elementary
event which belongs to A and not to B. However, the difference B - A contains the elements e2' e3, ... , en and is the event that the family has more than
one child.
Definition 1.2.8. '(he event A which contains those and only those
elements which belong to all the events Al' A2, is called the product
(or intersection) of these events.
We write
A

= Al

n A2 n ... ,

or

= AlA2'

.. ,

or

TI Ai
i

and read: Al and A2 and ....

The product of events is illustrated by Flg. 1.2.4, where square E
represents the set of elementary events, and circles Al' A2, A3 represent
three events; the shaded area represents the product AlA2A3'
In our definition the product of events Al' A2, ... , corresponds to the
set-theoretical product of subsets Ah A2, , of the set of elementary
events. A product of events occurs if and only if all these events occur.
We postulate the following property of Z.
Property 1.2.5. If a finite or denumerable number 01 events Al' A2,
belong to Z, then their product also belongs to Z.
It is easy to verify that for an arbitrary event A the following equalities
are true:
A n (O) = (O).
A nA = A,
A nE= A,
Example 1.2.4. Consider the random event A that a farm chosen at random
has at least one horse and at least one plow, with the additional condition that
the maximum number of plows as well as the maximum number of horses are
1

We shall discuss later the methods of making such a choice.

10

PROBABILITY THEOR y
E

Fig.1.2.4

Fig.1.2.5

two. Consider also the event B that on the farm there is exaetly one horse and at
most one plow. We find the produet of events A and B.
In this example the set of elementary events has 9 elements which are denoted
by the symbols
the first index denoting the number of horses, and the seeond the number of
plows.
The random event A eontains four elementary events, ell' e12, e21, e22 and the
random event B eontains two elementary events, e10and ell. The produet A n B
eontains one elementary event en, and hence the event A r, B oeeurs if and only
if on the ehosen farm there is exaetly one horse and exaetly one plow.
Definition 1.2.9. The difference of events E - A is called the complemen t
of the event A and is denoted by .
The complement of an event is illustrated by Fig. 1.2.5, where square
E represents the set of elementary events, and circle A denotes some
event; the shaded area represents the complement A of A.
This definition may also be formulated in the following way: Event A
occurs if and only if event A does not occur.
According to properties 1.2.1 and 1.2.4 of the set Z of random events,
the complement A of A is a random event.
Example 1.2.5. Suppose we have a number of eleetrie light bulbs. We are
interested in the time t that they gIow. We fix a certain value lo such that if the
bulb burns out in a time shorter than lo, we eonsider it to be defective. We
select a bulb at random. Consider the random event A that we select a defective bulbo Then the random event that we seleet a good one, that is, a bulb that
gIowsfor a time no shorter than 10' is the event A-, the complement of the event A.
We now give the definition (see Supplement) of the Borel field of events
which was mentioned earlier.
Definition 1.2.10. A set Z 9f subsets of the set E of elementary events
with properties 1.2.1 to 1.2.5 is called a Borel field of events, and its elements
are caIIed random events.
In the sequel we consider only random events, and often instead of
writing "random event" we simply write "event."

RANDOM

11

EVENTS

D
"The following definitions will facilitate some of the formulations
and proofs given in the subsequent parts of this book.
Definition 1.2.11. The sequence {An}(n = 1,2, ... ) of events is called
nonincreasing if for every n we have

An ~ An+1'
The product of a nonincreasing
limit of this sequence, We write

sequence of events

TI An

A =

Ano

= lim

n;;'l

{An} is called the

n++a:

Definition 1.2.12.. The sequence {An}(n = 1, 2, ... ) of events is called

nondecreasing if for every n we have

An+1
The sum of a nondecreasing
sequence.
We write
A =

Ano

sequence

2 An

n;;'l

= lim

{An} is called the Iimit of this

Ano

n-+oo

1.3 THE SYSTEM OFAXIOMS OF THE THEORY

OF PROBABILITY
A
In everyday Ianguage the notion of probability is used without a
precise definition of its meaning. However, probability theory, as a
mathematical discipline, must make this notion precise. This is done by
constructing a system ofaxioms which formalize sorne basic properties
of probability, or in brief, by the axiomatization
of the theory of
probability.! The additional properties of probability can be obtained as
con sequen ces of these axiorns.
In mathernatics, the notion of random event defined in the preceding
section corresponds to what is called a random event in everyday use.
The system ofaxiorns which is about to be formulated makes precise the
notion of the probability of a random event. It is the mathematical
formalization of certain regularities in the frequencies of occurrence of
1 Many works have been devoted to the axiomatization of the theory of probability.
We mention here the papers of Bernstein [1], Lomnick i [1], Rnyi [1], Steinhaus [1], and
the book by Mazurkiewicz [1]. The system ofaxioms given in this section was constructed by Kolmogorov [7]. (The numbers in brackets refer tothe number of the paper
quoted in the references at the end of the book.) The basic notions of probability theory
are also discussed in the distinguished work of Laplace [1], and by Hausdorf [1], Mises
[1, 2], Jeffreys [1], and Barankin [2].

12

PROBABILITY THEORY

random events (this last to be understood in the intuitive sense) observed

during a long series of triaIs performed under constant conditions.
Suppose we are given the set of elementary events E and a Borel field
Z of its subsets. As has already been mentioned (Section 1.1), it has been
observed that the frequencies of occurrence of random events oscilIate
about sorne fixed number when the number of experiments is Iarge.
This observed reguIarity of the frequency of random events and the fact
that the frequency is a non-negative fraction less or equal to one have led
us to accept the foIlowing axiom.
Axiom 1. To every random event A there corresponds a certain number
peA), called the probability of A, which satisfies the inequallty

< peA) < 1.

The following simple exampIe Ieads to the formuIation ofaxiom

n.

Example 1.3.1. Suppose there are only black balls in an urn. Let the random
experiment consist in drawing a ball from the urn. Let m/n denote, as before,
the frequency of appearance of the black ball. It is obvious that in this example
we shall always have m/n = 1. Here, drawing the black ball out of the urn is a
sure event and we see that its frequency equals one.

Taking into account this property of the sure event, we formuIate the
foIlowing axiom.
Axiom JI. The probability of the sure etent equals one.
We write
P(E) = 1.
We shaIl see in Section 2.3 that the converse ofaxiom II is not true:
if the probability of a random event A equaIs one, or peA) = 1, the set A
may not include all the elementary events of the set E.
\
We have already seen that the frequency of appearance of face 6 in
throwing a die oscillates about the number l. The same is true for face 2.
We notice that these two events are exclusive and that the frequency of
occurrence ofeither face 6 or face 2 (that is, the frequency ofthe aIternative
of these events), which equals the sum of their frequencies, oscilIates
about the number i
t = l
Experience shows that if a card is selected from a deck of 52 cards
(4 suits of 13 cards each) many times over, the frequency of appearance
of any one of the four aces equals about l2' and the frequency of appearance of any spade equaIs about H. Nevertheless, the frequency of
appearance of the aIternative, ace or spade, oscil1ates not about the
number 5~2 + H = H but about the number H. This phenomenon is
explained by the fact that ace and spade are not exclusive random .
events (we could select the ace of spades). Therefore the frequency of the

RANDOM

13

EVENTS

alternative, ace or spade, is not equal to the sum of the frequencies of ace
and spade. Taking into account this property of the frequency of the
alternative of events, we formulate the last axiom.
Axiom IIl. The probability 01 the alternative 01 a finite or denumerable
number 01pairwise exclusive events equals the sum 01 the probabilities 01
these events.
Thus, if we have a finite or countable sequence of pairwise exclusive
events {Ak}, k = 1,2, ... , then, according to axiom IlI, the following
formula holds:
(1.3.1)
In particular, if a random event contains a finite or countable number
of elementary events ek and (ek) E Z (k = 1,2, ... ),
P(e1, e2,

= P(e1)

P(e2)

+ ...

The property expressed by axiom III is called the countable (or complete)
additioity of probablity.'
Axiom III concerns only the sums of pairwise exclusive events. Now
let A and B be two arbitrary random events, exclusive or not. We shall
find the probability of their alternative.
We can write
A U B = A U (B - AB),
B

= AB

U (B

- AB).

The right sides of these expressions are alternatives

Therefore, according to axiom IlI, we have

Pi A

of exclusive events.

+ P(B - AB),
= P(AB) + P(B - AB).

U B) = peA)

P(B)

From these two equations we obtain the probability of the alternative of

two events
(1.3.2)

peA

U B)

= peA) + P(B)

- P(AB).

Let Al' A2, , An, where n ~ 3, be arbitrary

easy to deduce the formula (due to Poincar [ID
(1.3.2')

random events.

It is

PC~lAk)=J/(A -k,t/(Ak,A
k)

+ I

kl,1.2.k3 =1
k1<k2<k3

k,)

kl <i

P(AklAk2Ak)

+ ... + (-1)n+lp(A1

... An).

1 We could have said that the probability f(A), satisfying axioms 1 to IJI, is a normed,
non-negatioe, and countably additioe measure on the Borel field Z of subsets of E.

14

PROBABILITY THEORY

B
Consider a finite or countable number of random events Ak, where
k = 1, 2, . . .. lf every elementary event of the set E belongs to at least
one of the random events Ah A2, , we say that these events exhaust
the set 01 elementary eoents E. The alternative LAk contains all the
k

elementary events of the set E and therefore is the sure event. By axiom
II we obtain
Theorem 1.3.1. If the eoents Al' A2, exhaust the set 01 elementary
ecents E,
(1.3.3)
Example 1.3.2. Let the set of all non-negative integers form the set of
elementaryevents. Let (en) be the event of obtaining the number n, where n =
0, 1, 2, . . .. Suppose that

where e is some constant.

From theorem 1.3.1 and axiom III it follows that

r:IJ

P
<Xl)

),

r:IJ

Len

=cL

n=O

n=O

1
"
n.

ce

But P ( '/~oen = 1 and 1~O lln! = e, where e is the base of naturallogarithms.

We then have 1 = ce; hence
In the following chapters it turns out that in this example we have
considered a particular case of the Poisson distribution which appears
very often in practice.
We now prove the following theorem.
Theorem 1.3.2. The sum 01 the probabilities 01 any eoent A and its
complement A is one.
Proof. From the definition of A it follows that the alternative A U A
of A and A is the sure event; therefore, according to axiom II we have
peA
But since events A and

A) = 1.

A are exclusive, we have, by axiom lII,

peA

A) = peA)

+ peA)

and finaIly
(l.3.4)

peA)

peA)

1.

Let A be the impossible event. We prove the next theorem.

RANDOM

15

EVENTS

Theorem 1.3.3. The probability of the impossible event is zero.

Proof. For every random event A we have the equality

AuE=E.
If A is the impossible event (does not contain any ofthe elementary events),
A and E are exclusive because they have no common element. Applying
axiom IlI, we obtain
peA)

+ P(E) = P(E).

It folIows immediately that

peA) = O.

We shaIl see in Section 2.3 that the converse is not true ; from the fact
that the probability of sorne event equa]s zero it does not foIlow that this
event is impossible.
e
The folIowing two theorems have numerous applications.
Theorem 1.3.4. Let {An}, n = 1, 2, ... , be a nonincreasing sequence of
eoents and let A be their producto Then
(1.3.5)

peA) = lim P(An).

n .... oo

Proofi

If the sequence {An} is nonincreasing, then for every n we have

00

An = L AkAk+l
k=n

+ A.

It foIlows from formula (I.3.2) that

(1.3.6)

P(A

n) (Jn
=P

k+1) + peA)

AkA

- P (A

Jn

AkAk+1

).

We note that
00

AL

k=n

00

Ak'4,c+l = L AAkAk+l"
lc=n

For every k, the event AA;;AHl is the impossible event;

= O. By axiom IlI, we obtain

therefore

P(AAkAk+I)

Since the events under the summation sign on the right-hand side offormula
(I.3.6) are exclusive, we have
oc

(1.3.7)

P(An) = L P(AkAk+l)
k=n

+ peA).

16

PROBABILITY

THEORY

However, the series

ex)

I P(AkA

k+l)

k=l

is convergent, being a sum of non-negative terms whose partial sums are

bounded by one. 1t follows that as 11 -+ 00 the sum in (1.3.7) tends to
zero. Thus, finally,
lim P(An) = peA).
n-+ 00

Theorem 1.3.5. Let {An}, n = 1, 2, ... , be a nondecreasing sequence of

eoents and let A be their alternative. Then we hace

(1.3.8)

peA) = lim P(An).

n-+

ex)

Proof. Consider the sequence of events {ArJ which are the complements
of the events An' From the assumption that {An} is a nondecreasing
sequence it follows that {An} is a nonincreasing sequence. Let A be the
product of events An' From theorem 1.3.4 it follows that

peA)

= lim

P(An).

n-+ 00

Hence

peA) = 1 - peA) = 1 - lim P(An) = 1 - lim [1 - P(An)] = lim P(An)

n-+

n-+oo

ex)

n-+oo

and the theorem is proved.

We give one more simple theorem.
Theorem 1.3.6. If ecents A and B satisfy the condition
A e B,

then
peA)

Proof,

< P(B).

Let us write
B =A

(B - A).

Events A and B - A are exclusive; hence, according to axiom IlI,

P(B) = peA)

+ P(B

- A).

Since P(B - A) ~ 0, we ha ve P(B) ~ peA).

1.4 APPLICATION OF COMBINATORIA.L FORMULAS

FOR COMPUTING PROBABILITIES
In sorne problerns we can compute probabilities by applying combinatorial formulas. We illustrate this by sorne examples.

RANDOM

EVENTS

17

Example 1.4.1. Suppose we have 5 balls of different colors in an urn. Assume

that the probability of drawing any particular ball is the same for any ball and
equals p.
Here E consists of 5 elements and by hypothesis each has the same probability.
Hence by theorem 1.3.1, we have 5p = 1, or p = t.
Example 1.4.2. "Suppose we have in the urn 9 slips of paper with the numbers
1 to 9 written on them, and suppose there are no two slips rnarked with the same
number. Then E has 9 elementary events. Denote by A the event that on the
slip of paper selected at random an even number will appear. What is the probability of this event?
As before, we suppose that the probability of selecting any particular slip is the
same for any slip, and hence equals t. We shall obtain a slip with an even
number ir we draw one of the slips marked with 2, 4, 6 or 8. According to axiom
Ill, the required probability equals
peA)

= t + ~-+ t + t = -*.

If in the example considered we wish to compute the probability of selecting a

slip with an odd number, w~ may notice that this random event is the complement of A (we denote it by A) and, by theorem 1.3.2, we have
peA) = 1 - peA) = i.

Example 1.4.3. Let us toss a coin three times. What is the probability that
heads appear twiee?
The number of all possible combinations which may occur as a result of three
suceessive tosses equals 23 = 8. Denote the appearanee of heads by H and the
appearanee of tails by T. We have the following possible eombinations:
HHH, HHT, HTH, THH, HTT, THT, TTH, TTT.

Consider each of these combinations as an elementary event and the whole

colleetion of them as the set E. Suppose that the oeeurrenee of each of thern has
the same probability. We then have that the probability of eaeh particular cornbination equals 1/23 From the table we see that heads appear twce in three
elementary cvents (HHT, HTH, THH); henee by axiorn 111the required probability is ~-."
If in the example just considered we had 11 tos ses instead of 3 and looked
for the probability of obtaining heads m times, our reasoning would have
been as follows.
The number of all possible combinations with 11 tosses equals Z".
The number of combinations in which heads appear m times equals the
number of combinations of m elements from 11 elements given by

( mn)
If every possible result of
the required probability is
(1.4.1)

11

n!
=m!(I1-111)!'

successive tosses of a coin is equally likely,

n!

18

PROBABILITY THEORY

Example 1.4.4. Compute the probability that heads appear at least twice in
three successive tosses of a coin.
The random event under consideration will occur if in three tosses heads appear
two or three times. According to formula (1.4.1), the probability that heads
appear three times equals
3!
233! O! = 8'

and the probability that heads appear twice equals l, as we already know.
Hence, according to axiom Hl, the required probability is

i + i = .
In exarnples 1.4.1 to 1.4.4 the equiprobability of all elernentary events
was assurned. This assurnption was obviously satisfied in our exarnples,
but it is not always acceptable.

1.5 CONDITIONAL PROBABILITY

A

Let us first consider sorne examples.

Example 1.5.1. A. Markov [4] has investigated the probability of the appearance of these pairs of letters in Russian:
Vowel after vowel,
Vowel after coi.sonant.
To compute these probabilities he counted the corresponding pairs of letters in
Pushkin's poem Eugene Onegin on the basis of a text of 20,000 letters, and he
accepted the observed frequencies as probabilities.' The experiment yielded the
following results: there were 8638 vowels, and the pair "vowel after vowel"
appeared 1104 times.
Let us analyze this example. Denote a vowel by a and a consonant by b. As
elementary events we shall consider the pairs aa, ba, ab, bb, the set of elementary
events is then (aa, ab, ba, bb).
Consider event B that a pair of letters will appear in which a vowel is in
second place. Event B may be written as (aa, ba). It is known that a vowel
appears 8638 times. These vowels follow either another vowel (in the pairs aa)
or a consonant (in the pairs ba). Because no vowel appears at the beginning of
the text considered
"Mo ARAR causrx teCTHbIX npaunn ... ,"2

event B occurs 8638 times. Thus

8638

P(B)

= 20,000 = 0.432.

Consider now event A that the pair of letters occurs with a vowel in first place.
Event A may be written as (aa, ab).
1
2

The methods of verification of such hypotheses are given in Part 2 of this book.
It mean s, "My uncIe's shown his good intentions."

19

RANDOM EVENTS

The question "What is the frequency of a vowel followed by a vowel?" might

now be formulated as follows.
What is the probability of event A in cases when event B has already occurred?
We are not interested here in the probability of event A in the whole set E of
elementary events but in the conditional probability which would correspond to
the conditional frequency of event A provided event B has occurred ; in other
words, the probability of event A in the set (aa, ha) considered as the whole set
of elementary events.
In our example we are interested in the probability of the event (aa). The experiment showed that this event appeared 1104 times, and, since event B
appeared 8638 times, the probability we are looking
for equals
E
1104

8638 = 0.128.
B
In general, let B be an event in the set
of elementary events E. The set B is then an
element of the Borel field Z of subsets of the
set E of all elementary events. SupposeP(B)
O.
Let us consider B as a new set of elementary
events and denote by Z' the Borel field of all
Fig. 1.5.1
subsets of B which belong to the fieId Z.
Consider an arbitrary event A from the fieId Z. It may happen in
particular cases that the event A belongs to the field Z', namely, when A
is a subset of B. If, however, A contains any element of E which does not
belong to B, A is not an element of Z'; yet sorne part of A rnay be a
random event in Z', namely, when A and B have cornrnon eIernents, that
is, when the product AB is not empty.
Now let B denote a fixed elernent of the field Z, where P(B)
0, while
A runs over all possible elements of Z; then all elernents of Z' are products
of the form AB. To stress the fact that the product AB is now being
considered as an element of Z' (and not of Z) we denote it by the symboI
A I B and read: "A provided that B" or "A provided that B has occurred."
If A contains B, A IBis the sure event (in the field Z').
Event A IBis illustrated by Fig. 1.5.1. Here square E represents the
set of all elementary events, and circles A and B denote sorne random
events. The shaded area represents the random event B, and the doubly
shaded area represents the random event A B, that is, "event A provided
that B has occurred."
The probability of the event A I B in the field Z' will be denoted by
peA I B) and read: The conditional probability of A provided B has
occurred.
As will be shown shortly this probability can be defined by using the
probability in the field Z; hence there is no need to postulate separately .
the existence of the probability Pt A I B) and its properties.

>

>

20

PROBABILITY THEOR y

e To faciIitate the understanding of the definition of Pt A lB), let us

consider the following.
Suppose we have performed n random experiments and have obtained
the event B m times. Moreover, in k(k
m) of these experiments we also
obtained the random event A. The frequency of AB equaIs kln, and the
frequency of B equaIs m/n; the frequency of the random event A, provided
the random event B has occurred, equaIs klm.
AppIying the equaIity

<

k
m

(1.5.1)

k/n
m/n'

to the probabiIities instead of the frequencies, we accept the following

definition.
Definition 1.5.1. Let the probabiIity of an event B be positive. The
conditional probability 01 the event A provided B has occurred equaIs the
probability of AB divided by the probability of B.
Thus

(1.5.2)

peA B) = P(AB)
P(B) ,

where

P(B)

> O.

P(B I A) = P(AB)
peA) ,

where peA)

> O.

Similarly,
(1.5.3)

From (l.5.2) and (l.5.3) we obtain

(1.5.4)

P(AB) = P(B)P(A

I B) = P(A)P(B

lA).

This formula is to be read: The probability 01 the product AB 01 two events

equals the product 01 the probability 01 B times the conditional probability
01 A prooided B has occurred or, what amounts to the same thing, to the
probability 01 A times the probability 01 B provided A has occurred.
Let A, A2, A3 denote three events from the same fieId Z. Consider the
expression P(A3 I AA2), or the probabiIity of A3 provided the product
AA2 has occurred. According to (l.5.2) this probabiIity, assuming that
P(AA2)
O,equaIs

>

(1.5.5)

IA

peA
3

A ) = P(AA2A3) .
P(AA2)

From (l.5.5) and (l.5.3) we obtain for the probability of the product of
three events thc reIations
(l.5.6)

P(AA2A3) = P(AA2)P(A3 AA2)

= P(A)P(A2

I A)P(A3 I A1A2)

RANDOM

21

EVENTS

This formula is to be read: The probability 01 the product 01 three

events equals the probability of the first event times the conditional probability 01 the second event provided the first event has. occurred times the
probability 01 the third event provided the product 01 the first two events
has occurred.
Now let A, Az, ... ,An be random events. We could consider the
conditional probabilities peAk 1Ak 2 ... Ak r I Ak r+l ... Ak ) of the product
of sorne subgroup consisting of r events (1 ~ r ~ n - 1) provided the
product of the remaining n - r events has occurred. By a reasoning
similar to that stated we obtain
11

(1.5.7)

P(AlA2 ...

An)

= P(Al)P(A2

I Al)P(A3 I AlA2)

... P(An

D
We shall show that the conditional probability
to 111.
We notice that the folIowing inequality is true:
(1.5.8)

P(AB)

I Al

...

An-l)

satisfies axioms 1

< P(B).

In fact, event B may occur either when event A occurs, or when event
A does not occur; hence
B= AB UAB,
where A is the complement of A. Thus AB e B, and from theorem 1.3.6,
we obtain (1.5.8).
Since P(AB)
O and P(B)
O we obtain, from formula (1.5.8),

>

>

O~p(AIB)~l,

which is the property expressed by axiom 1.

Now let A I B be the sure event in fieId Z', that is, let AB = B. Then
P(AB) = P(B),

and hence
peA lB)

1.

This is the property expressed by axiom 11.

Consider now the alternative L(Ai B) of pairwise exclusive events.
We can write
i

tCA;1

B) = (tA,) lB,

and hence

p[ t CA; lB)] = p[ (tA,) lB J.

22

PROBABILITY THEORY

According to (1.5.2) and axiom 111we have

[( t Ai ) I B]

p[(tA,)BJ

P(tA,B)

P(B)

P(B)

= I P(AiB) = I P(Ai
P(B)

lB).

This formula expresses the countable additivity of conditional probability.

Since all the axioms are satisfied for the conditional probabilities, the
theorems derived from these axioms hold for the conditional probabilities.
1.6 BAYES THEOREM
A

Before we start the general consideration let us consider an example.

Example 1.6.1. We have two urns. There are 3 white and 2 black balls in the
first urn and 1 white and 4 black balls in the second. From an urn chosen at
random we select one ball at random. What is the probability of obtaining a
white ball if the probability of selecting each of the urns equals 0.5?
Denote by Al and A 2 respcctively, the events of selecting the first or second
urn, and by B the event of selecting a white ball. Event B may happen either
together with event Al or together with event A2; hence we have
B = AIB

+ A2B,

and since events AIB and A2B are exclusive, we have

P(B)

= P(AIB) + P(A2B).

Applying formula (1.5.4) we obtain

(1.6.1)

P(B) = P(AI)P(B Al)

+ P(A2)P(B I A2)

In this example we have peAl) = P(A2) = 0.5, P(B I Al) = 0.6, and
= 0.2. Placing these values into (1.6.1) we obtain P(B) = 0.4.

P(B I A 2)

Formula (1.6.1) obtained in this example is a speciaI case of the theorem

of absolute probability, which is now given.
.
Theorem 1.6.1. If the random events Al' A2, are pairwise exclusive
and exhaust the set E of elemen tary events, and ifP(Ai)
0for i = 1,2, ... ;
then for any random event B we have

>

(1.6.2)

P(B) = P(AI)P(B

I Al)

+ P(A2)P(B

IA

2)

+ ...

In fact, from the assumptions it follows that R may happen together

with one and only one of the events Ai' We then have
B = AIB

+ A2B + ...

and
(1.6.3)

P(B) = P(AIB)

+ P(A2B) + ...

RANDOM

23

EVENTS

According to (1.5.4) we obtain for every i,

(1.6.4)
Substituting values (1.6.4) into (1.6.3) we get (1.6.2).
B
Again let the events Ai satisfy the assumptions of theorem 1.6.1.
Suppose that the event B has occurred. Now what is the probability of
Ai? This question is answered by the following theorem due to Bayes.
Theorem 1.6.2. If the events Al' A2, satisfy the assumptions of the
theorem of absolute probability and P(B)
0, then for i = 1, 2, ... we hace

>

(1.6.5)

P(Ai

I B) =
P(Al)P(B

P(Ai)P(B I Ai)
I Al) + P(A )P(B I A + ...
2

2)

In fact, substituting Ai for A in formula (l.5.4), we obtain

and introducing in the denominator expression (1.6.2) for P(B), we

obtain (1.6.5).
Formula (1.6.5) is called Bayes formula or the formula for a posteriori
probability. The latter name is explained by the fact that this formula
gives us the probability of Ai after B has occurred. On the other hand,
the probabilities P(Ai) in this formula are called the a priori probabilities.
Bayes formula plays an important role in applications.
Example 1.6.2. Guns 1 and 2 are shooting at the same target. It has been
found that gun 1 shoots on the average nine shots during the same time gun 2
shoots ten shots. The precision of these two guns is not the same; on the average, out of ten shots from gun 1 eight hit the target, and from gun 2, only seven.
During the shooting the target has been hit by a bullet, but it is not known
which gun shot this bullet. What is the probability that the target was hit by
gun 2?
. Denote by Al and A 2 the events that a bullet is shot by gun 1 and gun 2,
respectively, Taking into consideration the ratio of the average number of shots
.made by gun 1 to the average number of shots made by gun 2, we can put
peAl) = 0.9P(A2V Denote by B the event that the target is hit by the bullet.
According to the data about the precision of the guns we have P(B I Al) = 0.8
and P(B I A2) = 0.7. According to Bayes formula
P(A21

B) = P(Al)P(B

P(A2)P(B I A2)
I Al) + P(A2)P(B

I A2)
0.7 P(A2)
0.9P(A2)

0.8

+ 0.7P(A2) = 0.493..

The methods of verifying such hypotheses will be given in Part 2.

24

PROBABILITY THEORY

1.7 INDEPENDENT EVENTS

A
In general, the conditional probability peA I B) differs from peA).
However, the case when we have the equality
peA I B) = peA)

(1.7.1)

is of special importance. Then the fact that B has occurred does not have
any influence on the probability of A, or we could say, the probability
of A is,independent of the occurrence of B.
We notice that if(l. 7.1) is satisfied, formula (1.5.4) gives
(1.7.2)

P(AB) = P(A)P(B).

Formula (1.7.2) is also satisfied if

(1.7.3)

P(B A) = P(B)

Formula (1.7.2) was derived from formula (1.5.4) where it was assumed
that peA) > O and P(B) > O; nevertheless this formula is also valid
when one of these probabilities equals zero.
We now define the notion of independence of two random events.
Definition 1.7.1. Two events A and B are called independent if their
probabilities satisfy (1.7.2), that is, if the probability of the product AB
is equal to the product of the probabilities of A and B.
It follows from this definition that the notion of independence of two
events is symmetric with respect to these events.
As we have already established, formula (1.7.2) can be obtained from
either of the formulas (1.7.1) and (1.7.3). We also notice that formulas
(1.5.4) and (1.7.2) with peA) > Oand P(B) > Ogive formulas (1.7.1) and
(1.7.3). We thus deduce that each of the last formulas is a necessary and
sufficient condition for
probabili 1ies.

the independence of two events with positive

B
The notion of independence of two events can be generalized to
the case of any finite number of events.
Definition 1.7.2. Events Al' A2, , An are independent iffor all integer
indices kl, k2, k, satisfying the conditions
1

< k < k < ... < k, < n

l

we have
(1.7.4)

Pt A;1Ak 2 ...

Ak .~.) = Pt A 1)P(Ak 2) ...

Pt A, ),
S

that is, if the probability of the product of every combination

Akl' Ak2' ... , Aks of events equals the product of the probabilities of these
events.

RANDOM

25

EVENTS

It is possible that Al' A2, , An are pairwise independent, that is, each
two events among Al' A2, , An are independent, that each three ofthem
are independent, and so on, and yet Ab A2, . , An are not independent.
This is illustrated by the example given by Bernstein.
Example 1.7.1. There are four slips of paper of identical size in an urn.
Each slip is marked with one of the numbers 110, 101,011,000 and there are no
two slips marked with the same number. Consider event Al that on the slip selected the number 1 appears in the first place, event A2 that 1 appears in the second
place, and A3 that 1 appears in the third place. The number of slips of each category is 2, the number of all slips is 4; hence if we assume that each slip has the
same probability of being selected, we have

= P(A2) = P(A3) = t .
Let A denote the product A1A2A3' P(A) = Osince event A is impossible (there
P(Al)

is no slip marked with 111) and since

P(Al)P(A2)P(A3) = i ~
O = P(A),
events Al' A2, and A3 are not independent. We shall show, however, that these
events are pairwise independent. In fact, for the pair Al' A2 we have
P(A21 Al) =

= P(A2),

since there are only two slips having 1 in the first place, and only one among them
with 1 in the second place. In a similar way we could show that the remaining
pairs are independent.
Independence of a countable number of events is defined in the following
way.
Definition 1.7.3. Events Al' A2, are independentifforevery n = 2,3, ...
events Ab A2, , An are independent.
From definitions 1.7.3 and 1.7.2 it follows that if the random events
Ab A2, are independent, for n = 2, 3, . . . and for arbitrary indices
kl, k2, , km events Ak 1, Ak 2, ... ,Ak n are independent.
To stress the fact that we consider the independence in the sense of
definitions 1.7.2 and 1.7.3, and not the independence of pairs, triples
and so on, the term independence en bloc, or mutual independence, is
used in probability theory. We shall avoid these terms, and independence
will always be understood in the sense of the given definitions.
Problems and Complements
1.1. Prove that the operations of addition and multiplication of random
events are commutative and satisfy the following associative and distributive
laws:

+ A2 = A2 + Ah
A1A2 = A2A,
Al + (A2 + A3) = (Al + A2)
Al(A2A3) = (A1A2)A3,
Al

Al(A2

+ A3,

+ A3) = A1A2 + A1A3'

26

PROBABILITY THEORY

1.2. Prove the relations:

A1A2 = Al - (Al - A2),
Al + A2 = Al + (A2 - Al),
Al - A2 = Al ,- A1A2,
Al(A2 - A3) = A1A2 - A3'
1.3. Prove that Al e A2 implies Al + A2 = A2 and A1A2 = Al'
1.4. (a) Prove the following two identities, called de Morgan's laws:
Al.+

A2

= A1A2,

A1A2

(h) Generalize these identities to the case of

1.5. (a) Prove that for n = 2, 3, ...
Al

+ ... + An = Al + (A2

- A1A2)

Note that the terms on the right-hand

(h) Show that

U Ai = Al + (A2

= Al + A2
n (n > 2) random

+ ... + (An

- A1An ...

events.
- An-1An)'

side are pairwise exclusive.

- A1A2)

+ (A3

- A1A3 - A2A3)

+ ....

i=l

1.6. Prove that properties 1.2.2 and 1.2.5 follow from properties 1.2.1, 1.2.3,
and 1.2.4.
1.7. Let {An} (n = 1,2, ... ) be an arbitrary sequence of random events. The
random event A * which contains all the elementary events which belong to an infinite nurnber of the events An will be called the upper limit of the sequence {An},
A* = lim sup An'
n---a)

The random event A* whch contains all the elementary events which belong to
all but a finite number of the events An will be called the lower Iimit of the sequence
{An},
n-a)

(a) Prove that

ce

a)

:L A
n=lk=n

A*

= TI

A*

a)

k,

ce

I TIAk'
n=lk=n

(h) Show that

A* e A*.
1.8. (Notation as in the preceding problern) lf A* = A*, then A
is called the limit of the sequence {An}' Then we write

= lim

Ano

(a) Prove that if {An} is a nondecreasing

sequence,
ce

A=A*=A*

=_LAn.
n=l

(h) Prove that if {An}

is a nonincreasing sequence, then

ce

A =A*

=A*

=TIAn.
n=l

A*

A*

RANDOM

EVENTS

27

1.9. (a) Prove that for an arbitrary sequence of random events {An}
P (lim sup An) ~ lim sup P(An),
n->-oo

n-oo

and
P (lim inf An) ~ lim inf P(An).
1t->-00

n->- 00

Hint: Use theorems 1.3.4 and 1.3.5.

(b) Deduce that if lim An exists, then
n->-oo

P ( lim An)
n->- 00

1.10. Using Problems

1.5(a)

= lim P(An).
n->- 00

and

(b),

Cl

A;) Si~/(A;),

PCl

prove the inequalities

A;) Si~/(Ai)'

In combinatorial Problems 1.11 to 1.16 assume that all the possible outcomes
have the same probability.
1.11. A deck of cards contains 52 cards. Player G has been dealt 13 of them.
Compute the probability that player G has
(a) exactIy 3 aces,
(b) at least 3 aces,
(e) any 3 face cards of the same face value,
(d) any 3 cards of the same face value from the five highest denominations,
(e) any 3 cards of the same face value from the eight lowest denominations,
(f) any 3 cards of the same face value,
(g) three successive spades,
(h) at least three successive spades,
(i) three successive cards of any suit,
(j) at least three successive cards of any suit.
1.12. Three dice are thrown once. Compute the probability of obtaining
(a) face 2 on one die,
(b) face 3 on at least one die,
(e) an even sum,
(d) a sum divisible by 3,
(e) a sum exceeding 7,
(f) a sum smaller than 12,
(g) a sum which is a prime number.
1.13. (Cheoalier de Mr's problem)
Find which of the following two events
is more likely to occur: (1) to obtain at least one ace in a simultaneous throw of
four dice; (2) at least one double ace in a series of 24 throws of a pair of dice.
1.14. (Banach's problem) A mathematician carries two boxes of matches,
each of which originally contained n matches. Each time he lights a cigarette
he selects one box at random. Compute the probability that when he eventually
selects an empty box, the other will contain r matches, where r = 0, 1, ... , n.

28

PROBABILITY THEORY

1.15. An urn contains m white and n-m black balls. Two players successively draw balls at random, putting the drawn ball back into the urn before the
next drawing. The player who first succeeds in drawing a white ball wins. Compute the probability of winning by the player who starts the game.
1.16. There are 28 slips of paper; on each of them one letter is written. The
letters and their frequencies are presented in the following table:
Letter

Number of slips

e
3

.i

lmn

The slips are then arranged in random order. What is the probability of obtaining the sentence.! "Sto lat sto lat niech iyje iyje nam"?
1.17. The famous poet Goethe once gave his guest, the famous chemist Runge,
a box of coffee beans. Runge used this gift-at that time very valuable-for
scientific experiments, and for the first time obtained pure caffeine. Is it possible
to compute the probability of this event? If so, is the answer unique? What are
the factors which determine the precise formulation of the random event whose
probability we compute?
1.18. (Bertrand's paradox)
A circle is drawn around an equilateral triangle
with side a. Then a random chord is drawn in this circle, The event A occurs if
and only if the length 1of this chord satisfies the relation 1 > a. State the conditions under which (a) P(A) = 0.5, (b) P(A) = 1/3, (e) P(A) = 1/4. Should these
resuIts be considered as paradoxical?
1.19. (Buffon's problem)
A needle oflength 21 is thrown at random on aplane
on which parallel lines are drawn at a distance 2a apart (a > 1). What is the
probability of the needle intersecting any of these lines?
. 1.20. The probability that both of a pair of twins are boys equals 0.32 and the
probability that both of them are girls equals 0.28. Find the conditional probability that
.
(a) the second twin is a boy, provided the first is a boy,
(b) the second twin is a girl, provided the first is a girl.
Hint: Use example 1.1.3.
1.21. (a) What should n be in order that the probability of obtaining the face 6
at least once in a series of n independent throws of a die will exceed 3/4?
(b) The events Al' A2, are independent and P(Aj) = P (j = 1, 2, ... ).
Find the least n such that

where Po is a given number.

1.22. (a) The events Al' A2, are independent and P(Ak) = h. What is the
probability that none of the events A,~occurs?
(b) Answer the question of Problem 1.22(a) without the assumption of independence.
1.23. Prove that if the evnts A and B are independent, the same is true for
the events A and [J.
1 The beginning of a Polish birthday songo It means something similar to, "May he
live a hundred years, a hundred years."

CHAPTER

Random Variables

2.1

THE CONCEPT

OF' A RANDOM

VARIABLE

We can assign a number to every elementary event from a set E of

elementary events. In the coin-tossing example we assigned the number 1
to the appearance of heads and the number O to the appearance of tails.
Then the probability of obtaining the number 1 as a result of an experiment
will be the same as the probability of obtaining a head, and the probability
of obtaining the number O will be the same as the probability of obtaining
a tail.
Similarly, in the example of throwing a die we can assign to every
result of a throw one ofthe numbers i (i = 1, ... , 6) corresponding to the
number of dots appearing on the face resulting from our throw. In
general, let e denote an elementary event of a set E of elementary events.
On the set E we define a single-valued real function X(e) such that, roughly
speaking, the probability that this function will assume certain values is
defined. To formulate precisely the conditions which are to be satisfied
by this function let us introduce the notion of an inverse image.
Definition 2.1.1. Let X(e) be a single-valued real function defined on the
set E of elementary events. The set A of all elementary events to which
the function X(e) assigns values in a given set S of real numbers is called
the inverse image 01 the set S.
It is clear that the inverse image of the set R of all real numbers is the
whole set E.
Definition 2.1.2. A single-valued real function X(e) defined on the set E
of elementary events is called a random variable! if the inverse image of
every interval 1 on the real axis of the form (- 00, x) is a random evento
We shall set the probability P(x>(l) that the random variable X(e) takes
on a value in the interval 1 equal to the probability peA) of the inverse
image A of l.
1 The notion of a random variable corresponds in the theory of real functions to the
notion of a [unction measurable with respecI lo the field 01 seIs being considered.

29