Module 1

Best Practices for Efficient Soil Sampling Designs

The Point of this Workshop

is not to teach you to be statisticians.
Rather, teach you basic concepts so able to ask for what you need & detect issues with statisticallybased soil sampling program designs.

is not to say statistics are wrong, never appropriate, or cannot be used.

Rather, to show that statistics cannot be used blindly or as a black box for data collection design/interpretation And identify pitfalls common to sampling programs, the problems they cause, and how to avoid them.
10June2008 Triad Investigations: New Approaches and Innovative Strategies 2

Workshop Agenda
Welcome Where does decision uncertainty come from? You cant find the answer if you dont know the question! Beware of statistics bearing assumptions The cure for sampling dilemmas: use emerging best practices
10June2008 Triad Investigations: New Approaches and Innovative Strategies 3

Instructors
Deana Crumbling, [email protected] Office of Superfund Remediation &Technology Innovation U.S. Environmental Protection Agency Washington, D.C. (703) 603-0643 Robert Johnson, [email protected] Environmental Science Division Argonne National Laboratory Argonne, Illinois (630) 252-7004
10June2008 Triad Investigations: New Approaches and Innovative Strategies 4

Software Resources and Disclaimer

Several software packages are referenced. References do not constitute an endorsement. For more information:
Visual Sampling Plan (http://dqo.pnl.gov/) ProUCL (http://www.epa.gov/esd/tsc/software.htm) BAASS (http://web.ead.anl.gov/baass/register2/)

10June2008

Triad Investigations: New Approaches and Innovative Strategies

Take Away Points

Statistics often are used to address cleanup decision-making uncertainty Care must be taken when applying traditional statistical approaches to soil sampling programs Be extremely cautious with black-box software Systematic planning, dynamic work plans, and real-time measurement techniques (the Triad) can greatly improve sampling program designs

10June2008

Triad Investigations: New Approaches and Innovative Strategies

Module 2

Where Does Uncertainty Come from When Making Restoration Decisions?

As we know, there are known knowns. There are things we know we know. We also know there are known unknowns. That is to say we know there are some things we do not know. But there are also unknown unknowns, the ones we don't know we don't know. Donald Rumsfeld, Feb. 12, 2002, Department of Defense news briefing
10June2008 Triad Investigations: New Approaches and Innovative Strategies 8

Decision Uncertainty Comes from a Variety of Sources

Political, economic, organizational & social uncertainty (outside scope of discussion) [know we know] Model uncertainty (also outside discussion scope, although approaches to be discussed may provide mechanisms for addressing this) [know we dont know] Data uncertainty. Data uncertainty refers to the uncertainty introduced into decision-making by uncertainty associated with data sets used to support decisions [Dont know we dont know] Data Uncertainty: primary focus of the trainingsampling programs play important role.
10June2008 Triad Investigations: New Approaches and Innovative Strategies 9

Decision Quality Only as Good as the Weakest Link in the Data Quality Chain
Sampling
Sampling Design SubSampling

Analysis
Extract Cleanup Method Result Reporting

Interpretive

Sample Support

Sample Preservation

Sample Prep Method

Determinative Method

Relationship between Measurement Parameter & Decision Parameter

Each link represents a variable contributing toward the quality of the analytical result. All links in the data quality chain must be intact for data to be of decision-making quality!
10June2008 Triad Investigations: New Approaches and Innovative Strategies 10

Taking a Sample for Analysis

Population Soil Core Sample Lab Subsamples (Duplicates) Analytical Sample Unit

Field 10June2008 Subsample

Analytical Sample Prep

GC

23.4567 ppm

Triad Investigations: New Approaches and Innovative Strategies

11

Historically the Focus Has Been Analytical Quality

Emphasis on fixed laboratory analyses following well-defined protocols Analytical costs driven to a large degree by QC/QC requirements Result:
analytical error typically on order of +/-30% for replicate analyses traditional laboratory data treated as definitivebut definitive (definite) about what?
10June2008 Triad Investigations: New Approaches and Innovative Strategies 12

The Biggest Cause of Misleading Data

10June2008

Triad Investigations: New Approaches and Innovative Strategies

13

Within-Sample Variability: Interaction between Contaminant & Matrix Materials

Firing Range Soil Grain Size Pb Concentration in (Std Sieve Mesh Size) fraction by AA (mg/kg)
Greater than 3/8 (0.375)
Adapted from ITRC (2003)

10 50 108 165 836 1,970 927 (wt-averaged)

14

Between 3/8 and 4-mesh Between 4- and 10-mesh Between 10- and 50-mesh Between 50- and 200-mesh Less than 200-mesh Bulk Total

10June2008

The decision determines representativeness Triad Investigations: New Approaches and Innovative Strategies

Concentrated Particles within Less Concentrated Matrix = Nugget Effect

Regulatory & field practices have long assumed that sample size/volume has no effect on analytical results The Nugget Effect
Soil Subsample

Sample Prep

2g

5g

Now we know the assumption is inaccurate because of microscale (within-sample) heterogeneity.

Sample volume affects the analytical result! 10June2008 Triad Investigations: New Approaches and Innovative Strategies
15

Micro-scale Heterogeneity Causes Highly Variable Data Results

True sample mean known to be 192 ppm
Replicate Range of results How many subsamples subsample weight to average so result is for 20 replicate taken from a large, close (+25%) to true subsamples ground & sieved soil sample mean? (ppm) sample [144 - 240 ppm]

1g 10 g 100 g

101 - 800 136 - 343 170 - 230

39 5 1

Adapted from DOE (1978) americium-241 study

10June2008 Triad Investigations: New Approaches and Innovative Strategies 16

Short-Scale Variability Can Also be Significant

286 ppm 7 416 ppm 2 41,400 ppm 1,220 ppm 6 1 2 ft
Figure adapted from Jenkins (CRREL), 1996

3 136 ppm

5 27,700 ppm
10June2008

4 42,800 ppm
17

Triad Investigations: New Approaches and Innovative Strategies

Heterogeneity Overwhelms Variability from Different Analytical Techniques

331 On-site 286 Lab

500 On-site 416 Lab

1,280 On-site 1,220 Lab

39,800 On-site 41,400 Lab

95% of data variability due to sample location over a 4 ft diameter

2 ft

164 On-site 136 Lab

24,400 On-site 27,700 Lab

10June2008

27,800 On-site 42,800 Lab

18

Triad Investigations: New Approaches and Innovative Strategies

Uncertainty Math Magnifies Weakest Links Effects in Data Quality Chain Uncertainties add according to (a2 + b2 = c2)
Analytical Uncertainty Total Uncertainty

Sampling Uncertainty Example: AU = 10 ppm, SU = 80 ppm: TU = 81 ppm

AU = 5 ppm, SU = 80 ppm: TU = 80 ppm AU = 10 ppm, SU = 40 ppm: TU = 41 ppm AU = 20 ppm, SU = 40 ppm: TU = 45 ppm
10June2008 Triad Investigations: New Approaches and Innovative Strategies 19

How Do We Reduce Data Uncertainty?

For analytical errors:
Modify current or switch to another analytical technique Improve QC on existing techniques

For sample prep and handling errors:

Improve sample preparation

For sampling errors:

Collect samples from more locations
10June2008 Triad Investigations: New Approaches and Innovative Strategies 20

We cant control the effects of uncertainty on our decisions if we dont know where it is coming from. Historically sampling programs have focused resources on the wrong sources of data uncertainty.

10June2008

Triad Investigations: New Approaches and Innovative Strategies

21

Module 3

You cant find the answer if you dont know the question!

To be [averaged] or not to be [averaged], that is the question

From William Shakespeare's Hamlet, Prince of Denmark, Act III, scene I, as translated by course instructors

10June2008

Triad Investigations: New Approaches and Innovative Strategies

23

Sometimes the Simplest Questions are the Most Complex

Does this site pose an unacceptable risk? Do groundwater concentrations exceed drinking water standards? Do soil concentrations exceed cleanup requirements?

10June2008

Triad Investigations: New Approaches and Innovative Strategies

24

Sample Support, Representativeness and Decision Unit Support are Intertwined

What sample support is most representative of the decision? #1 #2 #3

Surface layer of interest

Layer impacted by deposition

The decision driving sample collection: Can it be shown that atmospheric 10June2008 Triad Investigations: New Approaches and Innovative Strategies deposition caused contamination?

25

Advances in Sampling & Measurement Technologies Highlight Representativeness Issues

MIP = membraneinterface probe (w/ ECD detector)
GW data results HIGHLY dependent on sample support

10June2008

Triad Investigations: New adapted Approachesfrom and Innovative Strategies Graphic Columbia Technologies

26

The Same Holds True for Soils

0.080 0.080 0.070 0.070 0.060 0.060 Frequency Frequency 0.050 0.050 0.040 0.040 0.030 0.030 0.020 0.020 0.010 0.010 0.000 0.000 0 0 100 100 200 200 300 300
Homogenized Discrete Samples` XRF Readings Action Level Multi-Increment Samples

400

500 500

600 600

Concentration (ppm) Concentration (ppm)

10June2008

Triad Investigations: New Approaches and Innovative Strategies

27

The Decision Unit is Often Not WellDefined

Lead should not exceed 400 ppm in soils or TCE should not exceed 5 ppb in groundwater
Decisions are often ambiguous because cleanup criteria do not provide enough information to define the decision units.

10June2008

Triad Investigations: New Approaches and Innovative Strategies

28

Complete Cleanup Criteria Definitions

Cannot achieve data representativeness w/o a complete definition of cleanup criteria Incomplete criteria leads to confusionexample:
an in situ XRF Pb reading from a yard is 560 ppm, while a homogenized sample from same is 200 ppm, while the average for the yard is 50 ppm.

Different sample supports different concentration estimates that are all correct but lead to different conclusions

Must DEFINE population of interest to interpret data!!

10June2008 Triad Investigations: New Approaches and Innovative Strategies 29

For Soils, Three Cleanup Requirement Definitions are Most Common:

Never-to-Exceed Criteria: Lead concentrations cannot be > 400 ppm Hot-Spot Criteria: Lead concentrations cannot be > 400 ppm averaged over 100 m2 Averaged Criteria: The average concentration of lead over an exposure unit cannot be > 400 ppm

10June2008

Triad Investigations: New Approaches and Innovative Strategies

30

Technically Defensible Sampling Programs Require Complete Criteria

Technically defensible: making decisions with known level of confidence Impossible to design a defensible program for never-to-exceed criteria Hot spot criteria typically require the most intensive sampling to be technically defensible Exposure unit averages are the easiest

10June2008

Triad Investigations: New Approaches and Innovative Strategies

31

All Solid Samples are Composites!!

A data result = average for mass of soil digested/ extracted during analytical prep (the analytical sample support) Questions:
Is analytical sample support representative of original sample support as recd by lab? Is sample recd by lab representative of decision support as defined by project team? Is teams SAP representative of regulatory criterion?

Sample support critical, yet currently determined by convenience or whim of samplers & analysts. If so, data quality being left to chance!!
10June2008 Triad Investigations: New Approaches and Innovative Strategies 32

Defensible Statistical Sampling Program Design and Data Analysis Requires:

Clearly defined decision units and decisionmaking (e.g., action level) criteria Sample supports that are representative of the decision unit of interest Analytical method implementation consistent with required sample support

10June2008

Triad Investigations: New Approaches and Innovative Strategies

33

Module 4

Beware of Sampling Programs Built on Erroneous Statistical Assumptions

Statistics: The only science that enables

different experts using the same figures to draw different conclusions. Evan Esar: Esar's Comic Dictionary American Humorist (1899 - 1995)

10June2008

Triad Investigations: New Approaches and Innovative Strategies

35

Wed prefer to ignore statistics when they tell something we dont want to hear

10June2008

Triad Investigations: New Approaches and Innovative Strategies

36

Statistical Packages Can Give an Aura of Defensibility

But, if underlying assumptions are wrong, the conclusions are wrong!

10June2008 Triad Investigations: New Approaches and Innovative Strategies 37

Representativeness Assumed
SAP says: Representative samples will be collected. But provides no explanation of how or what the samples are supposed to represent. Non-representative data decision errors
Sample support mismatched to cleanup criteria (single grab vs. area average) Samples with different supports mixed together in databases & statistical analysis Use spatially clustered locations or biased samples when calculating average concentrations Mix different populations together
10June2008 Triad Investigations: New Approaches and Innovative Strategies 38

0.080 0.070 0.060 0.050 0.040 0.030

Action Level (AL)

Remember these Examples?

0.080 0.070

Frequency

0.020 0.010

0.060
0.000 Concentration (ppm)

Frequency

100

200

300

400

0.050 500 0.040 0.030 0.020 0.010 0.000 0

600

Sample support vs action level

0.080

100

200

300

400

500

0.070 600 0.060

Concentration (ppm)
Frequency

0.050 0.040 0.030 0.020 0.010 0.000

Lognormal seen with small sample supports & when data from different supports are mixed together 10June2008 Triad Investigations: New Approaches and Innovative Strategies
0 100 200 300

400

500

600

Concentration (ppm)

39

Bias Your Answer by Biasing Your Sampling Approach

Exposure Unit #1: Biased sampling & spatial clustering when goal is to calc the area average

EU#1

Dump

Exposure Unit #2: Mix 2 populations (cleaner area & dump) into the same sampling design & data set
10June2008

EU#2

Triad Investigations: New Approaches and Innovative Strategies

40

Pretend All Data Sets Are Normal

Normal distributions make statistics easy Can ignore complexities of spatial & non-random relationships Many common statistical tests (typical UCL calculation, Student t test, etc.) assume normality

10June2008

Triad Investigations: New Approaches and Innovative Strategies

41

But Contaminated Sites Are Rarely Normal

Distributions are usually heavily skewed to the right Often can be bimodal (i.e., two-humped) Usually reflect overlaying or mixed populations, e.g,: -- Contaminated over background, or -- Air deposition over discrete surface releases
10June2008 Triad Investigations: New Approaches and Innovative Strategies 42

Assuming Normality Can Under-estimate the 95% UCL on the Mean

400 ppm Pb requirement for exposure unit 4 lab lead results: 20, 24, 86, and 189 ppm Average of the 4 results: 80 ppm Too few samples to know whether normal or not.
If assume data distribution is the 95% UCL is Normal distribution Gamma distribution Lognormal distribution Non-parametric distribution
10June2008

Options for how ProUCL can calculate 95%UCL

172 ppm 434 ppm 246 33,835 ppm 144 472 ppm
43

Triad Investigations: New Approaches and Innovative Strategies

Assuming Normality Can Under-Estimate Sample Number Requirements

VSP example: How many samples recommended to demonstrate that mean concentration is less than action level?
Assuming normal distribution: 10 Assuming non-parametric distribution: 23

10June2008

Triad Investigations: New Approaches and Innovative Strategies

44

Consequences of Under-estimating Sample Numbers

Only know statistics were wrong when datas back from lab!
95%UCL is basis for many EPA decisions about risk & compliance. When calculate 95%UCL from the data, find that decisions cannot be made at desired (95%) confidence
95% UCL > action level, even if mean < action level
325
95%LCL

425
mean

500 525
AL 95%UCL

Need more samples if want to make confident decision about risk or compliance: redo project
10June2008 Triad Investigations: New Approaches and Innovative Strategies 45

Assume We Know Altho We Dont

What info is required to use statistics properly? VSP is a software pkg that calculates number of samples based on user inputs. For soil sampling projects, user must supply:
The conc variability that is present (recall that data variability is a function of sample volume) Underlying contaminant distribution (ditto) How statistically confident do you want to be?

10June2008

Triad Investigations: New Approaches and Innovative Strategies

46

0.080 0.070 0.060 0.050 0.040 0.030

Action Level (AL)

Variability & population distribution depends on sample support

0.080

Frequency

0.020

0.070
0.010

0.060
0.000 Concentration (ppm)

Frequency

100

200

300

400

0.050 500 0.040 0.030 0.020 0.010 0.000 0

600

0.080

100

200

300

400

500

0.070 600 0.060

Concentration (ppm)
Frequency

0.050 0.040 0.030 0.020 0.010 0.000 0 100 200 300 400 500 600

10June2008

Triad Investigations: New Approaches and Innovative Strategies

Concentration (ppm)

47

More that We Need to Know

What more is required to use statistics properly? User must also supply:
Width of gray region (requires prediction of the true mean for the area under investigation)
500
95%LCL

650
mean

700
AL

725
95%UCL

Recognize that these input values will be different for different contaminants on the same site
Different field concs, different ALs VSP may predict 10 samples for Zn, 500 for PAHs, 1000 for Hg
10June2008 Triad Investigations: New Approaches and Innovative Strategies 48

Characterize or verify cleanup? Statistical confidence desired How close to each other are the true mean & AL? How much variability is present in the soil concentrations

How choose the input values?

10June2008 Triad Investigations: New Approaches and Innovative Strategies 49

Choose wisely: It Makes a Big Difference in Sample Numbers!

Actual mean closer to AL --->

GR width: 350 ppm StDev 50 ppm 200 ppm 300 ppm

200 ppm

100 ppm

Increasing variability ----->

2 5 8

3 10 21

4 36 79
50

10June2008

Triad Investigations: New Approaches and Innovative Strategies

Fact is, if we knew everything we needed to know in order to design a statistical sampling program correctly, we wouldnt need to do the sampling!!

10June2008

Triad Investigations: New Approaches and Innovative Strategies

51

Dilemma Resolutions
How can good approximation inputs be chosen?
Data & experience from similar sites Historical data from your site Pilot study (efficient if part of dynamic field work)

How does a non-statistician ensure the quality of VSP applications?

When VSP used to justify sample numbers, require that submissions include explanations for how each input value was chosen Verify that the explanations are reasonable
10June2008 Triad Investigations: New Approaches and Innovative Strategies 52

There are three kinds of lies: lies, damned lies, and statistics
-- Attributed to Benjamin Disraeli (as popularized by Mark Twain)

Verifying Assumptions is a Cure for Statistical Misrepresentation

10June2008 Triad Investigations: New Approaches and Innovative Strategies 53

Module 5

The Cure for the Sampling Blues

uncertainty mgt

All truths are easy to understand once

they are discovered; the point is to discover them. Galileo

10June2008

Triad Investigations: New Approaches and Innovative Strategies

55

The Triad Framework

Systematic Project Planning

Uncertainty mgt

Dynamic Work Strategies

Real-time Measurement Technologies

10June2008

Triad Investigations: New Approaches and Innovative Strategies

56

Triad Data Collection Designs & Data Analysis Built On:

Planning systematically (CSM is central) Improving representativeness (& stats) Addressing the unknown (with dynamic work strategies)

10June2008

Triad Investigations: New Approaches and Innovative Strategies

57

Systematic Planning Addresses:

Planning Systematically
Defining sample representativeness Accurately delineating contaminant populations
Identifying populations requires using a CSM Helps design a sampling plan to cultivate data that conforms to a well-behaved statistical distribution

Discovering what we dont know

Systematic Project Planning
Triad Investigations: New Approaches and Innovative Strategies

Uncertainty mgt
58

10June2008

Planning Systematically

Systematic Planning & Data Collection Design

Planning must define decisions, decision units & sample support requirements Planning must identify sources of decision uncertainty & strategies for uncertainty management Planning must clearly define cleanup standards Conceptual Site Models (CSMs) play a foundational role All the above help define sampling populations
Triad Investigations: New Approaches and Innovative Strategies 59

10June2008

Planning Systematically

2 Fundamental Concepts for Sampling Design & Statistics: (1) Decision Unit
Decision Unit: Area, volume, or set of objects (e.g. -acre area, bin of soil, set of drums) All items treated as a single unit for decision-making Statistical goal: discover true mean for that single unit
Amount of variability w/in the unit creates uncertainty in estimating the true mean Therefore, statistics used to express amount of uncertainty around the estimate of the mean

Examples: exposure unit, survey unit, remediation unit

Triad Investigations: New Approaches and Innovative Strategies 60

10June2008

Valley of the Drums: These need to be characterized, transported, and disposed properly. What is the decision unit? How do you sample it?
10June2008 Triad Investigations: New Approaches and Innovative Strategies 61

Batch #1

Batch #2

Batch #3

Batch #4

40 drums were cleaned in batches of 20. You need to ensure the cleaning process worked. What is the decision unit and how would you sample it?

2nd Fundamental Concept: Population

Planning Systematically
Population: Set of objects or material volumes sharing a common characteristic; can be synonymous w/ decision unit.
2 populations (clean As one & dirty Pb one) overlap w/in a single decision unit (such as residential yard) A population is so large that more than 1 decision unit is needed to cover it. Example: Suspected clean population of 50 acres; but decision unit (exposure unit) is 1 acre.
Triad Investigations: New Approaches and Innovative Strategies 63

Examples where they are not synonymous:

10June2008

Dont Get Confused by Similar Terms!

Planning Systematically
Statistical Population Distribution: Number of times (frequency) that particular values occur in a data set drawn from the population (also called frequency distribution)
This data set is called a sample by statisticians & it contains >1 physical sample Example: How many times does the value 10 ppm occur? Spatial Population Distribution: A spatial pattern created by the locations of values (such as low, medium & high concentrations) across an area or within a volume Conversion of a spatial distribution to a statistical distribution results in loss of spatial & physical relationship information
Triad Investigations: New Approaches and Innovative Strategies 64

10June2008

Relationship between spatial population distribution & statistical population distribution

Different spatial distributions

Histogram
10 F re q u e n c y 5 0 0 5 10 15 20 25 More
65

but same statistical distribution

10June2008

Triad Investigations: New Approaches and Innovative Strategies

Bin

CSMs Should Articulate Decision Uncertainty Planning Systematically

CSM captures current understanding about site conditions
Identifies additional information needed for confident decision-making Data collection needs & design flow from the CSM

A well-articulated CSM serves as the point of stakeholder consensus. CSMs are livingas new data become available, incorporate into CSM.
CSM is mature when desired decision confidence is achieved
Triad Investigations: New Approaches and Innovative Strategies 66

10June2008

Statistics & the CSM

Planning Systematically
Spatial considerations are irrelevant to classical statistics
Inputs for calculating how many samples does not include area to be sampled If VSP inputs are the same, a 100-sq mi area will get same # samples as 1-sq ft area YOU must use the CSM to create decision units & select proper statistics for each decision unit

Statistics cannot be used properly w/o a CSM that defines the statistical populations!!
Triad Investigations: New Approaches and Innovative Strategies 67

10June2008

Improving Representativeness

Improving Representativeness & Statistical Performance

Sample support
Volume/dimensions of sample, for XRF in situ analysis, the sample support is the field of view Match it to decision needs & populations

Control within-sample heterogeneity

Appropriate sample preparation important (see EPA EPA/600/R-03/027 for additional detail) Uncertainty effects quantified by appropriate sub-sample replicate analyses

Control short-scale between-sample heterogeneity

Multi-increment sampling (physical averaging) Multiple readings (mathematical averaging)
Triad Investigations: New Approaches and Innovative Strategies 68

10June2008

Improving Representativeness

Guidances on Multi-Increment Sampling/Compositing May Differ Somewhat

Verification of PCB Spill Cleanup by Sampling and Analysis (EPA-560/5-85-026, August, 1985)
up to 10 adjacent samples allowed

Cleanup Standards for Groundwater and Soil, Interim Final Guidance (State of Maryland, 2001)
no more than 3 adjacent samples allowed

SW-846 Method 8330b (EPA Rev 2, October, 2006)

30 adjacent samples recommended

Draft Guidance on Multi-Increment Soil Sampling (State of Alaska, 2007)

30 50 samples recommended for compositing
Triad Investigations: New Approaches and Innovative Strategies 69

10June2008

Improving Representativeness

All Samples Are Composites at Some Scale

A soil subsample contains many particles that are extracted or digested & analyzed together A bulk heap of soil taken from a jar is a composite of many individuals particles Jar contents is a composite of many individual particles The source of the jar contents is a composite of many, many particles Compositing (from different locations at the between-sample level) is the same principle as stirring a jar to bring spatially separated particles into the same analytical sample at the within-sample scale
Triad Investigations: New Approaches and Innovative Strategies 70

10June2008

Improving Representativeness

Multi-Increment Sampling vs. Compositing

Multi-increment sampling: a strategy to costeffectively control the effects of heterogeneity multi-increment averaging Compositing: a strategy to reduce overall analytical costs when conditions are favorable while looking for contamination composite searching Same soil aggregation process, but done for very different reasons
Triad Investigations: New Approaches and Innovative Strategies 71

10June2008

Improving Representativeness

Multi-Increment Sampling vs. Compositing

Assumption: the cleanup criteria is averaged over decision unit
Multi-Increment Sampling
Decision Unit 1
Decision Unit 3 Decision Unit 4

Compositing
Decision Unit 1 Decision Unit 2

MI Sample

Decision Unit 5
sample

Decision Unit 6

Form one multi-increment sample for analysis

Form one composite sample for analysis

72

10June2008

Triad Investigations: New Approaches and Innovative Strategies

Improving Representativeness

Multi-Increment Sampling
Effective when cost of analysis is significantly greater than cost of sample acquisition/ handling How many increments?
Practical upper limit imposed by homogenization capacity, background concentration & magnitude of non-background concentration Enough to bring sampling error under control relative to other sources of error

10June2008

Triad Investigations: New Approaches and Innovative Strategies

73

How Many Increments per MI Sample?

Addressing the unknown
For statistically valid conclusions, the number of increments is matched to level of heterogeneity present within decision unit
At least 10 preliminary measurements across decision unit give good estimate of variability (SD) Use SD & mean of preliminary data in VSP to determine # increments needed for desired statistical confidence in the decision units mean estimate Dynamic strategies make this process highly efficient
Dynamic Work Strategies
74

Uncertainty mgt

10June2008

Triad Investigations: New Approaches and Innovative Strategies

How Many MI Samples Per DU?

Addressing the unknown
Only 1 MI sample per decision unit (DU) causes loss of spatial variability information
no means to QA (evaluate sufficiency) of increment #s difficult to implement statistical tests if DU average exceeds action level, cant identify where in the DU the problem is

Multiple MI samples per decision unit (e.g., 5)

provides the benefits of MI sampling (cost reduction, improved performance) while providing variability information to allow statistical tests (e.g., Student t test, Sign test, 95%UCL calculation, etc.) can help identify where contamination is in decision unit Dynamic work strategies highly efficient
Triad Investigations: New Approaches and Innovative Strategies 75

10June2008

Statistical & Decision Benefits of MI Sampling

0.080 0.070 0.060 0.050 0.040 0.030 0.020 0.010 0.000 0 100 200 300 400 Concentration (ppm)

MI sample

Frequency

0.080 0.070 0.060 Frequency 0.050 0.040 0.030 0.020 0.010 0.000 0 100 200 300 400 500 600 Concentration (ppm)
600

500

Large grab sample

Physical equivalent of averaging many individual sample results mathematicallyMI sampling creates larger sample supports & tends to normalize statistical data distributions
10June2008 Triad Investigations: New Approaches and Innovative Strategies 76

Improving Representativeness

Benefits & Limitations of MI Sampling

Significantly reduce analytical costs Significantly improve decision-making performance:
Reduce decision-making errors Much more likely that hot spots will be found and accounted for

Not as useful for subsurface & other sampling where sampling costs higher than analytical Requires special design & handling for volatile contaminants (Hg, VOCs, etc.) In situ & other cost-effective high density analyses (like XRF) potentially substitute or augment MIS
Triad Investigations: New Approaches and Innovative Strategies 77

10June2008

Addressing the Unknown

Addressing the Unknown through Dynamic Work Strategies

Optimize data collection design
Real-time testing of CSM & obtain statistical design parameters (SD, preliminary mean, sample support)

Adaptive analytics
Strategies to produce collaborative data sets with sufficient analytical & sampling QC checks

Adaptive sampling
Strategies for confident estimates of DUs mean Strategies for delineating contaminant populations

Adaptive compositing
Efficient strategies for searching for contamination
Triad Investigations: New Approaches and Innovative Strategies 78

10June2008

Adaptive Composite Searching

Addressing the Unknown
Goal: looking for contamination or demonstrating that large areas are compliant Assumptions:
Most of the area is clean/compliant Contamination is believed to be spotty Action level is significantly greater than background levels Sample acquisition/handling costs are less than analytical costs Appropriate methods exist for sample acquisition & aggregation
Triad Investigations: New Approaches and Innovative Strategies 79

10June2008

Composite Searching Designs

Addressing the Unknown
Must determine
appropriate number of samples to aggregated into composite develop decision criteria to indicate when analyses of contributing samples are necessary

Performance (cost/benefit) best when

contamination is spotty big difference between background & action level big difference between average concentration & AL Best case: no composite requires re-analysis Worst case: every composite requires re-analysis
Triad Investigations: New Approaches and Innovative Strategies 80

10June2008

Addressing the Unknown

Recipe for Adaptive Compositing

Determine appropriate number of samples to composite & decision criterion w/ this equation: Decision criteria = (action level - background)/(# of samples in composite) + background. Sample and split samples. Use one set of splits to composite and save other set. If:
composite result < decision criteria, done. composite result > decision criteria, analyze splits contributing to the pooled composite.
Triad Investigations: New Approaches and Innovative Strategies 81

10June2008

How Many Samples to Composite?

Addressing the Unknown
Normalized Expected Cost vs Composite Size
1.1 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3
s C t c p x E d e z i l a m r o N

Hit Prob = 0.2 Hit Prob = 0.1 Hit Prob = 0.05

How probable is it that contamination is present? The less likely it is that contamination is present, the larger the number of samples that can be composited

0.2 0.1 0.0 0 5 10

Hit Prob = 0.01

Hit Prob = 0.001
15 20

Number Contributing to Composite

Graph at left illustrates optimal sample numbers for different probabilities

82

10June2008

Triad Investigations: New Approaches and Innovative Strategies

Addressing the Unknown

Example Decision Criteria

Background: 10 ppm; Action Level: 100 ppm Determine decision criteria for a 2-sample, 3sample, 4-sample, 5-sample & 6-sample composite:
Decreasing Analytical Costs Increasing Chance of Failing

2-sample composite: 55 ppm 3-sample composite: 40 ppm 4-sample composite: 33 ppm 5-sample composite: 28 ppm 6-sample composite: 25 ppm

10June2008

Triad Investigations: New Approaches and Innovative Strategies

83

Addressing the Unknown

Performance (Cost/Benefit) Calculation

Compositing has a positive cost/benefit ratio as long as: Ff < 1 1/Nc where:
Nc = number contributing to composite Ff = fraction of composite samples failing (results above decision criteria)
Triad Investigations: New Approaches and Innovative Strategies 84

10June2008

Other Assorted Statistical Strategies

Useful classical statistics strategies
Stratified sampling designs Bernards sequential t-test sampling design Binomial Sequential Probability Ratio Test (SPRT) (sequential non-parametric sampling design) Adaptive cluster sampling Ranked set sampling

Geostatistics (free software, Google: SADA, geostatistics)

Probability maps [cleanup if Pr(non-compliance) > X%]

GeoBayesian (free BAASS software, see Bob Johnson)

10June2008 Triad Investigations: New Approaches and Innovative Strategies 85

USACE: Managing Uncertainty with an XRF & Geostatistics

Addressing the Unknown

Color coding for probabilities that 1-ft deep volumes > 250 ppm Pb (actual Pb conc not shown)

Decision plan: Any soil w/ Pr(Pb > 250 ppm) > 40% will be landfilled. 10June2008 Triad Investigations: New Approaches andbe Innovative Strategies 86 Soil with Pr(Pb > 250 ppm) < 40% will reused in new firing berm.

The Biggest Bang Comes from Combining

CSM knowledge, with Multi-increment sampling, with Collaborative data sets, with Adaptive analytics, with Adaptive QC & data uncertainty reduction, with Adaptive compositing, with Adaptive sample location selection.
10June2008 Triad Investigations: New Approaches and Innovative Strategies 87

Module 6

Project Case Study:

Adaptive X-ray Fluorescence (XRF) Sampling & Analysis Design to Achieve Decision Confidence for Residential Soil Lead Concentrations

10June2008

Triad Investigations Conference

88

This slimmed down case study illustrates how to determine & control data error in real-time to generate definitive data This project used a handheld X-ray fluorescence (XRF) instrument to measure Pb in minutes at the site of sample collection
Plastic bag of soil

10June2008

Triad Investigations: New Approaches and Innovative Strategies

89

This Projects Decisions

Is the Pb conc for each residential property below the 500 ppm risk-based AL? What is the greater source of data variability when reporting Pb conc results & how can it be reduced as needed?

10June2008

Triad Investigations: New Approaches and Innovative Strategies

90

Data Collection Design

An entire yard is an exposure unit Grassy yard initially stratified into 3 sections
Section F: Front of house Section S: Side of house Section B: Behind house

Divide each section into 5 equal area subsections The subsections will be sampled by taking 1 grab soil sample (~300 g) per subsection & placing it into a plastic bag for XRF analysis
10June2008 Triad Investigations: New Approaches and Innovative Strategies 91

Illustrative Sampling Design & Results

Property: 702 Main Street
Front Yard: 5 Samples Side Yard: 5 Samples
Area fraction = 0.25

Action Level = 500 ppm

1 bagged grab sample

Back Yard: 5 Samples

House Footprint
Area fx = 0.15

Area fx = 0.6

Total yard average determined statistically (& area-weighted) as 10June2008 Triad Investigations: New Approaches and Innovative Strategies 92 410 +/- 25 (385 435 ppm Pb)

Front yard average (at 95% statistical confidence) = 700 +/- 150 (550 850 ppm Pb) Side yard average (at 95% statistical confidence) = 500 +/- 100 (400 600 ppm Pb) Back yard average (at 95% statistical confidence) = 300 +/- 50 (250 350 ppm Pb)

XRF Bag Analysis

Take 4 30-sec XRF readings on each bag (2 on front & 2 on back of bag) Real-time Pb results (2 min to handle each bag) entered into pre-programmed spreadsheet Spreadsheet immediately calculates average & statistical uncertainty for Pb conc. for
each bag each yard section, and the entire yard
10June2008

Follow Decision Tree #1 Triad Investigations: New Approaches and Innovative Strategies

93

Decision Tree #1
Evaluate statistical results for the yard & compare to the 500 ppm AL
Is there statistical confidence that mean is below AL? Is there statistical confidence that mean is above AL?

yes

If neither condition is true

yes

Decide Pb conc for the yard is below AL

200 +/- 50 (150 250)
Confident that no action needed
10June2008

Decision too uncertain: more information needed

300 +/- 100 (150 520)

Decide Pb conc for the yard is above AL

700 +/- 150 (550 850) Confident that action is required
94

Triad Investigations: New Approaches and Innovative Strategies

Go to Decision Tree #2

Information from spreadsheet to feed into Decision Tree #2

to identify the most important source of data variability (aka, statistical error)
Average within-bag error (std dev, SD) for each of the 5 bags from a yard section Between-bag error SD for all bags from a yard section. Compare the average within-bag SD to the between-bag SD See example data set 10June2008 Triad Investigations: New Approaches and Innovative Strategies
95

Error Measurement in Data Results for a Backyard Section

Within-Bag Data Variability

Sect. Bag #1 Shot #1 700 #2 670 #3 740 #4 650 Bag Mean 690 W/in-Bag SD

Bag #2 #1 550 #2 534 654 #3 #4 590

39

582

Bag #3 #1 534 #2 440 #3 430 #4 420

54

456

Bag #4 #1 769 #2 710 #3 960 #4 800

53

810

Bag #5 #1 450 #2 400 #3 500 #4 550

475

107

65

Mean of within-bag SDs = (39+54+53+107+65)/5 = 63 Between-Bag Data Variability

690
10June2008

582

456

810

475
96

Between-bag error (SD) for and the 5 bag Strategies means = 150 Triad Investigations: New Approaches Innovative

Decision Tree #2
Determine the greater source of data variability (decision uncertainty) Is within-bag variability GREATER than between-bag variability? yes Go to Decision Tree # 3 no, they are ~equal Go to Decision Tree #5
10June2008

no Is within-bag variability LESS than between-bag variability? yes Go to Decision Tree #4

97

Triad Investigations: New Approaches and Innovative Strategies

Decision Tree #3
Major source of data error: heterogeneity within sample bag (subsampling error)
To control this source of variability: Re-shoot each bag another 4 times (total of 8 shots/bag. Add results to spreadsheet & recalculate stats for whole yard. Examine results.
Is within-bag variability sufficiently reduced? no Take addl corrective action
10June2008

yes Make decision at 500 ppm AL w/ desired statistical confidence

98

Triad Investigations: New Approaches and Innovative Strategies

Decision Tree #2
Determine the greater source of data variability (decision uncertainty) Is within-bag variability significantly GREATER than between-bag variability? yes Go to Decision Tree # 3 no, they are ~equal Go to Decision Tree #5 no Is within-bag variability significantly LESS than between-bag variability? yes Go to Decision Tree #4
99

10June2008

Triad Investigations: New Approaches and Innovative Strategies

Major source of data error is from concentration variations across the yard section area.
To control this source of variability: Collect another 5 bag samples from section area. Analyze 4 times/bag. Add results to spreadsheet & recalculate statistics for whole yard.
Is between-bag variability sufficiently reduced? no Take addl corrective action yes Make decision at 500 ppm AL w/ desired statistical confidence
100

Decision Tree #4

10June2008

Triad Investigations: New Approaches and Innovative Strategies

Decision Tree #2
Determine the greater source of data variability (decision uncertainty) Is within-bag variability significantly GREATER than between-bag variability? yes Go to Decision Tree # 3 no, they are ~equal Go to Decision Tree #5 no Is within-bag variability significantly LESS than between-bag variability? yes Go to Decision Tree #4
101

10June2008

Triad Investigations: New Approaches and Innovative Strategies

Decision Tree #5
Concentration variability across yard section & within samples about the same.
Analyze original bags an addl 4 times each. Also collect another 5 bag samples from the section & analyze 8 times each. Add all results to spreadsheet & recalculate statistics for whole yard.
Is statistical decision uncertainty now sufficiently resolved? no Take addl corrective action
10June2008

To control both sources simultaneously:

yes

Make decision at 500 ppm AL w/ desired statistical confidence Triad Investigations: New Approaches and Innovative Strategies 102

Benefits of the Dynamic XRF Strategy

Data gathered in real-time Data evaluated against decision goals in realtime Data uncertainty identified & measured in realtime (definition of definitive data) Decision tree guides actions to resolve data uncertainty in real-time Final decisions can be made in real-time Property owners informed of decisions in realtime
10June2008 Triad Investigations: New Approaches and Innovative Strategies 103