Determining Significant Effects in 2k Designs with a Single Replicate

[Editor's Note: This article has been updated since its original publication to reflect a more recent version of the software interface.]

Design of Experiments (DOE) is a systematic approach to experimentation that is used to investigate how various inputs (or factors) affect an output (or response) of interest. One of the useful results of the statistical analysis performed in a DOE is whether an investigated factor's effect on the response is significant. In order to do that, replicated runs are required. That means that all the combinations of factor levels (treatments) should be performed more than once. However, in many experiments running replicates of an experiment is not feasible due to a lack of time or resources.

In this article we will present three methodologies that can be applied in the case of 2k designs (where each factor has only two levels) to allow us to test for the significance of an effect without replicated runs. Those methodologies are:

• Normal probability plot of effects
• Pooling higher order interactions
• Using center point replicates

Introduction

In order to test for the significance of a factor or an effect, we use Analysis of Variance (ANOVA) and hypothesis tests. ANOVA is the procedure of splitting the variance of the response measurements into the variance caused by each factor and the variance caused by noise. The variance caused by factor A, for example, is represented by the factor sum of squares (SSA) and the variance caused by noise is represented by the error sum of squares (SSE). Once we have these two variances, we can calculate a test statistic that will be used in performing the hypothesis tests. The test statistic is based on the F distribution and is given by:

In the case of experiments with a single replicate, we do not have enough observations to calculate the error sum of squares. This is because we are essentially fitting a line to two observations, as shown in Figure 1.

Figure 1: Fitting a Line to Observations With and Without Replicates

Since we cannot calculate SSE, we also cannot calculate the test statistic; as a result, we cannot perform the hypothesis test in order to determine the significance of the factor.

Method 1: Normal Probability Plot of Effects

One methodology for identifying significant effects is constructing the normal probability plot of effects. The idea behind this methodology is that all non-significant effects will fall along the straight line representative of the normal distribution, N(0,σ2/2k∙m), where m is the number of replicates. However, in the case of unreplicated 2k designs, σ2 is unknown since SSE cannot be computed. In this case, we use Lenth’s method to estimate the variance of the effects. Under this methodology the estimated variance or the pseudo standard error (PSE) is calculated as 1.5 times the median value of all effects that are less than 2.5∙s0:

where s0 is 1.5 times the median of all effects.

Now given this estimate of variance, the normal probability line is the line N(0,PSE) and all the effects that show large deviations from this line will be significant.

Example 1

A team of engineers in a semiconductor fab that manufactures wafers is setting up a DOE to determine which factors affect the thickness of the wafers. Four factors are suspected to have a significant effect on the thickness, therefore a 24 design is chosen. Table 1 shows the four factors and the chosen levels for each factor.*

Because of time constraints, only one replicate of the experiment can be run. Figure 2 shows the experiment design and the measured response values as entered in DOE++.

Figure 2: Experiment Design

Since the experiment contains only one replicate, the significance of the effects to the response cannot be determined. Figure 3 shows the results of the analysis with all the individual terms selected to be displayed, where it can be seen that no hypothesis tests were performed (i.e., the F Ratio and P Value columns are not populated).

Figure 3: ANOVA Table of the Experiment Data

However, using Lenth’s method, a normal probability plot can be constructed and the effects that do not fall on the normal probability line can be deemed significant. Figure 4 shows the normal probability plot of the effects.

Figure 4: Normal Probability Plot of Effects

As the plot shows, the three main effects Temperature, Time and Pressure, as well as the interaction effects of Temperature/Pressure and Temperature/Time are determined to be significant.

Method 2: Pooling Higher Order Interactions

In general, higher order interactions can be assumed to be insignificant. Therefore, interactions between three or more factors can be dropped from the model. As a result, the sum of squares of those higher order interactions can be used as the error sum of squares.

Example 2

Consider the same experiment that was described in the previous example. The team of engineers has made the assumption that only effects up to two-way interactions are important and that higher order interactions can be dropped from the model. Figure 5 shows how the higher order interactions can be excluded from the model in DOE++.

Figure 5: Selecting Effects to Be Included in the Model

Once the higher order interactions have been removed, their sum of squares can be used as the error sum of squares and the hypothesis tests can be performed. Figure 6 shows the results of the analysis.

Figure 6: ANOVA Table After Removing Higher Order Interactions

Removing the higher order interactions from the model introduces a Lack of Fit error, since the model is now not a complete model. The sum of squares of the lack of fit error is now used to perform the hypothesis tests and calculate the p values that determine whether an effect is significant or not. The ANOVA table indicates that the Temperature, Time and Pressure, as well as the interactions between Temperature/Time and Temperature/Pressure are the significant effects (shown in red).

Method 3: Using Center Point Replicates

Another method that can be used to estimate the error sum of squares is to perform a few runs at the center point. The center point is a treatment where all factors are set exactly midway between their two levels. Obviously, in order to have runs at the center point, all factors need to be quantitative since there is no midpoint for a qualitative factor. The replicates at the center point can then provide an estimate of the pure error used to perform the hypothesis tests. The sum of squares of pure error is calculated as:

where:

• is the measured response at each center point.
• is the average of all responses at the center point.

Another benefit of using center point replicates is the ability to test for the presence of curvature. If curvature is found to be present, it would indicate that a higher order model might be necessary to accurately represent the response.

Example 3

Consider again the same scenario described in Example 1. The team of engineers has now decided to perform five runs at the center point. The design of the experiment and the measured response values are shown in Figure 7.

Figure 7: Experiment Design with Center Points

The center points are indicated with a zero in the Point Type column and each factor is set at the midpoint of its two levels. The results of the analysis are shown in Figure 8.

Figure 8: ANOVA Table for Experiment with Center Points

The measurements at the center point have introduced a pure error that is used in the calculation of the p value. In addition, the ANOVA table indicates that the curvature is not significant, so the order of the model that is being used is appropriate. Again, the significant effects are found to be Temperature, Time and Pressure, as well as the interactions between Temperature/Time and Temperature/Pressure.

* The figures in this example are for demonstration purposes only and are not intended to be realistic.

References

[1] ReliaSoft Corporation, Experiment Design and Analysis Reference, Tucson, AZ: ReliaSoft Publishing, 2008.