An Application of BlockSim's Log of Simulations

[Editor's Note: This article has been updated since its original publication to reflect a more recent version of the software interface.]

The need for understanding the risk of certain assumptions or predictions has resulted in the use of variability metrics in multiple disciplines, from the quality control field to financial applications. The most widely used metric to express variability is standard deviation (or variance, the square of the standard deviation). Given the expected value of a metric and a standard deviation, one can obtain the metric of interest at any confidence level (for more information on confidence bounds see [1]), sometimes expressed in terms of the risk level α. However, this assumes that the metric in question follows a normal distribution (the well-known, symmetric, bell-shaped curve [2]). In reality, many measures do not exhibit such behavior. This article will present a simple example of an asymmetric estimate using system availability and BlockSim. The simulation log feature of BlockSimwill then be used for further analysis of the mean availability via a non-parametric approach followed by a parametric approach in order to obtain bounds.

Example

Assume a system with two main failure modes with the properties presented in Table 1. We wish to estimate the mean availability for a mission time of 8,760 hours. After running 1,000 simulations with the settings shown in Figure 1, the simulation results at the system level are shown in Figure 2.

Table 1: Failure Mode Properties

 

Figure 1: Simulation settings

Figure 2: Simulation results, System Overview

The simulation results in BlockSim show a variety of results both at the system and block level (see [3] for more information on simulation results). In this example, the metric of interest is the mean availability, defined as follows:

For this example, the mean availability is equal to 0.529. In addition, the standard deviation of the mean availability is reported as 0.351.

Calculating Bounds Based on Assumption of a Normal Distribution with a Logit Transformation

A common assumption at this point is that the metric in question follows a normal distribution. The lower 1-sided, 90% confidence bound can then be obtained:

However, because of the large standard deviation, it is possible to obtain a lower bound smaller than 0 and an upper bound larger than 1 (this would also be the case if were close to 0 or 1). A better approximation here can be obtained by applying the logit transformation (logit(p) = log[p / (1 - p)]). This will ensure that the endpoints always fall between 0 and 1. We can then obtain the lower 1-sided, 90% confidence bound as follows [4]:

where:

In this example, A_L = 0.156 or 15.6%.

However, because of the nature of the failure modes and their properties, we expect that the mean availability will be highly asymmetrical and will display a mixed behavior. After all, if failure mode 2 occurs, the system is down for the remainder of the mission. We therefore choose to analyze the mean availability in more detail. This will require that we obtain the availability for each of the individual simulations so that additional statistical analyses can be performed.

Obtaining Individual Simulation Results in BlockSim

Obtaining the mean availability for each of 1,000 simulations could be a very time-consuming task. BlockSim allows saving specified results for individual simulations in an Excel® workbook or in a tab-delimited text (*.txt) file. It is possible to save results for specified blocks, crews and spare part pools in addition to individual results for the entire system by choosing Log of Simulations from the Advanced Options drop-down and using the settings shown in Figure 3.

Figure 3: Settings to save a log of simulations

We can also specify the results of interest by clicking Select Results. For this example, we select the mean availability, as shown in Figure 4.

Figure 4: Select Results window

The last step is to specify the file name and location where results will be saved, as shown previously in Figure 3. We are now ready to click Simulate. Note that during standard simulations BlockSim stores the most recent simulation results with the diagram as a summary of results averaged across all simulations. The log, on the other hand, saves individual simulation results in a file external to the software and will not affect any results that may have been saved with the diagram. Once the simulations have been completed, the user has the option to attach the file to the current project. Figure 5 shows the file, opened in Notepad.

Figure 5: Log of simulations for the mean availability

It will now be easy to perform further statistical analysis on these results using another application, such as Weibull++.

Obtaining Bounds Using a Non-Parametric Approach

We can use Weibull++ to analyze the data shown above and construct the histogram shown in Figure 6. In this plot, availability values are grouped in intervals of 0.1. The x-value shown is the number of times a mean availability within a range was observed divided by the size of the interval. For example, if we would like to obtain the percentage of times a mean availability from 0 to 0.1 was observed, we would take the size of the corresponding column (2.570) and multiply it by the interval size (0.1) to obtain 0.2570 or 25.7%.

Figure 6: Histogram of the mean availability

Figure 6 shows two distinct peaks, which results from the fact that the system displays two markedly different behaviors (i.e., failure modes). Attempting to use a mean and a standard deviation in this example (i.e., assuming a normal distribution), or even applying a transformation that will capture some skewness in the population, would largely misrepresent the actual behavior of the system.

The plot presented in Figure 6 is a non-parametric approach; that is, no model is assumed for the mean availability. In order to find the lower 1-sided, 90% confidence bound for such an approach, we can sort the mean availabilities in increasing order and pick the 100th value (out of the 1,000) as our estimate. For this example, this value is 0.00313 or 0.3%.

Non-parametric methods do not assume a model and hence are "safer" than parametric approaches, in which we run risks such as assuming the wrong model. However, non-parametric methods are limited in the results that they can provide. For example, no interpolations or extrapolations are possible.

Calculating Bounds Using a Parametric Approach

Should we decide to take a parametric approach instead, we might use a model that can represent the mixed behaviors of this particular system. Using Weibull++, we estimate the parameters for a 3-population, mixed Weibull distribution. The probability plot is shown in Figure 7. (Note that the choice of the number of populations in this example is based solely on the fit, which is usually not the case when using the mixed Weibull distribution.)

Figure 7: Probability plot for mean availability

We can then obtain 0.3% as the lower 1-sided, 90% confidence bound of the mean availability by either reading the mean availability value for a probability of 10% from the plot in Figure 7 or by using the Weibull++ Quick Calculation Pad (QCP).

In using any of these three methods, be aware that they account for both the simulation error and the intrinsic variability of the system. However they do not account for the variability due to the data used to calculate the parameters. In other words, we are treating the distribution parameters as constants even though there is uncertainty around those parameters due to the method or source used to calculate them.

Conclusion

In this article, we presented a simple example of an asymmetric estimate using the mean availability of a system with two failure modes. While the characteristics of the failure modes in this example were exaggerated for illustration purposes, it is not uncommon to see varying degrees of this behavior in actual analysis. The log of simulations feature of BlockSim was then used for further analysis of the mean availability via a non-parametric approach followed by a parametric approach in order to obtain bounds.

References

[1] https://help.reliasoft.com/reference/life_data_analysis/lda/confidence_bounds.html
[2] https://help.reliasoft.com/reference/life_data_analysis/lda/the_normal_distribution.html
[3] https://help.reliasoft.com/reference/system_analysis/sa/repairable_systems_analysis_through_simulation.html#GeneralSimulationResults
[4] W. Q. Meeker and L. A. Escobar, Statistical Methods for Reliability Data, New York: John Wiley & Sons, 1998.