Designing Reliability Tests with SimuMatic

One of the applications of SimuMatic is simulating the outcome from a particular test design that is intended to demonstrate a target reliability. You can specify various factors of the design, such as the test duration (for a time-terminated test), number of failures (for a failure-terminated test) and sample size. By running the simulations you can assess whether the planned test design can achieve the reliability target. Depending on the results, you can modify the design by adjusting these factors and repeating the simulation process—in effect, simulating a modified test design—until you arrive at a modified design that is capable of demonstrating the target reliability within the available time and sample size constraints.

Tip: SimuMatic is useful for designing tests where there will be enough failure times observed during the test to analyze the data and fit a life distribution (e.g., two or more observed failures are required for most 2-parameter distributions). To design a test that demonstrates the target reliability with minimal “allowable failures” (e.g., a zero-failure test), use the Reliability Demonstration Test Design tool.

Recommended Settings

Below are some recommended settings for using SimuMatic to design a reliability life test. The options described here can also be used for other types of what-if analyses designed to explore the life estimates that can be obtained from different types of data sets.

On the Settings tab:

If you are solving for the sample size, start with a large number of data points. You can later repeat the simulation with smaller values until you arrive at an acceptable test plan. (In the Stress-Dependent Monte Carlo utility, use the data sheet on the right side of the window to specify the number of data points for each stress level.)

Make sure an appropriate number of data sets (e.g., 1,000) has been specified in the Number of Data Sets field. Lower numbers may lead to less accurate results.

On the Censoring tab:

If you are designing a test that will end only after all units fail, select No censoring. All simulated time values will be considered failure times.
If you are designing a time-terminated test (i.e., a test that will end at a specified time), select Right censoring after a specific time and enter an estimated test duration in the Time field. All simulated time values that exceed this limit will be considered suspensions at the specified time (i.e., units that had not failed by the end of the test). If you are solving for the required test duration, enter a large estimate, and then you can repeat the simulation with smaller values until you arrive at an acceptable test plan.

Note: A short censoring time may create many data sets with too few failures to estimate an underlying life distribution. These data sets will display "N/A" for the parameters on the SimuMatic folio's data sheet. This can be due to an insufficient sample size or a test termination time that is too short. As a rule of thumb (and for 2-parameter distributions) the combined sample size and test duration should be sufficient to observe three or more failures. In other words, if you have a sample size of 10 then the test duration should be greater than the product's B30 life (i.e., the time at which unreliability = 30%).

If you are designing a failure-terminated test (i.e., a test that will end after a specified number of failures occur), select Right censoring after a specific number of failures and enter a value in the Number of Failures field. The tool will simulate a test that ends after this number of failures, with the surviving units marked as suspensions. If you are solving for the number of failures needed, enter a large estimate, and then you can repeat the simulation with smaller values until you arrive at an acceptable test plan.

Tip: In the Life Data SimuMatic utility, you can also choose the Random censoring option if you wish to explore the results that can be obtained from data sets that contain certain types of uncertainty. In this type of analysis, you can use the T1, T2 and DELTA values to estimate whether you'll be able to use the data set to demonstrate a required reliability target and evaluate how much the demonstrated life varies from the intrinsic reliability.

On the Test Design tab:

Select to Calculate test plan results and enter the Target Reliability you wish to demonstrate along with the Lower 1-Sided Confidence Level at which you want to demonstrate it.

Evaluating the Results

After generating the simulated data set, click the Summary (...) icon inside the Additional Results area of the control panel to view a report of the results. The Test Planning Results area of the report displays your test design inputs (reliability requirement, confidence level and sample size) along with the following results:

Expected Life (T1) is the lower one-sided confidence bound on the reliable life (i.e., the time for a given reliability) calculated based on the specified distribution/parameters. This value represents the intrinsic reliability of a product with the specified distribution/parameters. So if this value does not meet your reliability goal, then it is not possible to demonstrate the desired reliability for a product with your assumed life distribution, regardless of the test time or sample size that is employed.
Demonstrated Life at Given Confidence (T2) is the lower one-sided confidence bound on the reliable life calculated based on the simulated data sets. Specifically, SimuMatic calculates the time given reliability for each data set and then sorts the results to obtain the value associated with the specified confidence level percentile. You can then compare this value to the time for which you want to demonstrate the reliability (e.g., you might want to demonstrate that the product will have a reliability of 95% at 1,000 hours, at the 90% confidence level).
- If the demonstrated value is greater than your reliability goal, then SimuMatic predicts that the specified test plan would demonstrate the target. For example, if you wish to demonstrate that the product has a reliability of 95% at 1,000 hours and T2 = 1,200, then SimuMatic predicts that the test plan would demonstrate the target.
- If the demonstrated value is less than your reliability goal, then SimuMatic does not predict that your test plan would demonstrate the target. For example, if you wish to demonstrate that the product has a reliability of 95% at 1,000 hours and T2 = 800, then SimuMatic does not predict that the test plan would demonstrate the target.

Ratio of Expected Life/Demonstrated Life (DELTA) is a measure of the difference between the intrinsic reliability of your product (T1) and the life demonstrated by analyzing the simulated data (T2). This value is calculated by dividing the expected life by the demonstrated life. For example, if your test design would demonstrate a reliability of 90% at 50 hours (demonstrated life), and your product has an intrinsic reliability of 90% at 100 hours (expected life), then DELTA = 2.
Expected Test Duration (T3) is the average of the last failure or suspension time from each of the simulated data sets. In most contexts, this value provides an indication of how long you would need to run a test in order to demonstrate the reliable life calculated in T2. In the case of a time-terminated test, you can compare this value against the specified censoring time. If the censoring time (i.e., planned test time) is greater than T3, you may choose to repeat the simulation with smaller test times to see if it’s feasible to demonstrate the target with a less costly test plan. In addition, note that:
- T3 is an average of all the test runs.
- T3 is calculated based on the simulated data sets, and therefore it can never be greater than the specified censoring (test) time. If you set a test time that is too small, T3 may be identical to the censoring time but the test plan is still not adequate unless the demonstrated life (T2) also meets the requirement.

If the results indicate that you will not be able to demonstrate the target reliability, or if they indicate that the current test plan uses more samples or more test time than necessary to demonstrate the target, you can keep repeating the simulation with gradual adjustments until you arrive at an optimum plan.

Note that because the results are obtained through Monte Carlo simulation, you may arrive at slightly different answers each time you run the analysis.