Designation: E3023 − 15Standard Practice forProbability of Detection Analysis for â Versus a Data1This standard is issued under the fixed designation E3023; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon (´) indicates an editorial change since the last revision or reapproval.1. Scope1.1 This practice defines the procedure for performing astatistical analysis on Nondestructive Testing (NDT) â versus adata to determine the demonstrated probability of detection(POD) for a specific set of examination parameters. Topicscovered include the standard â versus a regressionmethodology, POD curve formulation, validation techniques,and correct interpretation of results.1.2 The values stated in inch-pound units are to be regardedas standard. The values given in parentheses are mathematicalconversions to SI units that are provided for information onlyand are not considered standard.1.3 This standard does not purport to address all of thesafety concerns, if any, associated with its use. It is theresponsibility of the user of this standard to establish appro-priate safety and health practices and determine the applica-bility of regulatory limitations prior to use.2. Referenced Documents2.1 ASTM Standards:2E178 Practice for Dealing With Outlying ObservationsE456 Terminology Relating to Quality and StatisticsE1316 Terminology for Nondestructive ExaminationsE1325 Terminology Relating to Design of ExperimentsE2586 Practice for Calculating and Using Basic StatisticsE2782 Guide for Measurement Systems Analysis (MSA)E2862 Practice for Probability of Detection Analysis forHit/Miss Data2.2 Department of Defense Document:3MIL-HDBK-1823A Nondestructive Evaluation System Re-liability Assessment3. Terminology3.1 Definitions of Terms Specific to This Standard:3.1.1 analyst, n—the person responsible for performing aPOD analysis on â versus a data resulting from a PODexamination.3.1.2 decision threshold, âdec,n—the value of â abovewhich the signal is interpreted as a find and below which thesignal is interpreted as a miss.3.1.2.1 Discussion—A decision threshold is required tocreate a POD curve. The decision threshold is always greaterthan or equal to the noise threshold and is the value of â thatcorresponds with the flaw size that can be detected with 50%POD.3.1.3 demonstrated probability of detection, n—the calcu-lated POD value resulting from the statistical analysis of the âversus a data.3.1.4 false call, n—– the perceived detection of a disconti-nuity that is identified as a find during a POD examinationwhen no discontinuity actually exists at the inspection site.3.1.5 noise, n—signal response containing no useful targetcharacterization information.3.1.6 noise threshold, ânoise,n—the value of â below whichthe signal is indistinguishable from noise.3.1.6.1 Discussion—The noise threshold is always less thanor equal to the decision threshold. The noise threshold is usedto determine left censored data.3.1.7 probability of detection, n—the fraction of nominaldiscontinuity sizes expected to be found given their existence.3.1.8 saturation threshold, âsat,n—the value of â associatedwith the maximum output of the system or the largest value ofâ that the system can record.3.1.8.1 Discussion—The saturation threshold is used todetermine right censored data.3.2 Symbols:3.2.1 a—discontinuity size.3.2.2 â—the measured signal response for a given disconti-nuity size, a.3.2.2.1 Discussion—The measured signal response is as-sumed to be continuous in nature. Units depend on the NDT1This test method is under the jurisdiction of ASTM Committee E07 onNondestructive Testing and is the direct responsibility of Subcommittee E07.10 onSpecialized NDT Methods.Current edition approved June 15, 2015. Published August 2015. DOI: 10.1520/E3023–15.2For referenced ASTM standards, visit the ASTM website, www.astm.org, orcontact ASTM Customer Service at

[email protected] For Annual Book of ASTMStandards volume information, refer to the standard’s Document Summary page onthe ASTM website.3Available from Standardization Documents Order Desk, DODSSP, Bldg. 4,Section D, 700 Robbins Ave., Philadelphia, PA 19111-5098, http://dodssp.daps.dla.mil.Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States1inspection system and can be, for example, scale divisions,number of contiguous illuminated pixels, or millivolts.3.2.3 ap—the discontinuity size that can be detected withprobability p.3.2.3.1 Discussion—Each discontinuity size has an indepen-dent probability of being detected and corresponding probabil-ity of being missed. For example, being able to detect a specificdiscontinuity size with probability p does not guarantee that alarger size discontinuity will be found.3.2.4 ap/c—the discontinuity size that can be detected withprobability p with a statistical confidence level of c.3.2.4.1 Discussion—According to the formula in MIL-HDBK-1823A, ap/cis a one-sided upper confidence bound onap. ap/crepresents how large the true apcould be given thestatistical uncertainty associated with limited sample data.Hence ap/c ap. Note that POD is equal to p for both ap/candap. apis based solely on the observed relationship between theâ and a data and represents a snapshot in time, whereas ap/caccounts for the uncertainty associated with limited sampledata.4. Summary of Practice4.1 This practice describes, step-by-step, the process foranalyzing nondestructive testing â versus a data resulting froma POD examination, including minimum requirements forvalidating the resulting POD curve.4.2 This practice also includes definitions and discussionsfor results of interest (e.g., a90/95) to provide for correctinterpretation of results.4.3 Definitions of statistical terminology used in the body ofthis practice can be found in Annex A1.5. Significance and Use5.1 The POD analysis method described herein is based onwell-known and well-established statistical methods. It shall beused to quantify the demonstrated POD for a specific set ofexamination parameters and known range of discontinuitysizes under the following conditions.5.1.1 The initial response from a nondestructive evaluationinspection system is measurable and can be classified as acontinuous variable.5.1.2 The relationship between discontinuity size (a) andmeasured signal response (â) exists and is best described by alinear regression model with an error structure that is normallydistributed with mean zero and constant variance, σ2. (Notethat “linear” refers to linear with respect to the model coeffi-cients. For example, a quadratic model yˆ 5β01β1·x1β2·x2is alinear model.)5.2 This practice does not limit the use of other statisticalmodels if justified as more appropriate for the â versus a data.5.3 This practice is not appropriate for data resulting from aPOD examination on nondestructive evaluation systems thatgenerate an initial response that is binary in nature (forexample, hit/miss). Practice E2862 is appropriate for systemsthat generate a hit/miss-type response (for example, fluorescentpenetrant).5.4 Prior to performing the analysis it is assumed that thediscontinuity of interest is clearly defined; the number anddistribution of induced discontinuity sizes in the POD speci-men set is known and well documented; the POD examinationadministration procedure (including data collection method) iswell designed, well defined, under control, and unbiased; theinitial inspection system response is measurable and continu-ous in nature; the inspection system is calibrated; and themeasurement error has been evaluated and deemed acceptable.The analysis results are only valid if the â versus a data areaccurate and precise and the linear model adequately representsthe â versus a data.5.5 The POD analysis method described herein is consistentwith the analysis method for â versus a data described inMIL-HDBK-1823A and is included in several widely utilizedPOD software packages to perform a POD analysis on â versusa data. It is also found in statistical software packages that havelinear regression capability. This practice requires that theanalyst has access to either POD software or other softwarewith linear regression capability.6. Procedure6.1 The POD analysis objective shall be clearly defined bythe responsible engineer or by the customer.6.2 The analyst shall obtain the â versus a data resultingfrom the POD examination, which shall include at a minimumthe documented known induced discontinuity sizes, the asso-ciated measured signal response, and any false calls.6.3 The analyst shall also obtain specific information aboutthe POD examination, which shall include at a minimum thespecimen standard geometry (e.g., flat panels), specimen stan-dard material (e.g., Nickel), examination date, number ofinspectors, type of inspection method (e.g., Eddy CurrentInspection), pertinent information about the instrument andinstructions for use (e.g., settings, probe type, scan path), andpertinent comments from the inspector(s) and test administra-tor.6.3.1 In general, the results of an experiment apply to theconditions under which the experiment was conducted. Hence,the POD analysis results apply to the conditions under whichthe POD examination was conducted.6.4 Prior to performing the analysis, the analyst shallconduct a preliminary review of the POD examination proce-dure to identify any issues with the administration of theexamination. The analyst shall identify and attempt to resolveany issues prior to conducting the POD analysis. Identifiedissues and their resolution shall be documented in the report.Examples of examination administration issues and possibleresolutions are outlined in the following subsections.6.4.1 If problems or interruptions occurred during the PODexamination that may bias the results, the POD examinationshould be re-administered.6.4.2 If the examination procedure was poorly designedand/or executed, the validity of the resulting data is question-able. In this case, the examination procedure design andexecution should be reevaluated. For design guidelines seeMIL-HDBK-1823A.E3023 − 1526.5 Prior to performing the analysis, the analyst shallconduct a preliminary review of the â versus a data to identifyany data issues. The analyst shall identify and attempt toresolve any issues prior to conducting the POD analysis.Identified issues and their resolution shall be documented in thereport. Examples of data issues and possible resolutions areoutlined in the following subsections.6.5.1 Any apparent outlying observations shall be reviewedfor correctness. If a typo is identified, the typo shall becorrected prior to performing the analysis. If the value iscorrect, it shall be retained in the analysis and its influence onthe â versus a model shall be evaluated during the modeldiagnostic assessment. The analyst should also reference Prac-tice E178.6.5.2 POD cannot be modeled as a continuous function ofdiscontinuity size if all the measured signal responses arebelow the noise threshold or above the saturation threshold. Ifthis occurs, the adequacy of the nondestructive testing systemshould be evaluated.6.6 Only â versus a data for induced discontinuities shall beused in the development of the linear regression model. Falsecall data shall not be included in the development of the linearmodel when using standard linear regression methods.6.7 The analyst in conjunction with the responsible engineershall determine the value of the noise threshold, ânoise, andsaturation threshold, âsat, prior to performing the analysis.6.7.1 The value of ânoiseis determined by performing anoise analysis. A noise analysis is typically accomplished byassessing the distribution of measured signal responses fromsites with no known discontinuity (false calls) and/or measuredsignal responses that are not influenced by the size of thediscontinuities (noise). Details on performing a noise analysiscan be found in MIL-HDBK-1823A.6.8 The analyst shall select an appropriate linear regressionmodel to establish the relationship between â and a. Selectionof a linear model may be an iterative process as the significanceof the predictor variable(s) and the appropriateness of theselected model are typically assessed after the model has beendeveloped.6.8.1 “Linear” refers to linear with respect to the modelcoefficients. For example, yˆi5b01b1·~x2! and yˆi5b01b1·x11b2·ln~x2! are linear regression models.6.8.2 In general, only significant and uncorrelated predictorvariables are included in a regression model. If more than onepredictor variable is being considered for inclusion in themodel, a preliminary graphical analysis of the response vari-able against each predictor variable may help identify whichpredictor variables appear to influence the response and thetype of relationship (for example, direct, inverse, quadratic). Inaddition, a preliminary graphical analysis of all possiblepairings of predictor variables shall be performed to verifyindependence of the predictor variables. When plotted againsteach other, there should be no apparent relationship betweenany two predictor variables.6.8.3 The appropriateness of a selected model is determinedby how well the model fits the observed data and how well theunderlying regression assumptions are met.6.9 The analyst shall use software that has the appropriatelinear regression capabilities to perform a linear regressionanalysis on the â versus a data.6.9.1 If censored data are present, the analyst shall do thefollowing:6.9.1.1 Include and identify the censored data in the analysis(according to the notation required by the software).6.9.1.2 Use the method of maximum likelihood to estimatethe model coefficients.6.9.1.3 Verify that convergence was achieved. If conver-gence is not achieved, the resulting â versus a model shall notbe used to develop a POD curve.6.9.1.4 Check the number of iterations it took to convergeprovided that information on convergence and the number ofiterations it took to converge is included in the analysissoftware output. If more than twenty iterations were needed toreach convergence, the model may not be reliable.6.9.1.5 Include a statement in the report indicating thatconvergence was achieved and the number of iterations neededto achieve convergence.6.9.2 If no censored data are present, the method of maxi-mum likelihood or the method of least squares shall be used.6.10 If included in the analysis software output, the analystshall assess the significance of the predictor variables in themodel. Only significant predictor variables should be includedin the model.6.11 Once the â versus a model is estimated, the analystshall use, at a minimum, the model diagnostic methods listedbelow to assess the underlying linear regression assumptions.The methods listed below shall be performed using onlynon-censored data. If