Designation: E2862 − 12Standard Practice forProbability of Detection Analysis for Hit/Miss Data1This standard is issued under the fixed designation E2862; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon (´) indicates an editorial change since the last revision or reapproval.1. Scope1.1 This practice defines the procedure for performing astatistical analysis on nondestructive testing hit/miss data todetermine the demonstrated probability of detection (POD) fora specific set of examination parameters. Topics coveredinclude the standard hit/miss POD curve formulation, valida-tion techniques, and correct interpretation of results.1.2 The values stated in inch-pound units are to be regardedas standard. The values given in parentheses are mathematicalconversions to SI units that are provided for information onlyand are not considered standard.1.3 This standard does not purport to address all of thesafety concerns, if any, associated with its use. It is theresponsibility of the user of this standard to establish appro-priate safety and health practices and determine the applica-bility of regulatory limitations prior to use.2. Referenced Documents2.1 ASTM Standards:2E1316 Terminology for Nondestructive Examinations2.2 Department of Defense Handbook:MIL-HDBK-1823A Nondestructive Evaluation System Re-liability Assessment33. Terminology3.1 Definitions of Terms Specific to This Standard:3.1.1 analyst, n—the person responsible for performing aPOD analysis on hit/miss data resulting from a POD examina-tion.3.1.2 demonstrated probability of detection, n—the calcu-lated POD value resulting from the statistical analysis on thehit miss data.3.1.3 false call, n—the perceived detection of a discontinu-ity that is identified as a find during a POD examination whenno discontinuity actually exists at the inspection site.3.1.4 hit, n—an existing discontinuity that is identified as afind during a POD demonstration examination.3.1.5 miss, n—an existing discontinuity that is missed dur-ing a POD examination.3.1.6 probability of detection, n—the fraction of nominaldiscontinuity sizes expected to be found given their existence.3.2 Symbols:3.2.1 a—discontinuity size.3.2.2 ap—the discontinuity size that can be detected withprobability p.3.2.2.1 Discussion—Each discontinuity size has an indepen-dent probability of being detected and corresponding probabil-ity of being missed. For example, being able to detect a specificdiscontinuity size with probability p does not guarantee that alarger size discontinuity will be found.3.2.3 ap/c—the discontinuity size that can be detected withprobability p with a statistical confidence level of c.3.2.3.1 Discussion—ap/cis calculated by applying a statis-tical uncertainty bound to ap. The uncertainty bound is afunction the amount of data, the scatter in the data, and thespecified level of statistical confidence. The resulting valuerepresents how large the discontinuity with POD equal to pcould be when uncertainty associated with estimating apisaccounted for. Hence ap/c ap. Note that POD is equal to p forboth ap/cand ap. apis based solely on the hit/miss data resultingfrom the examination and represents a snapshot in time,whereas ap/caccounts for the uncertainty associated withlimited sample data.4. Summary of Practice4.1 This practice describes step-by-step the process foranalyzing nondestructive testing hit/miss data resulting from aPOD examination, including minimum requirements for vali-dating the resulting POD curve.4.2 This practice also includes definitions and discussionsfor results of interest (for example, a90/95) to provide forcorrect interpretation of results.1This practice is under the jurisdiction of ASTM Committee E07 on Nonde-structive Testing and is the direct responsibility of Subcommittee E07.10 onSpecialized NDT Methods.Current edition approved Jan. 15, 2012. Published February 2012. DOI:10.1520/E2862-12.2For referenced ASTM standards, visit the ASTM website, www.astm.org, orcontact ASTM Customer Service at

[email protected] For Annual Book of ASTMStandards volume information, refer to the standard’s Document Summary page onthe ASTM website.3Available from Standardization Documents Order Desk, DODSSP, Bldg. 4,Section D, 700 Robbins Ave., Philadelphia, PA 19111-5098, http://dodssp.daps.dla.mil.Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States15. Significance and Use5.1 The POD analysis method described herein is based ona well-known and well established statistical method. It shallbe used to quantify the demonstrated POD for a specific set ofexamination parameters and known range of discontinuitysizes when the initial response from a nondestructive evalua-tion inspection system is ultimately binary in nature (that is, hitor miss). This method requires that a relationship betweendiscontinuity size and POD exists and is best described by ageneralized linear model with the appropriate link function forbinary outcomes.5.2 Prior to performing the analysis it is assumed that thediscontinuity of interest is clearly defined; the number anddistribution of induced discontinuity sizes in the POD speci-men set is known and well-documented; discontinuities in thePOD specimen set are unobstructed; the POD examinationadministration procedure (including data collection method) iswell-defined, under control, and unbiased; and the initialresponse is ultimately binary in nature (that is, hit or miss). Theanalysis results are only valid if convergence is achieved andthe model adequately represents the data.5.3 The POD analysis method described herein is consistentwith the analysis method for binary data described in MIL-HDBK-1823A, which is included in several widely utilizedPOD software packages to perform a POD analysis on hit/missdata. It is also found in statistical software packages that havegeneralized linear modeling capability. This practice requiresthat the analyst has access to either POD software or othersoftware with generalized linear modeling capability.6. Procedure6.1 The POD analysis objective shall be clearly defined bythe responsible engineer or by the customer.6.1.1 The analyst shall obtain the hit/miss data resultingfrom the POD examination, which shall include at a minimumthe documented known induced discontinuity sizes, whether ornot the discontinuity was found, and any false calls.6.2 The analyst shall also obtain specific information aboutthe POD examination, which shall include at a minimum thespecimen standard geometry (for example, flat panels), speci-men standard material (for example, Nickel), examination date,number of inspectors, type of inspection method (for example,line-of-site Level 3 Fluorescent Penetrant Inspection), andpertinent comments from the inspector(s) and test administra-tor.6.3 Prior to performing the analysis, the analyst shallconduct a preliminary review of the POD examination proce-dure and resulting hit/miss data to identify any examinationadministration or data issues. The analyst shall resolve anyissues prior to conducting the POD analysis. Examples ofexamination administration or data issues and possible resolu-tions are:6.3.1 If problems or interruptions occurred during the PODexamination that may bias the results, the POD examinationshould be re-administered. If this occurs, it shall be docu-mented in the report.6.3.2 If a discontinuity was missed because it was ob-structed (such as a clogged discontinuity), the discontinuityshall be removed from the POD analysis since there was not anopportunity for the discontinuity to be found. If a discontinuityis removed from the analysis, the specific discontinuity andrationale for removal shall be documented in the final report.6.3.3 POD cannot be modeled as a continuous function ofdiscontinuity size if there is a complete separation of missesand hits as crack size increases. If a complete separation ofmisses and hits is present in the data, the POD examinationmay be re-administered. If this occurs, it shall be documentedin the report. If a complete separation of misses and hits occurson a regular basis, the specimen set should be examined forsuitability as a POD examination specimen set.6.3.4 POD cannot be modeled as a continuous function ofdiscontinuity size if all the discontinuities are found or if all thediscontinuities are missed. If this occurs, the specimen set isinadequate for the POD examination.6.4 The analyst shall use a generalized linear model with theappropriate link function to establish the relationship betweenPOD and discontinuity size. For application to POD, thegeneralized linear model with discontinuity size as the singlepredictor variable is typically expressed as g(y) = b0+b1•a org(y) = b0+b1•ln(a), where a or ln(a) is the continuouspredictor variable, b0is the intercept, b1is the slope, y is thebinary response variable, and g(•) is the function that “links”the binary response with the predictor variable. If predictorvariables other than discontinuity size are quantifiable factors,a generalized linear model with more than one predictor maybe used.6.5 The analyst shall choose the appropriate link functionbased on how well the model fits the observed data. MIL-HDBK-1823A discusses four different link functions (Logit,Probit, Log-Log, Complementary-LogLog) and describesmethods for selecting the appropriate one. In general, the logitand probit link functions have worked well in practice formodeling hit/miss data.6.6 Only hit/miss data for induced discontinuities shall beused in the development of the generalized linear model. Falsecall data shall not be included in the development of thegeneralized linear model.6.7 The analyst shall conduct the analysis using softwarethat has generalized linear modeling capabilities.6.8 After running the analysis, the analyst shall verify thatconvergence has been achieved. The resulting POD curve shallnot be used if convergence has not been achieved.6.9 After verifying convergence, the analyst shall use at aminimum the informal model diagnostic methods listed belowto assess the reliability of the model and verify that the modeladequately fits the data.6.9.1 If included in the analysis output, the analyst shallcheck the number of iterations it took to meet the convergencecriterion. If more than twenty iterations were needed to reachconvergence, the model may not be reliable. A statementindicating that convergence was achieved and the number ofiterations needed to achieve convergence shall be included inthe report.E2862 − 1226.9.2 The analyst shall visually assess the shape of the PODcurve. (POD curves tend to be s-shaped.)6.9.3 The analyst shall visually assess how well the PODcurve fits the data by comparing how well the range over whichthe POD curve is rising matches the range over which missesbegin to overlap with and transition to hits as discontinuity sizeincreases.6.9.4 The analyst should also compare an empirical PODcurve to the POD curve based on the generalized linear model.The empirical POD curve shall be used for validation purposesonly. It shall not be used as a substitute for a POD curveresulting from a hit/miss analysis.6.9.4.1 To create an empirical POD curve, divide thediscontinuity sizes into bins. For example, (0.010 in., 0.020in.), (0.020 in., 0.030 in.), …, (0.100 in., 0.110 in.), etc.((0.0254 cm, 0.0508 cm), (0.0508 cm, 0.0762 cm), …, (0.2540cm, 0.2794 cm), etc.). For each bin, calculate the total numberof discontinuities contained in the bin and how many weredetected. Calculate the empirical POD in each bin by dividingthe number detected in the bin over the total number ofdiscontinuities in the bin. Plot the empirical POD versus themidpoint of the bin to obtain the empirical POD curve. Overlaythe POD curve based on the generalized linear model on theempirical POD curve to assess how well the generalized linearmodel fits the data by how well it matches the empirical PODcurve.6.9.5 If applicable, the analyst shall visually assess theshape of the confidence bound on the POD curve. Theconfidence bound should roughly follow the same shape as thePOD curve. If the confidence bound flares out significantly oneither or both ends or intersects the x-axis, the confidencebound should be viewed as suspect and may not be reliable.6.9.6 The analyst should assess the impact of data thatappears to be outlying (for example, an early hit in the smallsize range or a late miss in the large size range) by removingthe outlying value from the data and re-running the analysis toassess its influence on the shape of the POD curve andconfidence bound (if applicable). Both analysis results (withand without the outlying data) shall be included in the reportalong with a discussion of the impact to the POD curve andconfidence bound (if applicable).6.9.7 The analyst shall analyze any false call data and shallreport the false call rate at the 50 %, 90 %, and 95 % level ofstatistical confidence. Acceptable false call rates shall bedetermined by the responsible engineer or by the customer.6.9.7.1 The false call rate shall be defined as the number offalse calls divided by the number of opportunities in thespecimen set that do not contain discontinuity.6.9.7.2 What constitutes a false call shall be clearly definedby the responsible engineer or by the customer.6.9.7.3 What constitutes an opportunity in the specimen setthat does not contain a discontinuity shall be clearly defined bythe responsible engineer or by the customer.6.9.7.4 The Clopper-Pearson binomial method for con-structing confidence intervals for proportions should be used tocalculate the false call rate at the 50 %, 90 % and 95 % level ofstatistical confidence. The Clopper-Pearson upper 100•(1-α)%confidence bound for p is:PU5H11n 2 x~x11!·F~12α,2x12, 2n22x!J21where F(1–α, 2x+2, 2n–2x)is the F-statistics with degrees offreedom (2x+2, 2n–2x) and P[F F(1–α, 2x+2, 2n–2x)]=1–α.This method is consistent with that used in MIL-HDBK-1823A.7. Report7.1 At a minimum the following information about the PODanalysis shall be included in the report.7.1.1 The specimen standard geometry (for example, flatpanels).7.1.2 The specimen standard material (for example, Nickel).7.1.3 Examination date.7.1.4 Number of inspectors.7.1.5 Type of inspection method (for example, line-of-siteLevel 3 Fluorescent Penetrant Inspection).7.1.6 Any comments from the inspector(s) or test adminis-trator.7.1.7 The documented known induced discontinuity sizes.7.1.8 Which discontinuities were found and which weremissed.7.1.9 Any false calls.7.1.10 The selected link function.7.1.11 The generalized linear model coefficients.7.1.12 The variance-covariance matrix (if included in thesoftware output).7.1.13 A statement indicating that convergence wasachieved.7.1.14 The number of iterations needed to achieve conver-gence if