This is a defense motion to exclude fingerprint evidence in a criminal case under Frye (which was referred to as the Frye/Dyas test in D.C.). The actual document is sitting under the "Class 3" folder on Moodle - please go to Moodle to read the document. This motion:

(a) illustrates the application of the Frye test;

(b) considers the question of who counts as the relevant scientific community;

(c) describes the fingerprint identification process;

(d) describes the critique of that process; and

(e) illustrates the relationship of Rule 403 to Frye.

This is an excerpt from the government's response to the motion to exclude fingerprint evidence that you just read. This excerpt illustrates the different view the government takes on the Frye test, the definition of the relevant scientific community, and the meaning of the 2009 NRC Report.

The excerpt is a pdf posted on our Mooodle site under "Class 3."

Please go to our Moodle Page and under "Class 3" you will find the prompt and submission folder for Writing Reflection #3.

TABLE OF CONTENTS

Page

PRELIMINARY STATEMENT......................................................................... 3

ARGUMENT..................................................................................................... 4

I....... New York courts applying Frye should not use the “novelty test” to avoid further analysis of scientific evidence................................................................... 4

II...... New York courts should not mistake legal precedent for an analysis of reliability in evaluating scientific evidence under Frye............................................. 12

III..... This Court should provide guidance on the appropriate scope of the “relevant scientific community.”............................................................................ 18

IV..... New York’s interest in effectively scrutinizing scientific evidence for reliability could be aided by using the factors discussed in Daubert to conduct a Frye analysis.................................................................................................. 26

CONCLUSION................................................................................................ 30

   1. TABLE OF AUTHORITIES

Cases

In re Accutane Litig.,
234 N.J. 340 (2018)..................................................................................... 29

Ex parte Chaney,
563 S.W.3d 239 (Tex. Crim. App. 2018)........................................................ 8

Chesson v. Montgomery Mutual Ins. Co.,
75 A.3d 932 (Md. 2013)............................................................................... 16

Coble v. State,
330 S.W.3d 253 (Tex. Crim. App. 2010)...................................................... 16

State ex rel. Collins v. Superior Court,
644 P.2d 1266 (Ariz. 1982).......................................................................... 22

Commonwealth v. Foley,
38 A.3d 882 (Pa. Super. Ct. 2012).................................................................. 7

Commonwealth v. Shanley,
919 N.E.2d 1254 (Mass. 2010)................................................................. 8, 16

Contreras v. State,
718 P.2d 129 (Alaska 1986)......................................................................... 23

Cornell v. 360 W. 51st St. Realty, LLC,
22 N.Y.3d 762 (2014)............................................................................ 13, 19

Daubert v. Merrell Dow Pharm., Inc.,
509 U.S. 579 (1993).............................................................................. passim

Frye v. United States,
293 F. 1013 (D.C. Cir. 1923)................................................................. passim

Marso v. Novak,
42 A.D.3d 377 (1st Dep’t 2007)..................................................................... 9

Melendez-Diaz v. Massachusetts,
557 U.S. 305 (2009).................................................................................... 24

Motorola Inc. v. Murray,
147 A.3d 751 (D.C. 2016)...................................................................... 27, 28

Parker v. Mobil Oil Corp.,
7 N.Y.3d 434 (2006)................................................................... 14, 18, 25, 28

People v. Boone,
30 N.Y.3d 521 (2017).................................................................................. 29

People v. Calabro,
161 A.D.2d 375 (1st Dep’t 1990).............................................................. 9, 28

People v. Collins,
49 Misc. 3d 595 (Sup. Ct. Kings Cty. 2015)............................................ 19, 25

People v. Foster-Bey,
158 A.D.3d 641 (2d Dep’t 2018)............................................................ 15, 17

People v. Garcia,
39 Misc. 3d 482 (Sup. Ct. Bronx Cty. 2013)...................................... 10, 15, 18

People v. John.
27 N.Y.3d 294 (2016).................................................................................. 24

People v. Johnson,
27 N.Y.3d 199 (2016).................................................................................. 29

People v. LeGrand,
8 N.Y.3d 449 (2007).......................................................................... 5, 12, 13

People v. Luna,
989 N.E.2d 655 (Ill. App. Ct. 2013).............................................................. 14

People v. McKown,
875 N.E.2d 1029 (Ill. 2007).................................................................... 15, 16

People v. Rodriguez,
Ind. No. 5471/2009, Decision and Order (Sup. Ct. N.Y. Cty. May 1, 2012).. 6, 7

People v. Shreck,
22 P.3d 68 (Colo. 2001)............................................................................. 5, 6

People v. Slone,
76 Cal. App. 3d 611 (Cal. Ct. App. 1978)..................................................... 20

People v. Smith,
63 N.Y.2d 41 (1984).................................................................................... 20

People v. Vining,
28 N.Y.3d 686 (2017).................................................................................. 29

People v. Wesley,
83 N.Y.2d 417 (1994)........................................................................... passim

Reed v. State,
391 A.2d 364 (Md. 1978)............................................................................. 23

Starks v. City of Waukegan,
123 F. Supp. 3d 1036 (N.D. Ill. 2015)........................................................... 21

State v. Alberico,
861 P.2d 192 (N.M. 1993)........................................................................... 14

State v. Coon,
974 P.2d 386 (Alaska 1999)........................................................................... 6

State v. Hull,
788 N.W.2d 91 (Minn. 2010)....................................................................... 17

State v. Sharpe,
SP-7326 (Alaska, Jan. 4 2019)....................................................................... 8

State v. Ward,
694 S.E.2d 738 (N.C. 2010)......................................................................... 16

Sybers v. State,
841 So.2d 532 (Fla. Dist. Ct. App. 2003)...................................................... 22

Statutes & Rules

Federal Rule of Evidence 702....................................................................... 4, 27

Colorado Rule of Evidence 702........................................................................... 6

Other Authorities

David L. Faigman and Claire Lesikar, Organized Common Sense, 64 DePaul L. Rev. 421 (2014)........................................................................................... 13

David H. Kaye, Forensic Science, Statistics & the Law, “The New York City Medical Examiner’s Office ‘Under Fire’ for Low Template DNA Testing,” Sept. 11 2017, http://for-sci-law.blogspot.com/2017/09/the-new-york-city-medical-examiners.html................................................................................. 7

David H. Kaye, et al., The New Wigmore on Evidence, “Limiting Strict Scrutiny by Methodology,” § 9.5.1 (2018).................................................................. 16

DNA Exonerations in the United States, The Innocence Project, https://www.innocenceproject.org/dna-exonerations-in-the-united-states/ (last visited June 26, 2019).................................................................................... 1

Eric S. Lander, Fixing Rule 702: The PCAST Report and Steps to Ensure the Reliability of Forensic Feature-Comparison Methods in Criminal Courts, 86 Fordham L. Rev. 1661 (2018)................................................................. 11, 12

Harry T. Edwards, Solving the Problems That Plague the Forensic Science Community, 50 Jurimetrics 5 (2009)............................................................. 23

James E. Starrs, Frye, v. United States Restructured and Revitalized, 26 Jurimetrics J. 249 (1986)................................................................................ 7

Jane Campbell Moriarty, Deceptively Simple: Framing, Intuition, and Judicial Gatekeeping of Forensic Feature-Comparison Methods Evidence, 86 Fordham L. Rev. 1687 (2018)..................................................................................... 10

Lauren Kirchner, Traces of Crime: How New York’s DNA Techniques Became Tainted, NY Times, Sept. 4, 2017, https://www.nytimes.com/2017/09/04/nyregion/dna-analysis-evidence-new-york-disputed-techniques.html.......................................................... 17, 25, 26

Michael J. Saks, et al., Forensic bitemark identification: weak foundations, exaggerated claims, 3 J. L. Biosciences 538 (2016)......................................... 9

Misapplication of Forensic Science, The Innocence Project, https://innocenceproject.org/causes/misapplication-forensic-science/ (last visited June 26, 2019).................................................................................... 2

National Academy of Sciences, Strengthening Forensic Science in the United States (2009).......................................................................................... 21, 24

Peter W. Huber, Galileo's Revenge: Junk Science in the Courtroom (1991)........ 20

President’s Council of Advisors on Science and Technology, Report to the President: Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods (2016) ………………………………………passim

2. INTEREST OF THE AMICUS CURIAE

The Innocence Project is a national litigation and public policy organization dedicated to providing pro bono legal and related investigative services to indigent prisoners whose actual innocence may be established through post-conviction DNA evidence.  To date, the work of the Innocence Project and affiliated organizations has led to the exoneration of 365 individuals who post-conviction DNA testing has shown were wrongly convicted of crimes they did not commit. . . .

The Innocence Project submits this brief because the issues presented have serious implications for ensuring defendants are not wrongfully convicted through the admission of unreliable forensic science.  The use of unreliable forensic sciences occurs in 45 percent of exonerations of innocent defendants established through post-conviction DNA testing.  . . . .

3. PRELIMINARY STATEMENT

As currently applied in New York, the Frye standard for the admissibility of scientific evidence is susceptible to a level of confusion and complacency that poses a serious risk to the integrity of criminal proceedings in this state.  Therefore, this Court should take steps to provide the lower courts with clear guidance in order to establish an effective, consistent, and fair application of the “general acceptance” test for the admissibility of scientific evidence.  And the Court should remind the trial courts that trial courts must take seriously the responsibility of evaluating contested scientific evidence for reliability, particularly where a defendant’s freedom is at stake.

First, the Court should caution lower courts against rigidly applying “novelty” as a threshold test to bypass further analysis of scientific evidence, and provide specific guidance that underscores how “novelty” in the scientific fields can be a fluid concept.  Second, the Court should reiterate that legal precedent, while a useful tool, cannot be a substitute for examining whether a scientific technique is “generally accepted as reliable by the relevant scientific community.”  In particular, courts should be wary of relying on prior decisions where there is new evidence that the scientific consensus has changed.  Third, the Court should provide guidance on the definition of the “relevant scientific community” that makes clear that the opinions of a few experts, particularly those with financial or professional interest in the proffered methodology, are inadequate to represent the relevant scientific community.  Finally, [omitted].

4. ARGUMENT

I. New York courts applying Frye should not use the “novelty test” to avoid further analysis of scientific evidence.

In seminal cases applying Frye, this Court has discussed the concept of “novelty” only in general terms to explain what scientific evidence courts should evaluate for general acceptability.  See, e.g., People v. Wesley, 83 N.Y.2d 417, 435 (1994) (“[W]here the scientific evidence sought to be presented is novel, the test is that articulated in Frye.”); People v. LeGrand, 8 N.Y.3d 449, 455 (2007) (“[I]n recognition that expert testimony of this nature may involve novel scientific theories and techniques, a trial court may need to determine whether the proffered expert testimony is generally accepted by the relevant scientific community”) (citations omitted).  Many courts have interpreted this language to mean that “novelty” is a threshold question that determines whether scientific evidence is subject to Frye scrutiny at all.  The “novelty test” risks oversimplifying and abusing the concept of “novelty”—which is in fact a wholly fluid concept in science—to forego any real Frye analysis.  On one hand, courts may construe novelty too narrowly and refuse to scrutinize to new changes to an established methodology.  On the other hand, courts may construe novelty too broadly and refuse to scrutinize previously established methodologies that have since lost favor in the scientific community.  Compounding these problems, this Court has not provided guidance on what constitutes “novel” scientific evidence.   

Even where, as here, the underlying scientific theory is generally accepted, parties should be able to challenge truly novel applications of that underlying theory.  See People v. Shreck, 22 P.3d 68, 76 (Colo. 2001) (criticizing courts for not subjecting evidence previously admitted under Frye to new scrutiny, “despite improvements or other developments in scientific technologies”).[1]  This Court has recognized that, where a proposed methodology or hypothesis builds upon established scientific theory, courts must still exercise proper diligence before categorizing the proposed methodology or hypothesis as “not novel” and ending its analysis there. See Wesley, 83 N.Y.2d at 438 (Kaye, J.C., concurring) (finding that the addition of new steps to the traditional process for analyzing DNA, in order to compare two DNA samples, was “truly novel”).  Within the forensic sciences, dozens of new methodologies have emerged, many of which build on other non-novel methodologies.  As forensic sciences continue to innovate, courts must continue to scrutinize those innovations under Frye.

Courts should hold a Frye hearing where the methodology as a whole has not yet been proven reliable, even if some elements of the methodology have been proven reliable.  Some courts, however, have improperly focused on the non-novel elements of a proffered methodology, rather than evaluating the reliability of the methodology as a whole.  For example, in People v. Rodriguez, Ind. No. 5471/2009, Decision and Order (Sup. Ct. N.Y. Cty. May 1, 2012), the District Attorney argued that the likelihood ratios generated by the Forensic Statistical Tool (“FST”) should be admitted because likelihood ratios are not novel.  Id.  The proper focus of Frye inquiry though, is whether the methodology used to calculate the proffered likelihood ratios is generally accepted—not whether, in the abstract, likelihood ratios are a generally accepted statistical concept.  See David H. Kaye, Forensic Science, Statistics & the Law, “The New York City Medical Examiner’s Office ‘Under Fire’ for Low Template DNA Testing,” Sept. 11 2017, http://for-sci-law.blogspot.com/2017/09/the-new-york-city-medical-examiners.html (analogizing the Rodriguez court’s decision to admitting any linear regression model on the grounds that the method of least squares is not novel); see also James E. Starrs, Frye, v. United States Restructured and Revitalized, 26 Jurimetrics J. 249, 254 (1986) (approval of gas chromatographic evidence in one case should not be read as giving “carte blanche approval to all gas chromatographs,” especially as new types of detectors or sampling methods emerge).

At the same time, non-novel forensic techniques that once enjoyed “general acceptance” have since been challenged, or outright discredited, by new scientific developments.  As other Frye jurisdictions have recognized, it cannot be the law that a technique is insulated from new scientific criticism simply because it is not “novel.”  See, e.g., Commonwealth v. Foley, 38 A.3d 882, 888 (Pa. Super. Ct. 2012) (“[N]ovelty is not restricted to new science, and even ‘bedrock’ scientific principles may be subject to a Frye analysis if those principles become disputed.”) (internal quotation marks and citation omitted); Commonwealth v. Shanley, 919 N.E.2d 1254, 1264 n.15 (Mass. 2010) (“[T]he evolving nature of scientific and clinical studies of the brain and memory and the controversy surrounding those studies [on dissociative memory loss] made it prudent for the judge to proceed with a Lanigan hearing in this case.”).  Daubert jurisdictions have also recognized that new scientific evidence may necessitate a new evidentiary hearing.  See, e.g., State v. Sharpe, SP-7326, (Alaska, Jan. 4 2019) at 24 (“If an appellate court has made a Daubert determination and then new scientific research becomes available, or if a litigant identifies research that the appellate court overlooked, the trial court would be justified in holding an evidentiary hearing to make a complete record and rule in the alternative.”).  Similarly, New York courts should also ensure that “non-novel” methodologies and hypotheses continue to be accepted as reliable within the scientific community.  New York courts interpreting Frye, however, often fall short of this benchmark. 

For example, bite mark evidence and hair microscopy have both been effectively discredited.  See, e.g., Ex parte Chaney, 563 S.W.3d 239, 257 (Tex. Crim. App. 2018) (“[T]he body of scientific knowledge underlying the field of bite mark comparisons [has] evolved in a way that discredits almost all the probabilistic bite mark evidence at trial.”) (vacating conviction based on bite mark evidence); PCAST Report at 87 (“PCAST finds that bite mark analysis does not meet the scientific standards for foundational validity, and is far from meeting such standards.”); PCAST Report at 13 (“PCAST’s own review of the cited papers finds that these studies [of human hair comparisons] do not establish the foundational validity and reliability of hair analysis.”).  Yet despite a change in the scientific consensus, both have effectively evaded Frye scrutiny because neither technique is considered “novel.”  See Michael J. Saks, et al., Forensic bitemark identification: weak foundations, exaggerated claims, 3 J. L. Biosciences 538, 541 (2016) (“Despite the lack of empirical evidence to support its claims, to date no court in the United States has excluded [forensic odontology] expert evidence for failing to meet the requisite legal standard for admission of expert testimony.”); People v. Calabro, 161 A.D.2d 375 (1st Dep’t 1990) (finding sufficient evidence of guilt based upon testimony of three forensic odontologists and admission of similar testimony in another case nine years prior, without discussing whether forensic odontology was admissible evidence in the first instance). 

Lower courts faced with questions of where to draw the line between “novelty” and established science have ruled inconsistently.  Compare Marso v. Novak, 42 A.D.3d 377, 378 (1st Dep’t 2007) (granting judgment notwithstanding verdict where plaintiff’s expert could not show that her conclusions were generally accepted, even though the underlying methodology was generally accepted), with People v. Garcia, 39 Misc. 3d 482, 484 (Sup. Ct. Bronx Cty. 2013) (“The application of a generally accepted technique, even though its application in a specific case was unique or modified, does not require a Frye hearing.”).  This Court should remind trial courts to consider potential changes in the scientific consensus; otherwise, the temptation to evade a true Frye inquiry by simply checking the “not novel” box is as strong as it is problematic.  See Jane Campbell Moriarty, Deceptively Simple: Framing, Intuition, and Judicial Gatekeeping of Forensic Feature-Comparison Methods Evidence, 86 Fordham L. Rev. 1687, 1697 (2018) (“Rather than addressing the complexity head on and resolving [the pointed science-based critiques of forensic comparison methods], courts tend to use a variety of analysis-avoiding methods in evaluating the reliability of [forensic comparison method] evidence, even after learning of its shortcomings in the NRC Report.”).

Here, no party is challenging the admissibility of DNA evidence in general, which has been admissible in New York for over 20 years; nor does the Innocence Project suggest that a Frye hearing must be held in every instance of contested scientific evidence.  Rather, the challenges in this case are to two new techniques for analyzing DNA that were not subjected to a Frye hearing to evaluate whether they are generally accepted by the scientific community for the use to which they were put.  In situations like this one, the concept of “novelty” should not be used as an excuse to turn a blind eye to evidence that warrants additional examination. 

Finally, a widely used technique may still be “novel” where its use is primarily in non-scientific communities.  At minimum, use alone is not a proper factor in determining whether to forego a Frye hearing, as it is incumbent upon courts to look “under the hood” of even widely-used forensic methods.  Indeed, many unreliable forensic methods initially find their way into courts through their use as investigative techniques.  PCAST Report at 32 (“[M]any of these difficulties with forensic science may stem from the historical reality that many methods were devised as rough heuristics to aid criminal investigations and were not grounded in the validation practices of scientific research.”). 

The investigative origin of forensic methods poses two problems.  First, “fundamentally, the forensic sciences do not yet have a well-developed ‘research culture.’”  Id.  Although the forensic community has made significant strides to increase its scientific rigor in the past few years, courts should be wary of forensic techniques that are grounded in casework and investigative experience, as opposed to scientific research and knowledge.  See Eric S. Lander, Fixing Rule 702: The PCAST Report and Steps to Ensure the Reliability of Forensic Feature-Comparison Methods in Criminal Courts, 86 Fordham L. Rev. 1661 (2018) (describing how previously untested forensic methods are now undergoing empirical validity testing).[2]  Second, the standard of reliability required for just outcomes may be lower in an investigation, which is by its nature exploratory, than a prosecution, which seeks a final resolution.  See PCAST Report at 4 (“In investigations, insights and information may come from both well-established science and exploratory approaches.  In the prosecution phase, forensic science must satisfy a higher standard.”).  Although a technique’s “use” can serve as a helpful factor in determining whether court should treat it as “novel,” trial courts should be cautious not to conflate use outside of the courtroom with reliability. 

II. New York courts should not mistake legal precedent for an analysis of reliability in evaluating scientific evidence under Frye.

Directly related to the “novelty” inquiry is the role of precedent in Frye litigation.  This Court noted in LeGrand that “[o]nce a scientific procedure has been proved reliable,” scientific evidence may be considered not novel, thereby obviating the need for a Frye hearing.  8 N.Y.3d at 458 (citing Wesley, 83 N.Y.2d at 436 (Kaye, Ch. J., concurring)).  Thus, the test for determining when a Frye hearing should be held is whether prior proceedings have proved the proffered method’s reliability.  Once reliability has been proven, other courts “may take judicial notice of reliability of the general procedure.”  Id. at 458 (emphasis added).  In practice, similar to the problems inherent in the “novelty test,” courts have misconstrued this language and too often used legal precedent as a proxy for scientific acceptance, causing the most critical inquiry—reliability—to get lost in the shuffle.  Therefore, the Court should take this opportunity to provide crucial guidance to courts tasked with applying Frye, and, specifically, to instruct that (1) because science is constantly evolving, courts are not bound forever by a prior court’s Frye ruling; (2) a prior court’s ruling may be evidence of admissibility only where a Frye hearing was actually held; and (3) reliance on a prior court’s ruling may be improper where a party submits new evidence bearing on the challenged technique’s reliability or general acceptance within the scientific community.

Although Frye represents an intersection between law and science, these two fields approach “precedent” in fundamentally different ways.  See David L. Faigman and Claire Lesikar, Organized Common Sense, 64 DePaul L. Rev. 421, 422 (2014) (“[T]here is a basic disconnect between how scientists approach the empirical world and the way courts do so.”).  As this Court itself has recognized, “scientific understanding, unlike a trial record, is not by its nature static; the scientific consensus prevailing at the time of the Frye hearing in a particular case may or may not endure.”  Cornell v. 360 W. 51st St. Realty, LLC, 22 N.Y.3d 762, 786 (2014).  Similarly, the Supreme Court in Daubert acknowledged that, while reliance on precedent is axiomatic in our legal system, “[s]cientific conclusions are subject to perpetual revision.”  Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 596-97 (1993).  Other Frye jurisdictions have reached similar conclusions.  See, e.g., People v. Luna, 989 N.E.2d 655, 670 (Ill. App. Ct. 2013) (“[C]onstant scientific advances in our modern era may affect our inquiry as to the novelty of a particular methodology.”) (internal quotations omitted).  Therefore, when deciding the admissibility of scientific evidence, “courts should not be obliged to defer to past precedents: they should look afresh at the scientific issues.”  PCAST Report at 144.

  While a court may rely on a prior Frye hearing in the interest of judicial economy, a court should not rely on another court’s reliance on a third court’s Frye hearing.  Admissions without Frye hearings are only evidence of judges’ votes, as opposed to scientists’ votes.  See Parker, 7 N.Y.3d, 434, 447 (2006) (“[Frye] emphasizes counting scientists’ votes, rather than on verifying the soundness of a scientific conclusion.” (internal quotation marks omitted) (citing Wesley, 83 N.Y.2d at 439 (Kaye, C.J., concurring); see also State v. Alberico, 861 P.2d 192, 203 (N.M. 1993) (“It is improper to look for scientific acceptance only from reported case law because that amounts to finding a consensus in the legal community based on scientific evidence that is sometimes many years old.”).

When courts admit evidence based upon other courts’ findings without a Frye hearing—as was the case in Foster-Bey regarding the combination of FST and Low Copy Number (“LCN”) DNA evidence—courts compound the illusion that a methodology is generally accepted by the scientific community.  Courts should be wary of recycling prior courts’ decisions, especially when the scientific consensus has shifted.  For example, in People v. Garcia, the trial court cited judicial opinions admitting LCN DNA without a Frye hearing.  39 Misc. 3d at 487.  While the Garcia court correctly pointed out that LCN DNA profiling had been admitted in 125 cases, this did not mean that it had been proven reliable in 125 Frye hearings.  The Illinois Supreme Court recognized a similar species of error in People v. McKown, where the lower appellate court relied on a prior appellate decision that “merely reaffirmed” a third court’s decision to admit horizontal gaze nystagmus (“HGN”) evidence, [3] which itself relied heavily on a fourth court’s Frye hearing.  875 N.E.2d 1029, 1037–38 (Ill. 2007).  The Illinois Supreme Court held that this reliance on precedent was improper, and a Frye hearing should have been held, where the evidence of general acceptance was not “unequivocal or undisputed.”  Id. at 1046.  As evidence of such a dispute, the Illinois Supreme Court cited both more recent opinions from other states’ courts refusing to admit HGN evidence, id. at 1041-46, and more recent scientific articles questioning the reliability of HGN evidence, id. at 1044-47. 

Furthermore, where a trial court does rely on the findings of a previous hearing, that court should be convinced that the hearing was fair and thorough.  “Especially in the early days of a scientific technique, imbalanced hearings are not uncommon.”  David H. Kaye, et al., The New Wigmore on Evidence, “Limiting Strict Scrutiny by Methodology,” § 9.5.1 (2018). 

A number of appellate courts in other states have similarly recognized that trial courts should not blindly accept scientific evidence based on its admission in a prior case.  See, e.g., Coble v. State, 330 S.W.3d 253, 276 n.56 (Tex. Crim. App. 2010) (“[C]ourts do not ‘grandfather in’ expert testimony in a particular field or by a particular witness simply because the court has admitted expert testimony in that field or by that witness in the past.”); Shanley, 919 N.E.2dat 1264 n.15 (“[W]e have not ‘grandfathered’ any particular theories or methods for all time, especially in areas where knowledge is evolving.”); Chesson v. Montgomery Mutual Ins. Co., 75 A.3d 932, 938 (Md. 2013) (“Even scientific techniques once considered to be generally accepted are excluded when subsequent scientific studies bring their reliability and validity into question and show a fundamental controversy within the relevant scientific community.”); State v. Ward, 694 S.E.2d 738, 746 (N.C. 2010) (“[T]he length of time a method has been employed does not necessarily heighten its reliability or alleviate our concerns.”); c.f., State v. Hull, 788 N.W.2d 91, 103 n.3 (Minn. 2010) (noting that the “lengthy use of a method by law enforcement, and even lengthy unquestioning acceptance by courts, does not [by itself] exempt expert evidence from scrutiny under the first prong of Frye-Mack”) (alteration in original).

Published case law can provide a valuable tool in determining whether a scientific technique that was previously subjected to a Frye hearing is still generally accepted by the scientific community.  Courts, however, should be instructed not to lose sight of the fact that the Frye test emphasizes counting scientists’—not judges’—votes.  While published legal opinions may sit unaltered for decades, science is a field of ever-evolving developments.  Therefore, courts must be conscious of this fact and not simply admit evidence because previous courts have. 

In this case, none of courts cited by the trial court had ever subjected the contested evidence to a Frye hearing.  The trial court itself acknowledged that FST has never been subject to a Frye hearing, even while relying on other courts’ blind acceptance of that technique to support its own determination.  People v. Foster-Bey, 158 A.D.3d 641, 641 (2d Dep’t 2018).  In addition, the Office of Chief Medical Examiner (“OCME”), which introduced the contested evidence, no longer uses LCN or FST.  See Lauren Kirchner, Traces of Crime: How New York’s DNA Techniques Became Tainted, NY Times, Sept. 4, 2017, https://www.nytimes.com/2017/09/04/nyregion/dna-analysis-evidence-new-york-disputed-techniques.html. 

Thus, this case serves as a prime example of the dangers of a trial court deciding to use legal precedent as a substitute for its own determination of whether a technique is “generally accepted as reliable in the scientific community.”  Parker, 7 N.Y.3d at 449.  In light of the above, this Court should seize this opportunity to provide much-needed guidance on the appropriate use of precedent and under what circumstances reconsideration of previously-admitted scientific evidence is warranted under Frye.  As discussed in Section 4, infra, the adoption of Daubert principles, including encouraging courts to look to factors beyond “acceptability”—such as whether the technique in question can and has been tested, has been subjected to peer review, and has known or potential error rates—would further this objective.

III.This Court should provide guidance on the appropriate scope of the “relevant scientific community.”

In order to admit scientific evidence under Frye, a party must show that the evidence is generally accepted as reliable by the “relevant scientific community.”  As with “novelty,” this Court has not provided guidance to lower courts on how to interpret the “relevant scientific community” standard, leading to disparate and anomalous results.  Compare Garcia, 39 Misc. 3d at 487-88 (admitting testimony concerning LCN and FST based on studies conducted by the OCME despite the fact that it is the only government facility using these techniques), with People v. Collins, 49 Misc. 3d 595, 611, 613, 616, 618 (Sup. Ct. Kings Cty. 2015) (finding that the OCME’s research validating LCN and FST cannot establish general acceptance when no other laboratory uses these techniques as evidence in criminal cases).  Frye requires a careful balancing act when it comes to assessing the composition of the relevant community.  Because it is a general acceptance test, acceptance need not be universal; however, a court’s job must not end upon a finding of mere acceptance in the form of a single proponent of a technique. See, e.g., Cornell, 22 N.Y.3d at 783 (A showing that an expert’s opinion has “some support” is insufficient to establish general acceptance in the relevant scientific community). 

Here, the OCME, which developed the methodology in question, is the only entity in the United States that used the scientific techniques at issue to develop and analyze DNA profiles for criminal cases.  Collins, 49 Misc. 3d at 611, 618.  Moreover, the OCME is the only entity that used FST for any purpose.  Id.  Under these facts, it is likely that the trial court here did as Chief Judge Kaye cautioned against in Wesley: conflated the “judgment of the scientific community” with “the opinion of a few experts.”  Wesley, 83 N.Y.2d at 438 (Kaye, C.J., concurring). 

Despite this cautionary language, many courts have allowed small groups of experts who support a given technique to define themselves as the relevant scientific community, thus guaranteeing general acceptance.  One commentator referred to this practice as “gerrymandering” to create a majority by defining the scientific community “narrowly and uncritically.”  Peter W. Huber, Galileo's Revenge: Junk Science in the Courtroom 15 (1991).  For instance, in the case of now-discredited bite mark identification evidence, some courts have defined the relevant scientific community as the forensic dentists themselves, i.e., those whose careers are largely dependent on the validity of such evidence.  See, e.g., People v. Smith, 63 N.Y.2d 41, 63 (1984) (basing admission on the claim that the technique of comparing one photo of a bite mark to another was sufficiently reliable and had been “accepted by the scientific community,” comprised of prosecution and defense experts who together “acknowledged the reliability and acceptance of photographic comparisons”); People v. Slone, 76 Cal. App. 3d 611, 624-25 (Cal. Ct. App. 1978) (relying on testimony of three forensic odontologists which showed “bite-mark-identification technique had gained general acceptance in the scientific community of dentistry—the relevant scientific community involved”). 

By contrast, one federal court, considering the scientific validity of bite mark analysis in a civil suit against forensic dentists, brought by Benny Starks, who served over twenty years in prison after a bitemark “match” was used to convict him of a rape he did not commit, observed:

Starks argues that Dentist Defendants’ bite mark “analysis” was so far outside the norms of bite mark matching, such as they were in 1986, that it violated due process. For this assertion, Starks relies on the opinion of Senn, a forensic odontologist himself who has testified as a bite mark expert in many criminal cases. Doc. 313-20 at 48-50 (Senn’s CV). Eighty years ago, Upton Sinclair observed: “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” Upton Sinclair, I, Candidate for Governor: And How I Got Licked 109 (Univ. of Calif. Press 1994) (1935). Illustrating Sinclair’s point, Senn opines not that bite mark matching is inherently unreliable, but only that Dentist Defendants made analytical errors and overstated their conclusions in Starks’s criminal case.

Starks v. City of Waukegan, 123 F. Supp. 3d 1036, 1051-52 (N.D. Ill. 2015) (internal citations omitted).  The court went on to cite the National Academy of Sciences’s (“NAS”) observation, also relevant here, that “[a]lthough the majority of forensic odontologists are satisfied that bite marks can demonstrate sufficient detail for positive identification, no scientific studies support this assessment.”  Id. (quoting National Academy of Sciences, Strengthening Forensic Science in the United States 176 (2009)) (emphasis in original).

A broader view of the relevant scientific community is particularly important where certain experts, like the dentists discussed in Starks, have a vested interest in promoting a particular technique.  “If the field is too narrowly defined,” Chief Judge Kaye observed, “the judgment of the scientific community will devolve into the opinion of a few experts.  The field must still include scientists who would be expected to be familiar with the particular use of the evidence at issue, however, whether through actual or theoretical research.”   Wesley, 83 N.Y.2d at 438-39 (Kaye, J.C., concurring) (citation omitted).  “A Frye court should be particularly cautious when . . . the supporting research is conducted by someone with a professional or commercial interest in the technique.”  Id. at 440.  Indeed, the court in the original Frye decision, in examining the admissibility of systolic blood pressure deception testing, did not limit its view of the relevant scientific community to polygraph experts, but rather held that the technique had not gained general acceptance because it had “not yet gained such standing and scientific recognition among physiological and psychological authorities as would justify the courts in admitting expert testimony deduced from the discovery, development, and experiments thus far made.” Frye v. United States, 293 F. 1013, 1014 (D.C. Cir. 1923) (emphasis added).

Other jurisdictions have similarly recognized that courts should look to outside, disinterested experts with experience in the relevant field to confirm that such experts also accept those theories.  See, e.g., Sybers v. State, 841 So.2d 532, 543 (Fla. Dist. Ct. App. 2003) (“such assertions [that a technique is generally accepted] by experts who developed and performed the testing procedures are not, alone, sufficient.”); State ex rel. Collins v. Superior Court, 644 P.2d 1266, 1285 (Ariz. 1982) (“This requirement is not satisfied with testimony from a single expert or group of experts who personally believe the challenged procedure is accepted or is reliable.”); Reed v. State, 391 A.2d 364, 382 (Md. 1978) (“In general, members of the relevant scientific community will include those whose scientific background and training are sufficient to allow them to comprehend and understand the process and form a judgment about it”); Contreras v. State, 718 P.2d 129, 135 (Alaska 1986) (“We define the relevant scientific community as the academic, scientific, and medical or health-care professions which have studied and/or utilized hypnosis for clinical, therapeutic, research and investigative applications.”).  As such, this Court should instruct lower courts that under Frye, the “relevant scientific community” must be sufficiently broad as to include disinterested scientists with experience in related fields.

Courts should be wary of defining the “relevant scientific community” too narrowly where experts have a professional or pecuniary interest not only in the validity of a methodology, but also in the ultimate disposition of the case.  For example, Judge Edwards has warned that a close relationship between forensic scientists and law enforcement administrators may “inhibit good science and ultimately adversely affect the credibility of the field.”  Harry T. Edwards, Solving the Problems That Plague the Forensic Science Community, 50 Jurimetrics 5, 15 (2009); see also PCAST Report at 13-14 (to avoid potential bias and other issues, scientific evaluations are “best carried out by a science-based agency that is not itself involved in the application of forensic science within the legal system”).  Indeed, the Supreme Court in Melendez-Diaz rejected the State of Massachusetts’s arguments that forensic science was inherently “neutral” or “reliable.”  Melendez-Diaz v. Massachusetts, 557 U.S. 305, 318 (2009) (“Because forensic scientists often are driven in their work by a need to answer a particular question related to the issues of a particular case, they sometimes face pressure to sacrifice appropriate methodology for the sake of expediency.”) (quoting National Academy of Sciences, Strengthening Forensic Science in the United States 23–24 (2009)); see also Edwards, supra, at 19 (characterizing Melendez-Diaz as “a not very subtle indictment of our existing forensic science system”).  This Court similarly recognized the potential biases of forensic DNA analysts in People v. John.  27 N.Y.3d 294, 311 (2016) (“We will not indulge in the science fiction that DNA evidence is merely machine-generated, a concept that reduces DNA testing to an automated exercise requiring no skill set or application of expertise or judgment.”).

Likewise, enhanced scrutiny in defining the “relevant scientific community” might be appropriate where the proffered evidence relies on scientific techniques that are so new that the scientific community has not had a chance to determine whether to accept them.  Wesley, 83 N.Y.2d at 439 (Kaye, C.J., concurring) (“[A]bsence of controversy reflected not the endorsement perceived by our colleagues, but the prematurity of admitting this evidence. Insufficient time had passed for competing points of view to emerge.”).  Because the Frye test “emphasizes counting scientists’ votes, rather than on verifying the soundness of a scientific conclusion,” courts cannot determine whether a novel technique is generally accepted before scientists have been given enough time and information to decide how to cast their votes.  Parker, 7 N.Y.3d at 447 (internal quotation marks omitted) (citing Wesley, 83 N.Y.2d at 439 (Kaye, C.J., concurring).  That is not to say that recently introduced methodologies should be categorically rejected, but rather, that courts should recognize instances where “counting scientists votes” alone does not necessarily serve as an accurate proxy for reliability, such as where there has been insufficient time to establish a robust enough “community.”  

[omitted]

IV. New York’s interest in effectively scrutinizing scientific evidence for reliability could be aided by using the factors discussed in Daubert to conduct a Frye

[omitted]

CONCLUSION

For the reasons discussed herein, the time has come for this Court to speak on issues that have led to inconsistent and problematic results among courts when evaluating proffered scientific evidence under Frye.  This case provides an ideal opportunity for this Court to provide guidance that emphasizes the critical role of the courts in gatekeeping against evidence based on “novel” methodologies and hypotheses that have not yet been proven to be reliable, as well as previously accepted methodologies and hypotheses that have since been discredited by new scientific developments.  Incorporating the principles underlying the Daubert standard, which has been embraced by 40 states, would be a particularly effective tool to help courts bring Frye into the future.  Only with such guidance will courts achieve greater effectiveness, consistency, and fairness in their treatment of the admissibility of scientific evidence.     

Dated:         [, 2019]
                   New York, NY

                                                          Respectfully submitted,

                                                          ________________________________

CERTIFICATION

          I certify pursuant to §500.13(c) of the Rules of Practice of this Court that the total word count for all printed text in the body of the brief is 6,936 words.

Dated:         [, 2019]
                   New York, NY

                                                Respectfully submitted,

                                                          ___________________________________

                                                By:    Konrad Cailteux

[1] For this reason, the Colorado Supreme Court in Shreck rejected the Frye standard in favor of Colorado Rule of Evidence 702, which emphasizes “reliability and relevance of the scientific evidence.”  Id. at 78.  Other state supreme courts have also criticized Frye’s novelty test in adopting Daubert.  See, e.g., State v. Coon, 974 P.2d 386, 397–98 (Alaska 1999) (finding that Frye’s limitation to “novel” scientific evidence is not an advantage to Daubert’s application to “all scientific knowledge”), abrogated on other grounds by State v. Sharpe, 435 P.3d 887, 899 (Alaska 2019).

[2] Courts should also be wary of experts overstating the accuracy of otherwise reliable methods.  For example, Professor Lander describes how fingerprint identifications, once described by the Department of Justice as “infallible,” have since been shown to have a real world error rate as high as 1 in 24.  Lander, Fixing Rule 702, 86 Fordham L. Rev. at 1669–71.

[3] A type of field sobriety test that measures a subject’s eye movements as they track an object moving side to side.

IN THE CIRCUIT COURT OF COOK COUNTY

ILLINOIS                                                      )

                                                                        )           14 CR 16514-01

1. )

                                                                        )           JUDGE MALDONADO

[----]                             )           PRESIDING

MOTION TO EXCLUDE FIREARMS EXAMINATION OPINION TESTIMONY

NOW COMES the Defendant, [---], by his attorney Amy P. Campanelli, Cook County Public Defender, through her assistants Richard E. Gutierrez and Roger Warner, and brings this motion to bar testimony regarding firearms examination evidence. The Defendant is charged with murder for allegedly shooting Summer Moore. Officers of the Chicago Police Department recovered multiple spent cartridge cases at the scene of the purported shooting as well as a bullet from the body of the victim. The Defense expects that the State will attempt at trial to elicit opinions from a firearms examiner (Marc G. Pomerance) in order to link a later-recovered gun to the spent cartridges and bullet associated with Moore’s shooting. But such evidence, the Defense contends, possesses neither the general acceptance necessary to warrant admission under Frye, nor the reliability required to avoid the structures of Rule 403. In support thereof, the Defendant asserts the following:

1. INTRODUCTION

Firearms examination, in contrast to its high profile in criminal courts, has long benefitted from an existence in the shadow lands of science, where its pairing of grandiose claims and questionable tenets could escape unnoticed and uncritiqued. But in recent years, notorious misidentifications and wrongful convictions attributable to firearms examination and similar pattern-matching fields have at last compelled the broader scientific community to shed light on the suspect approaches of its forensic kin. The results have been sobering to say the least, for example the National Academy of Sciences (“NAS”) unequivocally and scathingly concluded that “no forensic method [besides DNA analysis] has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source,”[1] and the President’s Council of Advisers on Science and Technology (“PCAST”) even more recently and bitingly determined that firearms examination flat out “falls short of the scientific criteria for foundational validity.”[2] Thus, while little more than a decade ago acquiescence to the claims of the State’s expert would have met with few dissenters, at present it would fly in the face of a no-longer-silent majority of credentialed researchers and scientists.

Yet despite these admonitions by supremely-distinguished panels of scientists and law enforcement professionals, firearms examiners (members of a field without the benefit of sufficient scientific research validating its assumption of the uniqueness of fired bullets or spent cartridges, or affirming the reliability of its practitioners) have buried their heads in the sand and continue to report conclusions in nigh absolute terms with only token recognition of the potential for error stemming from the subjectivity of the practice and its lack of defined standards, the specter of cognitive bias, and the increasing uniformity of firearm components due to the evolution of manufacturing methods. This Court, however, should not follow the same misguided path by ignoring the criticisms of the field, emanating as they do from the highest scientific authorities organized by the federal government as well as from experts versed in the very disciplines (metrology and metallurgy/materials engineering) responsible for spawning the merely-applied practice of firearms examination. Instead, accounting for the broad consensus of experts positioned against the discipline and the substance of their critiques, firearms examination cannot be said to enjoy wide-spread scientific acceptance, nor to possess reliability sufficient to overcome the prejudicial and overblown statements of its practitioners. As such, this Court should exclude the subjective opinions of the State’s firearms examiner under Frye v. United States, 293 F. 1013 (D.C. Cir. 1923), or alternatively, Illinois Rule of Evidence 403.

1. FIREARMS EXAMINATION INVOLVES THE SUBJECTIVE ASSESSMENT OF MARKS DEPOSITED ON FIRED BULLETS & CARTRIDGE CASES.
   
   

Firearms examination at least begins with the scientifically sound premise that the inner-workings of guns, made from hard metals, may transfer their own markings to the softer metal of bullets and cartridges.[3] In other words, and without detailing ad nauseum the firing process itself: when bullets are propelled forward through a barrel they may take on the inverse of the lands and grooves (respectively peaks and valleys) of a its rifling as well as the imperfections/scratches (striations or striae) within those lands and grooves; cartridges may in turn be marked by the surfaces they impact, such as a gun's breech face and firing pin.[4] [The images below depict barrel rifling, the surfaces that may mark cartridges, and the placement of components within a gun.]

[5]

It is not the existence of such marks, however, with which this motion takes issue, but instead the methodology that firearms examiners use to derive meaning from whatever marks they may observe. Specifically, firearms examiners are generally asked to determine either (1) whether multiple recovered bullets or cartridges match (i.e. do the markings on the projectiles indicate that they were fired from the same gun), or (2) whether a recovered cartridge or bullet was fired from a specific recovered gun---in the latter instance examiners test fire the gun into a water tank and use the bullet or cartridge from that test fire for comparison.[6] And examiners have not updated their approach to answering those questions over the last 80 years: they use a comparison microscope to view two bullets or cartridges side by side, and make a determination based on the correspondence or lack thereof of the markings that they observe.[7] Said markings are divided into three categories: (1) class characteristics are the features predetermined by a manufacturer (and thus common to all guns of certain makes and models) such as the number of lands and grooves or the shape of a firing pin; (2) subclass characteristics are microscopic marks left behind by imperfections in gun parts and thus incidental to manufacture, but that are carried over and shared by multiple guns from the same batch; and finally (3) individual characteristics are marks produced by random irregularities of gun surfaces, which firearms examiners believe (without justification) are unique to each gun.[8]

            The Association of Firearm and Toolmark Examiners (AFTE), a trade organization whose membership is comprised exclusively of firearms examiners, [9] has established the ultimate range of conclusions for the discipline, and permits examiners to declare an identification (in other words a match) if they observe “sufficient agreement” between the individual characteristics of the bullets or cartridges they are comparing.[10] The definition offered for that vague term, however, scarcely clears things up, as AFTE describes the standard only by noting that agreement is sufficient when “it exceeds the best agreement demonstrated between toolmarks known to have been produced by different tools and is consistent with agreement demonstrated by toolmarks known to have been produced by the same tool.”[11] To boil things down, examiners may conclude that bullets or cartridges match when they look like a match.

Not surprisingly, AFTE admits even in its Theory of Identification that “the interpretation of individualization/identification is subjective in nature…and based on the examiner’s training and experience,”[12] meaning that “there will be some difference between examiners as to what constitutes the best-known non-match situation.”[13]  Nevertheless, AFTE still manages to claim that when examiners encounter sufficient agreement, that “means that the likelihood that another tool could have made the mark is so remote as to be considered a practical impossibility.”[14] But in contrast to AFTE’s unabashed self-confidence, this motion will demonstrate that “[s]ubjective methods [like firearms examination] require particularly careful scrutiny because their heavy reliance on human judgment means they are especially vulnerable to human error, inconsistency across examiners, and cognitive bias,”[15] and can only be evaluated as scientifically acceptable if vetted by multiple, appropriately designed, empirical studies of examiner reliability,[16] studies conspicuously absent at the base of the field of firearms examination.[17]

1. SCIENTIFIC AUTHORITIES HAVE ROUNDLY REJECTED FIREARMS EXAMINATION AS UNVALIDATED & BEREFT OF EMPIRICAL FOUNDATION.

Legitimate scientists have always understood that “valid scientific knowledge can only be gained through empirical testing of specific propositions.”[18] And although firearms examination (despite its longstanding use in courts) never developed such a foundation, it was not until the last few years that a harmonious and powerful consensus of scientific voices emerged to make unequivocally clear its doubts about of the discipline. Among the most influential of such voices, the National Academy of Sciences and its operating agency the National Research Council have twice joined the fray to chastise the field for its exaggerated claims, de minimus research, and vague/tautological standards.[19] This Court should accept its reports as authoritative. Not only have they been cited as such by the United States Supreme Court and other judges across the country,[20] but the mission and history of the NAS, which stands as the “leading scientific advisory body established by the Legislative Branch,”[21] ought to afford it ample reverence given that it has been tasked by Congress since the days of Abraham Lincoln “with providing independent, objective advice to the nation on matters related to science and technology” and has produced landscape-shifting studies of the forensic sciences (including the use of coroners offices, DNA statistics, and the shortcomings of bullet-lead analysis) since the 1920s. [22]

The NAS first approached the foundation of firearms examination when it set out (staffed by engineers, metallurgists, materials scientists, and others, as well as in consultation with firearms examiners) to evaluate the feasibility of operating a federal database of bullet and cartridge case images.[23] To do so it needed to study the underlying premises of any such database---the uniqueness and evidentiary value of bullet and cartridge case markings themselves---which it did through tireless review (eventually captured in over 80 pages of analysis) of a significant quotient of literature in the field of firearms examination, visits to manufacturing plants, and presentations from practitioners.[24] And though tasked to avoid the question of admissibility with regards to firearms evidence,[25] what the NAS discovered fell so short of valid science that the panel was compelled to nevertheless render several findings.[26]

Specifically, it emphasized that “the validity of the fundamental assumptions of uniqueness and reproducibility of firearms-related toolmarks has not yet been fully demonstrated,” and accordingly called for significant research to place even the basic premises of firearms examination on “solid scientific footing.”[27] Such work, the NAS conceded, would be arduous, necessitating “a designed program of experiments covering a wide range of sources of variability” while paying “careful attention to statistical experimental design issues, as well as intensive work on the underlying physics, engineering, and metallurgy of firearms.”[28] But because acceptable science, and derivatively acceptable testimony, requires a foundation of “established error rates” among other indicia of validity, the NAS viewed said research as “essential to the long-term viability” of firearms examination.[29]

Moreover, another panel of the NAS would echo those conclusions in a 300 page, meticulously-researched report published one year later. On this second go-round Congress directly authorized the NAS to investigate the status of several forensic science disciplines based on the recognition that “significant improvements are needed in forensic science.”[30] To that end NAS formed a team of acclaimed scientists, legal minds, and forensic specialists who for two years heard testimony from practitioners (including firearms examiners) and tirelessly “considered the peer-reviewed, scientific research purporting to support the validity and reliability of existing forensic disciplines.”[31] Ultimately, its authors reached unanimity with regard to the deficiencies of forensic identification (and especially pattern matching) approaches,[32] describing such methodologies as more akin to rough heuristics than validated science,[33] and noting in broad strokes that, as mentioned above, “no forensic method [other than DNA] has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source.”[34] 

The NAS report also pulled no punches when discussing firearms examination specifically. After adopting and incorporating the conclusions of the 2008 NAS panel discussed above, the report expressed concern that despite the “challenging” nature of distinguishing between marks left by the same or different firearms/tools, “the decision of the toolmark examiner remains a subjective decision based on unarticulated standards and no statistical foundation for estimation of error rates.”[35] Nor could the NAS discern any standards sufficient to guide examiners in that endeavor, noting that “a fundamental problem with toolmark and firearm analysis is the lack of a precisely defined process,” and criticizing the AFTE Theory of Identification for failing to “provide a specific protocol,” and “not even consider[ing], much less address[ing], questions regarding variability, reliability, repeatability, or the number of correlations needed to achieve a given degree of confidence.”[36] And, as to the research that could help flesh out such protocols, the NAS report could say only that (1) “sufficient studies have not been done to understand the reliability and repeatability of the methods,” and (2) “the scientific knowledge base for toolmark and firearms analysis is fairly limited.”[37] Thus its conclusion at bottom: firearms examination evidence lacks “any meaningful scientific validation, determination of error rates, or reliability testing to explain the limits of the discipline.”[38] In keeping with that rejection of the discipline, NAS allowed that examiners are capable of the fairly simple task of narrowing the pool of possible firearms matches using class characteristics, but did not evaluate the discipline and its methodology (in contrast to AFTE’s claims) as able to consistently link bullets or cartridges to a particular source.[39]

NAS, however, is but one member of an expansive coalition of the discontent comprised of academics and practitioners alike who view firearms examination with unmitigated skepticism and consider its claims as, at best, “plausible” but more realistically as “under researched, and oversold.”[40] In fact, article after article has appeared in the world’s preeminent scientific journals bemoaning the lack of research underlying firearms examination (and other forensic identification / pattern matching fields), the discipline’s lack of rigor, its failure to abide by any of the hallmarks associated with the very practice of science, and the overblown conclusions made by its practitioners.[41] Even the editorial board of Nature found “a disturbing degree of methodological sloppiness… [and] a poor empirical basis for estimating error rates.”[42] And statisticians (a group vital to the appropriate design of research studies and thus to any analysis of whether a discipline can lay claim to demonstrated validity) have widely endorsed the NAS reports and called for greater rigor in the design of experiments, increased transparency, and well-supported analysis and reporting or error rates.[43] Finally, scholars at the intersection of law and science have laid out the same concerns, and accordingly suggested outright exclusion of firearms examination testimony[44]

Nor have such admonishments been voiced merely by academics. Rather, forensic professionals admit that identification and pattern matching disciplines (like firearms examination) have “historically been troubled by a serious deficiency in that a heterogeneous assemblage of technical procedures … have frequently been submitted for basic theory and principles.”[45] And they emphasize that firearms examination “has always suffered from the fact that the examination of these types of evidence is highly subjective, and cannot fall back upon a body of independently-derived scientific knowledge … Despite three quarters of a century, no systematic and comprehensive attempt to codify standards for a minimum toolmark or firearms match has been published.”[46] Forensic self-critics have also included a past president of the American Academy of Forensic Sciences (who called the field of firearms examination wholly unvalidated, going so far as to suggest that threshold studies of the field’s underlying foundations might result in a finding that the entire field has always been invalid)[47] as well as a coalition of thirteen diverse authors who came together to support the recommendations of the NAS and issue a call for greater focus on the empirical underpinnings of pattern matching disciplines.[48]

Perhaps most pointedly, William Tobin (a materials scientist and retired FBI metallurgist) has railed against the discipline of firearms examination for (1) its “inherently vague and tautological” theory of identification, (2) its failure to conduct appropriate experiments to test its underlying foundations, (3) the lack of understanding by practitioners of the manufacturing processes so central to the production of toolmarks, (4) the absence of adequate proficiency testing, and finally (5) the unjustified grandeur of its conclusions.[49] And given his background, Mr. Tobin is ideally positioned to comment on the conclusions and methods of firearms examiners, because while those practitioners may possess an understanding of “the general manufacturing process for firearms,” the characteristics upon which they rely “are generated by a variety of metallurgical processes and entail complex tribological and microstructural (including atomic) interactions that can, and most often do, vary from product to product, and even from production lot to production lot.”[50] Thus firearms examiners, according to Tobin, should have (but have never) engaged with members of the field of metallurgy/materials science, “the most relevant true scientific domain … that understands the tribology of the manufacturing processes and their specific seminal effects on firearm components.”[51] The discipline’s failure to do so, coupled with its inability to “incorporate effective statistical methods,” ultimately leaves it bereft of “every critical cornerstone of the scientific method.”[52]

The same sentiments above, moreover, now echo through the various regulatory agencies for forensic science established in the wake on the NAS report. For example, the members of the Firearms and Toolmark subcommittee of the Organization of Scientific Area Committees (OSAC)[53] recently acknowledged the complete absence of appropriate studies concerning the reliability of firearms examination or the ability of examiners to characterize relevant markings, and described said absence as a “major gap” in understanding regarding the discipline.[54] Moreover, the National Commission on Forensic Science published a views document noting that “the underlying foundation of [a forensic discipline] and associated testimony must be supported by sound research that meets the standards of forensic practitioners, academic researchers, measurement scientists, and statisticians.”[55] Although the NCFS could not itself conduct such a review of firearms examination in particular, it identified “a clear and compelling need to address the technical merit of forensic science,” and proceeded to pass the torch by opining that “all forensic science methodologies should be evaluated by an independent scientific body to characterize their capabilities and limitations.”[56] That call (made explicitly by NCFS and so many of the other authorities cited throughout this section) has now been answered by PCAST, and firearms examination has been found wanting.

1. AFTER AN IMMENSELY DETAILED REVIEW OF FIREARMS EXAMINATION, THE PCAST DETERMINED THAT THE DISCIPLINE LACKS EVEN BASIC FOUNDATIONAL VALIDITY.

Despite the authority and scope of its critique of the forensic sciences, NAS readily admitted that, in part because of that expansive scope, it had “decided early in its work that it would not be feasible to develop a detailed evaluation of each discipline in terms of its scientific underpinning.”[57] As a result, AFTE ignored its comments and actively awaited the day when an independent body would evaluate the full breadth of its literature.[58] But that day has come, because the same concern motivated PCAST, in response to then-President Obama’s request to identify and provide insight regarding lingering deficiencies in forensics, to highlight two important gaps in scientific understanding regarding pattern matching disciplines: “(1) the need for clarity about the scientific standards for the validity and reliability of forensic methods and (2) the need to evaluate specific forensic methods to determine whether they have been scientifically established to be valid and reliable.”[59] Its resulting review went further than any before it by (1) compiling and reviewing over 2000 papers (more than 400 of which were specific to firearms examination); (2) consulting a diverse group of forensic scientists and practitioners (including those working with the FBI and the National Institute of Standards and Technology) as well as judges, attorneys, statisticians, and academic researchers; and (3) soliciting statements and bibliographies from all corners of the practitioner community.[60]

And this Court, as with the NAS, should consider the PCAST’s ultimate conclusions authoritative. Not only is PCAST “the leading scientific advisory body established by the Executive Branch,”[61] but the Obama-era-iteration of the PCAST consisted primarily of some of our nation’s leading and most-respected scientists, including: a geneticist from MIT/Harvard who was the principal contributor in efforts to map the human genome, an engineer and Vice President of the National Academy of Engineering, a mathematician and former CEO of The Aerospace Corporation, a doctor who was the first female president of the American College of Physicians, a chemist who directs the Institute for Nanotechnology at Northwestern University, the director of The Laboratory for Geochemical Oceanography at Harvard University, a doctor of biochemistry and professor emeritus at the University of California Berkeley, and a physicist who is a Senior Vice President at a leading aerospace and technology corporation (to name but a few).[62] For several decades, the PCAST has reported to the then-sitting U.S. President on a wide range of scientific issues, including, but not limited to, nanotechnology, internet broadband development, cloning, and the uses of science and technology to combat terrorism.[63]  In short, the PCAST represents one of the most important and authoritative collections of scientists in the country. And its final report on the pattern matching disciplines has, since its publication, been endorsed by the nation’s most prestigious forensic body (the American Academy of Forensic Sciences),[64] an international consortium of forensic experts,[65] and Judge Alex Kozinski of the United States Court of Appeals for the Ninth Circuit, who went so far as to say that the report “will fundamentally change the way many criminal trials are conducted” and “will likely upend many people’s beliefs” about once-trusted forensic disciplines.[66]

Turning to the substance of the report, PCAST viewed its primary mission as providing courts with an understanding of the “scientific standards for scientific validity” based on “the fundamental principles of the ‘scientific method’—applicable throughout science,” and more specifically the standards of metrology (the science of measurement), from which all pattern recognition disciplines derive.[67] But even as it began its work, PCAST had already identified numerous areas of concern, noting specifically (1) that many wrongful convictions discovered only through DNA testing and attributable to faulty forensic testimony “reflected a systemic problem—the testimony was based on methods and included claims of accuracy that were cloaked in purported scientific respectability but actually had never been subjected to meaningful scientific scrutiny,”[68] (2) “the historical reality that many methods were devised as rough heuristics to aid criminal investigations and were not grounded in the validation practices of scientific research,”[69] and (3) that “subjective methods require particularly careful scrutiny because their heavy reliance on human judgment means they are especially vulnerable to human error, inconsistency across examiners, and cognitive bias.”[70] Nevertheless, it did not simply dismiss pattern recognition disciplines outright based on that checkered record, but instead carefully explained that, because subjective methods like firearms examination rely on the skill of their practitioners, “without appropriate estimates of accuracy, an examiner’s statement that two samples are similar—or even indistinguishable—is scientifically meaningless.”[71] Thus based on general scientific standards: “Since the black box in the examiner’s head cannot be examined directly for its foundational basis in science, the foundational validity of subjective methods can be established only through empirical studies of examiner’s performance.”[72]

Accordingly, for a discipline to qualify as foundational valid, and therefore as worthy of scientific acceptance and legal admissibility, it would need at its base multiple black box studies that (1) are double blind (“neither the examiner nor those with whom the examiner interacts have any information about the correct answer”); (2) are “overseen by individuals or organizations that have no stake in the outcome of the studies”; and (3) “involve a sufficiently large number of examiners and [are] based on sufficiently large collections of known and representative samples from relevant populations to reflect the range of features or combinations of features that will occur in the application.”[73] Any divergent claims to validity, PCAST highlighted, would run “contrary to the fundamental principle of scientific validity in metrology—namely, that the claim that two objects have been compared and found to have the same property … is meaningless without quantitative information about the reliability of the comparison process.”[74]  And although these standards might appear taxing, such was simply not the case in practice: the NAS subjected fingerprint comparison, for example, to the same criticisms as firearms examination,[75] but because that discipline responded appropriately to those criticisms with empirical research, PCAST evaluated the field of fingerprint comparison as possessing foundational validity on the basis of merely two adequate studies.[76]

Firearms examination, in contrast, did not pass muster even under PCAST’s unexacting standards. As an initial matter, PCAST noted that the AFTE Theory of Identification (along with the methodology of firearms examination more generally) is “circular” and thus the discipline benefits from no rigorous, or objective criteria.[77] And although some studies promulgated by the discipline would seem to indicate that, despite that lack of guidance, “examiners can, under some circumstances, associate ammunition with the gun from which it was fired” those industry-funded and industry-implemented projects “involved designs that are not appropriate for assessing the scientific validity or estimating the reliability of the method as practiced … because of their design, many frequently cited studies seriously underestimate the false positive rate.”[78] In fact, the director of a leading forensic research institute analogized the design of such studies to “a ‘Sudoku’ puzzle, where initial answers can be used to help fill in subsequent answers.”[79] And because those efforts utterly failed to replicate casework by providing examiners with simplistic problems, PCAST discounted them, instead concluding that “there is only a single study that was appropriately designed to test foundational validity and estimate reliability,”[80] specifically one recently conducted by the independent AMES laboratory (which, by the way, discovered that 2% of examiners registered a disturbing misidentification rate of 40%).[81] As a result, it ultimately concluded that:

“The scientific criteria for foundational validity require appropriately designed studies by more than one group to ensure reproducibility…the current evidence [for firearms examination] falls short of the scientific criteria for foundational validity. There is thus a need for additional, appropriately designed black-box studies to provide estimates of reliability.”[82]

Worse still, the AMES study accomplishes far less for the discipline than any endorsement of its general design might indicate. Even PCAST noted concern because the article has never been published and thus subjected to peer review. But putting that issue aside, the false positive rate the study uncovered for examiners on the whole corresponds to an upper bound estimate of error in one of every 46 cases.[83] And that is without even mentioning that the samples involved simply did not reflect the difficulty of casework. Specifically, the study focused on cartridge cases fired from Ruger SR9 semiautomatics, a gun chosen precisely because the marks it produces are of average difficulty to interpret.[84] But recall that PCAST’s criteria require samples “representative of the quality of evidentiary samples seen in real cases. (For example…for distorted, partial, latent fingerprints; the random match probability for full scanned fingerprints, or even very high quality latent prints would not be relevant.)”[85] And by focusing on only cartridges of average difficulty, the AMES study could not hope to validate, or provide an idea of examiner reliability, when comparing bullets or more trying cartridge samples­—like ones that have been consecutively manufactured or feature subclass characteristics (as could well have been the case in this matter).[86] The actual and applicable error rates for the discipline may therefore far outstrip that reported by the AMES study, a reality that would further undercut the field’s already wanting claim to validity and admissibility.

1. FIREARMS EXAMINERS ARE IN NO POSITION TO DISPUTE THE CRITICISMS OF THEIR FIELD.

            The approach of the field of firearms examination to all the above criticisms, including even the PCAST report, has been to bury its collective head in the sand rather than strike out on a path of reform and confront said criticism with new and appropriately designed research studies.[87] In fact, at all levels, the discipline has even continued to express conclusions to a practical certainty,[88] an approach universally decried as scientifically indefensible and simply ludicrously devoid of support.[89] And those decisions should concern the Court, first because, as practitioners of a merely-applied science, firearms examiners are (by even the admission of other pattern matching specialists) simply not as qualified as the authorities cited above to design and conduct appropriate validation studies, or to opine more generally on the reliability of their profession.[90] But more than that because: (1) conscientious firearms examiners admit that their discipline lacks appropriate research and must therefore rely on tenets---the purported uniqueness of gun parts, as well as training and experience---that cannot serve scientifically as a means of validating the field, (2) the field has yet to grapple convincingly with the greatest challenges to reliability---coincidental similarity and subclass marks---facing examiners, and (3) unacceptable rates of error and misidentification have long-plagued the discipline. These deficiencies, alone and especially when taken together, serve as ample proof that PCAST and others critics of firearms examination got it right; firearms examination has not yet been adequately validated, and might be incapable of ever demonstrating sufficient reliability.

• Firearms Examiners Essentially Admit That their Field Lacks Validation.

Firearms examination practitioners acknowledge that, although every meaningful conclusion of “match” has at its foundation a calculation of probability,[91] they simply lack the data to provide any adequate measure of that foundation based on a lack of appropriate or varied studies.[92] That leaves practitioners only to guess at the likelihood that another firearm might be responsible for the marks they observe, to make essentially a “leap of faith”[93] based on nothing more than intuited (and altogether unjustified) feelings regarding uniqueness and the value of experience. And that reality is borne out by the history of research and self-reflection available for firearms examination.

It may be true that the discipline has been a fixture of criminal proceedings since the beginning of the 20^th century, but reviewing the field’s allegedly scientific literature in 1997, its preeminent defender, Ronald Nichols, found little reason to support the trust and reliance long-bestowed on firearms examination by the courts. He was forced to concede that most of the field’s research had been “very subjective in nature” and conducted in a manner only “analogous to” rather than in accordance with the scientific method.[94] Moreover, he admitted that “the most exhaustive, statistical empirical study ever published” in support of firearms examination dated back to 1959.[95] Troubling as it might seem that firearms examination had not conducted any scientific introspection since the United States numbered only 48, Eisenhower was president, and color television was a scarcely utilized medium, Nichols’s admission should further distress the Court given that the 1959 article’s author, Biasotti, held a much humbler view of the value of his own study (which by the way involved consideration of only one firearm model, .38 Special Smith & Wesson revolvers, and only analyzed fired bullets not cartridges). Specifically, he began his article by noting the “almost complete lack of factual and statistical data pertaining to the problem of establishing identity in the field of firearms examination,” but would say of his contribution to remedying that error only that “much more factual (statistical) data must be collected before any general verifiable laws can be formulated or before the data reported in this study can attain any real measure of practical significance.”[96]

Unfortunately, Biasotti’s calls to study and reflection would not come to fruition, and years later he would again lament that his chosen field of firearms examination was anything but “a highly developed science with well-defined criteria for evidence evaluation.”[97] Instead, Biasotti noted “a very superficial treatment of the basic problem of evaluating results and establishing identity,” and described firearms examination as “essentially an art limited by the intuitive ability of individual practitioners.”[98] Moreover, what studies have since been performed (mostly “10-gun” studies of consecutively manufactured tools) have been roundly lampooned not just by PCAST but also by a host of other outside scientists and firearms examiners alike for (among an even more extensive list of grievances) their gross lack of objectivity, minimal consideration of manufacturing variables, inappropriate design, minuscule sample sizes, and consideration of only a small fraction of gun make and models.[99] Thus, even as late as 2012, little had changed: firearms examiners could still point to “only a few numerically based studies” and admitted that their discipline “lacked scientific, statistical proof that would independently corroborate conclusions” of examiners performing casework.[100]

But perhaps nothing more perfectly captures the cavalier and unscientific bent of the discipline of firearms examination than the venue in which it has disseminated the overwhelming majority of its “research.”[101] This Court will surely notice myriad citations throughout this motion to the AFTE Journal. That periodical houses the vast majority of published research on firearms examination (although most such “research” is comprised of little more than two or three page anecdotes about casework). But it effectively stymies outside review of firearms examination because it wholly fails to meet professional standards regarding the publication of scientific literature,[102] only recently became inaccessible to the scientific and legal communities (formerly only firearms examiner could peruse the journal online), cannot be downloaded via most academic databases (i.e. PubMed), and is available in print at only a handful of libraries throughout the United States.[103] Moreover, while credible journals like the Journal of Forensic Science offer non-subscribers the opportunity to purchase all back issues for a reasonable price ($695),[104] AFTE allows purchase of its journal only on an article-by-article basis, necessitating the spending of thousands of dollars to obtain all its back issues. In fact, the Public Defender managed to obtain copies of the AFTE Journal (available publicly nowhere in Chicago) only by the fortunate happenstance of one assistant’s presence at a forensics conference in New York City (where the library of the John Jay College of Criminal Justice houses print copies); access nevertheless came at the cost of three full workdays’ worth of ceaseless scanning.[105] It should come as no surprise then, that both academics and forensic practitioners have concluded that the AFTE Journal, and thus the research contained within, wholly lack scientific indicia of trustworthiness.[106]

To attempt to claim credibility in the face of such a glaring deficiency in research, AFTE and others within the discipline have doubled down on their belief that the markings on fired bullets and cartridges are unique (and thus easily discerned and sorted by trained examiners), and pointed to the fact that the field has “stood the test of time.”[107] But the concept of uniqueness is not particularly relevant to the reliability of firearms examination,[108] and at all events is scientifically indefensible-at minimum it could never be proven through sampling (i.e. through failing to observe identical pairings during any period of casework).[109] In some sense, however, the discipline’s logical failure is expected. All human beings (including it would seem forensic professionals) are tempted to blindly accept the concept of uniqueness because “duplication is inconceivable to the rational mind.”[110] Unfortunately, our assumptions mislead us: despite the supposed truism that all snowflakes are unique, visually identical examples have been discovered,[111] and even in firearms examination (paradoxically considering the discipline’s commitment to the concept of uniqueness) examiners know that “each element of a firearm’s signature may be found in the signature of other firearms.”[112] Thus, because “the concept of uniqueness has more the qualities of a cultural meme than a scientific fact,”[113] firearms examiners have in fact conceded that “the erroneous conception of the ‘perfect match’ … is actually only a theoretical possibility and a practical impossibility.”[114]

Yet the assumption of uniqueness “although lacking theoretical or empirical foundations” perseveres in the field of firearms examination perhaps because “it offers important practical benefits” to the discipline (although not to the accused), as one expert explains:

“It enables forensic scientists to draw bold, definitive conclusions that can make or break cases. It excuses the forensic sciences from developing measures of object attributes, collecting population data on the frequencies of variations in those attributes, testing attribute independence, or calculating and explaining the probability that different objects share a common set of observable attributes. Without the discernable uniqueness assumption, far more scientific work would be needed, and criminalists would need to offer more tempered opinions.”[115]

Having so long relied on the broken notion of uniqueness, however, examiners can now only cite to casework and experience (rather than quantification) as proof of their abilities. “But whatever the courts' intuitive confidence in [firearms examination] ‘implicit testing’ and ‘casework validation’ set poor precedents that defy science and logic.”[116] At bottom, scientists have always been suspicious of claims grounded alone on longstanding use,[117] (perhaps given their experience with long-held beliefs of a flat world or longstanding use of such medieval methods as blood letting), and forensic use of the concept should meet with equal disapproval.[118]

Initially, belief in the validating power of casework relies on the absurd “assumption that every examiner remembers the details of every object ever examined, and even if only subconsciously, they have then ‘compared’ all of the objects they happened to examine with one another. Such a proposition is highly dubious, and relies on claims and observations that have never been recorded nor compiled in a systematic manner.”[119] And it requires as foundation the untenable notion that every examination comprising the test-of-time has been correct: “Because the ground truth is not known in casework, a case cannot serve as a test of the accuracy of a forensic assay used in it.”[120] But, because of the insufficiency of sampling as proof that has already been noted, casework fails as a pillar of support for firearms examination even if we assume that practitioners never err and possess superhuman powers of recollection. One readily understandable example posits that “a truly random sample of a large number of human beings may indicate that none of them have the same mother; but we know that to conclude that not one person on Earth shares the same mother defies common sense.”[121] And specific to the firearms context one expert has formulated an even more biting attack on the discipline’s belief in the power of experience to justify its methodology:

“suppose that exactly 100 pairs of firearms out of an estimated 100,000 guns in a Texas town share indistinguishable gun barrel markings. If each of 100 firearms experts examined 10 pairs of guns from the town's gun population every day for 10 years (n=3,650,000 gun pairs), there is about a 93% chance that none of the indistinguishable pairs will have come under examination. That is, despite 1,000 "collective years" of forensic science experience (100 experts multiplied by 10 years), the failure to find even a single pair of guns with indistinguishable markings would offer little basis for drawing conclusions about whether gun barrel markings, even in this single town, are unique.”[122]

As Tobin notes, “this is a striking and counterintuitive example of the folly of individual, or even collective, training and experience as basis for validation.”[123] It should compel this Court to link admissibility to real scientific research, and should dispel any feeling of comfort in the platitudes regarding years of experience and thousands of cases worked so often peddled by examiners.[124]

• Significant Pitfalls Confronting Firearms Examiners During Casework Undercut the Discipline’s Assertions of Reliability.

            And so we find firearms examiners, without standards to chart their course or research to keep them afloat, awash in a sea of roughly 310 million firearms,[125] claiming nonetheless the capability to navigate directly to the single and only gun that could have discharged a bullet or cartridge. But because their methodology calls for side-by-side, show-up style comparisons and welcomes investigators to provide examiners with task-irrelevant information, the distorting effects of cognitive bias may well lead them astray. And to further complicate matters, even assuming that practitioners persevere through these sources of cognitive corruption, the markings on the projectiles they examine, in part due to ever-advancing standardization and calibration of the manufacture of the weapons firing them, are not so easily sorted into matches and non-matches. Instead markings left by different firearms, whether based on the close proximity of the manufacture of such guns by the same production tool or even randomly, may exhibit striking similarity exceeding even that of markings produced by the same firearm.[126] Examiners, as the following sections will demonstrate, concede the dangers presented by these complications. And although they remain willing to contradict those admissions and stand by the reliability of their discipline, this Court should not permit such incongruity to distort its truth seeking function.

1. Cognitive bias can distort an examiner’s perception of evidence & produce error.

            At bottom, cognitive bias is merely the professional parlance for what laypeople, as evident from the long-unbroken parade of advertising focused on blind taste tests (i.e. the Pepsi Challenge), have long accepted: that outside influences, stimuli, or preconceptions can cloud subjective, human judgment.[127] Some psychologists have even more directly termed the concept “mental contamination,”[128] but regardless of the title given it, scientists have long acknowledged that cognitive bias “can lead to perceptual distortion, inaccurate judgment, or illogical interpretation,”[129] specifically because it causes decision makers to “seek information that they consider supportive of a favored hypothesis or existing beliefs and to interpret information in ways that are partial to those hypotheses or beliefs.”[130] So significant are the effects of cognitive bias that researchers have noted that “If one were to attempt to identify a single problematic aspect of human reasoning that deserves attention above all others, the confirmation bias would have to be among the candidates for consideration.”[131]

In sync with the warnings of these experts, studies have consistently shown that the misconceptions of laypeople and scientists alike can remain steadfast even in the face of clear evidence that their ideas are wrong. Laypeople have faltered due to cognitive bias and thereby ignored market trends,[132] inappropriately focused on “blossoming students” (thus inflating their performance relative to their peers),[133] and enacted sexism in hiring.[134] And the same distorting effects have infected scientific work for decades,[135] skewing even the results of clinical trials of novel, medical treatments.[136] Moreover, forensic practitioners are “not exempt from those cognitive biases that all interpreters of data and information face.”[137]  In fact, biasing contextual information (even when mundane) has been documented to cause serious mistakes and misidentifications across a wealth of forensic disciplines including the use of dogs for scent detection,[138] forensic anthropology,[139] arson investigation,[140] handwriting comparison,[141] hair comparison,[142] bite mark analysis,[143] bloodstain pattern analysis,[144] fingerprint comparison,[145] and even DNA.[146]

Because cognitive bias occurs at a subconscious level and “cannot be willed away,”[147] these findings in no way diminish the professionalism or virtue of scientists, both forensic and otherwise:

“contextual bias is by no means limited to cases of misconduct or bad intent. Rather, exposure to task-irrelevant information can bias the work of [examiners] who perform their job with utmost honesty and professional commitment. Moreover, the nonconscious nature of contextual bias also means that people cannot detect whether they are being influenced by it. It follows that task-irrelevant information can bias the work of [examiners] even when they earnestly and honestly believe they are operating with utmost objectivity.”[148]

Instead, experts (given the shortcuts in reasoning and perception that their experience affords them) may actually be at a greater risk of succumbing to cognitive bias (again unintentionally) than non-experts.[149] Thus the broader scientific community has, without any associated shame, adopted best practices to limit the impact of cognitive bias, such as double-blind trials and exposure control of extraneous information.[150] But forensic disciplines have been slow to follow. No wonder then, that the National Academy of Sciences, after stressing that “the findings of forensic science experts are vulnerable to cognitive and contextual biases,” expressed concern over the lack of “good evidence to indicate that the forensic science community has made a sufficient effort to address the bias issue,”[151] and recommended significant research into best practices for the forensic sciences.[152] And more recently, both the National Commission on Forensic Science, PCAST, and prominent forensic experts have stressed the need to keep examiners ignorant of extraneous case information (the identity of a suspect, the fact of an arrest or confession, etc…) and other biasing contexts.[153]

            But on theme, the discipline of firearms examination has failed, again unlike its peer disciplines, to conduct rigorous research into the effects of cognitive bias on its practitioners or to develop procedures to minimize its corrupting influence. In fact, the one study conducted on the topic (1) failed to satisfy OSAC-which still identifies the issue as direly in need of scientific investigation,[154] (2) utilized a sample of examiners so small that even the authors concede they easily could have failed to detect the effects of cognitive bias, and (3) presented examiners with biasing contexts far less extreme than even those encountered in routine casework.[155] Yet still it recorded at least some differences between biasing and non-biasing conditions.[156] Regardless, and as explained above, nothing in the scientific literature or the realm of common sense would suggest that firearms examiners are uniquely immune from mental distortion. Rather they have themselves admitted that outside influences can corrupt their decision making.[157] What they have failed to do is respond to the risk posed by cognitive bias.

Moving beyond the typical defense refrain of bias due to examiners’ work with law enforcement (although such effects have been documented to produce significant changes in expert opinion),[158] these practitioners continue to receive task-irrelevant information---the examiner in this case, for example, was unnecessarily informed that the submitted gun had already been linked by investigators to Mr. [---] (the State’s main suspect in the shooting of Summer Moore). Moreover, they conduct their analysis with the questioned and known bullets side-by-side, prompting them to reason back and forth between the two and creating the risk that they will “see in data the patterns for which they are looking, regardless of whether the patterns are really there.”[159] And finally, examiners are generally (and in this case were) provided with only one gun and asked to match it to a recovered projectile. Such examinations, as both district courts and forensic experts have noted, are “in effect, an evidentiary ‘show-up,’ not what scientists would regard as a ‘blind’ test.”[160] Illinois courts consider such practices “inherently suggestive and not favored as a means of identification” in the eyewitness context.[161] And given the equal susceptibility of subjective expert judgments to suggestion, there is ample reason to extend that logic and question the forensic evidence in this case, especially because the threat of cognitive bias actually reaches an apex in the context of firearms examination where, as the following section will demonstrate, practitioners, “must rely on data that are somewhat ambiguous.”[162]

1. Coincidental correspondence and subclass markings blur the line between matches and non-matches.

Initially, it bears mentioning that firearms examiners begin their work behind the eight ball, given that the bullets and cartridges they examine are most often damaged and deformed due to environmental factors or impacts against surfaces. The areas suitable for comparison may therefore be limited, and the relevant features necessary to identify or exclude a gun as the source of a projectile obscured.[163] But modern manufacturing techniques (meaning even the advent of mass production and its application to firearms in the 1920s)[164] cast doubt on examiners’ ability to reliably analyze even pristine samples. Such methods have resulted in increased calibration of the tools used to produce guns and thus heightened standardization of gun surfaces: “Modern mass-production methods for tools dictate the necessity of minimizing the manufacturing steps in order to make tool production as economical as possible. When this occurs, the manufacturing process could turn out consecutively manufactured parts that would have similar surface conditions.”[165] In fact, in some circumstances, examiners struggle to identify even the class characteristics of ammunition.[166] Because “variation due to manufacturing and individual wear patterns continues to be minimized by manufacturing processes” critics have gone so far as to note broadly that “there is simply no basis for the assumption, fundamental to classic toolmark identification theory and technique, that those markings previously classed as individual characteristics…are in fact unique to a particular gun.”[167] And examiners themselves admit that “as the techniques of firearms manufacture have evolved, following mostly commercial rather than forensic arguments, [their foundational assumptions] need to be verified on a regular basis.”[168]

Moreover, as alluded to earlier, the limited data produced by the minimal research that firearms examiners have performed over the years bears out that, given advances in manufacturing, no clear line actually divides matches from non-matches. Instead, different tools, including guns, can produce marks with striking similarity equivalent to that of marks produced by the same tool, especially if an examiner has only a small surface area available to consider.[169]

[170]

Biasotti, for example, noted as early as his 1959 article comparing testfires from Smith & Wesson revolvers, that “the average percent match for bullets from the same gun is low and the perfect match for bullets from different guns is high.”[171] In fact, “the total number of matching lines [in a match] is often no higher or even less than the number which could occur as the result of chance."[172] And William Tobin, summarizing Biasotti’s work and the studies replicating it would eventually emphasize that:

“Among those publications that hint at the nature and scope of the problem, one found up to 52% matching lines in a known non-match and another only 21-24% (steel-jacketed bullets) and 36-38% (non-jacketed bullets) concordance on bullets fired from the same gun.  It has been observed that there are typically 2 and 3 times more matching striations in known non-matches (fired in different guns) than in those fired in the same gun.”[173]

 Thus examiners may ultimately err because their assumptions about the level of similarity sufficient to constitute a match actually overlap with the level of similarity possible in non-matches.[174] This is likely a frequent occurrence given that, as the charts below demonstrate, the similarity between known, non-matches may exceed not just that found in some matches, but that found commonly in matches:

Percentage of Matching Two-Dimensional Striations in Single Land Impressions 

Percentage of Matching Three-Dimensional Striations in Single Land Impressions[175]

            Additionally, researchers have more recently taken advantage of computerized databases of bullet and cartridge images to confirm the troubling overlap of similarity as between known matches and known, non-matches. One such researcher, noting that prior to the availability of databases, close non-matches could be discovered only “sporadically” during the course of casework, concluded that when he retrieved non-matches that were highly ranked as hits by databases capable of searching immense numbers of samples simultaneously, he could observe “numerous two dimensional similarities.”[176] And such misattributions were not simply the result of some computerized error: “when using a comparison microscope, these similarities are still present and it is difficult to eliminate comparisons even though we know they are from different firearms."[177] Moreover, the National Institute of Standards and Technology, in its attempt to develop an even more-nuanced database by applying advanced surface topography imaging and comparison  algorithms  to the  evaluation of  cartridge cases  similarly  found that   non-matches

often exceed matches in terms of the similarity scores assigned them [see chart on next page].

[178]

All that, however, is before mentioning the disconcerting reality that firearms examiners will struggle even to appropriately identify which marks on bullets and cartridges are actually “individual” in nature. As noted at the outset of this brief, certain manufacturing processes, as well as imperfections on any number of manufacturing tools used to make guns, may produce what examiners term “subclass characteristics.”[179] Such marks are essentially, visually indistinguishable from individual marks, but in point of fact are shared by all guns of a batch rather than any one gun: “some machining processes are capable of reproducing remarkably similar surface characteristics…on the working surfaces of many consecutively produced tools which if not recognized and properly evaluated could lead to a false identification.”[180] Thus the potential for error should be clear: if examiners confuse subclass marks from their visibly indistinguishable cousins (individual marks) they will identify a single gun as the source of marks on a bullet or cartridge when in reality tens, hundreds, or even thousands of guns from a batch[181] would have produced the same patterns. And: “The danger that misidentifications will result from confusing subclass with individual characteristics is real, not theoretical. In the 1980s, this type of confusion was discovered to have produced misidentifications of striated toolmarks.”[182] One expert has even gone so far as to say that “The most seminal, but problematic, obstacle for toolmarks examiners, however, is discerning subclass from purported ‘individual’ characteristics.”[183]

                       [184]

The images above provide examples of such marks in configurations both easily mistaken for individual markings and of sufficient quality-if they were individual-to justify calling a match.

            AFTE may dismiss subclass characteristics as rare and easily discerned by trained examiners,[185] but even its theory of identification states: “caution should be exercised distinguishing subclass characteristics from individual characteristics.”[186] And its lack of concern flies in the face of statements of prominent firearms examiners who have studied the issue as well as nearly every article on the topic contained in its own journal. One examiner notes that “the specter of subclass characteristics has loomed over the field of firearms identification for a number of years,”[187] and another has said explicitly that “the difficulty of addressing subclass characteristics is not in debate.”[188] And for good reason, given that firearms examiners have described the appearance of such marks as prevalent,[189] and, in fact, at least one forensic scientist specializing in manufacturing and metallurgy believes that “most metal forming operations generally impart characteristics of forced contact on the work piece (firearm components in this case) that are overwhelmingly subclass [rather than individual] in nature.”[190] At all events, subclass marks can appear on all the gun surfaces firearms examiners consider in their analysis,  as well as the surfaces of even unfired cartridges; and article after article, for decades, has noted that such marks will readily cause misidentifications by examiners[191] and computers alike.[192]

            Worse still, subclass markings have generally been uncovered only at random,[193] and firearms examiners have no rules or credible guideposts to help them distinguish between them and individual characteristics, or even to alert them as to when to expect subclass markings.[194] Rather, the few nuggets of advice the literature has offered practitioners (i.e. consider whether parallel striations on cartridges appear within a firing pin impression, or utilize caution when considering groove impressions but feel safe considering land impressions)[195] have been negated by later discoveries.[196] In fact, even when Ronald Nichols was challenged by a critic of the discipline during a debate in the Journal of Forensic Science to delineate whatever rules he uses to address subclass characteristics, he could impart none, returning instead to broken-record-reliance on examiner training and experience,[197] a particularly concerning response given how significantly examiners struggle with subclass marks during proficiency testing.[198] Such failings to define standards or support examiner capability with regard to subclass characteristics have led one expert to question “how toolmark trainers communicate behind closed doors with trainees to recognize the difference between subclass and individual characteristics if instructors cannot articulate such differences in published articles.”[199]

• Known Errors by Firearms Examiners in Testing & Practice Should Disturb the Court.

DNA testing has exposed the forensic pattern matching disciplines as significant contributors to wrongful conviction.[200] Given that reality, especially when combined with all the weaknesses of firearms examination already discussed throughout this motion, it should come as no surprise to the Court that, when tested, firearms examiners simply cannot accede to the lofty ideals of practical certainty they proclaim. Instead, they must concede significant variability as between examiners[201] sometimes based on factors as unscientific as the geographic region where an examiner received training[202] and extending to even the most basic tasks (such as counting and matching-up striations on surfaces).[203] In fact, in perhaps the only empirical study that has ever endeavored to track and quantify analytical variations between examiners, participants observing the same tooled surfaces and asked to count lines, match lines, and calculate a total percent of matching lines differed in their final counts by as much as 21 total lines, 23 matching lines, and 34% matching lines.[204] And it bears mentioning that when experts from other forensic pattern matching disciplines have, unlike those in the field of firearms identification, actually undertaken research into the reproducibility of findings between examiners and the repeatability of findings by the same examiner, their results have done little to inspire confidence.[205]

At bottom, if accounting for the very real costs of examiner mistakes and accepting the higher-end of their error rates, accurate identifications are not just far from practically certain, but instead, scarcely qualify as particularly common. PCAST and other scientific authorities have long noted the limited value of the proficiency testing to which firearms examiners are subjected because such tests present examiners with far simpler tasks than does actual casework, allow test-takers more time than can be allotted to casework, can be worked on collaboratively, and are declared rather than blind (meaning examiners know they are being tested and may therefore exercise greater caution as to avoid incorrect responses).[206] Thus, any error rates generated by proficiency testing represent merely “lower-bound estimates.”[207] But despite the ease of the task before them, firearms examiners opt out of responding (by answering inconclusive) more than practitioners from any other discipline (at a rate, on the most trying tests, of up to 69% and 97%).[208] When they do provide answers they misidentify ammunition[209] at disturbing levels: (1) examiners taking a European proficiency misidentified bullets and cartridges 8.2% of the time {1 in 12 cases},[210] (2) standard American proficiency tests by have recorded misidentification rates as high as 10.5%,[211] and (3) a program designed to test the performance of crime labs recorded misidentification rates of up to 28.2% {1 in 4 cases}.[212]

Moreover, these rates rise even higher when firearms examiners confront more difficult samples. One of the studies reviewed by PCAST for example, uncovered an upper bound estimate of misidentification in the equivalent of one in nineteen cases.[213] And, in the one test to present examiners with the task of ferretting out subclass marks, 26.4% of examiners failed to classify the marks as subclass features and perpetrated misidentifications-a real world equivalent of an error in over 1 of every 4 such cases considered (this despite the fact that images of the specific marks had appeared in an AFTE Journal article just two years prior to the test cautioning practitioners to exercise caution).[214] And finally, the one audit[215] of actual casework undertaken regarding firearms examiners (specifically one involving the Detroit police laboratory) reported a misidentification rate of 10% based on a finding that 29 of the 250 cases reviewed involved misidentifications.[216]

            But however disturbing those error rates appear in the abstract, the real world cost of the arrogance of firearms examiners far out shadows them. Specifically, the 29 very real victims of the Detroit lab’s errors are by no means an anomaly. Misidentifications have ravaged defendants since the earliest days of firearms examination’s use in court[217] up through as recently as the investigation into a rash of unexplained shootings over the last year on the highways of Arizona.[218] They have occurred across the full geographic span of the United States,[219] threatened the livelihood of police officers,[220] afflicted both high-profile[221] and unheard of cases,[222] infected even the labs of major metropolitan areas,[223] and left defendants unjustly languishing on death row.[224] That the most candid of firearms experts concede the reality of such errors and decry their frequency[225] should not offer this Court any comfort given their continued assertions of reliability and certainty, as well as the fact that the exposed misidentifications likely pale in number compared to those left undiscovered.[226]

1. FIREARMS EXAMINATION ENJOYS NEITHER THE RELIABILITY NOR THE GENERAL ACCEPTANCE NECESSARY FOR ADMISSABILITY.

            The preceding sections of this motion demonstrate that the field of firearms examination deserves none of the aura of infallibility that has surrounded it for decades. In fact, practitioners themselves have long recognized their discipline’s failure to quantify and calibrate its standards or to credibly gauge the ultimate reliability of its methods when applied. But while firearms examiners may remain content to elide mention of such limitations and propound indefensible conclusions, experts from the disciplines at the foundation of that forensic methodology outright reject the ultimate validity of firearms examination as a proven and acceptable science.  At this point, the authoritative chorus of credentialed specialists criticizing the discipline simply overwhelms the minimal loyalty which it retains only amongst law-enforcement-engaged, forensic practitioners. Moreover, the discipline’s negligibly demonstrated reliability, when coupled with the substantial pitfalls confronting examiners, cannot justify the undue prejudice that would accompany the exaggerated and misleading statements of certainty peddled by practitioners. Thus, this Court should bar the testimony of the State’s firearms examiner as not generally accepted under Frye, or alternatively, as substantially more prejudicial than probative.

• The General Acceptance of Firearms Examination Did Not Survive the Last Decade’s Proliferation of Attacks By the Broader Scientific Community.

            In Illinois, the State as the proponent of scientific evidence bears the burden of satisfying the threshold inquiry set forth in Frye, which permits admission “only if the methodology or scientific principle upon which the opinion is based is sufficiently established to have gained general acceptance in the particular field in which it belongs.”[227] And admittedly, a merely passing review of the discipline of firearms examination might seem to confirm that it could survive review under that standard: it has enjoyed years of use in the courts of this State, and has never been excluded under Frye in the courts of any other. But even those recent decisions by Illinois appellate courts[228] endorsing the admissibility of firearms examination simply have not accounted for the scientific consensus presently positioned and crescendoing against firearms examination, ultimately failing to evaluate the effect of, among a plethora of other sources, the PCAST’s unambiguous and scathing conclusion that the discipline runs aground even under the unexacting criteria of foundational validity.[229] Thus, despite their fresh vintage, they in no way foreclose the more robust challenge to firearms examination presented by the Defense in this matter, and ultimately leave Illinois’s law regarding the legitimacy of the discipline unsettled. Left then on its own to weigh the significance of the addition of the PCAST report to the substantial chorus of doubts about firearms examination previously raised in scientific papers (doubts which could fairly be summarized as the field’s rejection by every authoritative, legitimately-scientific source to have evaluated it), this Court should at minimum conclude that  firearms examination no longer enjoys the “unequivocal and undisputed” accord necessary to admit such evidence via judicial notice, thus necessitating a pretrial hearing on its general acceptance.[230]

             As an initial matter, the State’s response to this motion will likely begin by arguing that firearms examination evidence, given its decades of use in criminal courts,  does not qualify as “new or novel” and is therefore beyond the reach of Frye.[231] But for this Court to reflexively foreclose its inquiry on that basis, it would have to ignore the truism that today’s scientific hypothesis may turn out to be tomorrow’s claim of a flat earth (or as PCAST frames the issue: “from a purely scientific standpoint, the resolution is clear. When new facts falsify old assumptions, courts should not be obliged to defer to past precedents: they should look afresh at the scientific issues.”)[232] And nothing from a legal rather than scientific standpoint would require this Court to walk the absurd path of turning a blind eye to the evolution of ideas central to the nature and purpose of science. Instead, myriad courts have described Frye as a test that can and must adapt to advances in understanding, acknowledging that “scientific developments may require that the court consider afresh whether a particular proffer meets the [Frye standard].”[233] Thus, by way of example, was bullet lead analysis eventually excluded under Frye, despite routine and decades-long admission in criminal trials, when scientific revelations, parallel to those facing firearms examination, undercut its reliability and validity.[234]

And in that regard, Illinois’s approach to the issue of novelty proves to be no outlier, incorporating the same recognition that “constant scientific advances in our modern era may affect our inquiry as to the novelty of a particular method.”[235] It was that reasoning that led the Illinois Supreme Court in McKown to conclude that HGN evidence, although used for years by police, qualified as novel “given the history of legal challenges to [its] admissibility … and the fact that a Frye hearing ha[d] never been held in Illinois on th[e] matter,”[236] as well as the First district to demand a hearing on the Gudjonsson Suggestibility Scale based on evolving scientific standards, holding that “acceptance of the GSS in the field of forensic psychology was unsettled despite its almost 30-year existence and, thus, remained a novel scientific methodology.”[237] And it is that same reasoning that should compel this Court to advance to the second stage of Frye and evaluate the general acceptance of firearms examination evidence. Not only has the field of firearms examination, like HGN testing at the time of McKown, never been vetted at a pretrial hearing in Illinois,[238] but it has also been subjected over the last decade to substantial scientific and legal challenges, challenges made materially more robust and impregnable by the PCAST report’s authority, scope of review, and direct rejection of the underlying validity of the discipline. No court (in Illinois or otherwise) has vetted the full scope of said challenges in the context of a hearing, and as such (even if paradoxically given the discipline’s age), firearms examination qualifies as new and novel for purposes of Frye.[239]

Turning then to the central issue of admissibility: the Frye standard aims to exclude methods “that undeservedly create a perception of certainty when the basis for the evidence or opinion is actually invalid.”[240]And even before delving into the test’s finer points, it should seem plain that firearms examination violates that animating spirit considering that its practitioners claim unsupported levels of certainty foreign to even the forensic gold standard of DNA, without any of that more rigorously-tested discipline’s empirical foundations. But more than that, firearms examination also runs afoul of Frye’s specific requirement of general acceptance, because although the field is not bereft of adherents (firearms practitioners themselves), experts from the relevant scientific communities of metrology and metallurgy have wholly rejected its methodology. Although the State need not demonstrate “universal acceptance,” that divide deprives the field of the “consensus” as opposed to “controversy” required for admissibility, and demonstrates that firearms examination possesses at best only “dubious validity.”[241] At all events, this Court could simply deny this motion and admit the testimony of the State’s expert only if “unequivocal and undisputed prior judicial decisions or technical writings on the subject” so dispose of the matter as to permit a determination via judicial notice.[242] But such a claim is belied on the scientific front by the PCAST report and other writings critical of firearms examination, and on the legal side by the simple truth that the judiciary has never had the opportunity to consider the full scope of those criticisms. Thus, at minimum, a pretrial hearing would be necessary to resolve the question of general acceptance.[243]

Although, as just stated, both scientific writings and prior judicial decisions may contribute to this Court’s understanding of general acceptance, Frye’s deference to the self-reflection and self-regulation of science clearly privileges the former.[244] And in evaluating that primary determinative issue it is vital to account for all the available and pertinent literature from the scientific fields on which firearms examination is based.[245] In fact, this Court should place a particular emphasis on ensuring that the relevant scientific community it considers extends beyond merely practitioners of firearms examination, thereby allowing room for disagreement.[246] Fortunately, authorities from the broader scientific community and practitioners themselves agree that the relevant foundational fields underlying firearms examination are metrology (the science and application of measuring features) and metallurgy/materials engineering.[247]

And writings from those fields overwhelmingly skew in favor of excluding firearms examination evidence. Given the exhaustive review specifically of firearms examination by PCAST’s metrologists (recall that it involved analysis of more than 400 bibliographic sources as well as exchanges with practicing firearms examiners) and that panel’s direct mandate to evaluate foundational validity, its indictment of the discipline obviously stands as the strongest indicator that firearms examination enjoys no general acceptance by its parent fields, much less the undisputed and unequivocal endorsement in technical writings necessary to avoid a full pretrial hearing. But the daunting coalition that has published about and coalesced around the nigh-unanimous view that firearms examination lacks the adequate empirical basis necessary for scientific acceptance wholly eclipses any support the discipline still retains, and includes, among others: (1) the metallurgists, materials engineers, and metrologists of the 2008 and 2009 NAS panels; (2) leading statisticians; (3) academics from all walks of the scientific community publishing in the most-prestigious of journals; (4) at least a majority quotient of the OSAC committee for firearms and toolmarks[248] as well as the full NCFS, (5) experts at the intersection of law and science, like Adina Schwartz, (6) a former metallurgist with the FBI, William Tobin, who has spent years analyzing the studies and methods of the discipline, and (7) the forensic organizations that have endorsed PCAST. The existence of such authoritative and diverse consensus, and the reality that many firearms examiners agree that their discipline needs but lacks a legitimate empirical foundation, demonstrate that the field is awash in controversy sufficient to disrupt judicial notice of its general acceptance.

But more than that, this Court could well dispense with a hearing and simply exclude the State’s firearms expert, because to rebut the authorities just cited and carry its burden of showing general acceptance, the State could and likely will present only the patently self-serving testimony of firearms examination practitioners. And permitting firearms examiners to establish the general acceptance of their own field would undercut the “scrutiny of the marketplace of general scientific opinion” central to Frye:

“To allow general scientific acceptance to be established on the testimony alone of witnesses whose livelihood is intimately connected with a new technique would eliminate the safeguard of scientific community approval implicit in the general scientific acceptance test. Scientific community approval is absent where those who have developed and whose reputation and livelihood depends on use of the new technique alone certify, in effect self-certify, the validity of the technique.”[249]

Instead, Frye’s criteria “requires the testimony of impartial experts or scientists,”[250] a truism recognized by the McKown court’s focus on objectivity of pundits in its evaluation of HGN evidence (as well as its conclusion that general acceptance must not turn on “the testimony or writings of law enforcement officers or agencies”),[251] and the First District’s holding that a discipline’s use in crime labs cannot “justify admission of evidence in the face of a bona fide scientific dispute.”[252] As law enforcement professionals whose very livelihoods would dissipate if their discipline were to be rejected by the courts, firearms examiners cannot qualify as objective experts with regard to general acceptance. And AFTE, as a trade organization representing the interests of such examiners, finds no stronger footing.[253] As such, unless the State in its response presents the opinion of a credentialed metrologist or metallurgist who accepts the validity of firearms examination (and the defense is aware of none) a pretrial hearing would simply expend judicial resources without avoiding the inevitability of excluding the State’s firearms expert.[254]

Moreover, prior judicial decisions do not diverge as sharply from the scientific opinions cited above as the lack of any case ultimately excluding firearms examination evidence might suggest, and thus provide insufficient support to warrant admission. True, the earliest of such cases permitted firearms examination testimony without pause.[255] But importantly, those bygone precedents arose before the past decade’s revelations regarding the weaknesses of forensic methods, in periods when such disciplines were “assumed rather than established to be foundationally valid.”[256] In contrast, the weight of recent judicial authority (of which the Robinson and Rodriguez decisions considered only a small percentage)[257] has (1) acknowledged serious deficiencies with regard to the reliability of and empirical support underlying firearms examination, and (2) almost uniformly imposed stringent limits on the scope of firearms examination evidence.[258] Those restrictions have often been so severe that they have approximated outright exclusion by effectively sapping firearms examination evidence of its probative value,[259] and have been justified by caustic rhetoric normally foreign to the niceties of legal discourse, with one judge going so far as to describe the probative value of firearms examination as akin to “the vision of a psychic” and emphasizing that  “it reflects nothing more than the individual's foundationless faith in what he believes to be true. This is not evidence on which we can in good conscience rely, particularly in criminal cases, where we demand proof—real proof—beyond a reasonable doubt, precisely because the stakes are so high.”[260]

Admittedly, however, none of the cases just referenced eclipse, in terms of their ultimate admissibility decisions, those reviewed by the Robinson or Rodriguez courts (although the sheer number of such decisions and their consistency across multiple forensic disciplines that share the limitations associated with firearms examination may well have troubled the First District had the full gamut of cases cited herein been available to or discovered by that appellate body).[261] But mere “[r]eliance upon other courts' opinions can be problematic,” because “[u]nless the question of general acceptance has been thoroughly and thoughtfully litigated in the previous cases, reliance on judicial practice is a hollow ritual…[and] could become a yellow brick road for judicial acceptance of bogus or at least unvalidated scientific theories or techniques.”[262]Thus, the Illinois Supreme Court in New refused to take judicial notice of the general acceptance of a paraphilic diagnosis in part because previous cases that had considered the issue had not had the opportunity to account for the most recent version of the DSM manual,[263] and the Fourth District in Kirk declined to follow past cases that had admitted HGN evidence given the existence of scientific articles published since they were decided as well as the lack of robust hearings supporting admission.[264] In fact, even the Robinson decision stressed that it could sanction the admission of firearms examination evidence absent a Frye hearing only because other courts had already extensively vetted all of the sources the defendant in that matter wished to use to challenge the discipline.[265]

But there’s the rub: no caselaw currently controls or settles the law regarding firearms examination evidence because the Robinson and Rodriguez cases reviewed the effect, essentially, of only the NAS reports, they did not (nor did those decisions they relied upon) confront the far more developed record, including the PCAST report amongst other materials, presented by the Defense in this motion.[266] In fact, in spite of often robust litigation concerning firearms examination (and the many doubts about its foundation and validity such litigation consistently produced) many past cases involving the discipline simply and ultimately cited back to even older cases from times before the weaknesses of the field surfaced to support admission.  And given the only-recent publication of the PCAST report, only one court has even evaluated its effect on the calculus of general acceptance in the setting of a Frye hearing.[267] Because that report (by evaluating thousands of sources, clearly defining the scientific criteria for validity, and thoroughly applying said criteria to firearms examination) went further than any prior critique of the discipline, cases that precede its publication have, to put it plainly, been thoroughly undermined. Moreover, all that is without accounting for the wide swath of materials presented in this motion that, although available at the time of previous decisions admitting firearms examination, have never been reviewed by judicial authorities.

This Court could only guess at whether that resulted from the oversights of other defense counsel, judges, or some combination of the two. But ultimately it should not deprive the Defendant, having gathered together such writings, or the general public, with its vested interest in the marriage of criminal convictions only to legitimate science, of the opportunity to gauge the validity of firearms examination against the most robust case possible. As the old judge’s adage goes “change the facts change the ruling.” Well, the Defense has changed the facts by developing a scathing record against firearms examination, and this Court should not resolve that challenge, one based on newly published materials, by reference solely to cases that never weighed the significance of said sources. To do so would constitute precisely the type of hollow ritual of “grandfathering in irrationality”[268] warned against time and time again by Illinois appellate courts. Instead, this Court should rise to the occasion and gauge thoroughly (in the context of a hearing, and with the benefit of expert testimony) the present legitimacy of firearms examination evidence.

• The Limited Probative Value of Firearms Examination Evidence Does Not Outweigh its Substantial & Unfair Prejudicial Effect.

Even if this Court remains convinced that its hands are tied under Frye, however, it should not dismiss the concerns regarding the validity of firearms examination raised throughout this motion as mere issues of weight as opposed to admissibility. Rather, the Illinois Supreme Court (along with other courts and commentators across the country) has emphasized that scientific evidence, even if deemed generally accepted, must still satisfy the strictures of Rule 403, which requires exclusion of evidence, “if its probative value is substantially outweighed by the danger of unfair prejudice, confusion of the issues, or misleading the jury.” [269] Thus, the court in Murray, after noting that “expert testimony, because of its powerful potential to mislead or confuse juries can be excluded under Rule 403 even if it would otherwise meet the standard for admissibility,” barred the testimony of an epidemiologist who, although claiming to utilize generally-accepted Weight of Evidence Analysis, in fact had distorted research conclusions while picking and choosing between studies to support her opinion.[270] And similarly, the Second District in Floyd, citing concerns about the unreliability of an expert’s underlying assumptions, refused to abandon for vetting by the adversarial process the expert’s testimony regarding a (normally generally accepted) retrograde extrapolation calculation.[271] Additional examples abound of courts excluding experts from generally-accepted disciplines as unreliable,[272] and one court has even relied specifically on Rule 403 to limit the testimony of a firearms examiner.[273]

In fact, both courts and commentators have noted that expert testimony actually requires heightened, rather than diminished, vigor with regard to applying Rule 403. The United States Supreme Court emphasized that "[e]xpert evidence can be both powerful and quite misleading because of the difficulty in evaluating it. Because of this risk, the judge … under Rule 403 of the present rules exercises more control over experts than over lay witnesses."[274] And other scholars from the judiciary have similarly noted the special deference juries grant expert testimony as well as the real possibility that even the best cross-examination may be insufficient to dispel the reverence afforded forensic experts. The warnings of these courts and scholars have come in the form of exhortations to “guard against complacency”[275] in admitting forensic testimony, the observation that “[b]ecause of the ‘talismanic significance’ and ‘authoritative quality’ that surrounds expert opinions, the court must be vigilant to prevent jury confusion caused by misleading testimony,”[276] and the recognition that “cross-examination is a minimal constitutional safeguard … But it is far from adequate."[277] On the home front, the Illinois Supreme Court has even highlighted the “natural inclination of the jury to equate science with truth and, therefore, accord undue significance to any evidence labeled scientific.”[278]

Moreover, the legal opinions just described find added support in scientific findings about the perceptions of jurors. The PCAST report, for example, concludes that “[c]ompared to many types of expert testimony, testimony based on forensic feature-comparison methods poses unique dangers of misleading jurors,” because “[t]he vast majority of jurors have no independent ability to interpret the probative value of results based on the detection, comparison, and frequency of scientific evidence…they would be completely dependent on expert statements garbed in the mantle of science.” [279] And a significant quotient of research underlies the PCAST’s conclusion, bearing out the troubling reality that neither cross-examination nor the conflicting opinion of a defense expert would likely be effective in exposing the weaknesses of firearms examination, and meaningfully impacting a jury’s perception of the strength of the State’s forensic evidence.[280]

Study after study shows that jurors struggle to assess the real value of forensic testimony, willingly defer to experts, and grossly underestimate the potential for misidentification.[281] Moreover, the same studies demonstrate that even robust and pointed cross-examination that is well-designed to expose weaknesses in forensic practitioners’ methods has little to no power to do so.[282] In fact, expert conclusions become even less susceptible to moderation by the adversarial process (and by the way, also impart the lowest levels of understanding to jurors) when premised on years of experience and framed in unshakeable terms like “match” (as of course is the case with firearms examiners).[283] Finally, if this Court fails to act as a gatekeeper, juror misconceptions about the firearms examination performed in this case may well persist even in the face of testimony from an expert favorable to the defense.[284] No wonder then that these researchers have themselves concluded that their “results should give pause to anyone who believes the adversarial process will always undo the effects of weak expert testimony.”[285]

Turning then to the specifics of Rule 403’s application to firearms examination, we find a methodology that, despite the absurd and indefensible claims of its practitioners to practical certainty in their conclusions, actually offers little in the way of probative value. Illinois courts have long tied the probative value of scientific evidence to its reliability.[286] And as previous sections of this motion have highlighted, issues like cognitive bias as well as confusion based on coincidental similarity and subclass characteristics have resulted in myriad errors by firearms examiners (both when tested and in real-world casework), thus calling into serious question the discipline’s reliability and validity. But more than that, “without appropriate empirical measurement of a method’s accuracy, the fact that two samples in a particular case show similar features [i.e. match] has no probative value,”[287] or as one court has phrased the matter: “Without [a] probability assessment, the jury does not know what to make of the fact that the patterns match: the jury does not know whether the patterns are as common as pictures with two eyes, or as unique as the Mona Lisa.”[288] And because the discipline of firearms examination has not subjected itself to rigorous empirical testing (recall that the PCAST report concluded that only one suitable study has ever been performed) the State’s firearms expert simply could not, through other than rank and disallowed conjecture,[289] establish that the probabilities of an accurate match are sufficient even to qualify as relevant, much less amply probative.[290]

The scientific studies cited above demonstrate, however, that in contrast to such minimal probative value, the testimony of the State’s firearms expert would impose a significant risk of unfair prejudice and juror confusion because “juries will likely incorrectly attach meaning to the observation” of an alleged match.[291] And that troubling possibility is only intensified given that (1) said expert’s unwarranted claims of certainty will elide mention of the various pitfalls that diminish the reliability and precision of firearms examination, and (2) cross examination will likely prove ineffective as a means of educating the jury about the weaknesses of the State’s forensic evidence. At bottom, the undeserved aura of infallibility cloaking firearms examination (especially when unfairly buttressed by the extreme conclusions of practitioners) is simply not amenable to correction by the normal workings of the adversarial process.[292] To prevent juror confusion and unfair prejudice, as well as to preserve the integrity of a trial’s truth-seeking function, this Court should therefore utilize its discretion under Rule 403, and bar testimony regarding firearms examination as substantially more prejudicial than probative.

1. CONCLUSION

            For too many years, the field of firearms examination and its sister forensic disciplines have resisted introspection, refused to develop rigorous criteria, and grossly overstated the probative value of findings. Moreover, bolstered by judicial decisions admitting the testimony of practitioners without conducting searching inquiries or demanding foundational validity, forensic communities have dismissed research that might uncover limitations as a “net loss,”[293] resulting in the present reality that “clinical laboratories must meet higher standards to be allowed to diagnose strep throat than forensic labs must meet to put a defendant on death row.”[294] But reform is coming to forensic science and likewise to courts that would ignore scientific shortcomings, its inevitability bolstered by the consistency and authority of the critics positioned against foundationally-lacking forensics as well as by the wholesale failure of the community of forensic practitioners to offer any legitimate rebuttal to their attacks.

In fact, at every opportunity afforded them over the last decade, prominent scholars have balked at the lack of validation, questionable research practices, and overblown conclusions infecting firearms examination and similar pattern matching disciplines. The PCAST report represents not simply another voice added to the fray, but the culmination of all those years of growing scientific discontent distilled into a straightforward, unequivocal, and authoritative excommunication of firearms examination from the realm of valid and reliable methodologies. In the face of such overwhelming evidence about the limitations of firearms examination, criminal courts can acquiesce to its admission only for so long. And the public simply cannot continue to bear the cost of delay as measured in innocents wrongfully convicted and the persistent harm perpetrated by the guilty left free. This Court’s decision must therefore shoulder far more than just the already-weighty burden of Mr. [---]’s fate. At stake instead is the very respect the public will accord the courts of Illinois, for as one judge has already framed the issue: “Why trust a justice system that imprisons and even executes people based on junk science?”[295] The voices of dissent and the concerns of reliability documented throughout this motion with regard to firearms examination amply support a decision to exclude the testimony of the State’s examiners under Frye or Rule 403. Thus, the only question that remains is whether this Court will have the courage to rise to the historical moment, carve out a path to progress, and cast out voodoo science as having no place in any hall with claims to justice.[296]

Wherefore, Mr. [---]  requests that this Court issue the following orders:

• Exclude the testimony of the State’s firearms examiners under Frye as not generally accepted by the relevant scientific community.

• Exclude the testimony of the State’s firearms examiners under Rule 403 as substantially more prejudicial than probative.

• Conduct a pretrial hearing to asses both the general acceptance and reliability of firearms examination pursuant to Frye and Rule 403.

• Limit the testimony of the State’s firearms examiner by precluding conclusions phrased in terms of “practical certainty,” instead permitting only testimony that the firearms examiner could not exclude any particular gun as the source of any particular bullet or cartridge casing.

                                                                        Respectfully Submitted,

Attorney for Mr. [---]

[1] National Academy of Sciences, “Strengthening Forensic Science in the United States: A Path Forward,” National Academies Press, at 7 (2009), available at https://www.ncjrs.gov/pdffiles1/nij/grants/228091.pdf.

[2] President’s Council of Advisors on Science & Technology, “Forensic Science in the Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods,” at 111 (Sept. 20, 2016), available at https://www.whitehouse.gov/administration/eop/ostp/pcast/docsreports.

[3] Robert Thompson, “Firearms Identification in the Forensic Laboratory,” at 7 (2010), available at http://www.crime-scene-investigator.net/firearm-Identification-in-the-forensic-laboratory.pdf.

[4] Id. at 7-8.

[5] Top Left: http://www.firearmsid.com/a_bulletidrifling.htm; Top Right: Thompson, “Firearms Identification in the Forensic Laboratory,” at 24; Bottom Center: David Kopel & H. Sterling Burnett, “Ballistic Imaging Not Ready for Prime Time,” National Center for Policy Analysis (April 3, 2003), available at http://www.ncpa.org/pub/bg160?pg=2.

[6] In situations where firearms examiners are presented with only crime scene samples and no suspect gun they may also utilize a database system (IBIS/NIBIN) to attempt and associate the crime scene evidence turned over to them to cold cases. See Thompson, “Firearms Identification in the Forensic Laboratory,” at 29-30. 

[7] Id. at 8.

[8] Id. at 8-9; AFTE, “Theory of Identification As it Relates to Toolmarks,” 30 AFTE J 86 (1998).

[9] AFTE, “The Response of the Association of Firearms & Tool Mark Examiners to the National Academy of Sciences 2008 Report Assessing the Feasibility, Accuracy, & Technical Capacity of a National Ballistics Database,” 40 AFTE J 234, 237 (2008).

[10] AFTE, “Theory of Identification As it Relates to Toolmarks,” 30 AFTE J 86 (1998).

[11] Id.

[12] Id.

[13] Ronald G. Nichols, “The Scientific Foundations of the Firearms & Toolmark Identification: Responding to Recent Challenges” CAC News, 2nd Quarter, at 26 (2006), available at http://www.forensicdna.com/assets/2ndq06.pdf.

[14] AFTE, “Theory of Identification As it Relates to Toolmarks,” 30 AFTE J 86 (1998).

[15] PCAST, “Forensic Science in the Criminal Courts,” at 5.

[16] Id. at 5-6.

[17] Id. at 111.

[18] Id. at 46; see also Jennifer L. Mnookin et al., “The Need for a Research Culture in the Forensic Sciences,” 58 UCLA L. Rev. 725, 732 (2011) (emphasizing the centrality of research culture to valid scientific endeavors meaning “a focus on empirical evidence, transparency, and a consistently critical and reflective perspective on claims of knowledge”).

[19] National Research Council. “Ballistic Imaging,” The National Academies Press (2008), available at https://www.nap.edu/catalog/12162/ballistic-imaging; National Academy of Sciences, “Strengthening Forensic Science in the United States: A Path Forward,” National Academies Press,” (2009).

[20] See e.g., Melendez-Diaz v. Massachusetts, 557 U.S. 305 (2009) (relying on The NAS Report for the finding that “serious deficiencies have been found in the forensic evidence used at criminal trials”); United States v. Mouzone, 696 F. Supp. 2d 536, 570 (D. Maryland 2009)  (“Suffice it to say that the concerns expressed by the NRC ought to be heeded by courts in the future regarding the limits of toolmark identification evidence, and courts should guard against complacency in admitting it just because, to date, no federal court has failed to do so”).

[21] See PCAST, “Forensic Science in Criminal Courts,” at 144.

[22] http://www.nasonline.org/about-nas/mission/; see also David Kaye, “The good, the bad, the ugly: The NAS report on strengthening forensic science in America,” 50 Science & Justice 8, 8-9 (2010).

[23] National Research Council. “Ballistic Imaging,” at 2.

[24] Id. at 3.

[25] Id. at 81.

[26] Id. at 3.

[27] Id. at 82.

[28] Id.

[29] Id. at 82, 85.

[30] National Academy of Sciences, “Strengthening Forensic Science in the United States: A Path Forward,” National Academies Press,” at xix.

[31] The Honorable Harry T. Edwards, “The National Academy of Sciences Report on Forensic Sciences: What it Means for the Bench & Bar,” Presentation to the Superior Court of DC, at 1-2 (2010).

[32] Id. at 1 (noting also that “I can now say that the substance of the Committee’s Report was really not hard to write. The problems that plague the forensic science community have been well understood for quite some time by thoughtful and skilled forensic professionals”).

[33] National Academy of Sciences, “Strengthening Forensic Science in the United States: A Path Forward,” National Academies Press, at 128 (2009).

[34] Id. at 7.

[35] Id. at 153-54.

[36] Id. at 155.

[37] Id. at 154.

[38] Id. at 107-108.

[39] Id. at 154 (“studies should be performed to make the process of individualization more precise and repeatable.”)

[40] Michael J. Saks & Jonathan L. Koehler, “The Coming Paradigm Shift in Forensic Identification Science,” 309 Science 892 (2005) (saying also that “Scientists have begun to question the core assumptions of numerous forensic sciences” & decrying forensic sciences for their lack of transparency and scientific rigor).

[41] Donald Kennedy, “Forensic Science: Oxymoron?” 302 Science 1625 (2003) (“…the analysis of bullet markings exemplifies kinds of ‘scientific’ evidence whose reliability may be exaggerated when presented to a jury”); David L. Faigman, “Is Science Different for Lawyers” 297 Science 339 (2002) (concluding that although research into forensic identification sciences would be easily accomplished, little if any has actually been conducted); Donald Kennedy & Richard A. Merrill, “Assessing Forensic Science” 20 Issues in Sci. & Tech. 1 (2003) (emphasizing that “the scientific foundation of many common forensic science techniques may be open to question” because they “have not undergone the type of extensive testing and verification that is the hallmark of science elsewhere”).

[42] NATURE Editorial Board, “Science in Court” 464 Nature 325 (2010).

[43] See American Statistical Association, “ASA Board Policy Statement on Forensic Science Reform,” (April 17, 2010), available at http://www.amstat.org/asa/files/pdfs/POL-Forensic_Science_Endorsement.pdf; Karen Kafadar, “Statistical Issues in Assessing Forensic Evidence,” Technical Report 11-01, Dep’t of Statistics-Indiana University (April 21, 2011), available at http://www.stat.indiana.edu/files/TR/TR-11-01.pdf; Alicia Carriquiry, “Declaration in Support of Defendant Joseph Blacknell’s Motion to Exclude Firearms & Toolmark Identification evidence Or, In the Alternative, for a Kelly Hearing,” (Nov. 21, 2011) (“In my opinion as a statistician with many years of experience, the studies that have been carried out and the (scant) data that have been collected in no way support the methods or the conclusions that are routinely drawn by firearms examiners”), available at https://afte.org/uploads/documents/swggun-cavblacknell-carriquiry.pdf.

[44] Paul C. Gianelli, “Forensic Science: Under the Microscope,” 34 Ohio N.U.L. Rev. 315 (2008) (noting an unfulfilled “need for comprehensive regulation of crime laboratories…there is a critical need for independent scientific validation of forensic techniques.”); Adina Schwartz, “A Systemic Challenge to the Reliability & Admissibility of Firearms & Toolmark Identification,” 6 Colum. Sci. & Tech. L. Rev. 2 (2005) (reviewing literature on firearms examination and concluding that such evidence fails to meet standards of admissibility).

[45] John Thorton, “The General Assumptions & Rationale of Forensic Identification,” In Modern Scientific Evidence: The Law & Science of Expert Testimony, at 3 (1997).

[46] Id. at 36.

[47] Thomas L. Bohan, “President’s Editorial- Strengthening Forensic Science: A Way Station on the Journey to Justice” 55 J. Forensic Sci. 5, 6 (2010).

[48] Mnookin, “The Need for a Research Culture in the Forensic Sciences,” at 732-35, 778.

[49] See William Tobin & Peter Blau, “Hypothesis Testing of the Critical Underlying Premise of Discernible Uniqueness in Firearms-Toolmark Forensic Practice,” 53 Jurimetrics 121 (2013); William A. Tobin & Clifford Spiegelman, “Analysis of Experiments in Forensic Firearms/Toolmark Practice Offered as Support for Low Rates of Practice Error & Claims of Inferential Certainty,” 12 L., Prob., & Risk 115 (2013); William A. Tobin, “Affidavit in Virginia v. Macumber,” (2011), available at https://afte.org/uploads/documents/swggun-azvmacumber-tobin.pdf.

[50] William Tobin, David Sheets, & Clifford Spiegelman, “Absence of Statistical and Scientific Ethos: The Common Denominator in Deficient Forensic Practices,” 4 Statistics & Public Policy 1, at 8 (2017).

[51] Id.

[52] Id. at 1, 10.

[53] A group of hundreds of forensic professionals charged by the National Institute of Standards and Technology to “to create a sustainable organizational infrastructure that produces consensus documentary standards and guidelines to improve quality and consistency of work in the forensic science community.” See “NIST Organization of Scientific Area Committees Roles and Responsibilities,” (2016), available at http://www.nist.gov/forensics/osacroles.cfm.

[54] See http://www.nist.gov/forensics/osac/upload/FATM-Research-Needs-Assessment_Class-and-individual-marks.pdf; http://www.nist.gov/forensics/osac/upload/FATM-Research-Needs-Assessment_Blackbox.pdf.

[55] NCFS, “Technical Merit Evaluation of Forensic Science Methods & Practices,” at 2 (2016), available at https://www.justice.gov/ncfs/file/881796/download.

[56] Id.

[57] NAS, “Strengthening Forensic Science in the United States” at 5.

[58] See AFTE, “The Response of the Association of Firearm and Tool Mark Examiners to the February 2009 National Academy of Science Report,” 41 AFTE J 204, 206 (2009); AFTE, “Comments on NCFS Views Document: ‘Scientific Literature in Support of Forensic Science and Practice,’” 47 AFTE J 109, 111 (2015).

[59] PCAST, “Forensic Science in Criminal Courts,” at x.

[60] Id. at 2, 23, 67.

[61] Id. at 144.

[62] https://obamawhitehouse.archives.gov/administration/eop/ostp/pcast/about/members.

[63] https://obamawhitehouse.archives.gov/administration/eop/ostp/pcast/docsreports.

[64] https://news.aafs.org/policy-statements/presidents-council-of-advisors-on-science-and-technology-pcast-report/.

[65] The Forensic Institute, “Commentary on PCAST 2016,” (last visited Jan. 19, 2017), available at http://www.theforensicinstitute.com/news-articles/views-and-opinions/commentary-of-pcast-2016.

[66] Kozinski, “Rejecting Voodoo Science in the Courtroom,” Wall Street Journal (Sept. 19, 2016) see also Motorola Inc. v. Murray, 147 A.3d 751, 759 (D.C. 2016) (J. Easterly concurring) (“Fortunately, in assessing the admissibility of forensic expert testimony, courts will have the aid of landmark reports [including PCAST’s]… These reports provide information about best practices for scientific testing, an objective yardstick against which proffered forensic evidence can be measured, as well as critiques of particular types of forensic evidence”).

[67] PCAST, “Forensic Science in Criminal Courts,” at 4, 44-46.

[68] Id. at 26.

[69] Id. at 32.

[70] Id. at 5.

[71] Id. at 46.

[72] Id. at 49.

[73] Id. at 52-53; see also PCAST, “An Addendum to the PCAST Report on Forensic Science in Criminal Courts,” at 2 & 4(Jan. 6, 2017) (“While scientists may debate the precise design of a study, there is no room for debate about the absolute requirement for empirical testing” & “there is no hierarchy in which empirical evidence is simply the best way to establish validity…in science, empirical testing is the only way to establish validity”).

[74] Id. at 62. It also bears mentioning that PCAST identified multiple forensic groups that would share its views, see id. at 63-65 & 105.

[75] See NAS, “Strengthening Forensic Science in the United States” at 154 (“Toolmark and firearm analysis suffers from the same limitations discussed above for impression evidence”).

[76] See PCAST, “Forensic Science in Criminal Courts,” at 101 (“Based largely on two recent appropriately designed black-box studies, PCAST finds that latent fingerprint analysis is a foundationally valid subjective methodology—albeit with a false positive rate that is substantial and is likely to be higher than expected by many jurors based on longstanding claims about the infallibility of fingerprint analysis.”)

[77] Id. at 60 (“[i]t declares that an examiner may state that two toolmarks have a ‘common origin’ when their features are in ‘sufficient agreement.’ It then defines ‘sufficient agreement’ as occurring when the examiner considers it a ‘practical impossibility’ that the toolmarks have different origins”). Importantly, PCAST provided the FBI lab with an opportunity to defend the theory and rebut their understanding of its circularity. It was unable to do, and merely restated the theory itself.

[78] Id. at 110-111.

[79] Id.

[80] Id.

[81] See David P. Baldwin, et al., “A Study of False-Positive & False-Negative Cartridge Case Comparisons” AMES Technical Report #IS-5207 (April 7, 2014) (the study notes that 5 of 218 participants [roughly 2%] committed 20 of the 22 total errors in the study, meaning those examiners erred 20 out of 50 times they considered evidence. The authors describe these results as demonstrating “a highly heterogeneous mixture of a few examiners with higher rates and most examiners with much lower rates” but identified no way to discriminate between the two.)

[82] PCAST, “Forensic Science in Criminal Courts,” at 111.

[83] Id. at 110. Before this Court dismisses that error rate as reasonable it should consider that PCAST, from a scientific standpoint, viewed the far lower error rates associated with fingerprint comparisons (as high as 1 in 306 cases for one study or 1 in 18 cases in a second) on the whole as “substantial.” See id. at 101.

[84] Baldwin, “A Study of False-Positive & False-Negative Cartridge Case Comparisons,” at 5.

[85] PCAST, “Forensic Science in Criminal Courts,” at 52.

[86] PCAST, “Forensic Science in Criminal Courts,” at 111 (acknowledging that the AMES study “did not involve consecutively manufactured guns” and that “Actual casework may involve more complex situations (for example, many different bullets from a crime scene)”).

[87] Jonathan J. Koehler, “Forensic Science Reform in the 21st Century: A Major Conference, a Blockbuster Report, and Reasons to be Pessimistic” 9 Law, Probability, & Risk 1, 4(2010) (accusing AFTE of “recoil[ing] from the [NRC] report’s conclusion that the existing science does not support the strong claims made by firearms and tool mark examiners at trial”); Nature Editorial Board, “Science in Court” 464 Nature 325 (2010) (many practitioners have closed themselves off from any open sharing of methods and information with the academic community.”); AFTE, “Response to the PCAST Report on Forensic Science,” (Oct. 31, 2016) (citing to no specific studies overlooked by PCAST, failing to respond to its criticisms of proficiency testing, and offering no legitimate retort to descriptions of its theory as circular), available at https://afte.org/uploads/documents/AFTE-PCAST-Response.pdf.

[88] AFTE, “The Response of the Association of Firearms & Tool Mark Examiners to the National Academy of Sciences 2008 Report Assessing the Feasibility, Accuracy, & Technical Capacity of a National Ballistics Database,” 40 AFTE J 234, 241-42 (2008); Ronald Nichols, “Defending the Scientific Foundation of the Firearms & Tool Mark Identification Discipline: Responding to Recent Challenges,” 52 J. Forensic Sci. 586, 590-91 (2007); Department of Justice, “Letter in Manning v. Mississippi, 2013-DR-00491-SCT” (May 6, 2013).

[89] See e.g., PCAST, “Forensic Science in Criminal Courts,” at 19 (describing as “scientifically indefensible” claims of: “‘zero,’ ‘vanishingly small,’ ‘essentially zero,’ ‘negligible,’ ‘minimal,’ or ‘microscopic’ error rates; ‘100 percent certainty’ or proof ‘to a reasonable degree of scientific certainty;’ identification ‘to the exclusion of all other sources;’ or a chance of error so remote as to be a ‘practical impossibility.’”); NAS, “Ballistic Imaging,” at 82 (rejecting certainty statements because they “‘cloak an inherently subjective assessment of a match with an extreme probability statement that has no firm grounding and unrealistically implies an error rate of zero”); NAS, “Strengthening Forensic Science,” at 142, 184 (concluding that practitioners should abandon absolutist claims of identification in favor of “modest claims about the meaning and significance of a ‘match,’”  as well as that “the concept of ‘uniquely associated with’ must be replaced with probabilistic association, and other sources of the crime scene evidence cannot be completely discounted”); NCFS, “Views of the Commission Regarding Use of the Term ‘Reasonable Scientific Certainty’,” Dep’t of Justice, at 3 (2016) (emphasizing that even the lesser term reasonable scientific certainty “cloaks” conclusions with unjustified levels of rigor and respectability and would confuse or mislead jurors concerning the weight owed forensic testimony), available at https://www.justice.gov/ncfs/file/839731/download.; Working Group on Human Factors in Latent Print Analysis, “Latent Print Examination and Human Factors: Improving the Practice through a Systems Approach,” National Institute of Justice, at 72 (2012) (rejecting extreme source attribution conclusions for fingerprint examiners as scientifically deficient); Tobin, “Hypothesis Testing the Critical Underlying Premise of Discernable Uniqueness in Firearms-Toolmarks Forensic Practice,” 53 Jurimetrics at 131 (calling on firearms examiners to “curb the excesses” of their conclusions and noting that “the switch to weaker forms of source attribution (such as ‘practical certainty’) is a cosmetic change that does nothing to remedy the underlying scientific shortcomings of F/TM practice”); Simon A. Cole, “Individualization is Dead, Long Live Individualization! Reforms of Reporting Practices for Fingerprint Analysis in the United States,” 13 Law, Probability, & Risk 117 (2014) (describing terms absolute and practical certainty as redundant and noting that practical certainty is “an obscure and seemingly nonsensical value for a probability” and concluding: “neither the Theory of Identification nor the toolmark literature provides a defensible justification for claims that toolmark analysis can reduce the probability that two impressions derive from different sources to ‘practical impossibility.’”); Budowle et al., “A Perspective on Errors, Bias, & Interpretation in the Forensic Sciences and Direction for Continuing Advancement,” 54 J. Forensic Sci. 798, 804 (2009) (conceding that with the use of terms like match there may be an “unintended contribution to bias (i.e., conveying more strength than intended)” and suggesting “instead the term ‘failure to exclude,’ which may seem to some more acceptable”); John M. Collins, “Stochastics-The Real Science Behind Forensic Pattern Identification,” The Crime Lab Report  (2009), available at http://forensicfoundations.com/Resources/Documents_CLR/Archive_Legacy/2009_1124_

Stochastics.pdf. (noting the scientific irresponsibility of extreme source attribution conclusions, suggesting instead that examiners more conservatively acknowledge the subjectivity of their work and state only: “I have never seen, nor would I expect to see, this amount of similarity in … bullet striations… that came from different sources”).

[90] See SWGGUN & AFTE, “Response to 25 foundational firearm and toolmark examination questions received from the Subcommittee on Forensic Science (SoFS), Research, Development, Testing, & Evaluation Interagency Working Group (RDT&E IWG),” at 1 (2011) (describing firearms examination as an applied science derived from the discipline of metallurgy); Mnookin, “The Need for a Research Culture in Forensic Sciences,” at 760 (“Even those with a BS in forensic science or some other scientific discipline have not typically received significant training in the development of research design. Experience may provide the basis for determining what questions to ask, but most pattern identification analysts, even with entirely noble intentions, would not be qualified to design or develop sophisticated research projects to answer those questions”); Tobin, “Absence of Statistical and Scientific Ethos,” at 4, 26 (criticizing discipline for being “insular” and failing to consult more specialized scientific authorities).

[91] Thorton, “The General Assumptions & Rationale of Forensic Identification,” at 20 (1997) (“Behind every opinion rendered by a forensic scientist there is a statistical basis. We may not know what that basis is, and we may have no feasible means of developing an understanding of that basis, but it is futile to deny one exists.”); A.A. Biasotti, “The Principles of Evidence Evaluation as Applied to Firearms & Tool Mark Identification,” 9 J. Forensic Sci. 428, 430 (1964) (“each time we claim identity we are giving an opinion based on the concept of statistical probability, whether or not we like to admit it”).

[92] A.A. Biasotti, “The Principles of Evidence Evaluation as Applied to Firearms & Tool Mark Identification,” 9 J. Forensic Sci. 428, 430 (1964) (firearms examiners “lack necessary statistical data which would permit [them] to formulate precise criteria for distinguishing between identity and nonidentity with a reasonable degree of certainty.”); see also David Howitt et al., “A Calculation of the Theoretical Significance of Matched Bullets,” 53 J. Forensic Sci. 868 (2008) (“The statistical likelihood that a particular correspondence of the striae will occur by chance has, however, never been properly assessed.”)

[93] National Research Council. “Ballistic Imaging,” at 60.

[94] Ronald G. Nichols, “Firearm & Toolmark Identification Criteria: A Review of the Literature,” 42 J. Forensic Sci. 466 (1997); see also Eliot Springer, “Toolmark Examinations-A Review of its Development in the Literature,” 40 J. Forensic Sci. 964 (1995) (reaching the same conclusion after reviewing literature specific to toolmarks and noting that despite decades of acknowledging a lack of objective research and standards for the field “no one had developed any of the methods for proper laboratory use”).

[95] Id. at 467 (citing Alfred A. Biasotti, “A Statistical Study of the Individual Characteristics of Fired Bullets,” 4 J. Forensic Sci. 34 (1959)).

[96] Biasotti, “A Statistical Study of the Individual Characteristics of Fired Bullets,” at 34, 47-48.

[97] A.A. Biasotti, “The Principles of Evidence Evaluation as Applied to Firearms and Toolmark Identification,” 9 J. Forenisc Sci. 428 (1964).

[98] Id. at 429.

[99] National Research Council. “Ballistic Imaging,” The National Academies Press, at 70-72 (2008) (attacking studies of consecutively manufactured guns for their small samples sizes and failure to consider whether sequential serial numbers actually indicate consecutive manufacture); National Academy of Sciences, “Strengthening Forensic Science in the United States: A Path Forward,” National Academies Press, at 155 (criticizing so-called ten gun studies for “a heavy reliance on the subjective findings of examiners rather than on the rigorous quantification and analysis of sources of variability.”); Alfred A. Biasotti & John Murdock, “Criteria for Identification or State of the Art of Firearm & Toolmark Identification” 16 AFTE J. 16, 19 (1984) (“Such studies are subjective evaluations based on criteria of identification which cannot readily be articulated or communicated to other examiners except through photography. The information gained from such studies is therefore only of value to the examiner who conducted the study; or to the examiners trained trained or supervised by that examiner.”); William A. Tobin & Clifford Spiegelman, “Analysis of Experiments in Forensic Firearms/Toolmark Practice Offered as Support for Low Rates of Practice Error & Claims of Inferential Certainty,” 12 L., Prob., & Risk 115 (2013) (finding substantial weaknesses such as small sample sizes, and failure to adequately consider manufacturing variables infect six of the most common studies cited by firearms examiners in support of their practice); William Tobin & Peter Blau, “Hypothesis Testing of the Critical Underlying Premise of Discernible Uniqueness in Firearms-Toolmark Forensic Practice” 53 Jurimetrics 121, 139 (2013) (“As it turns out, careful analysis for both internal and external validity of the various putative validation studies that currently exist reveals them to be nothing more than very limited proficiency tests of the participating examiners. . .in addition to the fact that they do not circumstantially mirror casework.”); Mark Page et al., “Uniqueness in the Forensic Identification Sciences-Fact or Fiction?” 206 Forensic Sci. Int. 12, 15 (2011) (noting that even if legitimate studies of consecutively manufactured guns fail entirely to address the issue of random matching); D. Michael Risinger & Michael J. Saks, “A House with No Foundation” 20 Issues in Sci. & Tech. 1 (2003) (arguing that most research into forensic sciences has been highly partisan, effectively overbilling positive findings and hiding negative ones); Stephen G. Bunch, “Consecutive Matching Striation Criteria: A General Critique,” 45 J. Forensic Sci. 955, 961 (2000) (“But what about the CMS research that already has been conducted? Is it useful? An honest answer is that it is only marginally so.” & “The existing research findings are directly relevant for only particular barrel manufacturing methods, barrel lengths, barrel hardness, bullet hardness, and bullet surface materials...so far there has been a paucity of published, empirical validity research …drawing conclusions from the limited existing data is unjustified”).

[100] Nicholas D.K. Petraco et al., “Addressing the National Academy of Sciences’ Challenge: A Method for Statistical Pattern Comparison of Striated Tool Marks,” 57 J. Forensic Sci. 900 (2012).

[101] See PCAST, “Forensic Science in Criminal Courts,” at 125 (we believe that the state of forensic science would be improved if papers on the foundational validity of forensic feature-comparison methods were published in leading scientific journals rather than in forensic-science journals, where, owing to weaknesses in the research culture of the forensic science community discussed in this report, the standards for peer review are less rigorous”).

[102] Compare National Commission on Forensic Science, “Scientific Literature in Support of Forensic Science and Practice,” (2015); with AFTE Editorial Committee, “Comments on NCFS Views Document: ‘Scientific Support of Forensic Science & Practice’,” 47 AFTE J. 109 (2015); & Dominic J. Denio, “The History of the AFTE Journal, the Peer Review Process, and Daubert Issues,” 34 AFTE J 210 (2003).

[103] Adina Schwartz, “Affidavit in N.C. vs. Vonzel Adams, No. 05CRS5889,” (2010), available at https://www.fd.org/ docs/trainingmaterials/2010/MT2010/MT10_Firearm_Toolmark_ID.pdf.

[104] http://www.astm.org/DIGITAL_LIBRARY/JOURNALS/FORENSIC/jofs_subscription.html.

[105] The gross lengths necessary to obtain the AFTE Journal present a particular problem of hypocrisy for the discipline giving that it has chided critics for failing to allegedly engage thoroughly enough with source material from said journal. See e.g., Ronald G. Nichols, “The Scientific Foundations of the Firearms & Toolmark Identification: Responding to Recent Challenges” CAC News, 2nd Quarter, at 9 (2006).

[106] See Mnookin, “The Need for a Research Culture in the Forensic Sciences,” at 755-56 (“This journal therefore appears to have extremely limited dissemination beyond the members of AFTE itself; completely lacks integration with any of the voluminous networks for the production and exchange of scientific research information; and engages in peer review that is neither blind nor draws upon an extensive network of researchers. None of this is compatible with an accessible, rigorous, transparent culture of research”); Simon A. Cole, “How Do We Trust the Scientific Literature,” in Forensic Science Research and Evaluation Workshop, at 88-89 (2015) (adopting the same assessment); Itiel Dror, “Recognition & Mitigation of Cognitive Bias in Forensic Science: From Crime Scene Investigation to Forensic research & Literature,” in Forensic Science Research and Evaluation Workshop, at 57-58 (2015) (noting bias in research published in forensic journals without review by outside scientists because of motivation to “underpin and justify the existing practices”).

[107] AFTE, “The Response of the Association of Firearms & Tool Mark Examiners to the National Academy of Sciences 2008 Report Assessing the Feasibility, Accuracy, & Technical Capacity of a National Ballistics Database,” 40 AFTE J at 238; Bruce Moran, “A Report on the AFTE Theory of Identification and Range of Conclusions for Tool Mark Identification & Resulting Approaches to Casework,” 34 AFTE J 227 (2002) (the traditional ‘pattern match’ approach … relies on … the uniqueness of tool surfaces”); Alfred Biasotti & John Murdock, “The Scientific Basis of Firearms & Toolmark Identification,” at 140.

[108] See PCAST, “Forensic Science in Criminal Courts,” at 62 (“The issue is not whether objects or features differ; they surely do if one looks at a fine enough level. The issue is how well and under what circumstances examiners applying a given metrological method can reliably detect relevant differences in features to reliably identify whether they share a common source. Uniqueness studies, which focus on the properties of features themselves, can therefore never establish whether a particular method for measuring and comparing features is foundationally valid. Only empirical studies can do so.”)

[109] Michael J. Saks & Jonathan J. Koehler, “The Individualization Fallacy in Forensic Science Evidence” 61 Vand. L. Rev. 199 (2008) (explaining that “the claim of unique individuality cannot be proven with samples, especially samples that are a tiny proportion of the relevant population” and emphasizing that uniqueness “exists only in a metaphysical or rhetorical sense. It has no scientific validity, and it is sustained largely by the faulty logic that equates infrequency with uniqueness.”); Tobin, “Hypothesis Testing the Critical Underlying Premise of Discernable Uniqueness in Firearms-Toolmarks Forensic Practice,” 53 Jurimetrics at 122-23 (“The cited scholarly essays suggest that forensic individualization based on the claim of uniqueness has a scientifically indefensible conceptual foundation and is a fallacy promulgated by the forensic community.  The authors, and relevant mainstream scientists and colleagues with specialized forensic expertise with whom the authors have collaborated, agree.”); Mark Page, Jane Taylor, & Matt Blenkin, “Uniqueness in the Forensic Identification Sciences-Fact or Fiction?” 206 Forensic Sci. Int. 12, 13 (2011) (“Accumulation of positive instances simple cannot lead to a conclusion of certainty.”); John Thorton, “The General Assumptions & Rationale of Forensic Identification” at 12 (uniqueness does “not seem susceptible of rigorous proof. But the general principle cannot be substituted for a systematic and thorough investigation of a physical evidence category”).

[110] National Research Council, “Ballistic Imaging,” at 60.

[111] Page, “Uniqueness in the Forensic Identification Sciences-Fact or Fiction?” 206 Forensic Sci. Int. at 16; John Thorton, “The General Assumptions & Rationale of Forensic Identification” at 11.

[112] Alfred A. Biasotti & John Murdock, “Criteria for Identification or State of the Art of Firearm & Toolmark Identification”16 AFTE J. 16 (1984).

[113] Page, “Uniqueness in the Forensic Identification Sciences-Fact or Fiction?” 206 Forensic Sci. Int. at 15; see also “The Individualization Fallacy in Forensic Science Evidence” 61 Vand. L. Rev. at 208-09 (noting lack of science behind uniqueness concept: “various arguments have been offered on behalf of the individualization hypothesis. None are scientifically compelling. Some arguments rely on the metaphysical notion that because no two objects can be the same object, they will inevitably manifest observable differences. Some rely on appeals to venerated authority (dead members of our field said it was so), contemporary authority (living members of our field say it is so), wishful thinking (because object variability has been observed, there will always be discernible differences between any two objects), or the personal experience of practitioners (as if by doing casework on pairs of objects the nature of the population and relationships within that population are revealed). These approaches amount to nothing more than faith and intuition.”

[114] Alfred A. Biasotti, “A Statistical Study of the Individual Characteristics of Fired Bullets,” 4 J. Forensic Sci. 34, 40 (1959).

[115] Michael J. Saks, Jonathan L. Koehler, “The Coming Paradigm Shift in Forensic Identification Science,” 309 Science 892 (2005); see also Page, “Uniqueness in the Forensic Identification Sciences-Fact or Fiction?” 206 Forensic Sci. Int. at 17 (same).

[116] Simon A. Cole, “Forensic Statistics, Part II ‘Implicit Testing’: Can Casework Validate Forensic Techniques?,”46 Jurimetrics J. 117, 128 (2006).

[117] Id. at 122; see also Tobin, “Absence of Statistical and Scientific Ethos,” at 20, 21.

[118] See PCAST, “Forensic Science in Criminal Courts,” at 55 (“By contrast, ‘experience’ or ‘judgment’ cannot be used to establish the scientific validity and reliability of a metrological method, such as a forensic feature-comparison method. The frequency with which a particular pattern or set of features will be observed in different samples, which is an essential element in drawing conclusions, is not a matter of ‘judgment.’ It is an empirical matter for which only empirical evidence is relevant. Moreover, a forensic examiner’s ‘experience’ from extensive casework is not informative—because the ‘right answers’ are not typically known in casework and thus examiners cannot accurately know how often they erroneously declare matches and cannot readily hone their accuracy by learning from their mistakes in the course of casework. Importantly, good professional practices—such as the existence of professional societies, certification programs, accreditation programs, peer-reviewed articles, standardized protocols, proficiency testing, and codes of ethics—cannot substitute for actual evidence of scientific validity and reliability. Similarly, an expert’s expression of confidence based on personal professional experience or expressions of consensus among practitioners about the accuracy of their field is no substitute for error rates estimated from relevant studies. For a method to be reliable, empirical evidence of validity, as described above, is required”); Mnookin, “The Need for a Research Culture in Forensic Sciences,” at 745-48 (similarly rejecting experience and longstanding use as surrogates for appropriately conducted research).

[119] Page, “Uniqueness in the Forensic Identification Sciences-Fact or Fiction?,” 206 Forensic Sci. Int. at 13; see also William Tobin, “Hypothesis Testing of the Critical Underlying Premise of Discernible Uniqueness in Firearms-Toolmark Forensic Practice,” 53 Jurimetrics at 134 (questioning whether “cognitive retention and subsequent recollection of the spatial relationships (patterns) of the tens of millions of nondescript lines on the many thousands or millions of specimens over a lengthy period of time was humanly possible”); Simon A. Cole, “Forensic Statistics, Part II ‘Implicit Testing’: Can Casework Validate Forensic Techniques?,”46 Jurimetrics J. 117, 123 (2006) (“Casework does not focus on searching the database for exact duplicates to either the mark or the prints. Moreover, even if latent-print examiners were searching for duplicate fingerprints, the sheer number of possible combinations would render it extremely unlikely that duplicates would be found if they did exist.”)

[120] Simon A. Cole, “Forensic Statistics, Part II ‘Implicit Testing’: Can Casework Validate Forensic Techniques?,”46 Jurimetrics J. 117, 123 (2006).

[121] Page, “Uniqueness in the Forensic Identification Sciences-Fact or Fiction?,” 206 Forensic Sci. Int. at 14.

[122] Saks, “The Individualization Fallacy in Forensic Science Evidence,” 61 Vand. L. Rev. at 213.

[123] William Tobin, “Hypothesis Testing of the Critical Underlying Premise of Discernible Uniqueness in Firearms-Toolmark Forensic Practice,” 53 Jurimetrics at 134.

[124] As a final indicator of the inadequacy of training, this Court should consider PCAST’s statement that: “The mere fact that an individual has been trained in a method does not mean that the method itself is scientifically valid nor that the individual is capable of producing reliable answers when applying the method.”  PCAST, “Forensic Science in Criminal Courts, at 61.

[125] William J. Krouse, Congressional Research Service, “Gun Control Legislation,” at 8 (2012), available at https://fas.org/sgp/crs/misc/RL32842.pdf.

[126] This brief, moreover, does not even have the scope necessary to discuss other sources of potential confusion such as the effects of finishing processes and bullet velocity. See John Fowler & Dave Brundage, “The Effects of Velocity on Bullet Striations” 15 AFTE J. 56 (1983); Jessica A. Winn, “The Effect of Vibratory Finishing on Broaching Marks as a Function of Time”45 AFTE J. 350 (2013).

[127] Bieber, “Fire Investigation and Cognitive Bias,” Encyclopedia of Forensic Science (2014) (“Cognitive bias is the tendency for an examiner to believe and express data that confirm their own expectations and to disbelieve, discard, or downgrade the corresponding data that appear to conflict with those expectations. The observer’s conclusions become contaminated with a pre-existing expectation and perception, reducing the observer’s objectivity and laying the groundwork for selective attention to evidence.”)

[128] Wilson & Brekke, “Mental Contamination and Mental Correction: Unwanted Influences on Judgments and Evaluations,” Psychological Bulletin, 116, p. 119, (1994).

[129] Working Group on Human Factors in Latent Print Analysis, “Latent Print Examination and Human Factors: Improving the Practice through a Systems Approach,” National Institute of Justice, at 10 (2012).

[130] Nickerson, “Confirmation Bias: A Ubiquitous Phenomenon in Many Guises,” Review of General Psychology 2, p. 177 (1998).

[131] Id. at 175; see also Evans, “Bias in Human Reasoning: Causes and Consequences,” Psychology Press, at 41 (1989) (describing cognitive bias as “the best known and most widely accepted notion of inferential error to come out of the literature on human reasoning.”); National Commission on Forensic Science, “Ensuring that Forensic Analysis is Based on Task-Relevant Information,” at 4 (2015) (“Contextual bias is not a problem that is unique to forensic science. It is a universal phenomenon that affects decision making by people from all walks of life and in all professional settings”).

[132] Cipriano & Gruca, “Power of Priors: How Confirmation Bias Impacts Market Prices,” Journal of Predictive Markets, Vol. 8 (2014).

[133] Rosenthal & Jacobson, “Pygmalion in the Classroom,” Crown House Publishing (1992); Rosenthal, “How Often are Numbers Wrong?” American Psychologist, at 1005-1008 (1978); Cahen, “An Experimental Manipulation of the Halo Effect: A Study of Teacher Bias,” Stanford University manuscript (1965).

[134] Claudia Goldin & Cecilia Rouse, “Orchestrating Impartiality: The Impact of ‘Blind’ Auditions on Female Musicians,” 90 A. Economic Rev. 715 (2000) (using a screen to blind judges to the gender of a musician significantly increased the probability of female hires into orchestras).

[135] Rosenthal, “How Often Are Our Numbers Wrong,” Rev. Gen. Psychol. Vol. 2, 1005 (1978) (meta-study looked at 140,000 findings in published scientific data, and found that the data was systematically infected by cognitive bias in favor of the preferred hypotheses); Nuzzo, “How Scientists Fool Themselves-And How They Can Stop,” Nature, (October 7, 2015) (detailing the failure of reproducibility in many areas of scientific research, attributable to cognitive bias);  Ionnidis, “Why Most Published Research Findings Are False,” PLOS.Medicine, (2005) (finding significant bias in the methodology and publication of psychological research).

[136] Hrobjartsson et al., “Observer Bias in Randomized Clinical Trials With Measurement Scale Outcomes: A Syetematic Review of the Trials with Both Blinded and Nonblinded Assessors,” Canadian Medical Association Journal, at  p. 201 (2013) (establishing the effects of observer bias in clinical trials and concluding that “failure to blind outcome assessors in such trials results in a high risk of substantial bias”).

[137] Working Group on Human Factors in Latent Print Analysis, “Latent Print Examination and Human Factors: Improving the Practice through a Systems Approach,” National Institute of Justice, at 40 (2012); see also  Forensic Science Regulator, “Draft Guidance: Cognitive Bias Effects Relevant to Forensic Science Examinations,” at 4 (2014) (emphasizing that every forensic discipline is “potentially susceptible to unconscious personal bias (cognitive contamination), which in turn could undermine the objectivity and impartiality of the forensic process.”).

[138] Lit, L., Schweitzer, J., & Oberbauer, A., “Handler Beliefs Affect Scent Detection Dog Outcomes,” 14 Animal Cognition 387 (2011).

[139] Nakhaeizadeh et al., “Cognitive Bias in Forensic Anthropology: Visual Assessment of Skeletal Remains is Susceptible to Confirmation Bias,” 54 Science and Justice 208 (2014).

[140] Bieber, “Fire Investigation and Cognitive Bias,” Encyclopedia of Forensic Science (2014); see also “NFPA Guide for Fire and Explosion Investigations,” 2014 (formally recognizing the effects of expectation bias and confirmation bias in forensic arson investigations).

[141] Jeff Kukucka & Saul Kassim, “Do Confessions Taint Perceptions of Handrwriting Evidence? An Empirical Test of the Forensic Confirmation Bias,” Am. Pych. Assoc. (2013); Reinoud D. Stoel, et al., “Bias Among Forensic Document Examiners: Still a Need For Procedural Changes,” Australian J. Forensic Sci. (2013).

[142] Miller, “Procedural Bias in Forensic Science Examinations of Human Hair,” 11 L. & Hum. Behav., 157 (1987).

[143] Nikola K.P. Osborne et al., “Does Contextual Information Bias Bitemark Comparisons,” 54 Science & Justice 267 (2014); Mark Page et al., “Context Effects & Observer Bias-Implications in Forensic Odontology,” 57 J. Forensic Sci. 108 (2012).

[144] Michael C. Taylor, “The Reliability of Pattern Classification in Bloodstain Pattern Analysis, Part I: Bloodstain Patterns on Rigid Non-Absorbent Surfaces,” J. Forensic Sci. 1 (2016).

[145] Dror & Charlton, “Why Experts Make Errors,” 56 J. Forensic Identification,’ at 600 (2006); Dror et al, “Contextual Information Renders Experts Vulnerable to making Erroneous Identifications,” 156 Forensic Sci. Int’l 74 (2006); Working Group on Human Factors in Latent Print Analysis, “Latent Print Examination and Human Factors: Improving the Practice through a Systems Approach,” National Institute of Justice, at 20 (2012)  (noting that “bias and error can occur in any process for making comparisons and drawing inference,” and further stating that “in the context of latent print examination, they can play a role in the final decision by an examiner”).

[146] Dror &Hampikian, “Subjectivity and Bias in Forensic DNA Mixture Interpretation,” 51 Science and Justice 204 (2011).

[147] National Academy of Sciences, “Strengthening Forensic Science in the United States: A Path Forward,” National Academies Press, at 122 (2009).

[148] National Commission on Forensic Science, “Ensuring that Forensic Analysis is Based on Task-Relevant Information,” at 4 (2015); see also M.J. Saks et al., “Context Effects in Forensic Science: A Review & Application of the Science of Science to Crime Laboratory Practice in the United States,” 43 Science & Justice 77, 78 (2003).

[149] Itiel Dror, “The Paradox of Human Expertise: Why Experts Get it Wrong,” in The Paradoxical Brain (2011).

[150] See, Nuzzo, “How Scientists Fool Themselves-And How They Can Stop,” Nature, October 7, 2015 (reporting on blind data analysis in physics); Wilson & Brekke, “Mental Contamination and Mental Correction: Unwanted Influences on Judgments and Evaluations,” Psychological Bulletin, 116, p. 134 (1994) (discussing blinding and exposure control procedures); MacCaun &Perlmutter, “Blind Analysis: Hide Results to Seek the Truth,” Nature (October 7, 2015).

[151] National Academy of Sciences, “Strengthening Forensic Science in the United States: A Path Forward,” National Academies Press, 2009, at 8-9, n.8.

[152] Id. at 184-85.

[153] National Commission on Forensic Science, “Ensuring that Forensic Analysis is Based on Task-Relevant Information,” (2015); Dan Krane et al., “Sequential Unmasking: A Means of Minimizing Observer Effects in Forensic DNA Interpretation,” 53 J. Forensic Sci. 1006 (2008); Itiel Dror, “Combating Bias: The Next Step in Fighting Cognitive and Pychological Contamination,” 57 J. Forensic Sci. 276 (2011); PCAST, “Forensic Science in Criminal Courts,” at 49, 51 (concluding that firearms examination is “especially vulnerable to human error, inconsistency across examiners, and cognitive bias” & noting that “Several strategies have been proposed for mitigating cognitive bias in forensic laboratories, including managing the flow of information in a crime laboratory to minimize exposure of the forensic analyst to irrelevant contextual information (such as confessions or eyewitness identification) and ensuring that examiners work in a linear fashion, documenting their finding about evidence from crime science before performing comparisons with samples from a suspect”).

[154] Organization of Scientific Area Committees, “OSAC Research Needs Assessment Form- Cognitive         Bias: To What Extent Does it Affect Firearm and Toolmark              Comparison Outcomes,” available at http://www.nist.gov/forensics /osac/ upload/FATM-Research-Needs-Assessment_Cognitive-Bias.pdf.

[155] Jose Kerstholt, et al, “Does Suggestive Information Cause a Confirmation Bias in Bullet Comparisons,” 198 Forensic Sci. Int’l 138, 139, 141 (2010).

[156] Id. at 140, Table 2.

[157] William C. Smith, “Who Me … Biased? Or ‘We Have Met the Enemy, and He is Us!’,” 25 AFTE J 260 (1993); Evan E. Hodge, “Guarding Against Error,” 20 AFTE J. 290 (1988) (acknowledging that all examiners are affected by outside pressures and noting an examination that occurred as a result); Budowle, “A Perspective on Errors, Bias, & Interpretation in the Forensic Sciences and Direction for Continuing Advancment,” at 803 (cognitive bias “might override sound judgment, may affect interpretations in certain circumstances, and need to be minimized”).

[158] Daniel C. Murrie, “Are Forensic Experts Biased by the Side that retained Them?,” 24 Pych. Sci. 1889 (2013); Michael Risinger et al., “The Daubert/Kumho Implications of Observer Effects in Forensic Science: Hidden Problems of Expectation and Suggestion,” 90 Calif. L. Rev. 1, 48 (2002) (noting that in crime lab study “fewer than 10% of all reports disassociated a suspect from the crime scene or from connection to the victim”).

[159] Nickerson, “Confirmation Bias: A Ubiquitous Phenomenon in Many Guises,” at 181. The effects of comparing known and questioned items side by side has been studied in fingerprint comparison, where researchers have concluded that the presence of a suspects prints will meaningfully change an examiner’s interpretation of a latent print, potentially causing them to see ridge patterns that are not actually present or to interpret ambiguous patterns in an inculpatory rather than exculpatory fashion. See Dror et al., “Cognitive issues in fingerprint analysis: inter- and intra-expert consistency and the effect of a 'target' comparison,” 208 Forensic Sci. Int’l 10 (2011).

[160] United States v. Green, 405 F.Supp.2d 104,  108 (D.C.Mass 2005); see also U.S. v. Taylor, 663 F.Supp.2d 1170, 1178-79 (D.C.N.M 2009) (The problem with this practice is the same kind of problem that has troubled courts with respect to show-up identifications of people: it creates a potentially significant "observer effect" whereby the examiner knows that he is testing a suspect weapon and may be predisposed to find a match.”); Michael Risinger et al., “The Daubert/Kumho Implications of Observer Effects in Forensic Science: Hidden Problems of Expectation and Suggestion,” 90 Calif. L. Rev. 1, 48 (2002).

[161] People v. Carrero, 345 Ill. App. 3d 1, 10 (1st Dist. 2003); see also Stovall v. Denno, 388 U.S. 293 (1967).

[162] National Commission on Forensic Science, “Ensuring that Forensic Analysis is Based on Task-Relevant Information,” at 4.

[163] See, e.g. Ronald G. Nichols, “The Scientific Foundations of the Firearms & Toolmark Identification: Responding to Recent Challenges” CAC News, 2nd Quarter, at 26 (2006) (noting the difficulties examiners face when damage from impact on bullets makes correspondence borderline at best).

[164] See People v. Robinson, 2013 IL App (1st) 102476 (2013) (ISP examiner testified that “the increased automation in firearms manufacture since 1929 creates potential carryover in subclass characteristics”).

[165] F.H. Cassidy, “Examination of Toolmarks from Sequentially Manufactured Tongue-and-Groove Pliers,” 25 J. Forensic Sci. 796, 797 (1980); see also William A. Tobin, “Affidavit in Virginia v. Macumber,” at 11,12 (2011) (“The effect of these motivating concerns [costs of manufacture] has been increasingly larger production lots before tooling changes are required. This consequently means that the subclass characteristics  (toolmarks) imparted to workpieces such as barrels, extractors, ejectors, and breech faces during production have tended to exist in larger production lots over time.” & “It can be expected that consecutively formed components could readily be confused in specific source attributions particularly when the examinations are temporally isolated”).

[166] Gil Hocherman et al, “Identification of Polygonal Barrel Sub-Family Characteristics,” 35 AFTE J 197, 200 (2003) (even after specialized training examiners erred up to 20% of the time identifying even the manufacturer of polygonally rifled barrels); National Research Council, “Strengthening Forensic Science in the United States: A Path Forward,” National Academies Press, 2008, at 46. (“generally speaking it is possible, although extremely difficult, to match bullets from polygonally rifled barrels.”); Jan DeKinder et al., “Reference Ballistics Imaging Database Performance,” 140 Forensic Sci. Int’l 207, 213 (2003).

[167] Joan Griffin & David LaMagna, “Daubert Challenges to Forensic Evidence: Ballistics Next on the Firing Line” The Champion, September/October, at 58 & 59 (2002).

[168] M.S. Bonfanti & J Dekinder, “The Influence of Manufacturing Processes on the Identification of Bullets & Cartridge Cases- A Review of the Literature” 39 Sci. & Justice 3, 4 (1999).

[169] John Murdock, “Some Suggested Court Questions to Test Criteria for Identification Qualifications,” 24 AFTE J 69, 70-71 (1992).

[170] David Baldwin et al., “Statistical Tools for Forensic Analysis of Toolmarks,” AMES Laboratory Final Report IS-5160, at 24 (2004).

[171] Alfred A. Biasotti, “A Statistical Study of the Individual Characteristics of Fired Bullets,” 4 J. Forensic Sci. 34, 38 (1959).

[172] Id. at 40.

[173] Tobin & Blau, “Hypothesis Testing of the Critical Underlying Premise of Discernible Uniqueness in Firearms-Toolmark Forensic Practice,” at 136; see also See Jerry Miller & Michael Neel “Criteria for Identification of Toolmarks Part III: Supporting the Conclusion,” 36 AFTE J 7, 9 (2000) (reporting that even more significant matching has been observed by the author in known, non-matches).

[174]Adina Schwartz, “A Systemic Challenge to the Reliability & Admissibility of Firearms & Toolmark Identification,” 6 Colum. Sci. & Tech. L. Rev. 2 (2005).

[175] Data for both graphs available in: Jerry Miller, “Criteria for Identification of Toolmarks Part II: Single Land Impression Comparisons,” 32 AFTE J 116, at 117, 121, & 124 (2000).

[176] Joseph J. Masson, “Confidence Level Variations In Firearms Identifications Through Computerized Technology,” 29 AFTE J. 42 (1997).

[177] Id.

[178] T.V. Vorburger et al., “Surface Topography Analysis for a Feasibility Assessment of a National Ballistics Imaging Database,” National Institute of Standards & technology Interagency/Internal Report (NISTIR) – 7362, at 94 (May 1, 2007), available at  http://www.nist.gov/manuscript-publication-search.cfm?pub_id=822733.

[179] Jerry Miller & Glen Beach, “Toolmarks: Examining the Possibility of Subclass Characteristics” 37 AFTE J 296 (2005).

[180] AFTE, “Theory of Identification as it Relates to Toolmarks” 30 AFTE J. 86 (1998); Alfred Biasotti & John Murdock, “Criteria for Identification or State of the Art of Firearm & Toolmark Identifiction,” 16 AFTE J 16, 17 (1984); see also Adina Schwartz, “A Systemic Challenge to the Reliability & Admissibility of Firearms & Toolmark Identification,” 6 Colum. Sci. & Tech. L. Rev. 2 (2005) (“Other manufacturing processes result in batches of such similar tools that their toolmarks have the same subclass characteristics, and may or may not also have individual characteristics”); David Q. Burd & Allan E. Gilmore, “Individual and Class Characteristics of Tools” 13 J. Forensic Sci. 390 (1968) (“Modern mass production methods used in industry often result in repetitive structural detail being left on tool surfaces. This is particularly true when such tools are formed in a mold, die stamped, or die forged”).

[181] Alfred Biasotti & John Murdock, “Criteria for Identification or State of the Art of Firearm & Toolmark Identification,” 16 AFTE J 16, 18 (1984) (“we can have remarkable reproduction on many hundred or even thousands of individual items”); M.S. Bonfanti & J Dekinder, “The Influence of Manufacturing Processes on the Identification of Bullets & Cartridge Cases- A Review of the Literature” 39 Sci. & Justice 3, 5 (1999) (noting that one tool, thanks to manufacturing improvements, may now make batches of hundreds or thousands of barrels); Gene C. Rivera, “Subclass Characteristics in Smith & Wesson SW40VE Sigma Pistols” 39 AFTE J 247, 250 (2007) (“anywhere between a couple of hundred to one thousand slides could be machined before the broach is resharpened”).

[182] Adina Schwartz, “A Systemic Challenge to the Reliability & Admissibility of Firearms & Toolmark Identification,” 6 Colum. Sci. & Tech. L. Rev. 2 (2005).

[183] William A. Tobin, “Affidavit in Virginia v. Macumber,” at 8.

[184] Top Left: Frederic A. Tulleners & James S. Hamiel, “Subclass Characteristics of Sequentially Rifled 38 Special S & W Revolver Barrels” 31 AFTE J. 117 (2004); Top Right: http://projects.nfstc.org/firearms/module13/fir_m13_t05

_04.htm.; Middle Left: Evan Thompson, “Possible NFEA research Project,” AFTE Forum (2011); Middle Right: Gene C. Rivera, “Subclass Characteristics in Smith & Wesson SW40VE Sigma Pistols” 39 AFTE J 247  (2007); Bottom Left: Michael Lee et al, “Subclass Carryover in Smith & Wesson M&P 15-22 Rifle Firing Pins,” 48 AFTE J. 27 (2016); Bottom Right: F.H. Cassidy, “Examination of Toolmarks from Sequentially Manufactured Tongue-and-Groove Pliers,” 25 J. Forensic Sci. 796 (1980).

[185] AFTE Committee for the Advancement of the Science of Firearm & Toolmark Identification, “The Response of the Association of Firearm and Toolmark Examiners to the National Academy of Sciences 2008 Report Assessing the Feasibility, Accuracy, and Technical Capability of a National Ballistics Database,” 40 AFTE J 234, 239 (2008).

[186] AFTE, “Theory of Identification as it Relates to Toolmarks” 30 AFTE J. 86 (1998).

[187] Gene C. Rivera, “Subclass Characteristics in Smith & Wesson SW40VE Sigma Pistols” 39 AFTE J 247 (2007).

[188] Ronald G. Nichols, “Defending the Scientific Foundations of the Firearms & Toolmark Identification Discipline: Responding to Recent Challenges,” 52 J. Forensic Sci. 586, 587 (2007).

[189] Ronald Nichols, “Firearm and Tool Mark Identification: The Scientific Reliability and Validity of the AFTE Theory of Identification Discussed Within the Framework of a Study of Ten Consecutively Manufactured Extractors,” 36 AFTE J 67, 77 (2004).

[190] William A. Tobin, “Affidavit in Virginia v. Macumber,” at 9; see also Clifford Spieglman & William Tobin, “Analysis of Experiments in Forensic Firearms/Toolmarks Practice Offered as Support for Low Rates of Practice Error and Claims of Inferential Certainty,” 12 Law, Probability, & Risk 115, 128 (2013) (“...one of the authors with relevant manufacturing experience has observed that the majority of manufacturing marks (other than grinding) imparted to work pieces are subclass in nature.”)

[191] William Matty & Torrey Johnson, “A Comparison of Manufacturing Marks on Smith & Wesson Firing Pins,”16 AFTE J 51 (1984) (describing concentric rings left by firing pins would be common to all pins produced by the same tool); Evan Thompson, “False Breech Face ID’s,” 28 AFTE J. 95 (1996) (“an examiner could miscall an identification based only on breechface markings” of Lorcin handguns); M.S. Bonfanti & J Dekinder, “The Influence of Manufacturing Processes on the Identification of Bullets & Cartridge Cases- A Review of the Literature,” 39 Sci. & Justice 3, 5 (1999) (because of subclass marks “a correct identification of the firearm on basis of breech face and firing pin impressions, respectively, turned out to be hardly possible”); Michael Lee et al., “Subclass Carryover in Smith & Wesson M&P 15-22 Rifle Firing Pins,” 48 AFTE J. 27, 29 (2016) (for firing pins “a false-positive identification could be made if no other marks were utilized in making the identification”); Vyacheslav Polosin, “Subclass Characteristics in Extractor Groove of Winchester Cartridges,” 48 AFTE J. 50 (2016); Alicia K. Welch “Breech Face Subclass Characteristics of the Jiminez JA Nine Pistol” 45 AFTE J. 336, 343 (Fall 2013) (calling breechface similarity due to subclass a “startling observation”); Frederic A. Tulleners & James S. Hamiel, “Subclass Characteristics of Sequentially Rifled 38 Special S & W Revolver Barrels,” 31 AFTE J 117 (1999) (“If these striae were not caused by subclass features of the rifling tool, the extent of this agreement would be sufficient for an identification.”); Ronald Nies, “Anvil Marks of the Ruger MKII Target Pistol-An Example of Subclass Characteristics,” 35 AFTE J 75 (2003) (“A surprisingly high degree of agreement could be found...even when the magnification was increased to 79x, enough agreement of the fine detail was present to possibly lead to the mistaken conclusion that the two cartridge cases could have been fired in the same barrel”); Patrick D. Ball, “Toolmarks Which May Lead to False Conclusions,” 32 AFTE J 292 (2000) (pre-firing marks on cartridges “could easily be identified as breechface impressions); Susan M. Komar & Gregory E. Scala, “Examiners Beware New Bolt Cutter Blades-Class or Individual,” 25 AFTE J 298 (1993) (subclass correspondence “could easily be mistaken for true matches”); Steve Kramer, “Subclass Characteristics on Firing Pins Manufactured by ‘Metal Injection Molding’,” 44 AFTE J 364 (2012); Salvatore LaCova, et al, “Subclass Characteristics on CCI Speer Cartridge Case Heads,” 42 AFTE J 281 (2010); Laura L. Lopez & Sally Grew, “Consecutively Machined Ruger Bolt Faces,” 32 AFTE J 19 (2000) (“Comparisons revealed a startlingly high correspondence of microscopic characteristics among the bolt faces examined”); Gene C. Rivera, “Subclass Characteristics in Smith & Wesson SW40VE Sigma Pistols,” 39 AFTE J 247 (2007) (“This article documents an alarming example of subclass characteristics that could easily be mistaken for individual characteristics, and might lead an examiner to make a false positive identification”); Peter Lardizabal, “Cartridge Case Study of the Heckler & Koch USP,” 27 AFTE J 49 (1995) (noting “excellent correspondence” between breech faces markings from different guns); E.J.A.T Mattijssen et al., “Subclass Characeristics in a Gamo Air Rifle Barrel,” 45 AFTE J 281 (2013); Evan Thompson, “Possible NFEA research Project,” AFTE Forum (2011) (describing subclass markings on land impressions of Ruger and Winchester firearms) available at http://forum.afte.org/index.php?topic=7455.0; National Forensic Science Technology Center, “Firearms Examiner Training: Physical Characteristics,” available at http://projects.nfstc.org/firearms/module13/fir_m13_t05_04.htm (same); Jason Flater, “Manufacturing Marks on Winchester USA Brand 9mm Luger Primers,” 34 AFTE J 315 (2002) (describing subclass marks on unfired cartridges); Bill Matty, “Lorcin L9mm & L380 Pistol Breechface Toolmark Patterns,” 31 AFTE J 134 (1999) (noting that because of subclass issues, breechfaces of Lorin fired cartridges alone are insufficient for identification); Evan Thompson & Rick Wyant, “9mm Smith & Wesson Ejectors,” 34 AFTE J 406 (2002) (because of subclass markings “more than just an ejector toolmark must be used before making an identification to a particular firearm); Michelle Hunsinger, “Metal Injection molded Strikers & Extractors in Smith & Wesson Model M&P Pistol,” 45 AFTE J 21 (2013) (noting especial concern after finding subclass marks because “the corresponding areas are small and irregular; not what examiners are taught to be subclass”); Fabiano Riva, “Objective Evaluation of Subclass Characteristics on Breech Face Marks,” 62 J. Forensic Sci. 417 (2017) (“recognizing subclass characteristics is not an easy task, and some have rightly indicated that the ability of examiners to detect them is not well established”).

[192] Stephen R. Garten, “The Effect of Subclass Characteristics Involving Shotgun Ammunition on IBIS Entries & Correlation Results,” 42 AFTE J 364 (2010).

[193] See e.g., Michael Lee et al., “Subclass Carryover in Smith & Wesson M&P 15-22 Rifle Firing Pins,” 48 AFTE J. 27 (2016); Vyacheslav Polosin, “Subclass Characteristics in Extractor Groove of Winchester Cartridges,” 48 AFTE J. 50 (2016); Alicia K. Welch “Breech Face Subclass Characteristics of the Jiminez JA Nine Pistol,” 45 AFTE J. 336 (2013).

[194] Alfred Biasotti & John Murdock, “Criteria for Identification or State of the Art of Firearm & Toolmark Identification,” 16 AFTE J 16, 18-19 (1984) (“Because what would constitute these subclass features is a function of the relative hardness of the tool, the material, and the dynamics of the cutting process, it is not currently possible to describe them in quantitative terms. ); Adina Schwartz, “A Systemic Challenge to the Reliability & Admissibility of Firearms & Toolmark Identification,” 6 Colum. Sci. & Tech. L. Rev. 2 (2005) (noting that firearms examiners have no rules or statistics for the frequency of subclass marks, how they can be identified, or how long they may last, so that “examiners can only rely on their personal familiarity with types of forming and finishing processes and their reflections in toolmarks.”); William A. Tobin, “Affidavit in Virginia v. Macumber,” at 9 (“The AFTE theory provides no guidance on this question”).

[195] See e.g., Richard K. Maruoka, “Guilty Before the Crime? The Potential for a Possible Misidentification or Elimination,” 26 AFTE J 206 (1994);

[196] See e.g., Richard K. Marouka, “Guilty Before the Crime II?” 27 AFTE J 20 (1995); Patrick D. Ball, “Toolmarks Which May Lead to False Conclusions,” 32 AFTE J 292, 293 (2000) (Those markings are only around the outer circumference of the primer, the center area was free of toolmarks. This is different from other reported manufacturing toolmarks on primers previously reported on in past issues of the journal”); E.J.A.T Mattijssen et al., “Subclass Characeristics in a Gamo Air Rifle Barrel,” 45 AFTE J 281 (2013); Evan Thompson, “Possible NFEA Research Project,” AFTE Forum (2011); Hunsinger, “Metal Injection molded Strikers & Extractors in Smith & Wesson Model M&P Pistol,” 45 AFTE J 21 (2013).

[197] Ronald G. Nichols, “Response to- Adina Schwartz, Commentary on: Defending the Scientific Foundations of the Firearms & Toolmark Identification Discipline: Responding to Recent Challenges,” 52 J. Forensic Sci. 1416 (2007).

[198] Petra Pauw-Vugts, et al, “FAID2009: Proficiency Test & Workshop,” 45 AFTE J 115, 124 & 126 (2013).

[199] William A. Tobin, “Affidavit in Virginia v. Macumber,” at 15.

[200] Michael J. Saks & Jonathan L. Koehler, “The Coming Paradigm Shift in Forensic Identification Science,”  309 Science 892  (2005) (“Erroneous forensic science expert testimony is the second most common contributing factor to wrongful convictions, found in 63% of those cases. These data likely understate the relative contribution of forensic science expert testimony to erroneous convictions.”); ABA Criminal Justice Section’s Ad Hoc Committee to Ensure the Integrity of the Criminal Process, “Achieving Justice: Freeing the Innocent & Convicting the Guilty,” (2006) (reporting that 1/3 of DNA exonerations resulted from tainted or fraudulent science); Craig Cooley & Gabriel Oberfield, “Symposium: Daubert, Innocence, and the Future of Forensic Science: Increasing Forensic Evidence’s Reliability and Minimizing Wrongful Convictions: Applying Daubert Isn’t the Only Problem,” 43 Tulsa L. Rev. 285 (2007) (describing similar percentages); PCAST, “Forensic Science in Criminal Courts,” at 3.

[201] Tasha P. Smith, et al., “A Validation Study of Bullet & Cartridge Case Comparisons Using Samples Representative of Actual Casework,” 69 J. Forensic Sci. 939 (2016) (“The ‘human factor’ in identification accounts for tremendous variability in analysis”); Angela Stroman, “Empirically Determined Frequency of Error in Cartridge Case Examinations Using a Declared Double-Blind Format,” 46 AFTE J. 157, 169-70 (2014) (documenting variations in the types of marks used by examiners during study to reach cartridge identifications as well as differing levels of knowledge concerning subclass characteristics); R.G. Nichols, “The Scientific Foundations of the Firearms & Toolmark Identification: Responding to Recent Challenges,” CAC News, 2nd Quarter, at 26 (2006) (“In the absence of a specific criterion such as CMS, there will be some difference between examiners as to what constitutes the best-known non-match situation…it is not necessarily unexpected that one examiner would reach an inconclusive determination while another might conclude a more positive association.”); Kertsholt, “Does Suggestive Information Cause a Confirmation Bias in Bullet Comparisons,” at 141 (when two sets of examiners evaluated the same ballistic evidence in 128 cases, they did not agree 16% of the time).

[202] Ronald G. Nichols, “Defending the Scientific Foundations of the Firearms & Toolmark Identification Discipline: Responding to Recent Challenges,” 52 J. Forensic Sci. 586, 590 (2007) (“while the concept of correspondence exceeding that observed in a best-known nonmatch situation is a standard ideal, the actual definition of that will be different between examiners because they have different experiences. For example, an examiner in California has access to certain training materials dealing with comparing known nonmatches that establish a baseline correspondence. It is very likely that an examiner in the Northeast has different materials and will therefore develop a different experiential concept of the best-known nonmatch.”)

[203] Werner Deinet, “Comments of the Application of Theoretical Probability Models Including Bayes Theorem in Forensic Science Relating Firearm and Tool Marks,” 39 AFTE J 4, 6 (2007) (“Very often, two independent experts will get different results concerning the total number of striae and the number of matching striae”); Schwartz, “A Systemic Challenge to the Reliability & Admissibility of Firearms & Toolmark Identification,” at  ¶42 (“different, well-qualified examiners are likely to count different numbers of striae on the same toolmark. This creates the possibility that different experienced examiners will reach different conclusions about whether the same toolmarks satisfy or do not satisfy the CMS criteria.”)

[204] Jerry Miller, “Criteria for Identification of Toolmarks Part III: Supporting the Conclusion,” 36 AFTE J 7, 9 (2004). Given that the total lines observed on any one surface reached only as high as 58, see id., differences in count between examiners of up to 21 and 23 lines  surely qualify as substantial.

[205] See e.g., Ulery, et al., “Repeatability and Reproducibility of Decisions By Latent Print Examiners,” Proceedings of the National Academy of Sciences (2012) (latent print examiners disagreed with each other about 50% of the time on difficult cases, and about 20% of the time on easier cases.  Examiners changed their own opinion about 30% of the time when taking a second look at fingerprint evidence identified as more difficult and about 10% of the time on easier cases); Neumann et al., “Improving the Understanding and the Reliability of the Concept of ‘Sufficiency’ in Friction Ridge Examination,” U.S Department of Justice, p. 56 (2013) (146 fingerprint examiners agreed regarding the suitability of prints for comparison on none of 15 sets of latent prints analyzed); Ulery et al., “Changes in Latent Fingerprint Examiners’ Markup Between Analysis and Comparison,” 247 Forensic Sci. Int’l 58 (2015) (fingerprint examiners agreed on just 23% of features in clear areas of latent prints when reaching association conclusions).

[206] PCAST, “Forensic Science in Criminal Courts,” at 57-59 (criticizing existing tests as not being blind and noting that even the designer of CTS testing admits that they are designed to be simplistic because the forensic industry prefers them that way); William Tobin & Peter Blau, “Hypothesis Testing of the Critical Underlying Premise of Discernible Uniqueness in Firearms-Toolmark Forensic Practice,” 53 Jurimetrics 121, 137 (2013); Adina Schwartz, “Challenging Firearms and Toolmark Identification- Part Two,” The Champion XXXII (9): 44-52, 47 (2008) (explaining that many CTS tests do not even require examiners to distinguish between guns of the same make or model and are described as incredibly easy by examiners, one of whom noted that he could complete the test “virtually with the naked eye”); Richard Grzybowski, et al, “Firearm ToolMark Identification: Passing the Reliability Test Under Federal and State Evidentiary Standards,” 35 AFTE J 209, 219 (2003) (“most [error rate tests] tend to be rather straightforward and of only moderate difficulty”);Michael J. Saks & Jonathan L. Koehler, “The Coming Paradigm Shift in Forensic Identification Science,” 309 Science 892 (2005) (describing proficiency testing as “infrequent, internal, and unrealistic”); Gianelli, “Reference Guide on Forensic Identification Expertise,” at 98 (tests are not representative or blind, error rates vary based on counting of inconclusive responses, and the rigor of exams has been questioned); Peterson & Markham, “Crime Laboratory Proficiency Testing Results, 1978-1991, I: Identification & Classification of Physical Evidence,” J. Forensic Sci. 994, 997 (1995); Angela Stroman, “Empirically Determined Frequency of Error in Cartridge Case Examinations Using a Declared Double-Blind Format,” 46 AFTE J. 157, 158 (2014) (When examiners knows they are being tested: “The examiner may feel more pressure to perform accurately and get the correct answer, which may lead the examiner to treat the test in a different manner than real casework” moreover notes of CTS tests that they are often taken collaboratively “typically do not accurately reflect the full range of difficulty experienced in real-life firearm and toolmark casework”); AFTE Committee for the Advancement of the Science of Firearm & Toolmark Identification, “AFTE Response to the NACDL Task Force on the Future of Forensic Science,” 42 AFTE J 102, 103 (2010) (says of CTS tests: “could be more representative of casework compared to how most of them are currently prepared and administered”); Donald Kennedy & Richard A. Merrill “Assessing Forensic Science” Issues in Sci. & Tech.: 20, 1 (Fall 2003) (“practitioners have not been subjected to rigorous proficiency testing, reliable error rates are not known”); Petra Pauw-Vugts, “FAID2009: Proficiency Test & Workshop,” at 117 (“Most of the participating examiners already participated in the CTS collaborative testing program, but found this test insufficiently challenging to be of use in demonstrating competence in microscopy skills”); Michigan State Police Forensic Science Division, “Audit of the Detroit Police Department Forensic Services Laboratory Firearms Unit,” at 27 (2008) (proficiency tests are taken as a group with consensus answers submitted to the test provider, management cannot determine an individual examiner’s proficiency level.”), available at http://www.sado.org/content/pub/10559_MSP-DCL-Audit.pdf.

[207] Adina Schwartz, “Challenging Firearms and Toolmark Identification- Part Two,” The Champion XXXII (9): 44-52, 47 (2008) (“results on the CTS tests provide an inflated, rather than an accurate, estimate of the competence of examiners”); Michael J. Saks & Jonathan L. Koehler, “The Coming Paradigm Shift in Forensic Identification Science,” 309 Science 892 (2005) (“Indeed these existing data [on error rates] are probably best regarded as lower- bound estimates of error rates. Because the tests are relatively easy …and because participants know that mistakes will be identified and punished, test error rates (particularly the false-positive error rate) probably are lower than those in everyday casework.”)

[208] Peterson & Markham, “Crime Laboratory Proficiency Testing Results, 1978-1991, II” at 1018-19 (inconclusive rate reached 69%, and increases “seemed to be a function of the difficulty of the test”); Gianelli, “Reference Guide on Forensic Identification Expertise,” at 98 (firearms examination “constitute the evidence category where evidence comparisons have the highest rates of inconclusive responses.”); Tobin & Blau, “Hypothesis Testing of the Critical Underlying Premise of Discernible Uniqueness in Firearms-Toolmark Forensic Practice,” at 137, n.39; Smith, “Cartridge case and bullet comparison validation study with firearms submitted in casework,” 37 AFTE J 130 (2005) (examiners declared 97% of the 704 different source samples they inspected to be inconclusives rather than exclusions).

[209] To the extent possible, and where the necessary data was available, the error rates noted above were calculated using the approach advocated by PCAST of discounting inconclusive responses. See PCAST, “Forensic Science in Criminal Courts,” at 51-52 (“When reporting a false positive rate to a jury, it is scientifically important to calculate the rate based on the proportion of conclusive examinations, rather than just the proportion of all examinations. This is appropriate because evidence used against a defendant will typically be based on conclusive, rather than inconclusive, examinations. To illustrate the point, consider an extreme case in which a method had been tested 1000 times and found to yield 990 inconclusive results, 10 false positives, and no correct results. It would be misleading to report that the false positive rate was 1 percent (10/1000 examinations). Rather, one should report that 100 percent of the conclusive results were false positives (10/10 examinations)”).

[210] Petra Pauw-Vugts, et al., “FAID2009: Proficiency Test & Workshop,” 45 AFTE J 115 (2013) (also reporting misidentification rate of 3% on a 2005 version of the same proficiency test).

[211] Joseph L. Peterson & Penelope Markham, “Crime Laboratory Proficiency Testing Results, 1978-1991, II. Resolving Questions of Common Origin,” 40 J. Forensic Sci. 994, 1009, 1018-19, 1024 (1995) (also noting that for toolmarks as opposed to firearms misidentification rate came in at as high as 13%).

[212] Paul C. Gianelli, et al., “Reference Guide on Forensic Identification Expertise,” In Reference Manual on Scientific Evidence (National Academies Press 2011), at 97 (also noting that administrators of the test viewed the error rates of firearms examiners as “particularly grave in nature”). And in fact, for the methodological twin of firearms examination (toolmark examination) one author has tabulated an error rate of 35%. See Michael J. Saks, “Merlin & Solomon: Lessons from the Law’s Formative Encounters with Forensic Identification Science,”49 Hastings L.J. 1069, 1089 (1998).

[213] PCAST, “Forensic Science in Criminal Courts,” at 109.

[214] Petra Pauw-Vugts, et al.,“FAID2009: Proficiency Test & Workshop,” 45 AFTE J 115, 124 & 126 (2013).

[215] Audits likely provide more reliable estimates of error even than testing of examiners. See “Report of the Governor’s Commission on Capital Punishment: Chapter 3, DNA & Forensic Testing,” at 53 (April 15, 2002) (advocating for routine, external audits whether both police and civilian crime laboratories); Janine Arvizu, “Forensic Labs: Shattering the Myth,” Champion Magazine, at 7 (May 2000) (former lab director and lab quality assurance specialist notes that “In recent years, review by external parties has been the most effective means of identifying forensic laboratory problems” as well as that “An independent on-site quality audit is the best means of assessing the quality of the field and laboratory operations”); Ronald G. Nichols, “Defending the Scientific Foundations of the Firearms & Toolmark Identification Discipline: Responding to Recent Challenges, ” 52 J. Forensic Sci. 586, 592-93 (2007) (“A better estimation of error rate in casework would be most rigorously achieved by the re-examination of several thousand cases where each case was examined by a panel of experts to achieve consensus”); Michael J. Saks, “Merlin & Solomon: Lessons from the Law’s Formative Encounters with Forensic Identification Science,” 49 Hastings L.J. 1069, 1089 (1998) (“the only way to find false matches would be to conduct special studies to look for them”).

[216] Michigan State Police Forensic Science Division, “Audit of the Detroit Police Department Forensic Services Laboratory Firearms Unit,” at 4 (2008). Similarly disturbing results have been documented in audits of other forensic disciplines. See Chris Swecker & Michael Wolf, “AN INDEPENDENT REVIEW OF THE SBI FORENSIC LABORATORY,” at 3 (2010), available at http://truthinjustice.org/sbi.audit.report.pdf; FBI Press Release, “FBI Testimony on Microscopic Hair Analysis Contained Errors in at Least 90 Percent of Cases in Ongoing Review,” (April 20, 2015).

[217] Craig Cooley & Gabriel Oberfield, “Symposium: Daubert, Innocence, and the Future of Forensic Science: Increasing Forensic Evidence’s Reliability and Minimizing Wrongful Convictions: Applying Daubert Isn’t the Only Problem,” 43 Tulsa L. Rev. 285, 337-38 (2007) (discussing wrongful conviction of Charles Stielow in 1915 due to firearms examination misidentification).

[218] Megan Cassidy, “A look inside the reports that unraveled the Phoenix freeway-shooting case,” The Arizona Republic (May 13, 2016) (reporting on the release of a suspect in a spree of highway shooting after a former AFTE president, reviewed the initial firearms examination report and concluded that no identification was warranted).

[219] Williams v. Quarterman, 551 F.3d 352 (5th Cir. 2008) (granting postconviction hearing based on State’s expert’s admission regarding firearms misidentification); People v. Williams, 2010 Mich. App. LEXIS 1955, at *16-17 (2010) (“The initial police report indicated that all bullet casings from the crime scene came from a single weapon. Subsequent testing showed that the bullet casings actually came from two different AK-47 assault rifles. At trial, expert testimony suggested that the discrepancy occurred because the original Detroit Crime  Lab examiners either lied, or were incompetent, or did not actually examine all 42 casings.”); Catherine Leahy Scott, “Investigation into the New York State Police Forensic Investigation Center,” (2014) (concluding that NY firearms examiners completed false reports to aid the prosecution’s case); Texas Forensic Science Comission, “FINAL REPORT FOR COMPLAINT FILED BY ATTORNEY FRANK BLAZEK REGARDING FIREARM/TOOL MARK ANALYSIS PERFORMED AT THE SOUTHWESTERN INSTITUTE OF FORENSIC SCIENCE,” (April 2016) (reporting on misidentification in murder case, which actually spurred on lab in question to adopt CMS rather than the subjective methodology).

[220] Craig Cooley & Gabriel Oberfield, “Symposium: Daubert, Innocence, and the Future of Forensic Science: Increasing Forensic Evidence’s Reliability and Minimizing Wrongful Convictions: Applying Daubert Isn’t the Only Problem,” 43 Tulsa L. Rev. 285, 338-39 (2007) (discussing false accusations against California police officer due to firearms misidentification).

[221] U.S. District Court for the Northern District of Illinois, Eastern Division, “Report of the 1970 Grand Jury,” at 79-90 (Jul. 28, 1970) (documenting misidentifications by the Chicago Police Department laboratory during the investigation into leading Black Panther, Fred Hampton’s death)  available at http://peopleslawoffice.com/wp-content/ uploads/2012/02/Hampton.-1970-FGJ-Report.pdf; Hinton v. Alabama, 134 S. Ct. 1081, 1086 (2014) (three postconviction experts concluded State’s firearms examiner had misidentified ballistic evidence, and State offered no rebuttal).

[222] Commonwealth v. Ellis, 364 N.E.2d 808 (Mass. Sup. Ct. 1977) (two State experts contradicted each other’s identifications); Trotter v. Missouri, 736 S.W.2d 536 (Ct. App. 1987) (State’s firearms expert changed his opinion identifying opinion after additional evidence was revealed post-trial).

[223] See Spencer S. Hsu & Keith OL. Alexander, “Forensic errors trigger reviews of D.C. crime lab ballistics unit, prosecutors say,” Washington Post (March 24, 2017) (three errors by multiple examiners at DC lab necessitated reexamination of over 150 cases), available at https://www.washingtonpost.com/local/public-safety/forensic-errors-trigger-reviews-of-dc-crime-lab-ballistics-unit-prosecutors-say/2017/03/24/2d67cdcc-0e75-11e7-ab07-07d9f521f6b5_story.html?utm_term=.43b68450f72a.

[224] Steve McVicker, “Ballistics lab results questioned in 3 death cases,” The Houston Chronicle, available at http://www.chron.com/news/houston-texas/article/Ballistics-lab-results-questioned-in-3-death-cases-1923892.ph.

[225] Bruce Moran, “A Report on the AFTE Theory of Identification & Range of Conclusions for Tool Mark Identification & Resulting Approaches to Casework,” 34 AFTE J 227 (2002) (“In the 1980s come striated toolmark mis-identifications resulting from a poor understanding of toolmark criteria for identification were experienced. An increasing need to address problems of applying subjective criteria became apparent.”); Evan E. Hodge, “Guarding Against Error,” 20 AFTE J. 290 (1988) (noting that “most of us [firearms examiners] know someone who has committed serious error” and describing misidentification by another examiner of the wrong .45 caliber firearm due to cognitive bias and pressure from prosecutors); Alfred Biasotti & John Murdock, “The Scientific Basis of Firearms & Toolmark Identification,” In Modern Scientific Evidence: The Law & Science of Expert Testimony at 143 (1997) (acknowledging misidentifications stemming “from one examiner ascribing too much significance to a small amount of matching striae and not appreciating that such agreement is achievable in known non-match comparisons.”); Lowell Bradford, “Forensic Firearms Identification: Competence or Incompetence,” 11(2) AFTE J (1979) (“An appalling number of misidentifications have been found in the firearm identification field”).

[226] Simon A. Cole, “Symposium: Forensic Statistics, Part II ‘Implicit Testing’: Can Casework Validate Forensic Techniques?” 46 Jurimetrics J. 117, 126-27 (2006) (“known misattributions are very likely to only be a small subset of actual misattributions”); Andre A. Moenssens, “Novel Scientific Evidence in Criminal Cases: Some Words of Caution,” 84 J. Crim. L. & Criminology 1, 12-13 (1993) (noting that misidentifications occur but “mistakes of this kind are not very likely to be discovered.”)

[227] People v. McKown, 226 Ill. 2d 245, 254 (2007); see People v. McKown, 236 Ill. 2d 278, 294-295 (2010).

[228] See People v. Rodriguez, 2017 IL App. (1st) 141379 (1st Dist. May 8, 2017); People v. Robinson, 2013 IL App. (1st) 102476 (1st Dist. Dec. 2, 2013).

[229] PCAST, “Forensic Science in the Criminal Courts,” at 112.

[230] McKown, 226 Ill. 2d at 254.

[231] See Donaldson v. Cent. Ill. Pub. Serv. Co., 199 Ill. 2d 63, 78-79 (2002) (“The trial judge applies the Frye test only if the scientific principle, technique or test offered by the expert to support his or her conclusion is new or novel.”)

[232] PCAST, “Forensic Science in the Criminal Courts,” at 144.

[233] Benn v. United States, 978 A.2d 1257, 1278 n.90 (D.C. 2009); see State v. Lucero, 85 P.3d 1059, 1062 (Ariz. App. 2004) (“This is not to say that, once admitted, scientific evidence is forever after unassailably admissible. After all, some theories once generally accepted ultimately have been rejected in favor of new ones”); People v. Kelly, 549 P.2d 1240, 1245 (Cal. 1976) (emphasizing that admissibility under Frye persists only “until new evidence is presented reflecting a change in the attitude of the scientific community”); Jones v. United States, 27 A.3d 1130, 1136 (D.C. 2011) (“we do not doubt that a technique that has previously been recognized in court as generally accepted may lose that wide acceptance”); Chesson v. Mongomery Mutual Insurance Company, 75 A.3d 932 (MD Ct App. 2013) (acknowledging that “even scientific techniques once considered to be generally accepted are excluded when subsequent scientific studies bring their reliability and validity into question and show a fundamental controversy within the relevant scientific community”); Trach v. Fellin, 817 A.2d 1102, 1110 (Pa. Super. Ct. 2003) (emphasizing that “a principle or discovery can fall by the wayside as science advances is just another way of saying it is not generally accepted”).

[234] See Clemons v. Maryland, 896 A.2d 1059, 1076 (MD Ct.  App. 2006) (finding that “the assumptions regarding that uniformity or homogeneity of the molten source and the uniqueness of each molten source that provide the foundation for CBLA have come under attack by the relevant scientific community of analytical chemists and metallurgists” and excluding such evidence as not generally accepted).

[235] Illinois v. Luna, 2013 IL App (1st) 072253, at ¶65 (1st Dist. 2013) (internal quotations & citations omitted); see also Donaldson, 199 Ill.2d at 78. In fact, even in the specific context of firearms examination, the First District has twice indicated, albeit in unpublished orders, that its decision in Robinson would not foreclose a subsequent Frye challenge if based on new authority. See People v. Smith, 2014 IL App (1st) 121062-U (1st Dist. Apr. 14, 2014); People v. Dupree, 2014 IL App (1st) 121179-U (1st Dist. Dec. 12, 2014). And the two states that have explicitly addressed the impact of appellate Frye rulings on subsequent trials in light of new scientific evidence, held that “precedent so established may control subsequent trials, at least until new evidence is presented reflecting a change in the attitude of the scientific community.” People v. Smith, 215 Cal. App. 3d 19, 26 (Cal. App. 1st Dist. 1989) (internal citations & quotations omitted); see also State v. Copeland, 922 P.2d 1304, 1333 (Wash. 1996) (“Only if a party presents new evidence seriously questioning continued general acceptance of use of the product rule will a Frye hearing be required”).

[236] McKown, 226 Ill. 2d at 258.

[237] See People v. Shanklin, 2014 IL App (1st) 120084, ¶80 (1st Dist. 2014).

[238] See Robinson, 2013 IL App (1st) 102476 at ¶70.

[239] The State will likely make much of Rodriguez’s statement that firearms examination “is not new or novel, either pursuant to the plain meaning of those words or in accordance with the analysis employed by our supreme court in McKown.” 2017 IL App. (1st) 141379, at ¶56. But that conclusion deserves only the weight due its context (which did not include a review of the PCAST report or other sources further challenging the legitimacy of firearms examination evidence). And its reference to the McKown approach to the novelty question actually demonstrates conclusively that said context matters: on the basis of new legal or scientific challenges, even longstanding forensic methods must again be labeled new and novel.

[240] Donaldson, 199 Ill.2d at 78.

[241] Id.

[242] McKown, 226 Ill. 2d at 254 (emphasis added).

[243] Id.

[244] See Paul C. Gianelli, “The Admissibility of Novel Scientific Evidence: Frye v. United States, a Half-Century Later,” 80 Coulm. L. Rev. 1197, 1218-19 (1980) (overreliance on prior cases as opposed to technical writings or expert testimony “undercuts the primary rationale supporting Frye-that those most qualified to judge the validity of a technique should have the determinative voice"); PCAST, “Forensic Science in the Criminal Courts,” at 144 (“from a scientific standpoint, subsequent events have indeed undermined the continuing validity of [judicial] conclusions that were not based on appropriate empirical evidence”).

[245] See McKown, 236 Ill. 2d at 300 (concluding that “the relevant scientific fields that embrace the testing for and observation of HGN include medicine, ophthalmology, and optometry… [t]hus, the question of general acceptance must be determined from the testimony of experts and the literature in these scientific fields.”); People v. Watson, 257 Ill. App. 3d 915, 926 (1st Dist. 1994) (“the proposed DNA profiling evidence should be evaluated by scientists in the fields of molecular biology, population genetics and forensic science”); PCAST, “Forensic Science in the Criminal Courts,” at 55 & 142 (“scientific validity of a method must be assessed within the framework of the broader scientific field of which it is a part (e.g., measurement science in the case of feature-comparison methods). The fact that bitemark examiners defend the validity of bitemark examination means little” & “the appropriate scientific field should be the larger scientific discipline to which [a forensic method] belongs”); Reed v. State, 283 Md. 374, 382 (Md. 1978) (“the relevant scientific community will include those whose scientific background and training are sufficient to allow them to comprehend and understand the process and form a judgment about it”).

[246] See Luna, 2013 IL App (1st) 072253 at ¶75 (“This court has counseled against too narrowly defining the relevant scientific community to those who share the views of the testifying expert.”); Bernardoni v. Indus. Comm'n (Huntsman Chem. Co.), 362 Ill. App. 3d 582, 595 (3d Dist. 2005) (“A court must not define the relevant field of experts so narrowly that the expert's opinion inevitably will be considered generally accepted. If the community is defined to include only those experts who subscribe to the same beliefs as the testifying expert, the opinion always will be admissible. The community of experts must include a sufficiently broad sample of experts so that the possibility of disagreement exists.”); United States v. Porter, 618 A.2d 629, 634 (D.C. 1992) (“It simply is not creditable to argue … that general acceptance may be premised simply on the opinion of forensic scientists”).

[247] See PCAST, “Forensic Science in the Criminal Courts,” at 42 & 44 (“As a scientific matter, the relevant scientific community for assessing the reliability of feature-comparison sciences includes metrologists (including statisticians) as well as other physical and life scientists from disciplines on which the specific methods are based. Importantly, the community is not limited to forensic scientists who practice the specific method” & “feature comparison is a common scientific activity, and science has clear standards for determining whether such methods are reliable. In particular, feature-comparison methods belong squarely to the discipline of metrology—the science of measurement and its application”); Brandon Giroux, “The Association of Firearm and Tool Mark Examiners (AFTE) December 23, 2015 response to seven questions related to forensic science posed on November 30, 2015 by The President’s Council of Advisors on Science and Technology (PCAST),” at 22 (2015) (“Metrology is a second discipline that has enhanced the science of firearm and toolmark identification.”); SWGGUN & AFTE, “response to 25 foundational firearm and toolmark examination questions,” at 1 (firearms examination “is based on previously established theories, principles and properties that were adapted in the material and engineering sciences”); Tobin, “Affidavit in Virginia v. Macumber,” at 17 (“Metallurgy is the most appropriate scientific discipline to address issues of metal to metal contact, such as occurs during the cycling of a firearm. This metal to metal contact produces the toolmarks on which firearms examiners base their conclusons.”); Gregory Klees (SWGGUN Chair), “Practice & Standards of the Scientific Working Group for Firearms & Toolmarks,” at slide 10 (acknowledging academic professionals, industry experts and engineers as subject matter experts), available at http://sites.nationalacademies.org/cs/groups/pgasite/documents/webpage/pga_049914.pdf.

[248] The State will surely point out that the subcommittee recently released a statement opposing the PCAST report’s ultimate conclusion regarding the validity of firearms examination. See OSAC Firearms & Toolmarks Subcommittee, “Response          to PCAST’s Call for Additional References Regarding its Report,” (Dec. 14, 2016). But that opinion piece (supported by an unknown number of the subcommittee’s members) in no way revokes its previous statements indicating a need for/but complete lack of appropriate black-box studies and other empirical measures of examiner accuracy. Instead, it more likely signals that the members of said committee who are themselves firearms examiners gave in to precisely the type of bias that a later portion of this brief notes is the underlying reason for discounting the opinions of practitioners from the Frye calculus. In other words, when dealing abstractly with the realities of available research the subcommittee found the discipline grossly lacking. When faced with the potential loss of their professions, it appears that some members suspiciously changed course. That leaves their first opinions as the most accurate and credible statement of their scientific as opposed to self-interested views.

[249] People v. Young, 425 Mich. 470, 483 n.24 (Mich. 1986); see also Ramirez v. State, 810 So.2d 836, 844 n.13 (Fla. 2001) (warning against reliance on experts who have a “personal stake” in the acceptance of a methodology or show “institutional bias”).

[250] Ramirez, 810 So.2d at 851 (emphasis added).

[251] 236 Ill.2d at 295 & 300 (also stating: “Law enforcement, however, is not a scientific field. Therefore, general acceptance within law enforcement circles cannot be the basis for finding scientific evidence admissible under Frye.”)

[252] People v. Harbold, 124 Ill. App. 3d 363, 379 (1st Dist. 1984).

[253] See McKown, 236 Ill.2d at 295 to 296 (giving no weight whatsoever to the stance of the American Optometric Association, a trade organization).

[254] It also bears mentioning that the counterarguments likely to be posited by any State expert at a Frye hearing (that training and experience when coupled with the alleged uniqueness of fired bullets and cartridges diminish the need for black box studies) would warrant little weight given that the PCAST’s conclusions “are based on the fundamental principles of the ‘scientific method’—applicable throughout science—that valid scientific knowledge can only be gained through empirical testing of specific propositions.” PCAST, “Forensic Science in Criminal Courts,” at 46. And firearms examiners, as mere practitioners of an applied science, simply lack the qualifications necessary to overcome the more specific expertise in validation and the appropriate methods for vetting scientific methods possessed by the PCAST panel. See McKown, 236 Ill.2d at 300-01 (Noting of a police specialist that his “years of experience, his professional credentials do not qualify him as an expert on the general acceptance of HGN testing for the purpose of alcohol impairment within these scientific fields.”)

[255] See e.g., People v. Fisher, 340 Ill. 216, 237-241 (1930).

[256] PCAST, “Forensic Science in the Criminal Courts,” at 143 (noting a serious tension between legitimate science and legal standards if “courts admit forensic feature-comparison methods based on longstanding precedents that were set before these fundamental problems were discovered”).

[257] 2013 IL App (1st) 102476 at ¶¶81-89; 2017 IL App. (1st) 141379, at ¶54.

[258] See Mark Page, et al., “Forensic Identification Science Evidence Since Daubert: Part I-A Quantitative Analysis of the Exclusion of Forensic Science Evidence,” 56 J. Forensic Sci. 1180, 1182 (2011) (identifying  total 37 challenges firearms examination testimony that resulted in either exclusion or limitation of the proffered evidence with reliability as the reason for exclusion in 20 of those); United States v. Mouzone, 696 F.Supp.2d 536, 569 & 572-73 (D. Maryland 2009) (concluding that neither conclusions of absolute nor practical certainty of a match were  factually warranted and noting that the most accurate reading of recent cases on firearms examination is that courts have recognized “as the NRC Forensic Science Report clearly did, that if firearms toolmark evidence is characterized exclusively as ‘science,’ it has a long way to go before it legitimately can claim this status ... the concerns expressed by the NRC ought to be heeded by courts in the future”); United States v. Willock, 696 F.Supp.2d 536, 546 (D. Maryland 2010) (adopting report and recommendation of magistrate in Mouzone, and enforcing “a complete restriction on the characterization of certainty”); United States v. Taylor, 663 F.Supp.2d 1170, 1180 (D. NM 2009) (“because of the limitations on the reliability of firearms identification evidence discussed above, Mr. Nichols will not be permitted to testify that his methodology allows him to reach this conclusion as a matter of scientific certainty. Mr. Nichols also will not be allowed to testify that he can conclude that there is a match to the exclusion, either practical or absolute, of all other guns.”); United States v. Ashburn, 88 F.Supp.3d 239, 249 (E.D.N.Y. 2015) (quoting the finding of the NAS Committee that forensic ballistic comparison “suffers from certain ‘limitations,’ including the lack of sufficient studies to understand the reliability and repeatability of examiners’ methods . . .” and precluding “expert witness from testifying that he is ‘certain’ or ‘100%’ sure of his conclusions that certain items match ... that a match he identified is to ‘the exclusion of all other firearms in the world,’ or that there is a ‘practical impossibility’ that any other gun could have fired the recovered materials.”); United States v. Green, 405 F. Supp. 2d 104, 124 (D. Mass. 2005) (permitting testimony regarding observations but NO ultimate opinion about source); United States v. Monteiro, 407 F.Supp.2d 351, 375 (D. Mass. 2006) (limiting testimony to “reasonable degree of ballistic certainty”); United States v. Diaz, 2007 U.S. Dist. LEXIS 13152, at *41-42 (N.D. Cal. 2007) (precluding matches to the exclusion of all other guns in the world); Massachusetts v. Pytou Heang, 942 N.E.2d 927, 945-46 (2010) (allowing testimony to a reasonable degree of ballistics certainty but precluding statements describing firearms examination as a science or phrasing of conclusions to an absolute or practical certainty); United States v. Love, No. 2:09-cr-20317-JPM (W.D. Tenn. Feb. 8, 2011) (excluding testimony regarding absolute or practical certainty); United States v. Alls, No. CR2-08-223(1) (S.D. Ohio Dec. 7, 2009) (forbidding any claim of a match to one firearm to the exclusion of all other guns and limiting examiner to descriptions of her methodology and observations of casings); United States v. St. Gerard, APO AE 09107, at 4 (U.S. Army Trial Judiciary, 5th Judicial Circuit June 7, 2010) (the probative value of [the expert’s] proffered testimony that it would be practically impossible for a tool other than the seized AK-47 to have made the marks on the cartridge case would be substantially outweighed by the unfair prejudice associated with its unreliability.”), available at http://www.swgfast.org/Resources/ 101126_US-v-Gerard.pdf.

[259] See e.g., United States v. Green, 405 F. Supp. 2d 104, 124 (D. Mass. 2005), citing United States v. Hines, 55 F. Supp. 2d 62 (D. Mass. 1999) (permitting testimony only regarding an examiner’s observations without any accompanying conclusions about the source of a projectile); United States v. Glynn, 578 F.Supp.2d 567 (S.D.N.Y. 2008) (noting that, given the lack of data supporting the discipline “ballistics lacked the rigor of science,” and limiting testimony of match to a conclusion of “more likely than not” instead of even “reasonable ballistics certainty” to ensure that “a conviction in a criminal case may not rest exclusively on ballistics testimony.”); Missouri v. Goodwin-Bey, No. 1531-CR00555-01 (Dec. 16, 2016) (limiting testimony “to the point this gun could not be eliminated as the source of the bullet.”)

[260] Williams v. United States, 130 A.3d 343, 355 (D.C. 2016) (J. Easterly concurring). Notably, even that most derisive of statements was discussed approvingly by PCAST. See “Forensic Science in the Criminal Courts,” at 55.

[261] In terms of similar decisions across other forensic disciplines: See Commonwealth v. Joyner, 4 N.E.3d 282, 289 (Mass. 2014) (holding that that fingerprint examiners should avoid expressing opinions of absolute certainty); United States v. Oskowitz, 294 F. Supp. 2d 379, 384 (E.D.N.Y. 2003) (“Many other district courts have similarly permitted a handwriting expert to analyze a writing sample for the jury without permitting the expert to offer an opinion on the ultimate question of authorship.”); United States v. Rutherford, 104 F. Supp. 2d 1190, 1194 (D. Neb. 2000) (expert limited to “explaining the similarities and dissimilarities between the known exemplars and the questioned documents” and “precluded from rendering any ultimate conclusions on authorship of the questioned documents and is similarly precluded from testifying to the degree of confidence or certainty on which his opinions are based”); United States v. Hidalgo, 229 F. Supp. 2d 961, 967 (D. Ariz. 2002) (“Because the principle of uniqueness is without empirical support, we conclude that a document examiner will not be permitted to testify that the maker of a known document is the maker of the questioned document. Nor will a document examiner be able to testify as to identity in terms of probabilities.”); U.S. v. McVeigh, 1997 WL 47724 3 (D. Colo. 1997) (holding that a pattern recognition expert could not testify to ultimate source attribution for unknown handwriting evidence).

[262] People v. Kirk, 289 Ill. App. 3d 326, 333 (4th Dist. 1997); see Donaldson, 199 Ill. 2d at 85 (same); In re Det. of New, 2014 IL 116306, ¶48 (2014) (same); McKown, 236 Ill. 2d at 303-04 (declining to rely on cases from other jurisdictions that had been undermined by later scientific developments or where the HGN issue had not been as fully litigated).

[263] 2014 IL 116306, at ¶51.

[264] 289 Ill. App. 3d at 333-34.

[265] See 2013 IL App. (1st) 102476, at ¶91 (“[I]n recent years, federal and state courts have had occasion to revisit the admission of expert testimony based on toolmark and firearms identification methodology. Such testimony has been the subject of lengthy and detailed hearings, and measured against the standards of both Frye and Daubert. Courts have considered scholarly criticism of the methodology, and occasionally placed limitations on the opinions experts may offer based on the methodology. Yet the judicial decisions uniformly conclude toolmark and firearms identification is generally accepted and admissible at trial”).

[266] See Rodriguez, 2017 IL App. (1st) 141379, at ¶56-57 (looking only at the NAS reports); Robinson, 2013 IL App (1st) 102476, at ¶90 (looking only at NAS reports and one other article by a firearms examiner).

[267] See Goodwin-Bey, No. 1531-CR00555-01 (Dec. 16, 2016) (ultimately and “reluctantly” admitting firearms examination evidence but noting that although “toolmark identification is a very valuable investigative tool. That is where it should stay, in the area of law enforcement, not in the courts.”) It does bear mentioning that despite admitting the evidence, Goodwin-Bey made no finding on general acceptance, instead admitting firearms examination testimony after concluding only that it required specialized knowledge.

[268] Green, 405 F.Supp.2d at 123.

[269] McKown, 236 Ill. 2d at 305; Luna, 2013 IL App (1st) 072253, at ¶72; People v. Floyd, 2014 IL App (2d) 120507, ¶22-24 (2d Dist. 2014); Murray v. Motorola, Inc., 2014 D.C. Super. LEXIS 16, 33-35, 56-58 (D.C. Super. Ct. 2014); United States v. Frazier,  387 F.3d 1244, 1263 (11th Cir. 2004);United States v. Van Wyk, 83 F. Supp. 2d 515 (D.N.J. 2000); United States v. Santillan, 1999 WL 1201765 (N.D. Ca 1999); United States v. Reynolds, 904 F.Supp. 1529, 1558 (E.D. Oka. 1995); People v. Shreck, 22 P.3d 68, 70 (Colo. 2001); Daubert v. Merrell Dow Pharms., 509 U.S. 579, 595 (1993); Bowers, “Forensic Testimony: Science, Law and Expert Evidence,” Academic Press (2014); Mnookin, “The Courts, NAS, & the Future of Forensic Sciences,” 75 Brooklyn L. R.  51-55 (2010).

[270] 2014 D.C. Super. LEXIS at ¶60.

[271] 2014 IL App (2d) 120507 at ¶23-24.

[272] See e.g., Ramirez, 810 So. 2d at 845 (barring the testimony of an expert who claimed to be able to identify the knife used in a murder by replicating marks on left on a victim’s cartridge, despite the general acceptance of the wider field of toolmark identification;); People v. Ferguson, 172 Ill. App. 3d 1, 8 & 12 (2d Dist. 1988) (excluded the testimony of an expert who claimed to be able to identify a suspect based on wear patterns repeated across multiple pairs of his shoes;); United States v. McCluskey, 954 F. Supp. 2d 1224, 1280-81 & 1286 (D.N.M. 2013) (analysis of low copy DNA insufficiently reliable despite the widespread use and reliability of other forms of DNA.); Sexton v. Texas, 93 S.W.3d 96, (2002) (“We conclude, based on the record before us, that the underlying theory of toolmark examination could be reliable in a given case, but that the State failed to produce evidence of the reliability of the technique [considering magazine marks alone] used in this case.”); Almeciga v. Ctr. for Investigative Reporting, Inc., 2016 U.S. Dist. LEXIS 60539, *54 (S.D.N.Y. May 6, 2016) (“It would be an abdication of this Court's gatekeeping role under Rule 702 to admit Carlson's testimony [regarding document examination] in light of its deficiencies and unreliability. Accordingly, Carlson's testimony must be excluded in its entirety.”).

[273] See United States v. St. Gerard, APO AE 09107 (2010).

[274] Daubert, 509 U.S. at 595 (internal quotations & citations omitted).

[275] Mouzone, 696 F.Supp.2d at 569.

[276] Murray, 2014 D.C. Super. LEXIS 16; see also Frazier, 387 F.3d at 1263; United States v. Monteiro, 407 F.Supp.2d 351, 358 (D. Mass. 2006) (“The court’s vigilant exercise of this gatekeeper role is critical because of the latitude given to expert witnesses to express their opinions on matters about which they have no firsthand knowledge, and because an expert’s testimony may be given greater weight by the jury due to the expert’s background and approach.”)

[277] The Honorable Harry T. Edwards, “The National Academy of Sciences Report on Forensic Sciences: What it Means for the Bench & Bar,” Presentation to the Superior Court of DC (2010) (also encouraging courts to rely on the NAS report when deciding questions of admissibility); see  Glynn, 578 F.Supp.2d 567 (S.D.N.Y. 2008) (“cross-examination is inherently handicapped by the jury’s own lack of background knowledge, so that the Court must play a greater role, not only in excluding unreliable testimony, but also in alerting the jury to the limitations of what is presented.”); Murray, 2014 D.C. Super. LEXIS at ¶60 (“the court cannot be confident that effective advocacy can eliminate the risk that a jury would be misled by [the expert’s] testimony and reach a result on an improper basis.”); Jennifer A. Mnookin, “Clueless Science,” L.A. TIMES (Feb. 19, 2009) (“[J]udges would be well advised to throw out forensic science altogether -- not forever, but until adequate research establishes, for example, that the conventional wisdom about evidence of arson is empirically valid, or until fingerprint and ballistics experts provide adequate proof that their real-world error rate is reasonably low.”); American Bar Association, “Forensic Sciences: Judges as Gatekeeper,” at 29-30 (2015).

[278] New, 2014 IL 116306, at ¶26; see also People v. Newberry, 166 Ill.2d 310, 316-17  (1995) (“The State asserts that [the defendant] is not without recourse because he can still assail the State's test results by…cross-examining the State's experts about the procedures they followed. While these opportunities may exist, the relief they offer is illusory.  Whatever the actual reliability of the tests performed in the lab -- and the reliability may not be great -- the laboratory analysis of the evidence will carry great weight with the jury”). 

[279] PCAST, “Forensic Science in Criminal Courts,” at 45.

[280] See e.g., Joseph Sanders, “Reliability Standards—Too High, Too Low, or Just Right? The Merits of the Paternalistic Justification for Restrictions on the Admissibility of Expert Evidence,” 33 Seton Hall L. Rev. 881, 936-938 (2003) (noting, in summary of the author’s analysis of a wide swath of literature, that the results “lend support to the argument that rulings excluding unreliable evidence promote jury accuracy even if we assume jurors are as good as judges in assessing reliability on jurors that” and that “the empirical research does lend some support to the paternalistic justification for restrictions on the admissibility of unreliable expert testimony”).

[281] See PCAST, “Forensic Science in Criminal Courts,” at 45(“In an online experiment, researchers asked mock jurors to estimate the frequency that a qualified, experienced forensic scientist would mistakenly conclude that two samples of specified types came from the same person when they actually came from two different people. The mock jurors believed such errors are likely to occur about 1 in 5.5 million for fingerprint analysis comparison; 1 in 1 million for bitemark comparison; 1 in 1 million for hair comparison; and 1 in 100 thousand for handwriting comparison. While precise error rates are not known for most of these techniques, all indications point to the actual error rates being orders of magnitude higher.”); William C. Thompson & Eryn J. Newman, “Lay Understanding of Forensic Statistics: Evaluation of Random Match Probabilities, Likelihood Ratios, & Verbal Equivalents,” 39 L. & Hum. Behav. 332 (2015) (juror evaluation of DNA evidence influenced by preconceived notions about the discipline & factfinders are susceptible to statistical fallacies, both prosecution and defense varieties); Jonathan J. Koehler, “If the Shoe Fits They Might Acquit: The Value of Forensic Science Testimony,” 8(s1) J. of Empirical Legal Studies 21-48 (2011) (“As detailed in the NRC report the ‘science’ part of forensic science has not kept pace with the extraordinary claims made on its behalf. As a result, jurors have little idea what the chance is that a forensic scientist’s conclusions are wrong, how often different objects share particular characteristics, or how much weight to give the forensic science as proof of identity.” Further noting that jurors “are slow to revise incorrect probabilistic hypotheses” “fall prey to logical fallacies” and “failed to appreciate the role that error plays in interpreting the value of a reported match”); Dawn McQuiston-Surrett & Michael J. Saks, “Communicating Opinion Evidence in the Forensic Identification Sciences: Accuracy & Impact,” 59 Hastings L.J. 1159, 1170 (2008) (“most jurors have an exaggerated view of the nature and capabilities of forensic identification”); Sanders, “Reliability Standards—Too High, Too Low, or Just Right?,” at 901, 919 (describing jurors as struggling with statistical information and unable to detect expert witness biases).

[282] Koehler, “If the Shoe Fits They Might Acquit,” (“Contrary to predictions, none of the source and guilt dependent measures in the main experiment were affected by the introduction of cross examination. There was no effect for cross examination on source confidence, source probability, guilt confidence, guilty probability, or verdict. Likewise there was no effect for cross examination across the two individualization conditions on any of the dependent measures.”); Sanders, “Reliability Standards—Too High, Too Low, or Just Right?,” at 913, 934-36 (Concluding that multiple studies bear out the sobering reality that even robust cross examination of experts affects neither ultimate verdicts nor even juror confidence in said verdicts); Saks, “The Testimony of Forensic Identification Science,” (Authors conducted a study and reviewed others, ultimately finding “little or no ability of cross-examination to undo the effects of an expert’s testimony on direct examination, even if the direct testimony is fraught with weaknesses and the cross is well designed to expose those weaknesses.” Interestingly, the authors conclude that cross examination can effect juror evaluation of expert evidence if it is presented honestly as a subjective guess, but that “...the unshakeableness of the traditional forms: match and similar-in-all-microscopic-characteristics produce something of a ceiling effect, which resist moderation by the presentation of other information.”);Shari Seidman Diamond, et al., “Juror Reactions to Attorneys At Trial,” 87 J. Crim. L. & Criminology 17, 41 (1996) (The author conducted an experiment, using 1925 jury-eligible residents of Cook County, which varied the strength of an attorney’s cross examination of an expert witness and found that: “Although juror perceptions of the attorney appear susceptible to influence by the attorney's efforts during cross-examination, the strong cross-examination had no effect on the verdict.”).

[283] PCAST, “Forensic Science in Criminal Courts,” at 45-46 (“The potential prejudicial impact is unusually high, because jurors are likely to overestimate the probative value of a “match” between samples” thus the term match conveys “inappropriately high probative value, a more neutral term should be used for an examiner’s belief that two samples come from the same source.”);  Koehler, “If the Shoe Fits They Might Acquit,” (“people are more persuaded by statistical testimony that ignores various error risks than by testimony that is objectively stronger by virtue of taking those risks into account”);Sanders, “Reliability Standards—Too High, Too Low, or Just Right?,” at 935 (Concluding that  testimony couched in terms of an expert’s experience, was “more impervious to cross-examination and opposing experts.”); Saks, “Communicating Opinion Evidence in the Forensic Identification Sciences,” at 1177 (“The conclusions of examiners in all areas of forensic identification other than DNA typing reach their conclusions on the basis of subjective guesstimations (clinical rather than actuarial), they present their opinions in nonquantitative, usually categorical, terms, and by all indications laypersons are generally quite persuaded by their testimony.”); Dawn McQuiston-Surrett & Michael J. Saks, “The Testimony of Forensic Identification Science: What Expert Witnesses Say & What Factfinders Hear,” 33 Law & Hum. Behav. 436 (2009) (“Participants in the conditions [hearing testimony in terms of a match or that targets were similar in all microscopic characteristics] which led to the highest estimates that the crime scene hair came from the defendant paradoxically gave the highest estimates of the incidence of the same hair traits in the reference population. This reinforces the inference that those two testimonial conditions lead to the least understanding of the basic concepts of forensic identification while leading to the highest inculpatory judgments” & “These data suggest that the two traditional forms in which forensic identification testimony is expressed [again referring to match of the similar-in-all-microscopic-characteristics language] are most damaging to the defense, while communicating a comfortingly simple and easily grasped (though not very informative and presumably misleading) understanding of the basis for the identification opinion.”); John Thorton, “The General Assumptions & Rationale of Forensic Identification,” In Modern Scientific Evidence: The Law & Science of Expert Testimony, at 16 (1997) (when an expert “and bases [an] opinion on ‘years of experience’ the practical result is that the witness is immunized against effective cross examination”).

[284] Sanders, “Reliability Standards—Too High, Too Low, or Just Right?,” at 934. Moreover, even the task of locating a favorable expert has more to do with luck than the ground truth of a bullet or cartridge’s actual source. To clarify, because an expert’s subjective conception of agreement sufficient to declare a match boils down to the prior known non-matches such an expert has personally examined, the defense could not simply request the assistance of any one examiner. Such an expert may well reach the same conclusion as the State’s hired-hand, but lurking just across the border, or down the street, or even in the same lab or agency may be another practitioner who by chance has come upon more guns of a similar make and model and encountered a better non-match than have his/her peers. Such knowledge may not even be correlated to experience. See Thomas Fadul, et al., “An Empirical Study to Improve the Scientific Foundation of Forensic Firearm and Tool Mark Identification Utilizing Consecutively Manufactured Glock EBIS Barrels with the Same EBIS Pattern,” DOJ Grant Project, at 30 (December 2013). But regardless, the relationship will never be absolute. Therefore, only by querying every firearms examiner across the globe could a defendant rest assured that he has not missed the few examiners with the relevant entry in the database of their mind’s eye necessary to reach the correct determination. But that collective concept of best known non-match could never be made available to the factfinder.

[285] McQuiston-Surrett & Saks, “Communicating Opinion Evidence in the Forensic Identification Sciences, at 1188; see also Sanders, Reliability Standards—Too High, Too Low, or Just Right?,” at 936 (same).

[286] See e.g., People v. Baynes, 88 Ill. 2d 225, 240 (Ill. 1981) (excluding polygraph evidence under the 403 calculus in part because, despite advances in the tool’s performance, its “recordings cannot be interpreted with the degree of accuracy that would render them reliable enough for the court to accept them into evidence”); Harbold, 124 Ill. App. 3d at 379 (questioning the probative value of genetic marker evidence on the basis of reliability and noting that: “One of the most important factors in Illinois' rejection of polygraph analysis, the subjectivity of interpretation of test results, is also involved in genetic marker analysis.”)

[287] PCAST, “Forensic Science in Criminal Courts,” at 53.

[288] United States v. Yee, 134 F.R.D. 161, 181 (N.D. Ohio 1990).

[289] See Modelski v. Navistar Int'l Transp. Corp., 302 Ill. App. 3d 879, 886 (1st Dist. 1999) (emphasizing that expert “testimony grounded in guess, surmise, or conjecture, not being regarded as proof of a fact, is irrelevant as it has no tendency to make the existence of a fact more or less probable.”); People v. Sargeant, 292 Ill. App. 3d 508, 511 (1st Dist. 1997) (excluding the “inconclusive, tentative, and speculative” testimony of a handwriting expert).

[290] See Harbold, 124 Ill.App.3d at 382-84 (expert testimony not relevant because “Jurors would be hard pressed to explain how the 1-in-500 chance of an accidental match did not equate with a 1-in-500 chance that defendant was innocent. Of course, the statistic means nothing of the sort. Absent a sound basis to limit the number of possible defendants, the defendant here is but one of thousands of people who share these same characteristics. Legion possibilities incapable of quantification, such as the potential for human error or fabrication, or the possibility of a frame-up, must be excluded from the probability calculation.”); People v. Pike, 2016 IL App (1st) 122626, ¶¶72-75 (1st Dist. 2016) (excluding DNA statistic that would include 50% of the population as irrelevant as well as more prejudicial than probative).

[291] PCAST, “Forensic Science in Criminal Courts,” at 53.

[292] See People v. Zayas, 131 Ill. 2d 284, 292 (1989) (in ruling hypnotically-assisted-recall testimony inadmissible court emphasized the likelihood and danger of prior juror exposure to misleading information about hypnosis); Baynes, 88 Ill.2d at 244 (“There is significant risk the jury will regard [polygraph] evidence as conclusive…It is questionable whether any jury would follow limiting instructions because the polygraph evidence is likely to be shrouded with an aura of near infallibility, akin to the ancient oracle of Delphi.”) (internal citations & quotations omitted).

[293] D. Michael Risinger & Michael J. Saks, “A House With No Foundation,” Issues in Science & Technology, Vol. XX, Issue I (2003); Compare PCAST, “Forensic Science in Criminal Courts,” at 26 (explaining that decisions excluding DNA evidence actually forced practitioners to team with molecular biologists and develop rigorously scientific standards and practices).

[294] Paul C. Gianelli, “Crime Labs Need Improvement,” Issues in Science & Technology, Vol. XX, Issue I (2003).

[295] Kozinski, “Rejecting Voodoo Science in the Courtroom,” Wall Street Journal (Sept. 19, 2016).

[296] Even if this Court determines that the proposed firearms examination testimony survives review under Frye and Rule 403, the shortcomings of such evidence expounded on throughout this motion (i.e. the lack of adequate empirical validation, questionable peer review, limited understanding of error rates, vague / tautological standards, and wholesale rejection by leading scientific authorities outside the insular community of firearms examiners) would render it inadmissible under the standards set forth in Daubert v. Merrell Dow Pharms., 509 U.S. 579 (1993) and Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999). And, “while our supreme court has recently noted that Illinois courts have not addressed the issue of whether Daubert should supplant Frye, it has continued to hint that this issue is ripe for its consideration.” See Donnellan v. First Student, Inc., 383 Ill. App. 3d 1040, 1057 (1st Dist. 2008). Mr. Jones acknowledges, however, that regardless of said intimation, this Court is bound to apply Frye and Rule 403, rather than Daubert, unless and until the Illinois Supreme Court adopts a new test for the admissibility of expert testimony. See Luna, 2013 IL App (1st) 072253 at ¶121. But the Frye standard, unlike that of Daubert, simply fails to comport with contemporary understanding of how to gauge the validity of scientific evidence (exemplified by the PCAST report’s rejection of mere training or longstanding use in favor of rigorous empirical testing). As such it does more harm than good and “is potentially capricious because it excludes scientifically reliable evidence which is not yet generally accepted, and admits scientifically unreliable evidence which although generally accepted, cannot meet rigorous scientific scrutiny.” State v. Coon, 974 P.2d 386, 393-394, (Alaska 1999). Therefore, Mr. Jones preserves for review by the Illinois Supreme Court the issues of whether sound public policy and respect for the due process rights of criminal defendants should compel adoption of the Daubert standard in criminal cases, as well as whether the firearms evidence in this case could possibly satisfy that standard.  See Manson v. Brathwaite, 432 U.S. 98, 114 (1977) (Under the Due Process Clause of the Fourteenth Amendment “reliability is the linchpin of determining the admissibility” of evidence.)

Please go to our Moodle Page and under "Class 4" you will find the prompt and submission folder for Writing Reflection #4.

This chapter covers rule 702 and the Supreme Court's decision in Daubert. If you do not have a copy of the textbook, please consult our Moodle course page.

. . .

A. Brave New World

Federal judges ruling on the admissibility of expert scientific testimony face a far more complex and daunting task in a postDaubert world than before. The judge’s task under Frye is relatively simple: to determine whether the method employed by the experts is generally accepted in the scientific community. Solomon, 753 F.2d at 1526. Under Daubert, we must engage in a difficult, two-part analysis. First, we must determine nothing less than whether the experts’ testimony reflects “scientific knowledge,” whether their findings are “derived by the scientific method,” and whether their work product amounts to “good science.” — U.S. at -, -, 113 S.Ct. at 2795, 2797. Second, we must ensure that the proposed expert testimony is “relevant to the task at hand,” id. at -, 113 S.Ct. at 2797, i.e., that it logically advances a material aspect of the proposing party’s ease. The Supreme Court referred to this second prong of the analysis as the “fit” requirement. Id. at -, 113 S.Ct. at 2796.

The first prong of Daubert puts federal judges in an uncomfortable position. The question of admissibility only arises if it is first established that the individuals whose testimony is being proffered are experts in a particular scientific field; here, for example, the Supreme Court waxed eloquent on the impressive qualifications of plaintiffs’ experts. Id. at - n. 2, 113 S.Ct. at 2791 n. 2. Yet something doesn’t become “scientific knowledge” just because it’s uttered by a *1316scientist; nor can an expert’s self-serving assertion that his conclusions were “derived by the scientific method” be deemed conclusive, else the Supreme Court’s opinion could have ended with footnote two. As we read the Supreme Court’s teaching in Daubert, therefore, though we are largely untrained in science and certainly no match for any of the witnesses whose testimony we are reviewing, it is our responsibility to determine whether those experts’ proposed testimony amounts to “scientific knowledge,” constitutes “good science,” and was “derived by the scientific method.”

The task before us is more daunting still when the dispute concerns matters at the very cutting edge of scientific research, where fact meets theory and certainty dissolves into probability. As the record in this case illustrates, scientists often have vigorous and sincere disagreements as to what research methodology is proper,- what should be accepted as sufficient proof for the existence of a “fact,” and whether information derived by a particular method can tell us anything useful about the subject under study.

Our responsibility, then, unless we badly misread the Supreme Court’s opinion, is to resolve disputes among respected, well-ere-dentialed scientists about matters squarely within their expertise, in areas where there is no scientific consensus as to what is and what is not “good science,” and occasionally to reject such expert testimony because it was not “derived by the scientific method.” Mindful of our position in the hierarchy of the federal judiciary, we take a deep breath and proceed with this heady task.

B. Deus ex Machina

The Supreme Court’s opinion in Daubert focuses closely on the language of Fed. R.Evid. 702, which permits opinion testimony by experts as to matters amounting to “scientific ... knowledge.” The Court recognized, however, that knowledge in this context does not mean absolute certainty. — U.S. at -, 113 S.Ct. at 2795. Rather, the Court said, “in order to qualify as ‘scientific knowledge,’ an inference or assertion must be derived by the scientific method.” Id. Elsewhere in its opinion, the Court noted that Rule 702 is satisfied where the proffered testimony is “based on scientifically valid principles.” Id. at -, 113 S.Ct. at 2799. Our task, then, is to analyze not what the experts say, but what basis they have for saying it.

Which raises the question: How do we figure out whether scientists have derived their findings through the scientific method or whether their testimony is based on scientifically valid principles? Each expert proffered by the plaintiffs assures us that he has “utiliz[ed] the type of data that is generally and reasonably relied upon by scientists” in the relevant field, see, e.g., Newman Aff. at 5, and that he has “utilized the methods and methodology that would generally and reasonably be accepted” by people who deal in these matters, see, e.g., Gross Aff. at 5. The Court held, however, that federal judges perform a “gatekeeping role,” Daubert, — U.S. at -, 113 S.Ct. at 2798; to do so they must satisfy themselves that scientific evidence meets a certain standard of reliability before it is admitted. This means that the expert’s bald assurance of validity is not enough. Rather, the party presenting the expert must show that the expert’s findings are based on sound science, and this will require some objective, independent validation of the expert’s methodology.

While declining to set forth a “definitive checklist or test,” id. at -, 113 S.Ct. at 2796, the Court did list several factors federal judges can consider in determining whether to admit expert scientific testimony under Fed.R.Evid. 702: whether the theory or technique employed by the expert is generally accepted in the scientific community; whether it’s been subjected to peer review and publication; whether it can be and has been tested; and whether the known or potential rate of error is acceptable. Id. at -, 113 S.Ct. at 2796-97.3 We read these *1317factors as illustrative rather than exhaustive; similarly, we do not deem each of them to be equally applicable (or applicable at all) in every ease.4 Rather, we read the Supreme Court as instructing us to determine whether the analysis undergirding the experts’ testimony falls within the range of accepted standards governing how scientists conduct their research and reach their conclusions.

One very significant fact to be considered is whether the experts are proposing to testify about matters growing naturally and directly out of research , they have conducted independent of the litigation, or whether they have developed their opinions expressly for purposes of testifying. That an expert testifies for money does not necessarily cast doubt on the reliability of his testimony, as few experts appear in court merely as an eleemosynary gesture. But in determining whether proposed expert testimony amounts to good science, we may not ignore the fact that a scientist’s normal workplace is the lab or the field, not the courtroom or the lawyer’s office.5

That an expert testifies based on research he has conducted independent of the litigation provides important, objective proof that the research comports with the dictates of good science. See Peter W. Huber, Galileo’s Revenge: Junk Science in the Courtroom 206-09 (1991) (describing how the prevalent practice of expert-shopping leads to bad science). For one thing, experts whose findings flow from existing research are less likely to have been biased toward a particular conclusion by the promise of remuneration; when an expert prepares reports and findings before being hired as a witness, that record will limit the degree to which he can tailor his testimony to serve a party’s interests. Then, too, independent research carries its own indicia of reliability, as it is conducted, so to speak, in the usual course of business and must normally satisfy a variety of standards to attract funding and institutional support. Finally, there is usually a limited number of scientists actively conducting research on the very subject that is germane to a particular case, which provides a natural constraint, on parties’ ability to shop for experts who will come to the desired conclusion. That the testimony proffered by an expert is based directly on legitimate, preexisting research unrelated to the litigation provides the most persuasive basis for concluding that the opinions he expresses were “derived by the scientific method.”

We have examined carefully the affidavits proffered by plaintiffs’ experts, as well as the testimony from prior trials that plaintiffs have introduced in support of that testimony, and find that none of the experts based his testimony on preexisting or independent research. While plaintiffs’ scientists are all experts in their respective fields, none claims to have studied the effect of Bendectin on limb reduction defects before being hired to testify in this or related cases.

If the proffered expert testimony is not based on independent research, the party *1318proffering it must come forward with other objective, verifiable evidence that the testimony is based on “scientifically valid principles.” One means of showing this is by proof that the research and analysis supporting the proffered conclusions have been subjected to normal scientific scrutiny through peer review and publication.6 Huber, Galileo’s Revenge at 209 (suggesting that “[t]he ultimate test of [a scientific expert’s] integrity is her readiness to publish and be damned”).

Peer review and publication do not, of course, guarantee that the conclusions reached are correct; much published scientific research is greeted with intense skepticism and is not borne out by further research. But the test under Daubert is not the correctness of the expert’s conclusions but the soundness of his methodology. See n. 11 infra. That the research is accepted for publication in a reputable scientific journal after being subjected to the usual rigors of peer review is a significant indication that it is taken seriously by other scientists, i.e., that it meets at least the minimal criteria of good science. Daubert, — U.S. at -, 113 S.Ct. at 2797 (“[Scrutiny of the scientific community is a component of ‘good science.’”). If nothing else, peer review and publication “increase the likelihood that substantive flaws in methodology will be detected.” Daubert, — U.S. at -, 113 S.Ct. at 2797.7

Bendectin litigation has been pending in the courts for over a decade, yet the only review the plaintiffs’ experts’ work has received has been by judges and juries, and the only place their theories and studies have been published is in the pages of federal and state reporters.8 None of the plaintiffs’ experts has published his work on Bendectin in a scientific journal or solicited formal review by his colleagues. Despite the many years the controversy has been brewing, no one in the scientific community — except defendant’s experts — has deemed these studies worthy of verification, refutation or even comment. It’s as if there were a tacit understanding within the scientific community that what’s going on here is not science at all, but litigation.9

Establishing that an expert’s proffered testimony grows out of pre-litigation research or that the expert’s research has been subjected to peer review are the two principal ways the proponent of expert testimony can show that the evidence satisfies the first prong of Rule 702.10 Where such evidence is *1319unavailable, the proponent of expert scientific testimony may attempt to satisfy its burden through the testimony of its own experts. For such a showing to be sufficient, the experts must explain precisely how they went about reaching their conclusions and point to some objective source — a learned treatise, the policy statement of a professional association, a published article in a reputable scientific journal or the like — to show that they have followed the scientific method, as it is practiced by (at least) a recognized minority of scientists in their field. See United States v. Rincon, 28 F.3d 921, 924 (9th Cir.1994) (research must be described “in sufficient detail that the district court [can] determine if the research was scientifically valid”).11

Plaintiffs have made no such showing. . . . 

A witness who is qualified as an expert by knowledge, skill, experience, training, or education may testify in the form of an opinion or otherwise if:

(a) the expert’s scientific, technical, or other specialized knowledge will help the trier of fact to understand the evidence or to determine a fact in issue;

(b) the testimony is based on sufficient facts or data;

(c) the testimony is the product of reliable principles and methods; and

(d) the expert has reliably applied the principles and methods to the facts of the case.

Rule 702. Testimony by Expert Witnesses

 [the proposed text is indicated in bold and strikethrough]

A witness who is qualified as an expert by knowledge, skill, experience, training, or education may testify in the form of an opinion or otherwise if the proponent has demonstrated by a preponderance of the evidence that:

(a) the expert’s scientific, technical, or other specialized knowledge will help the trier of fact to understand the evidence or to determine a fact in issue;

(b) the testimony is based on sufficient facts or data;

(c) the testimony is the product of reliable principles and methods; and

(d) the expert has reliably applied expert’s opinion reflects a reliable application of the principles and methods to the facts of the
case.

Committee Note:

Rule 702 has been amended in two respects:
(1) First, the rule has been amended to clarify and emphasize that expert testimony may not be admitted unless the proponent demonstrates to the court that it is more likely than not that the proffered testimony meets the admissibility requirements set forth in the rule. See Rule 104(a). This is the preponderance of the evidence standard that applies to most of the admissibility requirements set forth in the evidence rules. See Bourjaily v. United States, 483 U.S. 171, 175 (1987) (“The preponderance standard ensures that before admitting evidence, the court will have found it more likely than not that the technical issues and policy concerns addressed by the Federal Rules of Evidence have been afforded due consideration.”); Huddleston v. United States, 485 U.S. 681, 687 (1988) (“preliminary factual findings under Rule 104(a) are subject to the preponderance-of-theevidence standard”). But many courts have held that the critical questions of the sufficiency of an expert’s basis, and the application of the expert’s methodology, are questions of weight and not admissibility. These rulings are an incorrect application of Rules 702 and 104(a).

There is no intent to raise any negative inference regarding the applicability of the Rule 104(a) standard of proof for other rules. The Committee concluded that emphasizing the preponderance standard in Rule 702 specifically was made necessary by the courts that have failed to apply correctly the reliability requirements of that rule. Nor does the rule require that the court make a finding of reliability in the absence of objection.

The amendment clarifies that the preponderance standard applies to the three reliability-based requirements added in 2000—requirements that many courts have incorrectly determined to be governed by the more permissive Rule 104(b) standard. But it remains the case that other admissibility requirements in the rule (such as that the expert must be qualified and the expert’s testimony must help the trier of fact) are governed by the Rule 104(a) standard as well.

Some challenges to expert testimony will raise matters of weight rather than admissibility even under the Rule 104(a) standard. For example, if the court finds it more likely than not that an expert has a sufficient basis to support an opinion, the fact that the expert has not read every single study that exists may raise a question of weight and not admissibility. But this does not mean, as certain courts have held, that arguments about the sufficiency of an expert’s basis always go to weight and not admissibility. Rather it means that once the court has found it more likely than not that the admissibility requirement has been met, any attack by the opponent will go only to the weight of the evidence.

It will often occur that experts come to different conclusions based on contested sets of facts. Where that is so, the Rule 104(a) standard does not necessarily require exclusion of either side’s experts. Rather, by deciding the disputed facts, the jury can decide which side’s experts to credit. “[P]roponents ‘do not have to demonstrate to the judge by a preponderance of the evidence that the assessments of their experts are correct, they only have to demonstrate by a preponderance of evidence that their opinions are reliable. . . . The evidentiary requirement of reliability is lower than the merits standard of correctness.’” Committee Note to the 2000 amendment to Rule 702, quoting In re Paoli R.R. Yard PCB Litig., 35 F.3d 717, 744 (3d Cir. 1994).

Rule 702 requires that the expert’s knowledge “help” the trier of fact to understand the evidence or to determine a fact in issue. Unfortunately, some courts have required the expert’s testimony to “appreciably help” the trier of fact. Applying a higher standard than helpfulness to otherwise
reliable expert testimony is unnecessarily strict.

(2) Rule 702(d) has also been amended to emphasize that each expert opinion must stay within the bounds of what can be concluded from a reliable application of the expert’s basis and methodology. Judicial gatekeeping is essential because just as jurors may be unable, due to lack of specialized knowledge, to evaluate meaningfully the reliability of scientific and other methods underlying expert opinion, jurors may also lack the specialized knowledge to determine whether the conclusions of an expert go beyond what the expert’s basis and methodology may reliably support.

The amendment is especially pertinent to the testimony of forensic experts in both criminal and civil cases. Forensic experts should avoid assertions of absolute or one hundred percent certainty—or to a reasonable degree of scientific certainty—if the methodology is subjective and thus potentially subject to error. In deciding whether to admit forensic expert testimony, the judge should (where possible) receive an estimate of the known or potential rate of error of the methodology employed, based (where appropriate) on studies that reflect how often the method produces accurate results. Expert opinion testimony regarding the weight of feature comparison evidence (i.e., evidence that a set of features corresponds between two examined items) must be limited to those inferences that can reasonably be drawn from a reliable application of the principles and methods. This amendment does not, however, bar testimony that comports with substantive law requiring opinions to a particular degree of certainty.

Nothing in the amendment imposes any new, specific procedures. Rather, the amendment is simply intended to clarify that Rule 104(a)’s requirement applies to expert opinions under Rule 702. Similarly, nothing in the amendment requires the court to nitpick an expert’s opinion in order to reach a perfect expression of what the basis and methodology can support. The Rule 104(a) standard does not require perfection. On the other hand, it does not permit the expert to make claims that are unsupported by the expert’s basis and methodology.

Please go to our Moodle Page and under "Class 5" you will find the prompt and submission folder for Writing Reflection #5.

As stated earlier, the amendment does not distinguish between scientific and other forms of expert testimony. The trial court's gatekeeping function applies to testimony by any expert. See Kumho Tire Co. v. Carmichael, 119 S.Ct. 1167, 1171 (1999) (“We conclude that Daubert's general holding—setting forth the trial judge's general ‘gatekeeping’ obligation—applies not only to testimony based on ‘scientific’ knowledge, but also to testimony based on ‘technical’ and ‘other specialized’ knowledge.”). While the relevant factors for determining reliability will vary from expertise to expertise, the amendment rejects the premise that an expert's testimony should be treated more permissively simply because it is outside the realm of science. An opinion from an expert who is not a scientist should receive the same degree of scrutiny for reliability as an opinion from an expert who purports to be a scientist. See Watkins v. Telsmith, Inc., 121 F.3d 984, 991 (5th Cir. 1997) (“[I]t seems exactly backwards that experts who purport to rely on general engineering principles and practical experience might escape screening by the district court simply by stating that their conclusions were not reached by any particular method or technique.”). Some types of expert testimony will be more objectively verifiable, and subject to the expectations of falsifiability, peer review, and publication, than others. Some types of expert testimony will not rely on anything like a scientific method, and so will have to be evaluated by reference to other standard principles attendant to the particular area of expertise. The trial judge in all cases of proffered expert testimony must find that it is properly grounded, well-reasoned, and not speculative before it can be admitted. The expert's testimony must be grounded in an accepted body of learning or experience in the expert's field, and the expert must explain how the conclusion is so grounded. See, e.g., American College of Trial Lawyers, Standards and Procedures for Determining the Admissibility of Expert Testimony after Daubert, 157 F.R.D. 571, 579 (1994) (“[W]hether the testimony concerns economic principles, accounting standards, property valuation or other non-scientific subjects, it should be evaluated by reference to the ‘knowledge and experience’ of that particular field.”).

The amendment requires that the testimony must be the product of reliable principles and methods that are reliably applied to the facts of the case. While the terms “principles” and “methods” may convey a certain impression when applied to scientific knowledge, they remain relevant when applied to testimony based on technical or other specialized knowledge. For example, when a law enforcement agent testifies regarding the use of code words in a drug transaction, the principle used by the agent is that participants in such transactions regularly use code words to conceal the nature of their activities. The method used by the agent is the application of extensive experience to analyze the meaning of the conversations. So long as the principles and methods are reliable and applied reliably to the facts of the case, this type of testimony should be admitted.

Nothing in this amendment is intended to suggest that experience alone—or experience in conjunction with other knowledge, skill, training or education—may not provide a sufficient foundation for expert testimony. To the contrary, the text of Rule 702 expressly contemplates that an expert may be qualified on the basis of experience. In certain fields, experience is the predominant, if not sole, basis for a great deal of reliable expert testimony. See, e.g., United States v. Jones, 107 F.3d 1147 (6th Cir. 1997) (no abuse of discretion in admitting the testimony of a handwriting examiner who had years of practical experience and extensive training, and who explained his methodology in detail); Tassin v. Sears Roebuck, 946 F.Supp. 1241, 1248 (M.D.La. 1996) (design engineer's testimony can be admissible when the expert's opinions “are based on facts, a reasonable investigation, and traditional technical/mechanical expertise, and he provides a reasonable link between the information and procedures he uses and the conclusions he reaches”). See also Kumho Tire Co. v. Carmichael, 119 S.Ct. 1167, 1178 (1999) (stating that “no one denies that an expert might draw a conclusion from a set of observations based on extensive and specialized experience.”).

If the witness is relying solely or primarily on experience, then the witness must explain how that experience leads to the conclusion reached, why that experience is a sufficient basis for the opinion, and how that experience is reliably applied to the facts. The trial court's gatekeeping function requires more than simply “taking the expert's word for it.” See Daubert v. Merrell Dow Pharmaceuticals, Inc., 43 F.3d 1311, 1319 (9th Cir. 1995) (“We've been presented with only the experts’ qualifications, their conclusions and their assurances of reliability. Under Daubert, that's not enough.”). The more subjective and controversial the expert's inquiry, the more likely the testimony should be excluded as unreliable. See O'Conner v. Commonwealth Edison Co., 13 F.3d 1090 (7th Cir. 1994) (expert testimony based on a completely subjective methodology held properly excluded). See also Kumho Tire Co. v. Carmichael, 119 S.Ct. 1167, 1176 (1999) (“[I]t will at times be useful to ask even of a witness whose expertise is based purely on experience, say, a perfume tester able to distinguish among 140 odors at a sniff, whether his preparation is of a kind that others in the field would recognize as acceptable.”).

This document is posted on Moodle under "Class 5."

This study of 400 judges demonstrates how poorly many judges understand at least four of the five Daubert factors. I highlighted the assigned text (which you will find on pages 438, 444-45, 447-48, 452-53, & 455).

Please go to our Moodle Page and under "Class 6" you will find the prompt and submission folder for Writing Reflection #6.

This case (available at this link) demonstrates what can happen when a judge considers the scientific method. It is an unusual case for many reasons, not least of which is the inclusion of a visual aid

Read any one of these three short lists of the characteristics of science:

Ten Characteristics of Scientific Research or Knowledge 

Five Characteristics of the Scientific Method 

Nine Characteristics of Scientific Research 

This chapter covers the process for qualifying an expert witness.

This section, § 9.03, provides an overview of the rules applicable to expert testimony and a sample examination. It is a pdf so it posted on Moodle.

In this excerpt you can see a real-life example of the process of qualifying the prosecution’s firearm expert through direct examination and oral motion. (This is a pdf so it is posted on Moodle.).

You can access more of the transcript here if you are interested.

Please go to our Moodle Page and under "Class 8" you will find the prompt and submission folder for Writing Reflection #8.

Chapter 63 covers rules 703 and 705 and the proper bases for an expert’s opinion. You can skim or skip the last section on the Confrontation Clause.

An expert may base an opinion on facts or data in the case that the expert has been made aware of or personally observed. If experts in the particular field would reasonably rely on those kinds of facts or data in forming an opinion on the subject, they need not be admissible for the opinion to be admitted. But if the facts or data would otherwise be inadmissible, the proponent of the opinion may disclose them to the jury only if their probative value in helping the jury evaluate the opinion substantially outweighs their prejudicial effect.

Optional reading: Advisory Committee Notes: 

Rule 703 has been amended to emphasize that when an expert reasonably relies on inadmissible information to form an opinion or inference, the underlying information is not admissible simply because the opinion or inference is admitted. Courts have reached different results on how to treat inadmissible information when it is reasonably relied upon by an expert in forming an opinion or drawing an inference. Compare United States v. Rollins, 862 F.2d 1282 (7th Cir. 1988) (admitting, as part of the basis of an FBI agent's expert opinion on the meaning of code language, the hearsay statements of an informant), with United States v. 0.59 Acres of Land, 109 F.3d 1493 (9th Cir. 1997) (error to admit hearsay offered as the basis of an expert opinion, without a limiting instruction). Commentators have also taken differing views. See, e.g., Ronald Carlson, Policing the Bases of Modern Expert Testimony, 39 Vand.L.Rev. 577 (1986) (advocating limits on the jury's consideration of otherwise inadmissible evidence used as the basis for an expert opinion); Paul Rice, Inadmissible Evidence as a Basis for Expert Testimony: A Response to Professor Carlson, 40 Vand.L.Rev. 583 (1987) (advocating unrestricted use of information reasonably relied upon by an expert).

When information is reasonably relied upon by an expert and yet is admissible only for the purpose of assisting the jury in evaluating an expert's opinion, a trial court applying this Rule must consider the information's probative value in assisting the jury to weigh the expert's opinion on the one hand, and the risk of prejudice resulting from the jury's potential misuse of the information for substantive purposes on the other. The information may be disclosed to the jury, upon objection, only if the trial court finds that the probative value of the information in assisting the jury to evaluate the expert's opinion substantially outweighs its prejudicial effect. If the otherwise inadmissible information is admitted under this balancing test, the trial judge must give a limiting instruction upon request, informing the jury that the underlying information must not be used for substantive purposes. See Rule 105. In determining the appropriate course, the trial court should consider the probable effectiveness or lack of effectiveness of a limiting instruction under the particular circumstances.

The amendment governs only the disclosure to the jury of information that is reasonably relied on by an expert, when that information is not admissible for substantive purposes. It is not intended to affect the admissibility of an expert's testimony. Nor does the amendment prevent an expert from relying on information that is inadmissible for substantive purposes.

Nothing in this Rule restricts the presentation of underlying expert facts or data when offered by an adverse party. See Rule 705. Of course, an adversary's attack on an expert's basis will often open the door to a proponent's rebuttal with information that was reasonably relied upon by the expert, even if that information would not have been discloseable initially under the balancing test provided by this amendment. Moreover, in some circumstances the proponent might wish to disclose information that is relied upon by the expert in order to “remove the sting” from the opponent's anticipated attack, and thereby prevent the jury from drawing an unfair negative inference. The trial court should take this consideration into account in applying the balancing test provided by this amendment.

Unless the court orders otherwise, an expert may state an opinion — and give the reasons for it — without first testifying to the underlying facts or data. But the expert may be required to disclose those facts or data on cross-examination.

Optional: Advisory Committee Notes:

The hypothetical question has been the target of a great deal of criticism as encouraging partisan bias, affording an opportunity for summing up in the middle of the case, and as complex and time consuming. Ladd, Expert Testimony, 5 Vand.L.Rev. 414, 426–427 (1952). While the rule allows counsel to make disclosure of the underlying facts or data as a preliminary to the giving of an expert opinion, if he chooses, the instances in which he is required to do so are reduced. This is true whether the expert bases his opinion on data furnished him at secondhand or observed by him at firsthand.

The elimination of the requirement of preliminary disclosure at the trial of underlying facts or data has a long background of support. In 1937 the Commissioners on Uniform State Laws incorporated a provision to this effect in the Model Expert Testimony Act, which furnished the basis for Uniform Rules 57 and 58. Rule 4515, N.Y. CPLR (McKinney 1963), provides:

“Unless the court orders otherwise, questions calling for the opinion of an expert witness need not be hypothetical in form, and the witness may state his opinion and reasons without first specifying the data upon which it is based. Upon cross-examination, he may be required to specify the data * * *,”

See also California Evidence Code §802; Kansas Code of Civil Procedure §§60–456, 60–457; New Jersey Evidence Rules 57, 58.

If the objection is made that leaving it to the cross-examiner to bring out the supporting data is essentially unfair, the answer is that he is under no compulsion to bring out any facts or data except those unfavorable to the opinion. The answer assumes that the cross-examiner has the advance knowledge which is essential for effective cross-examination. This advance knowledge has been afforded, though imperfectly, by the traditional foundation requirement. Rule 26(b)(4) of the Rules of Civil Procedure, as revised, provides for substantial discovery in this area, obviating in large measure the obstacles which have been raised in some instances to discovery of findings, underlying data, and even the identity of the experts. Friedenthal, Discovery and Use of an Adverse Party's Expert Information, 14 Stan.L.Rev. 455 (1962).

These safeguards are reinforced by the discretionary power of the judge to require preliminary disclosure in any event.

Notes of Advisory Committee on Rules—1993 Amendment

This rule, which relates to the manner of presenting testimony at trial, is revised to avoid an arguable conflict with revised Rules 26(a)(2)(B) and 26(e)(1) of the Federal Rules of Civil Procedure or with revised Rule 16 of the Federal Rules of Criminal Procedure, which require disclosure in advance of trial of the basis and reasons for an expert's opinions.

If a serious question is raised under Rule 702 or 703 as to the admissibility of expert testimony, disclosure of the underlying facts or data on which opinions are based may, of course, be needed by the court before deciding whether, and to what extent, the person should be allowed to testify. This rule does not preclude such an inquiry.

This is the expert notice for Melissa Gische filed in the Frye fingerprint pleading you have read about. Ms. Gische was also featured in the portion of the film we watched together about fingerprinting. (This document is a pdf, so it is posted on Moodle under "Class 8.") You can skim the document - I just want you to see the types of information that are included. 

Please go to our Moodle Page and under "Class 9" you will find the prompt and submission folder for Writing Reflection #9.