Posts filed under Forensic Science (10)

March 16, 2014

Same number of workers being caught on drugs?

The Herald said, on Friday “Fewer workers stoned on the job

Information from the New Zealand Drug Detection Agency showed 81,410 on-site drug-screening tests were carried out last year, 16 per cent up from the previous year.

But only 5.5 per cent of tests showed the presence of drugs, down from 6.4 per cent in 2013

As usual, there’s no mention of the fact that NZDDA is just one of the private companies offering drug testing services. It took me a long time to realise this, until I was tipped off by a news story advertising one of their competitors.

Presumably NZDDA don’t think their customers choose them at random, and with no real reason for wanting testing. If customers were behaving even a little rationally you’d expect an expansion of drug testing to pull in lower-risk employees. If we look at the actual number of positive tests, using the quoted figures, it was about 4480 last year and about 4490 in the previous year. Given no change in the number of positive tests and a 1 percentage point change in the proportion of positive tests, from a single company, there’s not a lot of numerical evidence for an increase in number of workers with detectable cannabis in their systems.

More importantly, there’s no evidence whatsoever for the ‘stoned on the job’ headline: absolutely no information is given about this. One of the big problems with cannabis testing is that there is no useful biochemical assay for being stoned. Detectable levels persist long after impairment is over, and even when you’re actually stoned there is not a good relationship between drug concentration and impairment.  This is a real problem for Washington and Colorado, which have legalised cannabis and need to set driving laws. In contrast to alcohol, if you actually care about safe driving and cannabis, it’s really hard to get a useful and objective test.

The story ends with two examples of disasters. In one, cannabis was definitely ruled out as a contributing factor; in the other, the conclusion was only that “it could not be excluded”. The NZDDA  press release is at Scoop, and despite how the story reads, there is surprisingly little text in common.

February 13, 2014

How stats fool juries

Prof Peter Donnelly’s TED talk. You might want to skip over the first few minutes of vaguely joke-like objects

Consider the two (coin-tossing) patterns HTH and HTT. Which of the following is true:

  1. The average number of tosses until HTH is larger than the average number of tosses until HTT
  2. The average number of tosses until HTH is the same as  the average number of tosses until HTT
  3. The average number of tosses until HTH is smaller than the average number of tosses until HTT?

Before you answer, you should know that most people, even mathematicians, get this wrong.

Also, as Prof Donnelly doesn’t point out, if you have a programming language handy, you can find out the answer very easily.

September 13, 2013

Stats on Radio NZ

Last night’s edition of Our Changing World had two segments by University of Auckland statisticians called James.

Allison Ballance and James Russell talked about seabirds and rat eradication

 

Ruth Beran and James Curran talked about forensic statistics

 

 

July 31, 2013

“10 quadrillion times more likely to have done it”

Thomas Lumley, tipped off by Luis Apiolaza on Twitter, pointed me to this article in the NZ Herald.

The article is yet another example of the Herald’s inability to correctly report DNA statistics. It makes the following statement:
This article reports a quote from the Crown Prosecutor, paraphrased as follows:

A man charged with raping a woman during a kidnap has pleaded not guilty but Crown says DNA evidence shows the man was “10,000,000,000,000,000 times likely” to be responsible for the crime.

To be fair to the article’s author, this may have been the statement that the Crown prosecutor made, but nNo forensic scientist in New Zealand would say this. ESR scientists are trained to give statements of the following type:

“The evidence is 1016 (=10,000,000,000,000,000) times more likely if the defendant and the victim were contributors to the stain, rather than the victim and someone unrelated to the defendant.”

It is extremely important to note that This is a statement about the likelihood of the evidence given the hypotheses rather than the other way around. A forensic scientist is bought to court to comment on the strength of the evidence and specifically not on whether the defendant is guilty.

I have commented on this before., and sent correspondence to the NZ Herald numerous times. Perhaps a mention on StatsChat will inspire change.

Update: The NZ Herald reporter, James Ihaka, has contacted me and said “The statement came from a Crown prosecutor about the evidence that the forensic scientist will present later in the trial. Taking in to consideration what you have said however, it would probably be more accurate to rephrase this.” Good on you James!

Update 2: James Ihaka has contacted me again, with the following information:

This is the direct quote from Crown prosecutor Rebecca Mann: ( I checked with her)
“It is ten thousand million million times more likely for the DNA these samples originated from (the complainant) and Mr Martin rather than from (the complainant) and another unrelated individual selected at random from the general New Zealand population.”

I apologize unreservedly for attributing this to James Ihaka, and again congratulate him for following it up.

The statement Ms. Mann should have given is


The evidence (the DNA match) is ten thousand million million times more likely if these samples originated from (the complainant) and Mr Martin rather than if they originated from (the complainant) and another unrelated individual selected at random from the general New Zealand population.”

March 10, 2013

Your media on drugs

Last night, 3News had a scare story about positive drug tests at work.  The web headline is “Report: More NZers working on drugs”, but that’s not what they had information on:

New figures reveal more New Zealanders were caught with drugs in their system at work last year.

…new figures from the New Zealand Drug Detection Agency reveal 4300 people tested positive for drugs at work last year.

but

The New Zealand Drug Detection Agency says employers are doing a better job of self-regulating. The agency performed almost 70,000 tests last year, 30 percent more than in 2011.

If 30% more were tested, you’d expect more to be positive. The story doesn’t say how many tested positive the previous year, but with the help of the Google, I found last year’s press release, which says

8% of men tested “non-negative” compared with 6% of women tested in 2011.

Now, 8% of 70000 is 5600, and even 6% of 70000 is 4200. Given that the majority of the tests are in men, it looks like the proportion testing positive went down this year.

The worst part of the story statistically is when they report changes in proportions of which drug was found as if this was meaningful.  For example,

When it comes to industries, oil and gas had an 18 percent drop in positive tests for methamphetamine, but showed a marked increase in the use of opiates.

That’s an increase in the use of opiates as a proportion of those testing positive.  Since proportions have to add up to 100%, a decrease in the proportion positive tests that are for methamphetamine has to come with an increase in some other set of drugs — just as a matter of arithmetic.

Stuff‘s story from January just as bad, with the lead

Employers are becoming more aware of the dangers of drugs and alcohol in the workplace as well as the benefits of testing for them.

and quoting an employer as saying

“And, we have no fear of an employee turning up to work and operating in an unsafe way, putting themselves and others at risk.”

as if occasional drug tests were the answer to all occupational health and safety problems.

The other interesting thing about the Stuff story is that it’s about a different organisation: Drug Testing Services, not NZ DDA — there’s more than one of them out there! You might easily have thought from the 3News story that the figures they quoted referred to all workplace drug tests in NZ, rather than just those sold by one company.

Given the claims being made, the evidence for either financial or safety benefits is amazingly weak.   No-one in these stories even claims that introducing testing has actually reduced  on-the-job accidents in their company, for example, let alone presents any data.

If you look on PubMed, the database of published medical research, there are lots of papers on new testing methods and reproducibility of test results, and a few that show people who have accidents are more likely than others to test positive.  There’s very little even of before-after comparisons: a Cochrane review on this topic found three before-after comparisons. Two of the three found a small decrease in accident rates immediately after introducing testing; the third did not.  A different two of the three found that the long-term decreasing trend in injuries got faster after introducing testing; again, the third did not.   The review concluded that there was insufficient evidence to recommend for or against testing.

There’s better evidence for mandatory alcohol testing of truck drivers, but since those tests measure current blood alcohol concentrations, not past use, it doesn’t tell us much about other types of drug testing.

 

 

February 15, 2013

Genuine sasquatch DNA probably not found

There’s been a bunch of publicity recently over claims that Bigfoot really exists and that a group of forensic scientists have the DNA to prove it.

After being rejected from the top journals either because of prejudice and hide-bound conservatism or because of not having any worthwhile evidence, the researchers have managed to publish some results in a peer-reviewed journal. That they set up for the purpose. (unkind scientists on Twitter are making jokes about the next issue, some of which are quite funny)

Ars Technica has the closest to actual information about the paper that I’ve seen, and their analysis sounds right to me. The paper says that the Bigfoot mitochondrial DNA matches humans, so the creature is a hybrid between humans and some unknown primate.  However, the mitochondrial DNA matches are mostly to sequences from Europe and the Middle East, not to Native American sequences, which looks like contamination rather than hybridisation.  Similarly, the results for nuclear DNA should show fairly long sequences matching humans, and other fairly long sequences that look similar to but not identical to other known primates, but they don’t seem to.

The genome data has only been released in PDF format, not in any of the formats that scientists normally use for storing genome sequences. When someone gets around to converting it, and the full surplus power of the world’s sequence matching software is turned loose, the results will be obvious — so the fact this hasn’t happened is not encouraging.

Is this scientific fraud?  Given the real attempts the researchers have made to publish their results, I think we can repeat an answer quoted by physicist Bob Park after the first cold fusion press conference: “Not yet.” And let’s hope it stays that way.

January 23, 2013

Statistical evidence and cheating at chess

At the Zadar Open chess competition last month, a player who had previously been at the low end of the chess master range did extraordinarily well, playing the level of the world’s very best. Or at the level of a good computer program. There was absolutely no physical evidence to suggest that he had been cheating, but his level of improvement, and the agreement between his moves and those produced by top computer programs are striking.  On the other hand, if you are going to allow accusations in the absence of any corroborating physical evidence, it’s also essentially impossible for an innocent person to mount a defense.

KW Regan, who is a computer scientist and chess master has analysed historical chess competition data, looking at agreement between actual moves and those the computer would recommend, and he claims the Zadar Open results should happen less often than once in a million matches. In his letter to the Association of Chess Professionals, he raises the questions

1.What procedures should be instituted for carrying out statistical tests for cheating with computers at chess and for disseminating their results? Under whose jurisdiction should they be maintained?
2. How should the results of such tests be valued? Under what conditions can they be regarded as primary evidence? What standards should there be for informing di fferent stages of both investigative and judicial processes?

There’s a New York Times story, and Prof Regan also has a blog post. (via)

January 13, 2012

Drug driving: dodgy numbers in a good cause?

More than a year ago, ESR scientists produced a report on drugs and alcohol found in blood samples taken after fatal crashes.  Now, the Drug Foundation is launching a publicity campaign using the data.  Their website says “Nearly half of drivers killed on New Zealand roads are impaired by alcohol, other drugs, or both.” But that’s not what the ESR report found. [Edited to add: the Drug Foundation is launching a campaign, but the TV campaign isn't from them, it's from NZTA]

The ESR report defined someone as impaired by alcohol if they had blood alcohol greater than 0.03%, and said they tested positive for other drugs if the other drugs were detectable.   If you look at the report in more detail, although 351/1046 drivers had detectable alcohol in their blood, only 191/1046 had more than 0.08%.  At 0.03% blood alcohol concentration there may well be some impairment of driving, and near 0.08% there’s quite a lot, but we can’t attribute all those crashes to alcohol impairment rather than inexperience, fatigue, bad luck, or stupidity.  At least the blood alcohol concentrations are directly relevant to impairment.  An assay for other drugs can be positive long after the actual effect wears off. For example, a single use of cannabis will show up in a blood test for 2-3 days, and regular use for up to a week.  In  fact, the summary of the ESR report specifically warns “Furthermore, it is important to acknowledge that the presence of drugs and alcohol in the study samples does not necessarily infer significant impairment.”   Regular pot smokers who are scrupulously careful not to drive while high would still show up as affected by drugs in the ESR report.  In fact, the Drug Foundation makes this distinction when they talk about random roadside drug testing, pointing out the advantages of a test of actual impairment over a test of any presence of a drug.

The Drug Foundation also did a survey of community attitudes to driving while on drugs (also more than a year ago), and it is interesting how many people think that stimulants and cannabis don’t impair their driving.  However, if you look at the survey, it turns out that it was an online poll, and “Respondents were recruited to the online survey via an advertising and awareness campaign that aimed to stimulate interest and participation in the study.” Not surprisingly, younger people were over-represented “The mean age of respondents was 38.1 years”, as were people from Auckland and Wellington. Maori, Pasifika, and Asians were all under-represented.  36% of respondents had used cannabis in the past year, more than twice the proportion in the Kiwi population as a whole.  No attempt was made to standardise to the whole NZ population, which is the fundamental step in serious attempts at accurate online polling.  [If we could use the data as a teaching example, I'd be happy to do this for them and report whether it makes any difference to the conclusions]

And while it’s just irritating that news websites don’t link to primary sources, it is much less excusable that the Drug Foundation page referencing the two studies doesn’t provide links so you can easily read them. The study reports are much more carefully written and open about the limitations of the research than any of the press releases or front-line website material.[The NZTA referencing is substantially less helpful]

For all I know, the conclusions may be broadly correct. I wouldn’t be at all surprised if many drug users do believe silly things about their level of impairment. Before  the decades of advertising and enforcement, a lot of people believed silly things about the safety of drunk driving.  And the new TV ads are clever, even if they aren’t as good as the ‘ghost chips’ ad.  But the numbers used to advertise the campaign don’t mean what the people providing the money say they mean.  That’s not ok when it’s politicians or multinational companies, and it’s still not ok when the campaigners have good intentions. [Edited to add: I think this last sentence still stands, but should be directed at least equally at the NZTA].

 

[Update: Media links: TVNZ,  3 News, Stuff, NZ Herald, Radio NZ]

October 20, 2011

The use of Bayes’ Theorem in jeopardy in the United Kingdom?

A number of my colleagues have sent me this link from British newspaper The Guardian, and asked me to comment. In some sense I have done this. I am a signatory to an editorial published in the journal Science and Justice which protests the law lords’ ruling.

The Guardian article refers to a Court of Appeal ruling in the United Kingdom referred to as R v T. The original charge against Mr. T. is that of murder and, given the successful appeal, his name is suppressed. The nature of the appeal relates to whether an expert is permitted to use likelihood ratios in provision of evaluative opinion, whether an evaluative opinion based on an expert’s experience is permissible, and whether it is necessary for an expert to set out in a report the factors on which evaluative opinion based.

It is worthwhile noting before we proceed that to judge a case solely on one aspect of the whole trial is dangerous. Most trials are complex affairs with many pieces of evidence, and much more testimony that the small aspects we concentrate on here.

The issue of concern to members of the forensic community is the following part of the ruling:

In the light of the strong criticism by this court in the 1990s of using Bayes theorem before the jury in cases where there was no reliable statistical evidence, the practice of using a Bayesian approach and likelihood ratios to formulate opinions placed before a jury without that process being disclosed and debated in court is contrary to principles of open justice.

The practice of using likelihood ratios was justified as producing “balance, logic, robustness and transparency”, as we have set out at [54]. In our view, their use in this case was plainly not transparent. Although it was Mr Ryder’s evidence (which we accept), that he arrived at his opinion through experience, it would be difficult to see how an opinion of footwear marks arrived at through the application of a formula could be described as “logical”, or “balanced” or “robust”, when the data are as uncertain as we have set out and could produce such different results.

A Bayesian, or likelihood ratio (LR) approach to evidence interpretation, is a mathematical embodiment of three principles of evidence interpretation given by Ian Evett and Bruce Weir in their book Interpreting DNA Evidence: Statistical Genetics for Forensic Scientist. Sinauer, Sunderland, MA 1998. These principles are

  1. To evaluate the uncertainty of any given proposition it is necessary to consider at least one alternative proposition
  2. Scientific interpretation is based on questions of the kind “What is the probability of the evidence given the proposition?”
  3. Scientific interpretation is conditioned not only by the competing propositions, but also by the framework of circumstances within which they are to be evaluated

The likelihood ratio is the central part of the odds form of Bayes’ Theorem. That is
Bayes' Theorem

The likelihood ratio gives the ratio of the probability of the evidence given the prosecution hypothesis to the probability of the evidence given the defense hypothesis. It is favoured by members of my community because it allows the expert to comment solely on the evidence, which is all the court has asked her or him to do.

The basis for the appeal in R v T was that the forensic scientist, Mr Ryder, in the first instance computed a likelihood ratio, but did not explicitly tell the court he had done so. In the second instance, there was also criticism that the data needed to evaluate the LR was not available.

Mr Ryder considered four factors in his evaluation of the evidence. These were the pattern, the size, the wear and the damage.

The sole pattern is usually the most obvious feature of a shoe mark or impression. Patterns are generally distinct between manufacturers and to a lesser extent between different shoes that a manufacturer makes. Mr Ryder considered the probability of the evidence (the fact that the shoe impression “matches” the impression left by the defendant’s shoe) if it indeed was his shoe that left it. It is reasonable to assume that this probability is one or close to one. If the defendant’s shoe did not leave the mark, then we need a way of evaluating the probability of a “adventitious” match. That is, what’s the chance that the defendant’s shoe just happened to match by sheer bad luck alone? A reasonable estimate of this probability is the frequency of the pattern in the relevant population. Mr Ryder used a database of shoe pattern impressions found at crime scenes. Given that this mark was found at a crime scene this seems a reasonable population to consider. In this database the pattern was very common with a frequency of 0.2. The defense made much stock of the fact that the database represented only a tiny fraction of the shoes produced in the UK in a year (0.00006 per cent), and therefore it was not comprehensive enough to make the evaluation. In fact, the defense had done its own calculation which was much more damning for their client. Using the 0.2 frequency gives a LR of 5. That is, the evidence is 5 times more likely if Mr T.’s shoe left the mark rather than a shoe of a random member of the population.

The shoe size is also a commonly used feature in footwear examination. The shoe impression was judged to be size 11. Again the probability of the evidence if Mr T.’s shoe left the mark was judged to be one. It is hard to work out exactly what Mr Ryder did from the ruling, because a ruling is the judges’ recollection of proceedings, which is not actually an accurate record of what may, or may not, have been said. According to the ruling, Mr Ryder used a different database to assess the frequency of size. He estimated this to be 3%. The judges incorrectly equate this to 0.333, instead of 0.03 which would lead to an LR of 33.3. Mr Ryder used a “more conservative” figure to reflect to some uncertainty in size determination to 0.1, giving an LR of 10.

Wear on shoes can be different between different people. Take a look at the soles of your shoes and those of a friend. They will probably be different. To evaluate the LR, Mr Ryder considered that the wear on the trainers. He felt could exclude half of the trainers of this pattern type and approximate size/configuration. He therefore calculated the likelihood ratio for wear as 1/0.5 or 2. Note here that Mr Ryder appears to have calculated the probability of wear given pattern and size.

Finally, Mr Ryder considered the damage to the shoes. Little nicks and cuts accumulate on shoes over time and can be quite distinctive. Mr Ryder felt he could exclude very few pairs of shoes that could not previously have been excluded by the other factors. That is the defendant’s shoes were no more, or less, likely to have left the mark than any other pair in the database that had the same pattern, size and wear features. Therefore therefore calculated the likelihood ratio for damage as 1.

The overall LR was calculated by multiplying the four LRs together. This is acceptable if either the features were independent, or the appropriate conditional probabilities were considered. This multiplication gave an LR of 100, and that figure was converted using a “verbal scale” into the statement “the evidence provides moderate support for the proposition that the defendant’s shoe left the mark.” Verbal scales are used by many forensic agencies who employ an LR approach because they are “more easily understood” by the jury and the court.

The appeal judges ruled that this statement, without the explicit inclusion of information explaining that it was based on an LR, was misleading. Furthermore, they ruled that the data used to calculate the LR was insufficient. I, and many of my colleagues, disagree with this conclusion.

So what are the consequences of this ruling? It remains to be seen. In the first instance I think it will be an opening shot for many defense cases in the same way that they try to take down the LR because it is “based on biased Bayesian reasoning.” I do think that it will force forensic agencies to be more open about their calculations, but I might add that Mr Ryder didn’t seek to conceal anything from the court. He was simply following the guidelines set out by the Association of Footwear, Tool marks, and Firearms Examiners guidelines.

It would be very foolish of the courts to dismiss the Bayesian approach. After all, Bayes’ Theorem simply says (in mathematical notation) that you should update your belief about the hypotheses based on the evidence. No judge would argue that against that.

September 14, 2011

Reefer madness

The factoid of dramatically increasing cannabis potency has popped up again, with a claim that cannabis used to be 1-2% THC and is now up to 33%.    The most comprehensive and consistent data on cannabis potency come from a long-term project at the University of Mississippi. Their 2010 paper is based on analysis of 46,000 confiscated samples from 1993 to 2008.    Over this time period, the percentage of THC in marijuana (leaves and buds with seeds) increased from about 3.5% to about 6%.  The percentage in sinsemilla (buds without seeds) increased from about 6% to about 11%.   Since the more-recent samples were more likely to be sinsemilla, the percentage over all confiscated samples increased a bit more, from about 3.5% to about 9%.  A small fraction of the samples had much higher concentrations, but this fraction didn’t change much over time. So, yes, the average used to be about 3% in 1993 and may have been as low as 2% in earlier decades, and, yes, the concentration is now ‘up to‘ 33%, but the trend is nothing like as strong as that suggests.   A New Zealand paper , by ESR researchers (who are hardly pot-sympathising hippies), says that there was no real change in THC concentration in cannabis plant material from 1976 to 1996, and the concentration in cannabis oil actually fell.

The Southland Times article also reports a claim that 90% of first-term methamphetamine users continue to use the drug. If this just means that 90% of them go on to have a second dose at some time it might well be true, but if it is implying long-term addiction the figure seems implausible. It’s certainly not what is found in other countries.  For example, the most recent results from the US National Survey on Drug Use and Health (NSDUH) estimate that 364000 people in the US had dependence/abuse of illegal stimulants in 2010. If we assume that all of these were methamphetamine, and that the other illegal stimulants didn’t cause any dependence/abuse problems, that’s still only 20% of the estimated 1.8 million people who first tried methamphetamine in the period 2002-2010. In fact, since NSDUH has a nice online table generator we can do a more specialized query and find out that an estimated 118000 people currently had dependence on stimulants out of the estimated 10 million people who had ever tried methamphetamine. That’s more like 1% than 90%.   Amphetamines are clearly something you want to stay well away from, but there’s no way that they addict 90% of the people who try them. In any case, if we believe the drug warriors, New Zealand’s P epidemic has already been solved by banning pseudoephedrine without a prescription.

I’m all for getting teenagers to appreciate the risks of drug use, but we need to remember teenagers can use Google too.