Tests only performed on adult helmets: As the Virginia Tech researchers themselves admit, the STAR evaluation system was developed based on head impact exposure of collegiate football players, which research shows is different from that of high school football players, which, in turn, is different from youth football players. [3]
Said NOCSAE in its July 2013 statement, "Only adult-size large helmets were evaluated. There are no data to support applying the same ratings to other sized helmets of the same model, including youth football helmets. No adult X-Large, Medium, Small, X-Small or any youth-size helmets were tested. The STAR value itself is a theoretical calculation that is based on a probablistic analysis of impact exposure and injury risk at the collegiate level." NOCSAE repeated that criticism in its May 2014 statement.
The Virginia Tech researchers agree: "It is not safe to extrapolate the[ir] findings to youth football helmets," they admit. "Youth football players are likely to experience a different head impact exposure, which will require a modified evaluation system."
"There are many questions left to be answered regarding youth helmets," conceded Duma in a recent email to MomsTEAM. "In a general sense, lower head acceleration is better. I think that everyone would agree to that statement. However, we still have much research to perform on the youth population."
Sales of 5-STAR helmets soaring
The criticism by NOCSAE and the helmet manufacturers of the STAR ratings system, however, has recently been replaced, at least according to the July 2014 Times article [12] with grudging support. "After initially voicing skepticism, football helmet manufacturers [have] embraced the STAR system's ratings, and the N.F.L. posts the ratings in every team's locker room," the article reports.
The reason? Apparently, because money talks. "There has been a dramatic change in sales in football helmets since 2011 [when the first STAR ratings were issued]," Oliver told The Times' Jeff Klein. "I see high school football athletic directors submitting purchase orders for 500 five-star helmets. Parents are saying, 'I don't want a four-star helmet, I want the best for my kid.'"
Absence of proof of protective effect not same as no protective effect
To be fair, all the major football helmet manufacturers (Riddell, Schutt, Rawlings, and Xenith), and newcomer, Simpson-Gnassi, are clearly attempting to engineer helmets that will reduce the risk of concussions, for which they deserve credit. As the 2013 international consensus statement on concussions [4] observes, although there is "no good clinical evidence that currently available protective equipment will prevent concussions, ... biomechanical studies [such as those performed by the engineers at Virginia Tech] have shown a reduction in impact forces to the brain with the use of helmets." In other words, that there aren't more studies showing a concussion reduction effect under field conditions, is not the same as saying that helmets aren't reducing that risk; just that proving that they do with the methodological and statistical rigor required by a peer-reviewed journal is is "complex, challenging methodologically, resources intensive and expensive."[5]
Are 5-star helmets actually more protective?
In apparent reference to the Virginia Tech ratings, the American Academy of Neurology concluded in its 2013 guidelines on concussion management [6] that the data was "insufficient to support or refute the superiority of one type of football helmet in preventing concussions. In other words, while Virginia Tech's rating system would lead one to conclude that a 5-star helmet is "better" than a 4-star helmet, and so forth, the AAN wasn't buying into the ratings as providing that kind of information (which is no doubt why Virginia Tech itself says any 5- or 4-star helmet is recommended).
NOCSAE's statement echoed the AAN's view, asserting that, "According to an independent statistical review of the scores and categories upon which the STAR number is based, there is no significant statistical difference between helmets in the 5 STAR, 4 STAR, and 3 STAR categories."
Indeed, NOCSAE suggested in 2013 that the STAR rating methodology might be fundamentally flawed, expressing "concern" that Xenith's X2 helmet jumped after re-testing from a 3-STAR rating to a 5-STAR rating without a change in design or materials, and about the fact that not all brands/models were retested.
NOCSAE also noted that another helmet model retested in 2013 (the Schutt DNA-Pro) went from a 4 STAR to a 3 STAR with no design changes.
From these results, NOCSAE surmised that "If all helmets had been retested, it appears reasonable to believe that other helmets also may have moved into and out of different STAR categories."
In an email to concussion blogger Dustin Fink published on his website in August 2013, [7] Duma responded to NOCSAE's criticisms, characterizing them as "incorrect and misleading" in two respects (facts that he claims to have told NOCSAE before it released its statement but that they chose not to make): "First, only one helmet, the Xenith X2 increased two stars, and most importantly it did have substantial design changes. The 2013 [version] added much more padding and it performed much better. I have no idea why NOCSAE is misleading the public about this."
"Second, we did re-test all of the helmets. I have no idea why NOCSAE is implying that we did not. Each year we re-test samples from all makes and models. If the values are within the range of previous years, we kept them the same. But, if the values are outside of the previous range, we use the new values. In this light, some helmets moved down. When you examine those, there are clear differences in materials. I relayed all of this to NOCSAE, and am disappointed they made the statement so misleading and inaccurate."
