- Ben Straub (GSK)
- Eric Nantz (Eli Lilly)
- HyeSoo Cho (FDA)
- Joel Laxamana (Roche/Genentech)
- Joseph Rickert (R Consortium)
- Kui Schen (Bayer)
- Lei Zhao (Roche/Genentech)
- Nan Xiao (Merck)
- Paul Schuette (FDA)
- Renping Zhang (Johnson & Johnson)
- Robert Devine (Johnson & Johnson)
- Saghir Bashir (Argenx)
- Sam Parmar (Pfizer)
- Sean Healey (Pfizer)
The meeting was recorded and the video is available.
Due to a problem with zoom permissions, the meeting started 15 minutes late. Apologize to everyone who was inconvenienced.
(At 1:37 into the video) HyeSoo Cho begins sharing here screen and providing her comments on the recent Pilot 2 submission.
A bug in the submission code caused population subsetting in the Kaplan Meier Curve to propagate silently throughout the shiny app. During the course of the meeting Eric Nantz was able to identify the problem as a incorrectly set option and correct it.
JBR noted that turning off subgrouping for everything but the KM plot was a special requirement for the pilot submission and asked if under normal operation whether the FDA reviewers would like the shiny app to display this default behavior. (The alternative would be for app users to remember to turn on the same subgrouping for every plot or table in order to make comparisons.) HeySoo and Paul Schuette provided an explanation of the FDA’s need to be very careful of subgroup analysis.
Paul emphasized that it is very important to distinguish between “exploratory analysis” and analysis for “inferential analysis”. Subgrouping that causes p-values to change cross the line between the two types of analyses. The FDA is very concerned that the misuse of subgrouping could lead people to improperly argue for evidence of efficacy based on restricted subpopulations. When asked if he was advising that submissions should not contain interactive capabilities Paul replied “No”. He elaborated that nothing interactive should produce a p-value.
(10:48 into the video) Ning Leng asked if subpopulation analyses associated with predefined endpoints would be permissible as this would distinguish between inferential and exploratory analyses. Paul replied this is correct. He noted that prespecified subgroup analyses will often product confidence intervals but not p-values.
(At 19:56) Eric announced the resolution to the parameter problem and the group agreed that we have a path forward for the current Pilot 2 submission. Eric also suggested that the interactivity may be the biggest topic for the group to discuss. Paul asked to defer the discussion of this bigger issue and suggested that the working group look to the ASA statement on p-values for guidance.
(18:36) The FDA does not want to be seen as endorsing the practice of rescuing a “failed” clinical trial by torturing the data. He noted that even now there is a problem with JMP and other tools that that are interactive but do not produce an open and fully transparent analysis true.
Interactive tool must be open, transparent, reproducible, and consistent with good statistical practices. Paul suggested that an interactive tool that produces interactive tools on demand may not be consistent with the ASA statement on p-values.
In response to a question, Paul affirmed that the ASA statement in p-values is a good foundation for understanding FDA practices.
(22:57) Begins a discussion on shinymeta and reproducibility. It as also noted that the teal project is also addressing this reproducibility issue.
(27:43) HeySoo resumes her comments on the Pilot 2 submission by commenting on the warning and error messages generated by the submission code, and points out that referencing a repository other than CRAN is problematic. Paul noted that FDA IT security is insisting on strong security checks, but that a case can be made that CRAN provides sufficient checks. He said that his group at the FDA is floating the notion of “curated” packages.
HeySoo suggested that providing a README file that discusses all the potential error and warning messages and what needs to be done to run the app properly. Paul noted that some of this information should be included in the app to make it as stand alone as possible.
The working group agreed that the next step should be to produce an updated Pilot 2 submission package and resubmit it through the gateway. This will make it for the FDA to produce a written response. We will proceed with Pilot 2 take 2.
Eric will deploy will a new version on our internal host so that everyone can review it before the second submission.
When asked if the FDA security work Paul mentioned would lead to a guidance document, Paul replied that it may and noted that there is an internal FDA working group addressing the problem, and that currently Python appears to be a bigger security problem than R.
(44:26) Paul informed the group that the FDA is being approached by multiple groups, including PhUSE and Transcelerate, about using R and validation issues. The FDA would coordinate the activities of these groups to prevent duplication of effort and decide who is going to do what. He also suggested that we should broaden our base within the FDA and seek to get John Scott and CBER involved.
Action JBR will invite someone from the R Validation Working Group to attend our next meeting and meet with Paul to identify a representative from PhUSE to invite.
The next meeting of the Submissions Working Group will be on Friday, April 7, 2023 at 9AM PST.