The creation of these meeting minutes was supported by the use of the Zoom meeting transcript.
Conference Recap: R/Pharma Summit
Eric opened the meeting with a recap of the September conference circuit (R/Pharma Summit, Posit Conf, and RISW). The R/Pharma Summit roundtable summaries are now available and cover six broad topics, including two dedicated to AI (best practices as well as trust and safeguards), reproducible environments, and leveraging open source in regulatory submissions. Key themes echoed Joe Cheng’s morning presentation: clear examples, proactive and early communication with reviewers, and the reminder that FDA is agnostic to the programming language used. Nicole from Denali shared a successful R-based submission story, with early communication being the major factor. Joe Cheng shared that Kevin Ushey (maintainer of {renv}) is motivated to address issues with restoring {renv} environments and is asking for specific, reproducible examples (such as lock files that fail to restore). Issues can be filed directly or sent to Joe. Adrian asked about NDAs for proprietary environments, and Joe confirmed Posit is willing to handle any necessary paperwork. Eric asked Hye Soo, Youn Kyeong, and Gabriel to share any specific {renv} restoration issues encountered in previous pilots (such as packages requiring compilation from source) so they can be triaged.
Conference Recap: RISW and CAMIS Panel
Eric and Hye Soo participated in a panel at RISW focused on raising awareness of the CAMIS project. The session was not recorded, but Eric offered to share his slides. Hye Soo presented the working group’s blog post on best practices for R-based submissions, which was well received and generally agreed upon by attendees, including a number of FDA colleagues in the audience.
R Package Validation: R Validation Hub and OpenVal
Hye Soo asked whether OpenVal from Atorus could be leveraged for validating R packages. Eric shared that Lilly has used OpenVal for about a year, but noted that it is a commercial (not open source) product that validates snapshots of CRAN packages, so there is a lag between new CRAN releases and their availability in OpenVal. Because the Pilot 7 statistical verification effort aims to be fully transparent and developed in the open, OpenVal is not a good fit for that framework. Adrian suggested the group could build its own open test suite comparing SAS and R results (potentially in collaboration with CAMIS) that could be run against any environment, giving both FDA reviewers and sponsor quality/IT departments a way to trust an assembled environment.
In response to a question from Youn Kyeong, Ellis explained that the R Validation Hub provides a centralized set of ideas, guidance, and tools around validation, whereas OpenVal is one vendor’s implementation of a risk-based approach that writes tests based on identified risks. Validation is ultimately organization-dependent, driven by company SOPs and QA expectations. Ning summarized that the Validation Hub is a playbook with tools for companies who have the resources to implement it in-house, while OpenVal is one of several commercial options for companies who prefer to outsource. Joe noted in the chat that OpenVal is more focused on validating container images to ensure specific package versions work together. Eric pointed out that proprietary solutions do not solve the end-to-end distribution problem, since FDA reviewers still install packages from CRAN rather than a sponsor’s internal validated repository. Phil shared two PHUSE presentations on how organizations implement this internally (2024, 2025).
Pilot 7 Updates: AI Skills Collaboration with FDA
Following a talk at RISW, Laura Thompson from FDA joined the most recent Pilot 7 meeting. Laura has been developing AI skills internally, including agents for reviewing study design and submission packages, and for independently recreating analyses (similar to double programming / QC review). She was impressed with the rigor of the Pilot 7 pharma-skills repository and benchmarking process, and is receptive to contributing some of her skills to the repository. Eric emphasized that early-stage skills are welcome since the goal is to iterate as a community. Adrian and Ning highlighted the value of testing both sponsor-side and agency-side skills against the same suite of community-contributed benchmarks. Michael Kane expressed interest in joining the effort, as he is doing similar work with registry data at MD Anderson, and Eric shared the working group joining information.
Pilot 7 Updates: Skill Evaluation and Benchmarking
In response to Gabriel’s question about how skills are evaluated, Eric walked through the process documented in the skill lifecycle. Community members contribute benchmark test cases through GitHub issues, and a GitHub Actions pipeline runs an LLM against each test case both with and without the skill loaded, comparing the output to the expected results. Results vary by case; in some instances (such as a non-inferiority example) the model performed better without the skill, which is valuable feedback for improving the skill. Keaven raised a concern that protocols are often inadequately specified to reproduce a design; Eric noted that benchmark quality depends on the information provided in the intake template, though the IPD skill retrieves the protocol and analysis plan using the trial’s NCT ID.
Gabriel raised the challenge of validating LLM agents and the circularity of using an LLM to evaluate an LLM. Eric clarified that the exercise assumes an LLM will be used, and the question is whether a skill adds enough value to recommend it over the base model. Ellis asked what “validation” means for a skill, given that a skill is ultimately a set of markdown files and resources. Adrian emphasized that skills help get closer to the right answer faster, but a qualified human reviewer is still required to judge the output. Michael noted that standardized CDISC data appearing earlier in the analysis pipeline also strengthens the reproducibility and automation case. Eric also highlighted Yilong’s update on the YAMAA framework for benchmarking EDC to SDTM and ADaM data conversion, which now includes over 150 benchmarks (example), and the team is seeking additional test case contributions.
R Release Schedule Alignment Proposal
Ning revisited a proposal for a non-binding community agreement among sponsors on a time window for adopting new major R releases (typically anchored to the summer release of R in the June/July timeframe, with an optional winter release). The motivation stems from Pilot 3, where the R version used by the working group was not yet installed on FDA workstations, and installation required a lengthy process. Gabriel, who helps maintain the Office of Biostatistics scientific workstations, stated that a predictable industry schedule would make maintaining a large number of distributed workstations easier, while acknowledging that FDA will continue to work with whatever versions applicants use.
Ellis questioned whether a formal agreement is the right solution, noting that the R version rarely impacts an analysis, long-running studies often use older versions, and a published agreement could become a perceived requirement that sponsors feel they must follow. He suggested simply sharing what sponsors generally do might suffice. Adrian agreed with the concerns and suggested the ideal solution is for reviewers to have a ready-to-use environment for each submission, which is easiest when sponsors pin all packages to a specific CRAN/Bioconductor snapshot date so binaries can be installed from a public package manager. Internal, non-open-source packages complicate this considerably. Adrian also suggested tools such as Lmod, EasyBuild, and Spack for managing environments on shared servers, and noted that installing on Windows laptops would require either a binary build machine or full build tools for each reviewer.
Paul noted that proprietary software such as SAS changes very slowly (version 9.2 to 9.4 over about 18 years), while R and especially R packages change much more rapidly. A key pain point for reviewers is undisclosed breaking changes in package updates, which can break internal review scripts. Keaven emphasized that package maintainers should avoid breaking changes and document them when they occur. Adrian added that packages change daily, and proposed that health authorities could maintain a regular cadence of shared environments (for example, monthly snapshots of a curated set of CRAN and Bioconductor packages), allowing sponsors to target the closest available environment, with any mismatches resolved through reviewer-sponsor discussion.
Nix for Reproducible Environments
Eric shared that he and Bruno Rodrigues gave a workshop at R/Pharma on using Nix for reproducible environments (slides). As part of this, Eric created a version of the Pilot 2 Shiny application powered by Nix instead of {renv}. With WSL and Nix available on a Windows machine, the environment can be reproduced and the app launched with only three commands, which is considerably simpler than the Docker and standard approaches used in previous pilots. Eric noted this would require adoption by both sponsors and reviewers to avoid mismatches. Adrian also shared his Administering R workshop materials from R/Pharma.
Next Steps
- All: Share specific, reproducible examples of
{renv}environment restoration failures (e.g., lock files) with Joe Cheng or via issues so they can be triaged with Kevin Ushey. - Hye Soo, Youn Kyeong, Gabriel: Review notes from previous pilots for specific
{renv}restoration issues encountered at FDA. - Eric: Share slides from the RISW CAMIS panel.
- Eric, Ning: Continue collaborating with Laura Thompson on contributing FDA-developed AI skills to the Pilot 7 pharma-skills repository.
- Michael Kane: Join the R Consortium Slack and Pilot 7 efforts to share learnings from MD Anderson’s work with registry data.
- Ning: Continue refining the R release schedule alignment proposal, taking into account the feedback from Ellis and Adrian.
- Gabriel: Follow up asynchronously on the tools and infrastructure available for managing FDA workstation environments, so the group can identify how best to help.
- Adrian: Discuss with Gabriel and Eric potential approaches for front-loading environment provisioning on shared servers, and potentially develop a demonstration repository.
- Hye Soo: Send an email update on the Precision FDA exploration.