Is the Hype Real? Real-Life User Experience of an Artificial Intelligence Tool for Clinical Study Report Production

We all know that artificial intelligence (AI) is changing the world we live in, and the world of medical writing is no different. The explosion of AI is creating significant opportunities in identifying data signals, document creation, and supporting the work of medical writers. This article summarizes how the medical writers in my company, Trilogy Writing & Consulting, have been using a rule-based AI tool (that we purchase as a Software as a Service license) to augment their writing of clinical study reports (CSRs) and is a summary of a talk that I gave at AMWA’s 2023 Medical Writing & Communication Conference in Baltimore, Maryland, on October 26, 2023.

First, a brief introduction to the tool we are using. It is called TriloDocs and is what is called an expert system. This is a form of AI that uses rules to simulate how a human would apply their expertise to do something. In this case, it is simulating how medical writers make decisions to select the information to be used in a CSR. The rules in the system were developed by experienced medical writers to recreate our thought processes.

The system does not apply machine learning or generative AI (á la ChatGPT). This was an important decision in developing the tool because it was critical that we could understand exactly how the system makes all its decisions; namely, why it chooses which data, which cutoffs it uses, and where the material comes from. The system was designed to be completely transparent about what it does and to inform the user about the decisions made.

The user works in a browser-based working environment in which they upload all source files (the protocol, statistical analysis plan, and the tables and figures) in Word or Rich Text Format. The system generates the output from these, and the source files are removed. Nothing is stored in the system. This was another important feature because the tool was not intended to be a repository of any kind. In fact, it functions much like a photocopier! You put the material in, it creates the output, and you take the material out.

The output itself is a Word file based on the Transcelerate template. This means the medical writers continue to work in Word as usual, without needing to learn a new system. Using the Transcelerate template makes it as universal as is possible in an industry full of unique interpretations of the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use E3 guideline.

We wanted a tool that would support how we work as medical writers. Its main purpose is to pull information together quickly and accurately—leaving the interesting part of writing to us. The initial draft it generates is not complete and is not intended to be sent to the team for review. The medical writer takes what it has pulled together and refines it, giving it the color and context needed to tell the full story. But by saving days’ worth of time normally spent copying and pasting the source information together, we can better focus on the intellectual part of the exercise. This is critical in understanding the benefits and limits of an AI tool. AI cannot interpret data; it can only either repeat information that is already summarized, or it can describe data based on rules. Humans look at the signals and decide what they actually mean in the context of everything else we know.

TriloDocs does two main things in generating the initial draft of a report. The first is to compile the data-independent sections by pulling information from the protocol and statistical analysis plan. Some things are used verbatim and left for the medical writer (in collaboration with the rest of the authoring team) to decide how much to adapt it for this particular CSR. Some things are woven together from multiple sections of the source material. Anything that happened during the study and is thus not described in the protocol (eg, audits and their findings) will still need to be added manually by the medical writer.

It also writes text and produces in-text tables and figures for the data-driven sections (the results). The tool applies good medical writing methods by using lean text focused on the data signals and not just a repetition of the data from the tables. The text identifies data that are likely to have clinical relevance (ie, notable differences between treatment groups) or indicates if there are no differences. It is the equivalent of having someone sit down with a highlighter pen and mark up all notable differences between groups on the tables—only now these data have been captured in the text already. The system does not make a judgement call on these differences, it simply identifies them, leaving the medical writer (together with the clinical team) to decide if they are important and if more detail or explanation needs to be added. In this way that initial draft is equivalent to giving the medical writer the tables with all the potentially interesting findings already laid out in text.

So, the output is not a complete final draft—it is a solid starting point that then requires the human element to refine it. That’s why an experienced medical writer is needed who can take the initial text provided, craft it (in collaboration with the clinical team) to interpret what those findings mean, and hone the report into a well-written document that tells the scientific story of what these data mean. In fact, the system was designed to be used by an experienced medical writer from the start. The user must have reviewed the source material and have an overview of the study intention and purpose to be able to guide the tool to generate a meaningful output. They must understand what to present in the report and inform the tool accordingly.

This happens by answering a series of questions posed by the tool about the study before creating the initial draft. After uploading the source files (protocol, statistical analysis plan, tables and figures), the medical writer steers the decisions that TriloDocs will make through these questions. The questions consider things that have implications for what should be considered important (like size of the population, is it a comparative study, how many treatment arms are there).

Because it only takes about half a day for a medical writer to generate that initial draft, they have much more time free to focus on discussing with the authoring team what the messaging is around the findings. By removing the grunt work, medical writers can now focus on the strategic input to the CSR. The role and the perception of the medical writer will be that of true subject matter experts in their own right who aid the team to craft well-honed content.

In addition, the tool reduces or eliminates the risk that teams will overlook important signals in the data. The rules were created to err on the side of showing more rather than less, leaving the teams to decide what needs to be retained. By always indicating in the text what the cutoffs were for selecting data (eg, how the most common adverse events were selected), the users and reviewers are fully informed about what has been presented.

Are there limitations to the tool? Definitely. It was designed by medical writers, so it only knows what we know in our combined knowledge base. If there is something highly atypical (eg, a new type of analysis or an unusual data presentation), it may not have a rule to process that, in which case it informs the user that it did not know what to do and will leave that section blank. The writer will have to look at that material and populate the section manually. However, the rest of the report will be populated, so it still eliminates the drudgery of filling in the rest of the document.

Feedback from the medical writers using the tool has shown that they particularly appreciate the way the tool guides them to think about the context of the study and the report before they start writing. One writer said, “The way the tool guides me to consider different things meant I was thinking about aspects of the structure and presentation of the CSR that I wouldn’t necessarily do or even think of while writing manually.” We have also seen that the guided nature of generating the report means there is greater consistency in how things are reported. It prevents key things (eg, regulatory requirements) from being forgotten or overlooked in haste.

“The part I loved about TriloDocs is that it read the prohibitively long, complex, and poorly designed lab tables and gave me a simple statement that there were no clinically significant findings. In my whole experience, this saved me most time and effort and was a real blessing.”

Another learning has been that the tool finds important signals that clinical teams miss. We have done several comparisons of the TriloDocs initial draft to reports that were previously written manually and had completed full team review. In many cases the draft results produced by the tool highlighted findings that had not been reported in the manually generated report. For example, in one pivotal study report two key safety findings had not been reported in the original report but had been summarized in the report generated by TriloDocs.

Another quote from a medical writer highlights how medical writers benefit from using the tool. “The part I loved about TriloDocs is that it read the prohibitively long, complex, and poorly designed lab tables and gave me a simple statement that there were no clinically significant findings. In my whole experience, this saved me the most time and effort and was a real blessing.” Using the tool saves the time normally needed for wading through the data, freeing up the writer to think about what the signals mean rather than struggling to find them. In addition, while humans get tired and lose focus as they pour through hundreds of pages of data, an AI tool does not.

So, to consider the original question “Is the hype real?” I can say that from our experience, these tools are dramatically changing the way medical writers work. They enable us to focus on the messages by eliminating the tiring and errorprone tasks (copy and pasting, typing in the data, identifying signals, etc). Yet, the elimination of these tasks does not diminish the value and need for a medical writer. Very much like how automatic pilot systems on airplanes have not eliminated the need for pilots, this technology brings advantages to the user that allows them to focus on the essential (and human) aspects they bring to the role. The tasks we do will change, but in an industry in which medical writers are overloaded with work, removing some parts of the job could be a welcome respite. Ultimately, we see how using AI tools will elevate further the medical writer’s role to a strategic level of focusing on critical thinking around the messaging of the documents. From here, the next step in TriloDocs development is to focus on populating Module 2 summaries, which will bring all these advantages to dossier writing and, ultimately, aid in getting drugs to patients faster—a goal we are eager to embrace.

Author declaration and disclosures: The author notes that she is a partial owner of the rule-based AI tool mentioned in this article, but this does not pose a conflict of interest in relation to this article.
Author contact: julia@trilogywriting.com