For any service, the end user wants a way to measure the performance of the service provider to understand if they are getting good value for money. While medical writing is no different, measuring the service provided for medical writing is not as straightforward.
How do you define a good-quality document? How do you measure the myriad of intangible activities performed by medical writers? How do you reduce the noise arising from the many interfaces with other functions that impact on the outcome of the medical writer’s product? Certainly there are numerous things that can be measured, some of which can be meaningful. But in our zeal to find the ultimate data points to declare a document and its writing process a success, there is the risk of overburdening those involved with labour-intensive activities, which may not provide us with any more than what is already obvious – and therein lies the challenge.
WHY METRICS
It is not hard to understand the impetus behind collecting metrics. They help compare how different people do the same job, offer insight as to whether there are standards across an industry, and make it possible to identify ways of enhancing workflows (1). In any sector, this kind of information can be helpful to improve the product and streamline efficiencies in the production process. However, this becomes more of a challenge when it is difficult to find things that can easily be measured. Without a well-defined, objective measurement, how can the quality of deliverables be clearly assessed and compared?
In the world of medical writing, most companies focus on productivity metrics because these are the easiest to measure (2). Predominantly, people look at budget and timelines, and at first glance, these seem fairly objective and have clear-cut answers (yes or no). However, since medical writing is rarely (or certainly should not be) done in isolation, can we be sure that either answer is truly a measure of the medical writer’s ability, or of the writing process applied? And can we be sure that the goal posts set at the start – in terms of timelines and costs – were accurate for the documents being produced?
Consider timelines, for example: how much time is enough for writing a well-crafted document? In the past 20 years, industry standards have continued to shrink regarding how much time is available for a team to produce a document (whether it is a clinical study report (CSR) or a clinical summary for a common technical document submission). These tend to be ever-more frequently determined at management level and dictated down to the teams producing them. The timelines are often based on industry ideals about how long it should take to get the dossier out the door, which are driven by comparisons to what other companies have claimed to achieve. They are rarely based on an understanding of the complexity of the documents at hand and the processes involved in preparing them, nor do they consider the fact that increasing output rate is likely to decrease quality. A study on the metrics of writing a CSR found that medical writers whose work rate exceeded the standard by 1.5 times, were more likely to relate that major sections of the draft CSR required reworking than medical writers whose work rate did not exceed the standard (3).
TIMELINES AND PROCESSES
Across the industry, there is a fairly commonly applied approach to timelines and processes. Many companies plan to write a shell, then two drafts with results for review, and then deliver the final document. In an ideal world, this is perfectly reasonable. While reviewing the shell, the team can focus its efforts on background information and methods, and on strategically thinking through how to present the results before the data arrives. Then, after the final tables, listings and figures are delivered, the team can focus on developing the messages.
This process relies on the core authoring team reviewing the data carefully and discussing the key messages with the writer, so that the first draft already reflects their input. The team then has two rounds of review to tweak and tailor the document. Who needs more than that? Almost every team I have ever met.
TEAMWORK CHALLENGES
The fact that most documents today are authored by multinational teams working on numerous, conflicting projects, is often not factored into assumed timelines or budgets. In many companies, the core authors (from areas such as clinical, statistics, regulatory and sometimes pharmacovigilance) do not actually sit down together and discuss the data as a group. Even when a company has a well-oiled project management function coordinating meetings and bringing these authors together, they may not agree on what the data says. This could be due to cultural reasons (different medical dogmas in different regions, for instance), political reasons, or just due to different degrees of industry experience of putting things into context. Sometimes it is just because some groups of people simply do not get on well together.
The cost of writing a document with a team that is cohesive, has a clear, well-planned strategy, who delivers their comments on time, and that does not have too many conflicting other projects, will always be much less than when several of these factors do not apply. Take, for example, a team with numerous conflicting projects. Due to competing priorities, their efforts on one document may get put on hold for several weeks at a time. When people have to revisit something after several weeks of focusing on something else, it takes them more time to get back into the details and flow of thought than if they had only been working on the one activity. As a result, agreements that had been reached previously are re-hashed, background material may need to be reread to remember the exact context, and everyone involved spends more time than if there had been no interruption.
Similarly, if some team members deliver comments on a draft several days (or weeks) after the rest of the team, or if they only provide an in-depth review of the document on a final draft instead of early drafts, the writer must account for new or possibly conflicting ideas raised after having already dealt with the comments of everyone else. This can result in rewriting or restructuring sections of text that the writer had already revised based on other comments. All of this extra time – for coming back up to speed after delays, or for revising the same text twice instead of only once after a round of reviewing – results in added costs, none of which are due to the ability of the medical writer. They are process-driven fall-out.
COMPLEXITY GRIDS
It is possible, however, to define known factors that influence the complexity of a document. For example, in general: how many variables will need to be reported, or how many treatment groups are in the studies; and – for dossiers in particular – how many studies will be included; will these be integrated; and, if so, how many pools will be presented and for how many variables (efficacy and safety)? These aspects directly determine how much needs to be written. Taking such factors into account, it is possible to develop complexity grids to more objectively assess how complex a writing project will be. If the industry were to use a standardised system for estimating the complexity of documents, and were to consider predictable complexities arising from the expected writing process for each particular project, everyone – management included – would be better able to make more realistic goals for timelines and budgets, and better able to make comparisons across projects.
Therefore, using budget and timelines to assess the output of a medical writer can be appropriate – as long as the assumptions used to define what these should be are accurate reflections of the task at hand. For many projects, these assumptions probably need to be reviewed along the way to take into account project creep and team dynamics. At the end of a project, if it becomes apparent that the estimates used were not accurate, the first question should be why, and the second should be how assumptions can be improved to get a better estimate next time, based on these learnings.
MEASURING BOTH SIDES
A true understanding of what is hindering the writing process (for example, is it lack of writer expertise, delayed delivery of statistical outputs, team dynamics, competing resources, and so forth) can only be gained if all parties involved are measured over the course of the project. This can be done by using a spreadsheet of deliverables to track each milestone.
By simply monitoring all deliverables (drafts of the document, team deliverables such as review comments, data outputs, other team action items), and recording what was provided on the agreed date and what was delayed, it is possible to gain a fairly quick overview of why a project took longer than expected and where the delays arose. This is a very effective metric that can provide meaningful information while the project is running, and can be used to identify problems before a project gets hideously off-track.
QUALITY METRICS
The simplest way to assess if the medical writer was skilled enough to do the job is to simply ask other people on the team, who would have been most directly involved with the writer. A survey with no more than five questions, graded on a scale of one to five, will quickly tell if the team was satisfied with the work done. The questions should address the main elements that a medical writer is expected to be competent in, such as: did the writer understand the document requirements, and were they able to advise on developing the optimum structure and presentation? Did the writer work effectively with the team to get the information needed and resolve open issues? Were they able to crystallise messages and explain them clearly? By obtaining a quantified measure of these elements, it can be observed which writers repeatedly receive excellent grades and which do not. This can help managers identify areas for further training and development of their writers, and can help teams and clients recognise which writers are worth developing long-term partnerships with.
Other known quality metrics include the number of quality control findings in a document, the number of unplanned review cycles needed, and the number and type of unexpected questions raised by authorities on a submission dossier (not the questions everyone knew would be raised, but questions asking for clarification of a concept or how to find something in the dossier, for example). All of these can be a meaningful measure of what constitutes a well-written document.
But, for all the reasons mentioned above, these must also be used with caution. They can only be used to assess a writer’s ability to produce a well-written document if that writer was truly empowered in the decisions made on the document. Did the writer have enough time to produce something meaningful and without mistakes, or were they adding data in at the last minute, close to midnight, without an opportunity to go back and check it? Was the team having trouble agreeing on things? Did they listen to the writer’s suggestions or overrule them, insisting on presenting something in a way no medical writer with any pride would ever agree to unless under duress? If damning metrics come in, make sure to step back and understand the full picture before deciding who or what is truly at fault.
LESS IS MORE
In a world driven by measurements, and with more and more metrics taking over our lives, is it really necessary to measure everything we can? By taking the attitude of ‘what we can measure, we can manage’, we are making people fill in more and more spreadsheets, and track more and more parameters. But, after spending all that time monitoring every step along the way, do we really know anything more? Or have we just wasted hours of valuable time that could have been spent crafting some wonderful texts? Already, there are sufficient metrics in place to understand what is good and what is not, if these are wielded intelligently. It would be wise not to collect more just for the sake of it – and instead, collect data that are a true reflection of what the medical writer does and contributes to the process.
REFERENCES
- Grom T, Metrics: It’s all about performance, PharmaVOICE, October 2006
- Hudson SE et al. Research report: Medical communication practice and trends in pharmaceutical and biotechnology companies, AMWA Journal 26: pp56-60, 2011
- Purcell TJ, The application of metrics to determine the work rate for producing the first draft of the clinical study report: A case study, Drug Information Journal 39: pp171-175, 2005