Matt Herper at Forbes has a post on new pharmaceutical research productivity data from Richard Evans, a former Roche executive and Wall Street analyst.  Evans calculates metrics such as (1) Economic Returns to R&D spending, (2) Patents / $1M R&D spend, (3) Average Relative Quality of Innovation, (4) Average Rank (by share of innovation) in Target Research Areas, and (5) Internal Bias Index.  It’s an ambitious attempt to quantify R&D productivity, and the data is though-provoking, but there are a lot of caveats.  Economic returns to R&D is the most accurate calculation, but can still be confounded by several factors, including sampling bias.  Young companies with single blockbusters look much more profitable.  The power law distribution of performance makes an older company with a bigger portfolio (with a longer tail of less lucrative products) look less productive, overall.  But small companies with non-block busters are likely to go out of business or be acquired, and so don’t show up in the data.

My main complaint, however, is that several of the parameters are derived from data on the number of patents and citations of patents. Unlike software or engineering patents which can be reasonably assumed to be more proportional to research productivity, I don’t think drug and therapeutic patents, publications and citations are at all necessarily proportional to R&D economic productivity.  The numbers of compound patents filed and issued can be quite variable, and target patents are quite often issued to academic centers, which aren’t included in the data.  Numbers of published papers and citations of course are highly variable (some companies have lenient rules for publications, some clamp down and limit publications), and often depend on the trendiness of a target or the ease at which multiple investigators can work on related projects yet still carve out publishable material. The citation data is also highly biased toward who is first to patent, but not necessarily who is getting the greatest economic benefit.  Coming late to the statin field, Liptor patents were probably not as highly cited as, say, Mevacor patents.

The internal bias index is particularly flawed in its logic.  Relying on more externally-derived programs is not a priori an indication of better R&D productivity, especially when that productivity is measured in units of patents and citations.  A more accurate metric would be the ratio of economic returns from internal vs. external sources, but all that would tell you is the relative value of internal vs. external R&D.  The total economic returns parameter is still the bottom line.