The Devil in the Detail

In Queen Mary: nobody expects the Spanish Inquisition John Allen and I wrote:

The same double standard follows, now, in our School of Biological and Chemical Sciences. For example, one of the “metrics” for research output at professorial level is to have published at least two papers in journals with impact factors of 7 or more. This is ludicrous, of course—a triumph of vanity as sensible as selecting athletes on the basis of their brand of track suit. But let us follow this “metric” for a moment. How does the Head of School fair? Zero, actually. He fails. Just consult Web of Science. Take care though, the result is classified information. HR’s “data” are marked Private and Confidential. Some things must be believed. To question them is heresy.

We hope to report back on our Head’s one-to-one interview with himself. After all, we have his word, and that of College senior management, that the restructuring is proceeding with complete fairness and transparency. Perhaps he’ll use a mirror?

This passage has been scrutinized already by two Employment Tribunals who have put to themselves the task to determine whether Professor Evans, who is the Head of School referred to above, met the “Research Output – quality” restructuring criterion, which is described immediately below in his own words in his restructuring proposal document.

Research Output – quality

The following minimum thresholds will be used for high publication quality between 1st January 2008 and 31st December 2011 for staff at each academic level.

Professor/Reader – 2 published items

Senior Lecturer/Lecturer – 1 published items

Publications must be in journals rated in the top 5%. The top 5% of journals will be assessed as either attaining an ARC quality rating of A* or an impact factor ≥ 7. Publications must be as a significant author (defined as sole, first, last or corresponding) and contain original research or review. Evidence of a publication being in press before 31st December 2011 will be accepted.

One advantage of this straightforward measure of quality is its objectivity. In April 2012, a search in Web of Science (see below why the search was restricted) for the publication record of Professor Evans returned no evidence of any paper published by this author after the 1st of January 2008 in a Journal that met the criteria listed above.

I asked Professor Evans about this problem given his written declaration that “all staff in SBCS” would be subject to these criteria, including himself. It looked strange that he would have written a set of criteria that he would so evidently fail, hence the insistence to find out what was going on. Repeated requests for this information were declined citing confidentiality.

It was during the Employment Tribunal hearing (Dr F Missirlis vs Queen Mary University of London, Case Number: 3202937/2012) that I first understood how Professor Evans claimed to have met the criterion listed above. He relied on two papers:

Predictive ecology: systems approaches

Matthew R. Evans, Ken J. Norris, Tim G. Benton

Phil. Trans. R. Soc. B (2012) 367, 163-169

Modelling ecological systems in a changing world

Phil. Trans. R. Soc. B (2012) 367, 181–190

The argument then goes that this athlete has a track suit of the right brand. For Philosophical Transactions of the Royal Society B may have an impact factor < 7, but it carries the A* distinction from the Australian Research Council. Furthermore, Professor Evans produced two publications in this venue that first appeared online on the 5th of December 2011 and were formally published in January 2012.
– – –
Let us dig slightly deeper into what actually happened in this fair and transparent process. As “all staff” were included, there was a need to compile a publication record for each and every one of us. This work was instructed by Professor Evans. He produced outputs in the following format (the example is my own record):Screen Shot 2015-06-09 at 10.04.57 PM

These data were used to assess whether we met two “Research Output” criteria. The first one being:

Research Output – quantity

The following minimum thresholds will be used for high publication quality between 1st January 2008 and 31st December 2011 for staff at each academic level.

Professor – 11 published items

Reader – 9 published items

Senior Lecturer – 7 published items

Lecturer – 5 published items

Publications must be cited in Web of Sciences (document types categories – ‘article’, ‘proceedings paper’ or ‘review’). Publications not on Web of Science will not be included, nor will publications in Web of Science document categories not listed above, for example ‘Meeting abstract’, ‘Letter’, ‘Note’, ‘Editorial Material’, ‘Book review’ or ‘Correction/addition’. Evidence of a publication being in press before 31st December 2011 will be accepted.

The rules were applied to a subset of 61 academics. (Professor Evans’ claim at the time that they had been applied to all staff was false). Each member of staff only received their own table under the heading “confidential”. A significant number of errors was discovered and this caused stress.

One member of staff had produced a publication containing original research or review in a journal rated in the top 5%, which was listed in Web of Science category ‘Editorial Material’. He inquired why this publication was not listed in his table and he was told that it was not listed because the category ‘Editorial Material’ was not part of the “metrics”.

The table above requires some explanation. For example, if you look under ‘notes’ I have collected six zeros. That is because in this column the six papers I produced during my first four years at Queen Mary are considered for “quality”. None of my papers was published in a journal rated in the top 5% so I get rated as zero quality.

In other tables, each publication in a journal rated in the top 5% received the score of 1 under the same column, subject to the author having made a ‘significant’ contribution.

You can now consider the table Professor Evans produced for himself. He listed his publications from Web or Science: a collection of eleven zeros.

He then added the two publications mentioned above. The journal had classified one of the two as “Introduction”, which Web of Science identifies as ‘Editorial Material’.

In the circumstances, would you say that Professor Evans had met the  “Research Output – quality” restructuring criterion?
The reader can make up their own mind, of course, but others with ‘Editorial Material’ publications in high impact factor journals did not have theirs counted. I close this post by recalling a similar question over the Principal’s performance. Who should be called to answer?

8 thoughts on “The Devil in the Detail

  1. Paul Jump reports today in the Times Higher Education on the Employment Tribunal Judgment in Dr J F Allen vs Queen Mary University of London. He says “The tribunal also rules that… Professor Evans met the metrics threshold”.

    This conclusion predisposes the Tribunal, which also found that John Allen’s dismissal was unfair because Evans and QMUL acted unreasonably both in removing all his teaching and in failing to address his grievance in respect with these decisions. The Tribunal, however, didn’t accept that it was within John’s right to refuse further teaching prior to reaching a conclusion on the former matters. The Tribunal found that our Lancet letter contributed to all of the above.

    Therefore the true position is very important. If ‘editorial material’ was excluded by the metrics, then the fact is that Evans failed to meet the metrics threshold. Queen Mary managers continue to insist otherwise. Hence, a Tribunal that would have found that the Lancet disclosure was substantially true, would have likely reached a very different conclusion as to the reasons that led John to resist.

  2. I am reading again the full “restructuring proposals” and I spotted another relevant passage:

    “Research Outputs: Only including outputs that can be accessed via Web of Science is justified as this academic citation index is the most widely accepted and provides an objective, independent and transparent means to determine output numbers. In order to appropriately recognise the range of outputs that may be considered to represent peer-reviewed research activity that may vary across the broad subject range covered in SBCS both full journal articles and proceedings papers are included, but other outputs such as abstracts and editorial comments etc are not included.”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s