by Dr A. Mesut Erzurumluoğlu | Principal Bioinformatician at Bicycle Therapeutics (formerly at Boehringer Ingelheim, and Univs. of Cambridge, Leicester & Bristol) – blogging since 2006. All views mine unless stated otherwise
Many of today’s scientists (incl. myself a lot of the time) have probably lost touch with some of the central tenets of being a scientist – instead titles, number of published papers and grant money brought in becoming more important than the societal impact of their publications and how much they contributed to human knowledge. A shoddy paper published in Nature/Science/Cell (especially if cited/talked about a lot) carries far more weight than a solid paper in a less glamorous journal. An academic who brings in grant money – doesn’t matter if he/she wastes it on shoddy or average research – is far more important (i.e. they will be promoted and bring in further funding easier as they already brought in some before) than one who chooses to concentrate on producing solid research but struggles to bring in money e.g. due to a lack of funding in their specific field or publishing papers in non-glamorous journals due to ‘non-exciting’ results as they didn’t add a spin to their conclusions (click here for other examples). Some of the papers published in prestigious journals in my field would not have been accepted if the senior authors of the same papers were the reviewers – many seem to apply a less stringent criteria to their own papers. The relationship between editors and some senior scientists is also opaque which is ultimately damaging to science. Image source:naturalphilosophy.orgHell for academics and researchers (NB: The list is loosely ordered and is not an exhaustive one). Of course, inspired by Dante’s Nine levels/layers/circles of Hell
A few months ago, I spent almost a week trying to replicate a published “causal” association which had received >500 citations in the last 5 years. My aim was to provide a better effect estimate and to do this, I used two different datasets, one with similar and another with a larger sample size. However, both of my analyses returned null results (i.e. no effect of exposure on outcome). Positive controls were carried out to make sure the analysis pipeline was working correctly. Ultimately, I moved on to other ‘more interesting’ projects as there was no point spending time writing a paper that was probably going to end up in a ‘not-so-prestigious’ journal and never going to get >500 citations or be weighted heavily when I apply for grants/fellowships.
Consequently, inadvertently I contributed to publication bias on this issue – and no other analyses on the subject matter were published since the original publication, so I am sure others have found similar results and chose not to publish.
State of academia (very generally speaking): Really talented and successful people working like slaves for unimportant academic titles and average salaries. What’s worse is that the job market is so fierce that most are perfectly happy(!) to just get on with their ‘jobs‘ and do what they’ve always been doing (Note: this is my first attempt at drawing using Paint 🙂 )
However, I have changed my mind about publishing null/negative results after encountering Russell, Wittgenstein and others’ long debates on proving ‘negative’ truths/facts (and in a nutshell, how hard it is to prove negatives – which should make it especially important to publish conclusive null findings). These giants of philosophy thought it was an important issue and spent years structuring their ideas but here I am, not seeing my conclusive null results worthy of publication. I (and the others who found similar results) should have at least published a preprint to right a wrong – and this sentiment doesn’t just apply to the scientific literature. I also think academics should spend some time on social media to issue corrections to common misconceptions in the general public.
Recommended podcast on philosophy: from the pre-Socratic period to the master himself (incl. Plato & Aristotle), from Islamic Golden Age (e.g. Avicenna, Averroes) to Aquinas, Descarte, Kant…https://t.co/fFeS6YnQtL
This also got me thinking about my university education: I was not taught any philosophy other than bioethics during my undergraduate course in biological sciences (specialising in Genetics in the final year). I am now more convinced than ever that ‘relevant’ philosophy (e.g. importance of publishing all results, taking a step back and revisiting what ‘knowledge’ is and how to attain ‘truth’, how to construct an argument1, critical thinking/logical fallacies, what is an academic’s intellectual responsibility?) should be embedded and mandatory in all ‘natural science’ courses. This way, I believe future scientists and journal editors would appreciate the importance of publishing negative/null results more and allow well-done experiments to be published in ‘prestigious’ journals more. This way, hopefully, less published research findings are going to be false2.
References/Further reading:
Think Again I: How to Understand Arguments (Coursera MOOC)
Ioannidis JPA. Why Most Published Research Findings Are False. PLoS Med. 2(8): e124 (2015)
How Life Sciences Actually Work: Findings of a Year-Long Investigation (Blog post)
A gross generalisation but unfortunately there is some truth behind this table – and it’s not even a comprehensive list (e.g. gatekeepers, cherry picking of results). Incentives need to change asap – and more idealists are needed in academic circles!
*the title comes from the fact that today’s natural scientists would have been called ‘natural philosophers’ back in the day
BBC news article published on the 18th March 2018. According to the article, men with low sperm counts are at a higher risk of disease/health problems. However, this is unlikely to be a causal relationship and more likely to be a spurious correlation. May even turn out to be the other way round due to “reverse causality”, a bias we encounter a lot in epidemiological studies. The following sounds more plausible (to me at least!): “Men with disease/health problems are likely to have low sperm counts” (likely cause: men with health problems tended to smoke more in general and this caused low sperm counts in those individuals).
As an enthusiastic genetic epidemiologist (keyword here: epidemiologist), I try to keep in touch with the latest developments in medicine and epidemiology. However, it is impossible to read all articles that come out as there is a lot of epidemiology and/or medicine papers published daily (in fact, too much!). For this reason, instead of reading the original academic papers (excluding papers in my specific field), I try to skim read from reputable news outlets such as the BBC, The Guardian and Medscape (mostly via Twitter). However, health news even in these respectable media outlets are full of wrong and/or oversensationalised titles: they either oversensationalise what the scientist has said or take the word of the scientist they contact – who are not infallible and can sometimes believe in their own hypotheses too much.
It wouldn’t harm us too much if the message of an astrophysics related publication is misinterpreted but we couldn’t say the same with health related news. Many people take these news articles as gospel truth and make lifestyle changes accordingly. Probably the best example for this is the Andrew Wakefield scandal in 1998 – where he claimed that the MMR vaccine caused autism and gastro-intestinal disease but later investigations showed that he had undeclared conflicts of interest and had faked most of the results (click here for a detailed article in the scandal). Many “anti-vaccination” (aka anti-vax) groups used his paper to strengthen their arguments and – although now retracted – the paper’s influence can still be felt today as many people, including my friends, do not allow their children to be vaccinated as they falsely think they might succumb to diseases like autism because of it.
The first thing we’re taught in our epidemiology course is “correlation does not mean causation.” However, a great deal of epidemiology papers published today report correlations (aka associations) without bringing in other lines of evidence to provide evidence for a causal relationship. Some of the “interesting ones” amongst these findings are then picked up by the media and we see a great deal of news articles with titles such as “coffee causes cancer” or “chocolate eaters are more successful in life”. There have been instances when I read the opposite in the same paper a couple of months later (example: wine drinking is protective/harmful for pregnant women). The problem isn’t caused only due to a lack of scientific method training on the media side, but also due to health scientists who are eager to make a name for themselves in the lay media without making sure that they have done everything they could to ensure that the message they’re giving is correct (e.g. triangulating using different methods). As a scientist who analyses a lot of genetic and phenotypic data, it is relatively easier for me to observe that the size of the data that we’re analysing has grown massively in the last 5-10 years. However, in general, we scientists haven’t been able to receive the computational and statistical training required to handle these ‘big data’. Today’s datasets are so massive that if we take the approach of “let’s analyse everything we got!”, we will find a tonne of correlations in our data whether they make sense or not.
To provide a simple example for illustrative purposes: let’s say that amongst the data we have in our hands, we also have each person’s coffee consumption and lung cancer diagnosis data. If we were to do a simple linear regression analysis between the two, we’d most probably find a positive correlation (i.e. increased coffee consumption means increased risk of lung cancer). 10 more scientists will identify the same correlation if they also get their hands on the same dataset; 3 of them will believe that the correlation is worthy of publication and submit a manuscript to a scientific journal; and one (other two are rejected) will make it past the “peer review” stage of the journal – and this will probably be picked up by a newspaper. Result: “coffee drinking causes lung cancer!”
However, there’s no causal relationship between coffee consumption and lung cancer (not that I know of anyway :D). The reason we find a positive correlation is because there is a third (confounding) factor that is associated with both of them: smoking. Since coffee drinkers smoke more in general and smoking causes lung cancer, if we do not control for smoking in our statistical model, we will find a correlation between coffee drinking and lung cancer. Unfortunately, it is not very easy to eliminate such spurious correlations, therefore health scientists must make sure they use several different methods to support their claims – and not try to publish everything they find (see “publish or perish” for an unfortunate pressure to publish more in scientific circles).
A figure showing the incredible correlation between countries’ annual per capita chocolate consumption and the number of Nobel laureates per 10 million population. Should we then give out chocolate in schools to ensure that the UK wins more Nobel prizes? However, this is likely not a causal relationship as it makes more sense that there is a (confounding) factor that is related to both of them: (most likely) GDP per capita at purchasing power parity. To view even quirkier correlations, I’d recommend this website (by Tyler Vigen). Image source: http://www.nejm.org/doi/full/10.1056/NEJMon1211064.
As a general rule, I keep repeating to friends: the more ‘interesting’ a ‘discovery’ sounds, the more likely it is to be false.
Hard to explain why I think like this but I’ll try: for a result to sound ‘interesting’ to me, it should be an unexpected finding as a result of a radical idea. There are just so many brilliant scientists today that finding unexpected things is becoming less and less likely – as almost every conceivable idea arises and is being tested in several groups around the world, especially in well researched areas such as cancer research. For this reason, the idea of a ‘discovery’ has changed from the days of Newtons and Einsteins. Today, ‘big discoveries’ (e.g. Mendel’s pea experimets, Einstein’s general relativity, Newton’s law of motion) have given way to incremental discoveries, which can be as valuable. So with each (well-designed) study, we’re getting closer and closer to cures/therapies or to a full understanding of underlying biology of diseases. There are still big discoveries made (e.g. CRISPR-Cas9 gene editing technique), but if they weren’t discovered by that respective group, they probably would have been discovered within a short space of time by another group as the discoverers built their research on a lot of other previously published papers. Before, elite scientists such as Newton and Einstein were generations ahead of their time and did most things on their own, but today, even the top scientists are probably not too ahead of a good postdoc as most science literature is out there for all to read in a timely manner (and more democratic compared to the not-so-distant past) and is advancing so fast that everyone is left behind – and we’re all dependent on each other to make discoveries. The days of lone wolves is virtually over as they will get left behind those who work in groups.
To conclude, without carefully reading the scientific paper that the newspaper article is referring to – hopefully they’ve included a link/citation at the bottom of the page! – or seeking what an impartial epidemiologist is saying about it, it’d be wise to take any health-related finding we read in newspapers with a pinch of salt as there are many things that can go wrong when looking for causal relationships – even scientists struggle to make the distinction between correlations and causal relationships.
Amy Cuddy’s very famous ‘Power posing’ talk, which was the most watched video on the TED website for some time. In short, she states that if you give powerful/dominant looking poses, this will induce hormonal changes which will make you confident and relieve stress. However, subsequent studies showed that her ‘finding’ could not be replicated and she that did not analyse her data in the manner expected of a scientist. If a respectable scientist had found such a result, they would have tried to replicate their results; at least would have followed it up with studies which bring other lines of concrete evidence. What does she do? Write a book about it by bringing in anecdotal evidence at best and give a TED talk as if it’s all proven – as becoming famous (by any means necessary) is the ultimate aim for many people; and many academics are no different. Details can be found here. TED talk URL: https://www.ted.com/talks/amy_cuddy_your_body_language_shapes_who_you_are
PS: For readers interested in reading a bit more, I’d like to add a few more sentences. We should apply the below four criteria – as much as we can – to any health news that we read:
(i) Is it evidence based? (e.g. supported by a clinical trial, different experiments) – homeopathy is a bad example in this regard as they’re not supported by clinical trials, hence the name “alternative medicine” (not saying they’re all ineffective and further research is always required but most are very likely to be);
(ii) Does it make sense epidemiologically? (e.g. the example mentioned above i.e. the correlation observed between coffee consumption and lung cancer due to smoking);
(iii) Does it make sense biologically? (e.g. if gene “X” causes eye cancer but the gene is only expressed in the pancreatic cells, then we’ve most probably found the wrong gene)
(iv) Does it make sense statistically? (e.g. was the correct data quality control protocol and statistical method used? See figure below for a data quality problem and how it can cause a spurious correlation in a simple linear regression analysis)
Wrong use of a statistical (linear regression) model. If we were to ignore the outlier data point at the top right of the plot, it becomes easy to see that there is no correlation between the two variables on the X and Y axes. However, since this outlier data point has been left in and a linear regression model has been used, the model identifies a positive correlation between the two variables – we would not have seen that this was a spurious correlation had we not visualised the data.
We now know that, through studies carried out by many natural scientists over decades, smoking is a (considerable) risk factor for many cancers and respiratory diseases; but the public ignore these findings and keep smoking, which is where social scientists can help facilitate in getting the message across. Just one example of where the social sciences can have a massive (positive) impact on society. Image taken from stopcancer.support
“Scientists focus relentlessly on the future. Once a fact is firmly established, the circuitous path that led to its discovery is seen as a distraction.” – Eric Lander in the Cell journal (Jan 2016)
As scientists in the ‘natural’ sciences (e.g. genetics, physics, chemistry, geology), we have to make observations in the real world and think of hypotheses and models to make sense of it all. To test our hypotheses, we then have to collect (sufficient amounts of) data and see if the data collected fit the results that our proposed model predicted. Our hypotheses could be described as our ‘prejudice’ towards the data. However, we then have to try and counteract (and hopefully eliminate) our biases towards the data by performing well-designed experiments. If the results backup our predictions, we of course become (very!) happy and try to (replicate and then) publish our results. Even then (i.e. after a paper has been submitted to a journal), there is a lot left to do as the publication process is a long-winded one with many rounds of ‘peer-reviewing’ (an important quality control mechanism), where we have to reply fully to all the questions, suggestions and concerns the reviewers throw at us about the importance of the results, reliability of the data, the methods used, and the language of the manuscript submitted (e.g. are the results presented in an easy-to-understand way, are we over-sensationalising the results?). If all goes well, the published results from the analyses can help us (as the research community) understand the mechanisms behind the phenomenon analysed (e.g. biological pathways relating to disease, underlying mechanism of a new technology) and provide a solid foundation for other scientists to take the work forward.
If the results are not what we expected, a true scientist also feels fortunate and becomes more driven as a new challenge has now been set, igniting the curious side of the scientist; and strives to understand if anything may have gone wrong with the analysis or that whether the hypothesis was wrong. A (natural) scientist who is conscious and aware of the evolution and history of science knows that many discoveries have been made through ‘happy accidents’ (e.g. penicillin, x-ray scan, microwave oven, post-it notes) since it is in the nature of science to be serendipitous; and that a wrong hypothesis and/or an unexpected result can also lead to a breakthrough. Hopefully without losing any of our excitement, we go back to square one and start off with a brand new hypothesis (NB: the research paradigm in some fields is also changing, with ‘hypothesis-free’ approaches already been, and are being developed). This process (i.e. from generating the hypothesis to data collection to analysis to publication of results) usually takes years, even with some of the brightest people collaborating and working full-time on a research question.
“The first time you do something, it’s science.The second time, it’s engineering.A third time, it’s just being a technician.I’m a scientist. Once I do something, I do something else.” – Cliff Stoll in his TED talk (Feb 2006)
Natural scientists take great pride in exploring nature (living and non-living) and the laws that govern it in a creative, objective and transparent way. One of the most important characteristics of publications in the natural sciences is repeatability of the methods and replication of the results. I do not want to paint a picture where everything is perfect with regards to the literature in the natural sciences, as there has always been, and will be, problems in the way some research questions have been tackled (e.g. due to poor use of statistical methods, over-sensationalisation of results in lay media, fraud, selective reporting, sad truth of ‘publish or perish’, unnecessary number of co-authors on papers). However science evolves through mistakes, being open-minded about accepting new ideas and being transparent about the methods used. Natural scientists are especially blessed with regards to there being many respectable journals (with relatively high impact factors, 2 or more reviewers involved in the peer-reviewing process) in virtually all fields within the natural sciences, where a large number of great scientific papers are published; and these have clearly (positively) affected the quality of life of our species (e.g. increasing crop yield, facilitating understanding of diseases and preventive measures, curative drugs/therapies, underlying principles of modern technology).
I wrote all the above to come to the main point of this post: I believe the abovementioned ‘experiment-centric’ (well-designed, statistically well-powered), efficient (has real implications) and reliable (replicable and repeatable) characteristics of the studies carried out within the natural sciences should be made more use of in (and probably become a benchmark for) the social sciences. There should be a more stringent process before a paper/book is published similar to the natural sciences, and a social scientist must work harder (than they are doing at current) to alleviate their own prejudices before starting to write-up for publication (and not get away with papers which are full of speculation and sentences containing “may be due/related to”). I am not even going to delve into the technicalities of some of the horrendously implemented statistical methods and the bold inferences/claims made as a result of them (e.g. correlations/associations still being reported as ‘causation’, P-values of <0.05 used as 'proof').
Of course there are great social scientists out there who publish some policy-changing work and try to be as objective as a human being can possibly be, however I have to say that (from my experience at least!) they seem to be a great minority in an ocean of bad sociologists. Social sciences seem (to me!) to be characterised by subjective, incoherent and inconsistent findings (e.g. due to diverse ideologies, region-specific effects, lack of collaboration, lack of replication); and a comprehensive quality control mechanism does not seem to be in place to prevent bad literature from being published. A sociologist friend had once told me “you can find a reference for any idea in the social sciences”, which I think sums up the field's current state for me in one sentence.
“The scientist is not a person who gives the right answers, he’s one who asks the right questions.” – Claude Lévi-Strauss, an anthropologist (I would humbly update it as “The scientist is not necessarily a person who gives the right answers, but one who asks the right questions”)
Social sciences should not be the place where ones who could not (get the grades and/or) be successful in the natural sciences go to and get a (relatively) easier ride; and publish tens of papers/books which go insufficiently peer-reviewed, unread and uncited for life; but get a lecturer post at a university much quicker in relation to a natural scientist. Social scientists should not be any different from natural scientists with regards to the general aspects of research, so they should also spend years (just like most natural scientists) trying to develop their hypotheses and debunk their own prejudices; work in collaboration with other talented social scientists who will guide them in the right way; and be held accountable to a stringent peer-reviewing process before they can claim to have made a contribution (via books/papers) to their respective fields. Instead of publishing loads of bad papers, they should be encouraged to and concentrate on publishing fewer but much better papers/books.
Social sciences have a lot to offer to society (see the above figure about smoking for an example), but unfortunately (in my opinion) the representatives have let the field down. I believe universities and maybe even the governments all around the world should make it their objective to develop great sociologists by not only engaging them with the techniques used in the social sciences (and its accompanying literature), but also by funding them to travel to other laboratories/research institutions and get a flavour of the way natural scientists work.
Addition to post: For an academically better (and much harsher!) criticism of the social sciences than mines, see Roberto Unger’s interview at the Social Science Bites website (click on link).
Moon landing – a momentous achievement of mankind, and the natural sciences (and engineering)
PS: I must state here that I have vastly generalised about the social sciences; and mostly cherry picked and pointed out the negative sides. However every sociologist knows within them whether they really are motivated to find out the truth about sociological phenomena; and are not just in it for the respect that being an academic brings, or for the titles (e.g. Dr., Prof.). I personally have many respectable sociologist friends/colleagues myself (including my father) who are driven to understand and dissect sociological problems/issues and look for ways to solve real-life problems. They give me hope in that sense…
PPS: I am not an expert in the natural sciences nor in the social sciences. Just sharing my (maybe not so!) humble opinions on the subject matter as I get increasingly frustrated with the lack of quality I observe throughout the social sciences. Many of my friends/colleagues in the social sciences would attest to some or all of the things I stated above (gathering from my personal communications). I value the social sciences a lot and want it to live up to its potential in making our communities better…
An ordinary blog by an ordinary bloke 