Why I read the network nonsense papers

February 12, 2014 in q-bio.GN, reviews, sophistry | Tags: Benjamini-Hochberg, binomial coefficient, FDR, Joshua Grochow, Luke Ward, Manolis Kellis, methodsmatter, network motif

This is the third and final post in a series (part1, part2) of posts on two back-to-back papers published in Nature Biotechnology in August 2013:

Baruch Barzel & Albert-László Barabási, Network link prediction by global silencing of indirect correlations, Nature Biotechnology 31(8), 2013, p 720–725. doi:10.1038/nbt.2601
Soheil Feizi, Daniel Marbach, Muriel Médard & Manolis Kellis, Network deconvolution as a general method to distinguish direct dependencies in networks, Nature Biotechnology 31(8), 2013, p 726–733. doi:10.1038/nbt.2635

An inconvenient request

One of the great things about conferences is that there is time to chat in person with distant friends and collaborators. Last July, at the ISMB conference in Berlin, I found myself doing just that during one of the coffee breaks. Suddenly, Manolis Kellis approached me and asked to talk in private. The reason for his interruption: he came to request that I remove an arXiv post of mine, namely “Comment on ‘Evidence of Abundant and Purifying Selection in Humans for Recently Acquired Regulatory Functions“, a response to a paper by Ward and Kellis. Why? He pointed out that my arXiv post was ranking highly on Google. This was inconvenient for him, he explained, while insisting that it was wrong of me to post a criticism of his work on a forum where he could not directly respond. He suggested that it would be best to work out any issues I might have with his paper offline. Unfortunately, there was the inconvenient truth that arXiv postings cannot be removed. Unlike some journals, where, say, a supplement can be revised while having the original removed (see the figure switching of Feizi et al.), arXiv preprints are permanent.

My initial confusion quickly turned to anger. After all, my arXiv comment had been rejected from Science where I had submitted it as a technical comment on the Ward-Kellis paper. I had then put it on the arXiv as a last resort measure to at least have some record of my concerns publicly accessible. How is this wrong? Can one not critique the work of Manolis Kellis?

Network nonsense begins

My first review of a Manolis Kellis paper was in the fall of 2006, in my capacity as a program committee member of the Research in Computational Molecular Biology (RECOMB) conference held in Oakland, CA in 2007. Because Oakland is right next to Berkeley, a number of Berkeley professors were involved in organizing and running the conference. Terry Speed was chair of the program committee. I was out of the country that year on sabbatical at the University of Oxford, so I could not participate, or even attend, the conference, but I volunteered to serve on the program committee. For those not familiar with the RECOMB review process, it is modeled after the standard Computer Science conferences. The program committee chair forms the program committee, who are then assigned a handful of papers to review and score. Reviews are entered on a website, and after a brief period of online discussion about borderline papers, scores are revised and accept/reject decisions are made. Authors can revise their manuscripts, but the reviewers never see the papers again before publication in the proceedings.

One of the papers I was assigned to review was by a student named Joshua Grochow and his advisor Manolis Kellis. The paper was titled “Network Motif Discovery Using Subgraph Enumeration and Symmetry-Breaking“. Although networks were not my research focus at the time, and “symmetry-breaking” evoked in me nightmares from the physics underworld, I agreed to the review. The paper seemed to contain some interesting algorithms, appeared to have a combinatorial flavor, and potentially important applications- a good mix for RECOMB.

The problem addressed by Grochow & Kellis was that of identifying “network motifs” in biological networks. “Motifs” can be defined in a variety of ways, and the Grochow-Kellis objective was simple. In graph theoretic terms, given a graph G, the goal was to find subgraphs occurring with high multiplicity to an extent unlikely in a random graph. There are many models for random graphs, and the one that the results in Grochow-Kellis are based on is the Erdös-Renyi model (each edge chosen independently with some fixed probability). The reason this definition might be of biological interest, is that recurrent motifs interspersed in a graph are likely to represent evolutionarily conserved interaction modules.

The paper begins with a description of the method. I won’t go into the details here, except to say that everything seemed well until I read the caption of Figure 3. There the number 27,720 caught my eye.

During my first few years of graduate school I took many courses on enumerative and algebraic combinatorics. There are some numbers that combinatorialists just “know”. For example, seeing 42 emerge as the answer to a counting problem does not bring to mind Douglas Adams, but rather the vast literature on Catalan numbers and their connections to dozens of well-known counting problems. Similarly, a number such as 126 brings to mind binomial coefficients ( $126={9 \choose 4}$ ), and with them the idea of counting the number of subsets of fixed size from a larger set. When I saw the number 27,720 I had a hunch that somehow some canonical combinatorial set had been enumerated. The idea may have entered my mind because of the picture of the “motif” in which I immediately recognized a clique (all vertices mutually connected) and a stable set (no pair of vertices connected). In any case, I realized that

$27,720 = 220 \cdot 126 = {12 \choose 3} \cdot {9 \choose 4}$ .

The significance of this is that the “motif” on the left-hand side of Figure 3 had appeared many times because of a type of double- or rather thousandfold- counting. Instead of representing statistically significant recurring independent motifs, this “motif” arises because of a combinatorial artifact. In the specific example of Figure 3, the motif occurred once for any choice of 4 nodes from the clique of size 9, and any choice of 3 nodes from the stable set of size 12.

The point is that in a graph, any subgraph attached to a large clique (or stable set) will occur many times. This simple observation is a result of the fact that there are many subgraphs of a clique (or stable set) that are identical. I realized that this meant that the Grochow-Kellis method was essentially a heuristic for finding cliques and stable sets in graphs. The particular “network motifs” they were pulling out were just subgraphs that happened to be connected to some large cliques and stable sets. There are two problems with this: first, a clique or a stable set can hardly be considered an interesting “network motif”. Moreover, the fact that they appear in biological networks much more than in Erdös-Renyi random graphs is not surprising. Second, there is a large literature on finding cliques in graphs, none of which Grochow-Kellis cited or seemed to be familiar with.

The question of the performance of the Grochow-Kellis algorithm is answered in their Figure 3 as well. There is a slightly larger motif consisting of 6 nodes from the stable set of size 12, instead of 3. That motif occurs in all ${12 \choose 6}$ subsets of the stable set instead of ${12 \choose 3}$ subsets which means that there is a motif that occurs 116,424 times! Grochow and Kellis’s algorithm did not even achieve its stated goal. It really ought to have outputted the left hand side figure with six nodes in the stable set on the left, and not three. In other words, this was a paper providing uninteresting solutions from a biological point of view, and doing so poorly to boot.

I wrote up a detailed report on the paper, and posted it on the RECOMB review website together with poor scores reflecting my opinion that the paper had to be rejected. How could RECOMB, ostensibly the premier computer science conference on computational and algorithmic biology, publish a paper with neither a computational nor biological result? Not to mention an algorithm that demonstratably did not find the most frequently occurring motif.

As you might already guess, my rejection was subsequently overruled. I don’t know who made the final decision to accept the Grochow & Kellis paper to the RECOMB conference, although presumably it was the program committee chair. The decision jarred with my sense of scientific integrity. I had put considerable effort into reviewing the paper and understanding it, and I felt that I had provided a compelling objective argument for why the paper was fundamentally flawed- the fact that the results were trivial (and incorrect!) was not a subjective statement. At this point I need to point out that the RECOMB conference is quite difficult to get into. The acceptance rate for papers in 2007, consistent with other years, was 21.8%. I knew this meant that even a single very negative review, especially one with a compelling argument against the paper, almost certainly would lead to rejection of the paper. So I couldn’t understand then, nor do I still understand now, on what basis the decision was made to accept the paper. This bothered me greatly, and after much deliberation I started boycotting the conference. Despite publishing five RECOMB papers from 2000 to 2006 and regularly attending the meeting during that time, the continued poor decisions and haphazard standards for papers selected have led me to not return in almost 8 years.

Grochow and Kellis obviously received my review and considered how to “deal with it”. They added a section titled “The role of combinatorial effects”, in which they explained the origins of the number 27,720 that they gleaned from my report, but then spun the bad news they had received as “resulting from combinatorial connectivity patterns prevalent in larger network structures.” They then added that “…this combinatorial clustering effect brings into question the current definition of network motif” and proposed that “additional statistics…might well be suited to identify larger meaningful networks.” This is a lot like someone claiming to discover a bacteria whose DNA is arsenic-based and upon being told by others that the “discovery” is incorrect – in fact, that very bacteria seeks out phosphorous – responding that this is “really helpful” and that it “raises lots of new interesting open questions” about how arsenate gets into cells. Chutzpah. When you discover your work is flawed, the correct response is to retract it.

I don’t think people read papers very carefully. Joshua Grochow went on to win the MIT Charles and Jennifer Johnson Outstanding M. Eng. Thesis Award for his RECOMB work on network motif discovery. [Added February 18: Grochow and Kellis have posted a reply here].

The nature of man

I have to admit that after the Grochow-Kellis paper I was a bit skeptical of Kellis’ work. Not because of the paper itself (everyone makes mistakes), but because of the way he responded to my review. So a year and a half ago, when Manolis Kellis published a paper in an area I care about and am involved in, I may have had a negative prior. The paper was Luke Ward and Manolis Kellis “Evidence for Abundant and Purifying Selection in Humans for Recently Acquired Regulatory Functions”, Science 337 (2012) . Having been involved with the ENCODE pilot, where I contributed to the multiple alignment sub-project, I was curious what comparative genomics insights the full-scale $130 million dollar project revealed. The press releases accompanying the Ward-Kellis paper (e.g. The Nature of Man, The Economist) were suggesting that Ward and Kellis had figured out what makes a human a human; my curiosity was understandably piqued.

Ward and Kellis combined population genomic data from the 1000 Genomes Project with biochemical data from the ENCODE project to look for signatures of human constraint in regulatory elements. Their analysis was based on measuring three different proxies for constraint: SNP density, heterozygosity and derived allele frequency. To identify specific classes of regulatory regions under constraint, aggregated regions associated with specific gene ontology (GO) categories were tested for significance. Reading the paper I was amazed to discover they found precisely two categories: retinal cone cell development and nerve growth factor receptor signaling. It was only upon reading the supplement that I discovered that their tests had produced 53 other GO categories as well (Table S5).

Despite the fact that the listed categories were required to pass a false discovery rate (FDR) threshold for both the heterozygosity and derived allele frequency (DAF) measures, it was statistically invalid for them to highlight any specific GO category. FDR control merely guarantees a low false discovery rate among the entries in the entire list. Moreover, there was no obvious explanation for why categories such as chromatin binding (which had a smaller DAF than nerve growth) or protein binding (with the smallest p-value) appeared to be under purifying selection. As with the Feizi et al. paper, the supplement produced a story much less clean than the one presented in the main body of the paper. In fact, retinal cone cell development and nerve growth factor were 33 and 34 out of the 55 listed GO categories when sorted by the DAF p-value (42 and 54 when sorted by heterozygosity p-value). In other words, the story being sold in the paper was based on blatant statistically invalid cherry picking.

The other result of the paper was an estimate that in addition to the 5% of the human genome conserved across mammalian genomes, at least another 4% has been subject to lineage-specific constraint. This result was based on adding up the estimates of constrained nucleotides from their Table S6 (using the derived allele frequency measure). These were calculated using a statistic that was computed as follows: for each one of ten bins determined according to estimated background selection strength, and for every feature F, the average DAF value D_F was rescaled to

$PUC_F = \frac{(D_F - D_{CNDC})}{(D_{NCNE}-D_{CNDC})}$ ,

where D_CNDC and D_NCNE were the bin-specific average DAFs of conserved non-degenerate coding regions and non-conserved non-ENCODE regions respectively. One problem with the statistic is that the non-conserved regions contain nucleotides not conserved in all mammals, which is not the same as nucleotides not conserved in any mammals. The latter would have been needed in order to identify human specific constraint. Second, the statistic PUC_Fwas used as a proxy for the proportion under constraint even though, as defined, it could be less than zero or greater than one. Indeed, in Table S6 there were four values among the confidence intervals for the estimated proportions using DAF that included values less than 0% or above 100%:

Ward and Kellis were therefore proposing that some features might have a negative number of nucleotides under constraint. Moreover, while it is possible that after further rescaling PUC_Fmight have correlated with the true proportion of nucleotides under constraint, there was no argument provided in the paper. Thus, while Ward and Kellis claimed to have estimated the proportion of nucleotides under constraint, they had only computed a statistic named “proportion under constraint”.

Nicolas Bray and I wrote up these points in a short technical comment and submitted it to the journal Science early in November 2012. The comment was summarily rejected with a curt reply by senior editor Laura Zahn stating that “relative to other Technical Comments we have recently received we feel that the scope and focus of your comment make it more suitable for the Online Comments facility at Science, rather than as a candidate for publication as a Technical Comment.” It is worth noting that Science did decide to publish another comment: Phil Green and Brent Ewing’s, “Comment on’Evidence of Abundant and Purifying Selection in Humans for Recently Acquired Regulatory Functions‘”, Science 10 (2013). Green and Ewing’s comment is biological in nature. Their concern is that “… the polymorphism trends are primarily attributable to mutational variation and technical artifacts rather than selection.” Its fine that Science decided to host a debate on a biology question on its pages, but how can one debate the interpretation of results from a method, when the method is fundamentally flawed to begin with? After all, our problem with PUC was much deeper than a “technical flaw”.

We decided at the end to place the comment in the arXiv. After doing so, it became apparent that it had little impact. Indeed, I have never received any feedback about it from anyone. Apparently even this was too much for Manolis Kellis.

Methods matter

By the time I noticed the Feizi et al. paper in the journal Nature Biotechnology early in August 2013, my experiences reading Kellis’ papers had subtly altered the dynamic between myself and the printed word. Usually, when I read a paper and I don’t understand something, I assume the fault lies with me. I think most people are like this. But now, when the Feizi et al. paper started to not make sense, I didn’t presume the problem was with me. I tried hard to give the paper a fair reading, but after a few paragraphs the spell of the authors was already broken. And so it is that Nicolas Bray and I came to figure out what was really going on in Feizi et al., a project that eventually led us to also look at Barzel-Barabási.

Speaking frankly, it was difficult work to write the blog posts about these articles. In addition to the time it took, it was exhausting and exasperating to discover the flaws, fallacies and frauds. Both Nick and I prefer to do research. But we felt a responsibility to spell out in detail what had happened here. Manolis Kellis is not just any scientist. He has, and continues to play leading roles in major consortium projects such as mod-ENCODE and ENCODE, and he has served on numerous advisory committees for the NHGRI. He is a member of the GCAT (Genomics, Computational Biology and Technology) study section until 2018. That any person would swap out a key figure in a published paper without publishing a correction, and without informing the editor is astonishing. That a person with great responsibility towards scientists is an abuser of science is unacceptable.

Manolis Kellis’ behavior is part of a systemic problem in computational biology. The cross-fertilization of ideas between mathematics, statistics, computer science and biology is both an opportunity and a danger. It is not hard to peddle incoherent math to biologists, many of whom are literally math phobic. For example, a number of responses I’ve received to the Feizi et al. blog post have started with comments such as

“I don’t have the expertise to judge the math, …”

Similarly, it isn’t hard to fool mathematicians into believing biological fables. Many mathematicians throughout the country were recently convinced by Jonathan Rothberg to donate samples of their DNA so that they might find out “what makes them a genius”. Such mathematicians, and their colleagues in computer science and statistics, take at face value statements such as “we have figured out what makes a human human”. In the midst of such confusion, it is easy for an enterprising “computational person” to take advantage of the situation, and Kellis has.

I believe the solution for this problem is for computational biologists to start taking themselves more seriously. Whether serving as reviewers for journals, as panel members for funding agencies, on hiring/tenure committees, or writing articles, all of us have to tone down the hype and pay closer attention to the science. There are many examples of what this means: a review of a math/stats containing paper cannot be a single paragraph long and based on a hunch, and similarly computational biologists shouldn’t claim, as have many of the authors of papers I’ve reviewed in these posts, pathways to cure disease and explanations for what makes humans human. Don’t fool the biologists. Don’t fool the computer scientists, statisticians, and mathematicians.

The possibilities for computational methods in biology are unlimited. The future is exciting, and there are possibilities for significant advances in areas ranging from molecular and evolutionary biology to medicine. But money, citations and fame cannot rule the day. The details of the #methodsmatter.

78 comments

Comments feed for this article

February 12, 2014 at 7:43 am

Marc RobinsonRechavi (@marc_rr)

Thank you for writing these posts. I realize that it’s a lot of work which won’t count on your CV and will only gain you enemies.

I have a similar experience, that showing a methodological flaw which renders a biology paper moot is “too technical” for Nature. I discussed this only much later on my blog, after publishing a follow-up paper, and I regret not posting earlier to ArXiv and not blogging about it early also.

http://people.unil.ch/marcrobinsonrechavi/2013/07/story-behind-the-paper-the-hourglass-and-the-early-conservation-models-co-existing-evolutionary-patterns-in-vertebrate-development/

I’ve seen that people attack you on your tone. I probably wouldn’t have written these posts the same way as you did, but I am amazed that there is a culture among some scientists of thinking that the appearance of respect for colleagues is more important than actual respect for facts and logic. If you are wrong, this should be shown, but implying that it’s impolite to call out perceived errors in a argumented manner is opposite of what we should stand for as scientists.

February 12, 2014 at 7:45 am

Lior Pachter

Thank you. I truly appreciate it.

February 12, 2014 at 3:35 pm

Yoda

@Lior – You have made your case but the most appropriate will be an independent body to look in to this matter. You will not come across as biased in this matter. Why don’t you send your findings to MIT, Nature Biotech and NIH (who funded that work to Kellis) to investigate. They are the right bodies to decide if there was a fraud or ethical academic violations

February 12, 2014 at 8:59 am

edermitzakis

BTW Marc, your story of getting the paper in PLOS Genetics after a discussion with Greg Barsh can be perceived in two different ways. 1. A good editor can change their mind and see the value of the paper (which is the case since GB is a very good editor in chief); 2. Lobbying on your end pushed the paper in. Had it been Nature instead of PLOS Genetics even Lior would not approve and say it is no 2.

February 12, 2014 at 9:11 am

Marc RobinsonRechavi (@marc_rr)

I see your point. We were contacted directly by Greg after writing a rather standard appeal letter, so I would not call that lobbying. I certainly did not appel to my (inexistant) clout.

Relevant to the discussion here on tone, I had to insist to not have our paper written as an attack on the “hourglass model”, which is what several editors seemed to expect or want.

February 12, 2014 at 9:15 am

edermitzakis

My point was just to show that the exact same process would be perceived completely differently based on (correct or incorrect) prior perceptions.

May 4, 2017 at 10:49 am

Yolanda

I read it and I’m in Lior’s team! So he gains more than enemies, I’m a fan! I’m surprised there are so many weeds in research nowadays, it became such a huge business with lots of money and the greedy rub their hands together at how profitable it is.

February 12, 2014 at 8:12 am

UBN

Bravo Dr. Pachter! This is all the result of the debasing science has gone through in the last 30 years. I am encouraged that there are still people like you who care enough to go through the hassle of uncovering the BS.

February 12, 2014 at 8:43 am

edermitzakis

I also agree that it is great that Lior is taking the time to go that deep into the methods of papers and reveal problems. However, I am not impressed by the personal accusations and overall tone. I am not yet in a position to say if there is anything wrong in the two papers but I find it very hard to believe that the investigators mentioned are committing fraud. If the authors made mistakes or if they do analysis based on assumptions that we/you don’t agree that is no fraud. This is a very serious accusation that is not only far from a scientific argument but goes to other areas not necessary to mention here. What is important is that this accusation does not only raise flags for the person being accused but also equally for the person making the accusations. Lior carries a lot of responsibility here and when called to prove his point (not that there are mistakes – if there are – but that they committed fraud) I hope he has proof and not just personal impressions and biases. Such type of accusations depending on where the truth lies can ruin the careers of those accused but equally of those accusing.

February 12, 2014 at 1:26 pm

Marc RobinsonRechavi (@marc_rr)

The specific quote from the previous post is:

In academia the word fraudulent is usually reserved for outright forgery. However given what appears to be deliberate hiding, twisting and torturing of the facts by Feizi et al., we think that fraud (“deception deliberately practiced in order to secure unfair or unlawful gain”) is a reasonable characterization.

I agree that Lior carries a lot of responsability here, yet it seems to me that the case was made with proper clarity and documentation, and cannot be summarized nor dismissed as an ad hominem attack solely on the character or personality of colleagues. Of note, by writing under his true name and answering critics, Lior is taking that responsability.

So it remains my opinion that focusing on the tone and not on the issues raised by these posts is not producitive. If the issues raised are mistaken, then the accusations should clearly be retracted, with apologies. If these issues are correct, then it is important and should be discussed publicly. At present, Lior’s case appears strong and the answers are not very convincing.

I would also like to note that if more scientists behaved as Lior does with his expertise of published results and methods, and open discussion of his findings, I think that science overall would be better. Maybe I’d also be a victim, who knows? The same can be send about reviewers who sometimes raise difficult but correct issues.

I sincerely believe that the blog of Lior contributes to the aim of learning honestly about the world, which is what science does or should do. Even though, as I wrote previously, I would not personally have written these posts this way.

February 12, 2014 at 7:23 pm

James Rustle

What a vacuous comment. Seriously, no one cares if you find it “hard to believe” given that you like so many others didn’t put in any time or effort into evaluation of the content of the arguments. Do the math or stfu.

February 12, 2014 at 10:17 am

homolog.us

> I hope he has proof and not just personal impressions and biases.

Edermitzakis,

Can you not read English text, or is it another case of ”It is difficult to get a man to understand something, when his salary depends on his not understanding it’? Don’t you think switching figures of a peer-reviewed paper without mentioning anywhere deceptive and fraudulent?

Let me reproduce my comment from the previous blog post –

“Manolis, What you did with the supplementary figure is clearly fraudulent. You replaced Fig S4 in the updated version post-publication, which changed the qualitative nature of the figure and paper. However, there is no mention of that in the text that you provided related to revision (which I reproduce below).

“In the version of this file originally posted online, in equation (12) in Supplementary Note 1, the word “max” should have been “min.” The correct
formula was implemented in the source code. Clarification has been made to Supplementary Notes 1.1, 1.3, 1.6 and Supplementary Figure 4 about
the practical implementation of network deconvolution and parameter selection for application to the examples used in the paper. In Supplementary Data, in the file “ND.m” on line 55, the parameter delta should have been set to 1 – epsilon (that is, delta = 1 – 0.01), rather than 1. These errors have been corrected in this file and in the Supplementary Data zip file as of 26 August 2013.”

That means the reviewers saw one figure and the current version of the figure has a completely different version. , You may argue that qualitative change in Fig S4 has no effect on the main paper (or rather it is an uninformative figure). However, you were supposed to make that argument with the editor before replacing the figure. I would recommend that you withdraw and resubmit the paper, because the currently published paper is not peer-reviewed for the reasons stated above.”

February 12, 2014 at 11:00 am

Yoda

@Lior – You have made your case but the most appropriate will be an independent body to look in to this matter. You will not come across as unbiased in this matter. Why don’t you send your findings to MIT, Nature Biotech magazine and NIH (who funded that work) to investigate. They are the right bodies to decide if there was fraud or ethical academic violations

February 12, 2014 at 12:37 pm

Daniel Falush

Aristotle said: “to be angry with the right person and to the right degree and at the right time and for the right purpose, and in the right way – that is not within everybody’s power and is not easy”. One observation I would like to make about these posts is that they describe many different types of misbehaviour from many different people. It is legitimate to be angry with journal editors and reviewers because they give famous people an easy time. It is legitimate to get angry with people because they switch figures without providing clearcut explanation. It is legitimate to be angry because people bury technical details to too great a degree. But mixing all of these qualitatively different things together with several other seems problematic from everyone’s perspective. Effective anger is staccato. A contained fire.

February 12, 2014 at 4:15 pm

Erik van Nimwegen

Dear Lior,
just wanted to let you know that, although I may not agree with everything you say, I do think I understand very well why you are doing this and I sympathise very much with what you are trying to do. I also find it highly courageous.
Reading your posts got me thinking why I don’t write blog posts like this. Am I just less courageous than you, or are there other reasons I don’t invest my time doing this?
Here are a couple of my thoughts for whatever they’re worth:
Having worked a on a closely-related problem (disentangling direct from indirect co-evolutionary correlations in protein alignments to identify interacting residues) my interest was obviously aroused by the title and abstract of the Feizi et al paper but I already felt skeptical at the rather grand claims of general applicability in the abstract (i.e. domain-specific knowledge is almost always required to get to the best approaches).
Then I saw the formulas in figure 1. Now, for anybody that has done any work on in the general area of counting paths on graphs, or diffusion on graphs, or calculating things like first-passage time distributions of Markov chains, etcetera, seeing that formula you immediately know that something is not kosher here. That kind of inversion simply does not work for any matrix and that is fairly elementary stuff. Looking in the main text the paper the main text does read like the authors are beating around the bush and are essentially playing hide-the-ball with what they are actually doing. At that point I pretty much concluded that probably the entire work is misleading (call me cynical).
The fact of the matter is that I have become used to seeing published, in high profile journals, methodologically incredibly sloppy, and misleadingly presented work in computational biology, and I guess I have basically come to accept that these things are among us. As you rightly point out, given that most biologists are either math-clueless or math-phobic and that most mathematicians have no inkling about biology, there is a niche for this kind of this stuff. One part of me says: this is no use bothering with. These kinds of methods won’t stand the test of time anyway. Attacking them is like shovelling snow. It will anyway disappear by itself eventually.
On the other hand, I do sometimes worry that this kind of stuff may actually PUSH OUT more honest and rigorous work in computational biology and that there is even a chance that it may bring the whole field down. So another part of me says that we should indeed invest time to fight this.

The question is what the best way to do this. I am not sure going after particular people is the most effective way. Yes, I agree there are repeat offenders that repeatedly peddle poor stuff in high profile journals but I don’t think particular individual are the problem. I think ultimately education is the key problem: people that want to study molecular biology these days should get a solid education in math. Molecular biology should no longer be a place for students that don’t like math.

I do think that too many senior people in the field don’t speak up enough to some of the poor work that is being done. I know there are lots of people that at meetings gossip with each other about how appalled they were that this or that nonsense got published in a high profile journal, but the large majority of them do not speak up about it in public. Especially for young people coming into the field it is important to realise that there is a lot of nonsense out there, and they should be very critical.

February 12, 2014 at 8:40 pm

Saheli Datta (@SaheliDatta)

I think ultimately education is the key problem: people that want to study molecular biology these days should get a solid education in math. Molecular biology should no longer be a place for students that don’t like math.

As a long-time friend and staff person at Berkeley who currently works for Lior on educational projects, allow me to step in, and note that I believe he passionately agrees with you about the long term solution. He has dedicated an enormous amount of his personal time and effort–and procured mine and that of several others–to making that happen at Berkeley. But the freshman biology majors he taught math to last fall, and the freshman biology majors we are currently plotting to prevent from developing math phobia next fall, are not going to be among main targeted consumers for these kinds of papers for at least five, if not ten, years. They are not going to be respected authorities in biology for another 15 or 20 years. The best case scenario of full reform of the undergraduate biological curriculum can’t be fully implemented in less than five years. A lot can happen to a field in 5-25 years, and a lot of public money and wasted youth can be spent on pursuing ideas build on a foundation of unreliable insight generated by poorly understood algorithms. He wants those students to inherit a healthy field filled with robust, dependable findings upon which to build their own work. So please keep that in mind when you consider his short-term strategy.

February 13, 2014 at 8:18 am

Nikolay Nikolov

“I know there are lots of people that at meetings gossip with each other about how appalled they were that this or that nonsense got published in a high profile journal, but the large majority of them do not speak up about it in public. ”

Which is exactly why blog posts like Lior’s are so valuable. Computational Biology as a field has a low signal-to-noise ratio precisely because lots of computational or math garbage can be peddled to biologists and vice versa. It takes both a lot of hard work and a lot of courage to speak up.

February 17, 2014 at 3:38 am

Nicolas Bray

“Attacking them is like shovelling snow. It will anyway disappear by itself eventually.”

We will all disappear eventually, it’s true, but I’d note that many places where it snows have laws *requiring* people to remove it. The reason being that snow which is not removed will often end up as ice: impossible to remove and dangerous for those who encounter it unawares.

I’m sure there’s a metaphor for tenure in there somewhere.

February 12, 2014 at 4:22 pm

Charles Warden

I’d consider it a relatively minor issue, but I think there are some caveats to the GO enrichment critique:

Sometimes the top ranked GO enrichment category is truly not the most important. For example, I see enrichment for olfactory receptor genes in a non-trivial amount of data sets that I analyze, and I wouldn’t consider those to the driving factor for most of the phenotypes being studied (my guess that this may have to do with being in a similar genomic location, possible issues with non-specific hybridization / alignment, etc). Likewise, I think biologists can have a genuine intuition based upon experience that isn’t captured in the statistical analysis strategy.

For example, I point out in this blog post on an unrelated paper that the TGF-beta signaling pathway gets a lot of focus but it wasn’t the most statistically significant result:

http://cdwscience.blogspot.com/2011/03/artilce-review-epigenetic-suppression.html

However, I think an important distinction is that the enrichment wasn’t a final result in that case. There was further validation done to characterize the behavior of the TGF-beta signaling pathway (in fact, perhaps the feasibility of validation was a factor in deciding what to focus on in subsequent analysis). In other words, I wouldn’t have a problem with not selecting the top GO category if there is independent evidence to support the results.

That said, I don’t think detailed characterization is always possible, and I would also say there is value for presenting functional enrichment results as a hypothesis (so long as those results are presented in a way that shows less confidence than some other more carefully characterized results).

Of course, there can be debate about whether such preliminary explanations should be presented in the abstract of a Science article (and/or whether phrases like “intriguing” in the main text were sufficient to indicate an observation is a hypothesis that may or may not be correct), but I don’t think failure to focus on the most statistically significant result should be considered unacceptable in all circumstances.

February 12, 2014 at 6:38 pm

Razib Khan (@razibkhan)

thanks for this. charges against m. kellis in specifics seem damning, but i’ll hold off any final judgment. but the general problem is surely true, and many people discuss it privately. so important and worthwhile that this was highlighted and is getting attention.

February 13, 2014 at 4:25 am

Student

Respected scientists, please inspect all Manolis Kellis’ papers!!! We don’t want this kind of “knowledge” to propagate to our textbooks. Please help to save us!

February 14, 2014 at 6:03 pm

Anon

Someone on the Reddit thread posted below pointed out data quality issues in Kheradpour et al., Systematic dissection of regulatory motifs in 2000 predicted human enhancers using a massively parallel reporter assay

February 13, 2014 at 4:37 am

Nikolay Nikolov

Thank you, Lior, for the hard work that you have done in analysing these papers and in sharing your concerns. I find the response by Manolis Kellis and his co-authors troubling. It does, indeed, make a mockery of the peer-review process. I also find troubling the lack of response by Nature Biotechnology. Clearly, the paper should be retracted.

February 13, 2014 at 6:46 am

Edward Lear

Has anyone pointed out this thread to the editors at Nature Biotech?

February 13, 2014 at 7:10 am

Manolis Kellis

(1) Lior’s criticism of our RECOMB paper with Joshua Grochow is complete news to me and Josh, is completely baseless, and we will respond more at length.
(2) Lior re-iterates his criticisms of our Ward and Kellis 2012 paper, which have already been the subject of multiple posts by Lior. His critiques lack any substance, which makes it difficult to respond, but we will do so nonetheless.
(3) Lior claims that our work in Feizi et al 2012 is irreproducible and fraudulent. We have responded to these allegations here http://compbio.mit.edu/nd/Response_to_Nonsense_Blog_Post.pdf and invite readers of his blog to try out the ND code here http://compbio.mit.edu/nd/try_it_out.html
(4) Our corrections to the NBT supplement were reviewed by the Nature Biotechnology editors and staff at all stages and do not affect any of the results of the paper. The original correction notice lacked sufficient detail, which has now been rectified.
(5) Lior admits that ‘any good blogger’ uses hyperbole to attract readers, but allegations of fraud are extremely serious, can be damaging, and have no place in the list of tricks to attract readers to a blog.
(6) As for the public admission that he has held a grudge against both me and RECOMB for 7 years, i am frankly shocked, saddened, and troubled, but at least it helps explain the anger in his baseless attacks

February 13, 2014 at 1:00 pm

Anon

Lior claims that the RECOMB paper was modified by you & Josh to “[add] a section titled “The role of combinatorial effects”” after Lior reviewed it. You claim that his criticism is “complete news to [you] and Josh”. These two statements don’t fit together very well.

First of all, did you add the section in question, and if so, when? Second, if you didn’t add it in response to Lior’s review, why did you add it?

February 13, 2014 at 1:10 pm

Anon

The section in question is present in the copy of the paper available from Josh’s site, including all the quotations Lior provided: http://www.people.cs.uchicago.edu/~joshuag/Grochow_Kellis_RECOMB_07_Network_Motifs.pdf

The only question was whether or not it was added in response to Lior’s review, which Kellis claims was “complete news” to him.

February 13, 2014 at 8:12 am

Joe Pickrell

Ward and Kellis were therefore proposing that some features might have a negative number of nucleotides under constraint.

This is potentially misleading. As Lior is surely aware, an estimator of a parameter can take values outside the range of “reasonable” values for that parameter.

For example, a standard estimator of the genetic distance parameter Fst is often estimated as negative. (see this BioStars post for details). But of course a distance can’t be negative! What’s happening is that the estimator is designed to be unbiased even at low values (try constructing an unbiased estimator of Fst when Fst = 0 without allowing the estimator to take negative values).

I agree that clarifying the behavior of the statistic in the Ward/Kellis paper and the assumptions that go into it seems like a reasonable thing to do. But the fact that it can be negative is not necessarily a problem.

February 13, 2014 at 9:34 am

Lior Pachter

I completely agree that some good estimators can take on meaningless values. The problem here is not that the Ward-Kellis one can. It is that it does.

February 13, 2014 at 10:23 am

Joe Pickrell

Thanks Lior. I think on reflection (see conversation w Nick below) I agree with the criticism that the statistic is not well-motivated.

I guess our main disagreement is that I’m hesitant to call using a poorly-motivated statistic an example of an “abuse of science”.

February 13, 2014 at 10:24 am

Lior Pachter

My abuse comment refers to the totality of what I reported on in all the posts.

February 13, 2014 at 10:35 am

Joe Pickrell

My abuse comment refers to the totality of what I reported on in all the posts.

I understand what you mean. But I guess I feel like each individual example should be able stand–at least to a large extent–on its own.

February 14, 2014 at 2:06 am

Erik van Nimwegen

I know this is not really the topic at hand but I am surprised to see how many people apparently think it is completely ok that estimators can take on values that are nonsensical. Why would we find that acceptable in a supposdely coherent mathematical theory?
Instead, I would say that fact that unbiased estimators can take on nonsense values strongly indicates that there is something fundamentally wrong with the orthodox statistical practice of using unbiased estimators.
Moreover, there is no need to use such estimators. Bayesian statistics gives you ways of estimating parameters that don’t exhibit any of these pathologies. And this is not really a recent finding.
For a useful discussion, with examples of unbiased estimators that given nonsense results for essentially all possible datasets, see chapter 17 of ET Jaynes “probability theory, the logic of science”. An early version of this chapter you can read here:

Click to access cc17h.pdf

February 14, 2014 at 7:52 am

Joe Pickrell

This would be a great topic for Lior’s next blog post. Tentative title: “The field of statistics is fundamentally wrong” 🙂

February 13, 2014 at 9:21 am

Nick Eriksson

It’s a good point that there are estimators that are very useful but can give nonsense values (my favorite being that you can usually just use linear regression instead of logistic without substantially changing p-values).

I think the more damning things about the paper are (1) the cherry picking and (2) the fact that they don’t give an argument that the statistic measures the right thing (I can’t see how it does).

February 13, 2014 at 9:50 am

Joe Pickrell

Thanks Nick. Taking a second look, I think you can interpret the statistic as an attempt at a scaled value of “evolutionary constraint”, where 0 is nucleotides outside of any region of the genome that looks “interesting”, and 1 is nucleotides that alter protein sequence. I agree I don’t see any obvious argument for mapping this onto percentages in a linear way.

Also agree the GO results are not overwhelming.

February 13, 2014 at 2:31 pm

A random visitor

I’ll preface this by saying that I am not from this field: I stumbled into this by reading the Metafilter post. First of all, I think these posts argue compellingly that the standard for reviewing papers that rely on technical methods clearly has to be raised. I do not think this is an easy task. In my understanding, the expectation for biology papers is that they should be reviewed within, say, a month of the request. That can be done if it’s all standard methods and small data sets, but that duration is ill-suited to, for example, consortium-produced behemoths. Thus one solution, as these posts exemplify, is to take more time and care when reviewing papers in the first place. In mathematics, for example, it is not unheard of for papers to spend months or even years in review. (Nevertheless, anyone who has done math research will know that still does not guarantee that published papers are free of errors or gaps in logic!) Computational biology seems to face an even more difficult task, as you must review both an algorithm (presumably something whose validity can be supported by rigorous arguments) and its utility (which requires more experimentally-flavored confirmations). I think that many people tend to rely on the latter arguments rather than the former, which can lead to methodological problems going unnoticed once the focus shifts to the results. So on that point you’ve got me entirely on board.

Like others, however, the manner in which some of these points were raised was distracting for me. I think science is both a search for truth, but like all human activities, it is also a social enterprise. This article in particular suggests a long-term dislike for Manolis Kellis played a role in these investigations. Perhaps the continued criticism is warranted, but as a naïve reader I have to say this dogged pursuit of one person left a bad taste in my mouth. (E.g. Was it really necessary to call out some poor Master’s student’s thesis award?) I realize that you’re trying to draw attention to a structural problem in your field, but I can’t help but feel some sympathy for the people who are singled out, as I’m sure you could have written similar write-ups on many other papers in the literature. Retracting a paper is an extremely painful experience for most scientists, so if that’s what you want to encourage, perhaps a measure of compassion is in order.

Of course, several of the other commenters appear to have no problem with the tone, so maybe it’s all in my reading of the posts. I think that distinction is worth noting: what one person perceives as “passionate” another might perceive as “aggressive.” The problem with the latter is that it reduces the impact of the original, valid criticism, because people start to write you off as bombastic, or worse are simply inclined to ostracize you completely.

For the record, I think the net benefits here still outweigh any concerns I raise. Results should be challenged and scrutinized, and this can lead to tough situations for the authors. But the questions about civility, tone, etc. do deserve some thought. We obviously want to maintain a healthy, open debate about science as it unfolds so that we can all make the most efficient use of our time and resources. As the role that blog posts, anonymous peer comments, Twitter, etc. play in doing this is still evolving, we should make sure that we end up with a system that people actually want to participate in.

February 13, 2014 at 4:23 pm

RobertA

I understand and mostly agree with your comment. However, I personally interpret the fact that there is a ‘history’ between Lior and Manolis less as showing a long term dislike between the two, but more as proving that Lior has been trying to get them to fix the problems through other means than these posts, which we can all agree is a necessary thing to do. So I feel that the lack of history would, in some sense, be more damaging.

February 13, 2014 at 7:32 pm

Also random

+1. Particularly agree the swipe at the student thesis award from 7 years ago looks gratuitously resentful and mean-spirited. Much of the remainder of the post is also resentful and mean-spirited, perhaps just not gratuitously so 🙂

February 13, 2014 at 4:03 pm

Anon Student

Lior has really done comp. bio. a huge service here. If computational biologists don’t take their own work seriously, why should anyone else? If after coming to know about the misleading work of M. Kellis there is no response, that is a disservice to the field. We can look at secondary attributes such as tone and manners, etc. w.r.t. the way his message was conveyed, but let’s not lose sight of the big picture here. After reading Lior’s post, I have even more respect for the computational biolgists that are out there because they clearly take their own work seriously and strive for rigor no less than any other scientist can or should. While passionate tone can be misconstrued for aggression as stated in a previous post, there is the added benefit that it draws attention to a very serious and critical problem in academia today: the inability to maintain adequate standards in the peer-review process.

February 14, 2014 at 1:12 am

Someone

I think that it would be nice to start a more constructive approach and to extract the true questions from the noise.
As random visitor was saying, there are personal feelings and scientific questions mixed. This is not good for your point.
So, start a new blog post with a neutral tone saying that you want to understand this paper completely. Imagine that you have no a-priori about the authors or methods, you just have questions.
What are the three most important questions that you would like to ask the authors? Write them down in that neutral blog post and start from a new perspective.
You need to remove the burden from the authors as much as possible. So, explain why these questions are important and where you got stuck. For instance, if you try to reproduce some of the figures but did not succeed, show what you got, show where you got stuck in the process, etc…
That would definitely help the authors to answer your queries.

February 14, 2014 at 8:01 am

Curious Wavefunction

“The first principle is that you must not fool yourself, and you are the easiest person to fool” – Richard Feynman.

I don’t know enough about the original study to comment on it but the kind of brouhaha described here seems to me to be typical of the birthing pains that interdisciplinary research often suffers from. It’s especially the case in biology which has recently been invaded by computer scientists, mathematicians, statisticians and other non-biologists. But this is hardly a new trend; after all there has been a highly successful tradition of physicists in biology, starting with Francis Crick and continuing on through Max Delbruck, Walter Gilbert and venki Ramakrishnan.

The fact is that biology is too complex and fascinating to be left to the biologists and in many cases it’s only outsiders who can bring a fresh perspective to it. So this is all good as long as, as you indicated, the concerned parties don’t peddle incoherent principles from their own disciplines to each other.

February 14, 2014 at 10:52 am

Claudiu Bandea

I think most people would agree with ‘A random visitor’ (see above) that “science is both a search for truth, but like all human activities, it is also a social enterprise”.

And, I also think that most people would be sympathetic to his/her call for civility.

However, we are in the midst of a revolution in the science enterprise, including science communication, and applying ‘civility’ at this critical stage might not be a wise approach.

You see, those who have prospered under the current system are not enthusiastic about changes. However, they are sophisticated enough to realize that blatantly opposing these changes is not a viable solution, as it might put them at risk of losing their ‘privileges’ under the new system.

So, obviously, a smart strategy would be to join the revolution, but call for ‘civility’, which would assure a quiet and smooth transition right back at the top of the new enterprise.

Go Lior! Go Dan! Go Joe!

February 17, 2014 at 11:59 am

A N

The wisest of approaches more frequently call for civility and diplomacy than anything else.

February 14, 2014 at 1:05 pm

RedditReader

There’s a discussion on Reddit about this blog post http://www.reddit.com/r/bioinformatics/comments/1xmdzn/the_network_nonsense_of_manolis_kellis_lior/

February 15, 2014 at 11:47 am

Centrist Biostatistician

There’s a problem with your boldfaced statement, “FDR control merely guarantees a low false discovery rate among the entries in the entire list.” The type of procedure used here is formulated to control FDR in expectation, over repeated applications of the procedure. It does not actually provide a guarantee about the false discovery rate in any one list. The objection raised in this blog post can thus be taken even further – even the list itself cannot be trusted!

This however recalls the infamous gaffe of Neyman and Pearson, “as far as a particular hypothesis is concerned, no test based upon the theory of probability can by itself provide any valuable evidence of the truth or falsehood of that hypothesis.” Frequentist extremism renders the statistical enterprise nearly worthless.

There’s another way to look at it. Storey (2003, Annals) showed that FDR values computed by a variant of the procedure (which differs in a corner case that is usually negligible in high-throughput applications) can be interpreted as approximate empirical Bayesian belief values under simple assumptions. When viewed in this way, it’s not “statistically invalid” per se to select examples passing the FDR threshold.

Bottom line, where you see “blatantly statistically invalid cherry picking” there’s actually a fair degree of texture. Because I’d like you to present the strongest possible arguments, I’d suggest the objection should be focused on the sin of omission rather than the statistical arcana.

February 16, 2014 at 9:25 pm

Manolis Kellis

We provide here a response to the incorrect criticisms of the Grochow and Kellis RECOMB 2007 paper, which can also be found at:

Click to access GrochowKellis_ResponseToPachter.pdf

Lior Pachter reveals that he was one of the reviewers of our RECOMB 2007 paper, and claims that the paper is incorrect and trivial. We respectfully disagree, and we refute each of his criticisms below:

Criticism 1 (subjective): Pachter claims our results are trivial.
Our algorithm advanced the state of the art in motif finding, enabling the identification of larger network motifs than ever before. It achieved that by introducing several key ideas, which have now been built upon in multiple algorithms to identify even larger motifs:
1) We reversed the traditional directionality of the network motif discovery problem, to start with subgraphs and then search for them in the network, rather than scanning the network and enumerating all subraphs of a given size.
2) We provided a solution to the subgraph isomorphism and enumeration problem that runs exponentially faster on real-world networks, by recognizing self-symmetries of network motifs and imposing symmetry-breaking conditions that eliminated repeated instances.
3) In addition, using our algorithm we discovered the role of combinatorial effects in larger motifs, and introduced a motif clustering score and computed the clustering properties of all discovered motifs. We explicitly described these both in our original submission and in our published paper.
We stand by the novelty and contribution of our work, which has not only stood the test of time since 2007 (see our response to Criticism 6 below), but has since been replicated and built upon, and is now part of the cited literature in the field of network motif research.
Prior to our algorithm, no one could find even a single motif of 15 nodes, let alone dream of looking for motifs of 20 nodes, 30 nodes, or more (we successfully searched for 31-node subgraphs, for example, although we didn’t find any of statistical significance). Even today, only two algorithms are able to reach these motif sizes: (1) [Omidi, Schreiber, Masoudi-Nejad. MODA. Genes Genet. Syst. 2009 Oct, 84(5):385-95]; and (2) [Ribeiro and Silva. G-Tries: an efficient data structure for discovering network motifs. ACM Symposium on Applied Computing (SAC ’10)]. Both of these algorithms build on our work, and would not be able to reach such motif sizes without the ideas first introduced in our paper.

Criticism 2 (technical): Pachter claims that the 15-node motif in Fig. 3 is solely due to combinatorial effects due to the abundance of cliques relative to Erdös-Renyi random graphs.
As explicitly stated in our paper, the enrichment of the motif in Fig. 3 is due to combinatorial effects. However, it is not as simple as Pachter suggests in his blog:
1) First, Pachter’s claim that the high frequency of the 15-node graph of Fig. 3 is merely due to the presence of a large clique and independent set is mathematically incorrect. It is due to the presence of a vertex v that is connected to every node in a large independent set, such that none of the nodes in the independent set are connected to any of the other 14 nodes in the graph on the right of the figure (and similar descriptions for other nodes in the graph). Even in a random graph model that was somehow rigged to have large cliques and large independent sets, it would be highly unlikely to have many independent sets with that property.
2) Second, Pachter states that we define motifs relative to subgraph frequency in an Erdös-Renyi model, which is factually incorrect, and a key to our next point (3) below. Erdös-Renyi random graphs exhibit with very high probability a Poisson degree distribution, which is very different from the long-tailed degree distribution observed in most real-world networks, including those examined in our paper. For that reason, we used a standard Markov chain Monte Carlo edge-swapping technique in our null model, similar to nearly all network motif papers. In order to preserve not just the degree distribution but also the distribution of 3-node motifs, we combined the MCMC edge-swapping with a simulated annealing technique.
3) Pachter’s criticism hinges on the assumption that both 9-node cliques and 12-node independent sets are over-represented in the real network relative to our control network. However, this is incorrect. Although cliques would be over-represented in our network relative to an Erdös-Renyi control, the data show that cliques are neither over- nor under-represented in the real network relative to the control we actually use.
Thus, Pachter’s specific criticism about Fig. 3, on which hinges his criticism of combinatorial effects, is incorrect.
More generally, not all large motifs arise from combinatorial clustering effects, and our clustering score helps distinguish large motifs that arise in a combinatorial manner from those that do not. Biologically, the discovery of combinatorial effects for some of the large network motifs has taught us something new about the nature of some large network motifs and the organization of large networks. Mathematically, the presence of combinatorial effects doesn’t mean our algorithm is flawed. On the contrary, it shows that the definition of network motifs needs to incorporate clustering properties, which was directly addressed in our paper, both in the original submission and in the published form.
These properties, which in retrospect make a lot of sense, had not been previously reported, and were to our knowledge first recognized as a direct result of our work, as no other algorithms had managed to recognize large network motifs and identify all their instances.

Criticism 3 (technical): Pachter claims our algorithm is essentially a heuristic for finding cliques and stable sets.
Pachter claims that “the Grochow-Kellis method is essentially a heuristic for finding cliques and stable sets in graphs.” This is a gross misrepresentation of our algorithm, both at a theoretical level and at the level of specific examples:
1) Pachter claims to have inferred this claim based on his interpretation of the example of Fig. 3, which we refuted in the previous section. Moreover, even if combinatorial effects accounted for the motif enrichment in that figure, it is logically incorrect to infer from this one example that our algorithm merely finds cliques and stable sets.
2) As Pachter himself points out, the combinatorial effects present in Fig. 3 would also be present not just near cliques or independent sets, but near any subgraph H that itself contained many isomorphic subgraphs, which our algorithm would also find. There are many possibilities for such an H: notably any vertex-transitive graph, or any graph that differs from a vertex-transitive graph in only a few edges. The class of vertex-transitive graphs is quite large (it includes all Cayley graphs of groups), and the presence or absence of such graphs in real-world networks would beg for a theoretical explanation.
3) A different example, in Figure 3-6 from Joshua Grochow’s thesis (from August 10, 2006, a month before we even submitted to RECOMB), is an anti-motif of 10-nodes with 95,754 instances covering 472 nodes, that is clearly not due to the presence of cliques or independent sets. Although the instances of this 10-node graph are heavily clustered, they are much less clustered than the example in Fig. 3. Furthermore, as an anti-motif, it appears less frequently in the real network than in the null model, and hence its clustering and significance in the real network cannot be due merely to the presence of cliques or independent sets in the real network compared to the null model.

Criticism 4 (review process): Pachter claims that we got the idea for clustering and the explanation of 27,720 from his review.
Pachter spends three paragraphs explaining how he “took many courses on enumerative and algebraic combinatorics” and “there are some numbers that combinatorialists just know”. He continues: “When I saw the number 27,720 I had a hunch”, but gives us credit that “the idea may have entered my mind because of the picture of the motif”. And then concludes, four paragraphs later that “Grochow and Kellis […] added a section titled ‘The role of combinatorial effects”, in which they explained the origins of the number 27,720 that they gleaned from my report”.
Even if we had gotten these ideas from his report, it is expected that the authors follow the reviewer’s recommendations when revising their paper; although sometimes this is acknowledged by thanking an anonymous reviewer, this is not universally practiced and in some fields even discouraged. But in this case, we didn’t get these ideas from the referee report:
1) First, we had explicitly drawn the motif in a way that reveals the combinatorial nature of the cliques, which Pachter semi-acknowledges, saying “the idea may have entered my mind because of the picture of the motif”.
2) Second, what Pachter claims was ‘gleaned’ from his report in the revised version of our paper was in fact included in Joshua Grochow’s thesis which was submitted on August 10, 2006 to MIT, 1 month before we even submitted the paper to RECOMB and 4 months before we received Pachter’s review. Months before Pachter’s review of our paper, Josh’s thesis wrote: “the frequency of the motif is due to choosing 3 nodes from the large independent set on the left and four from the large clique at the bottom of the figure,” and includes the exact same equation “(12 choose 3)(9 choose 4)=27,720” (Figure 3-5 legend on page 50), while the number or the equation didn’t even appear in Pachter’s critique.
In any case, it should be clear to the reader that we knew of these ideas prior to receiving any reviews. Moreover, the suggestions that somehow we had not recognized the combinatorial nature of these examples, that it was news to us, that we had to somehow deal with them, and that his review was somehow a revelation to us are clearly refuted by the discussion in Section 3.4.1 of Joshua Grochow’s thesis, finalized and published months before we received any reviews and one month before we even submitted our paper to RECOMB.
As for the more general recognition that large network motifs cluster due to combinatorial effects, that very property would have remained unrecognized without our paper. The role of clustering only became apparent thanks to our algorithm, which could for the first time find all instances of such large motifs. Moreover, we had already developed and applied a motif clustering score to specifically address that point ever since our original submission. In responding to the reviews, we expanded our discussion of these ideas and included the section titled “The role of combinatorial effects”, which was directly responsive to the critiques, as is expected for RECOMB and any other scientific conference or journal.

Criticism 5 (technical): Pachter claims our algorithm is incorrect and doesn’t achieve its stated goal.
The stated goal of our algorithm is to find all instances of a query graph H in a graph G, and our algorithm correctly achieves this stated goal, as a simple proof of correctness shows. This can then be used to evaluate the statistical significance of the frequency of occurrences of H in G relative to various null models, according to the standard definition of network motifs. We applied this to discover network motifs, but many other applications are enabled by our algorithm, such as the study of motif clustering, which we demonstrated in our paper.
For small graphs H up to 8 nodes, our algorithm can also be used to exhaustively evaluate all subgraphs of that size and their significance (this is larger than anything that had been achieved ever before). For larger sizes, we do not claim that we can find all motifs (the sheer number of possible subgraphs simply prohibits this), nor do we claim to find the most significant ones (given the enormous search space, even our sampling approach is not guaranteed to encounter all over-represented subgraphs).
Pachter claims that our algorithm is incorrect because it found a 15-node motif but not a more significant 18-node motif which can be formed combinatorially from the same nodes that make up our 15-node motif. This criticism is analogous to blaming the tallest building in the world for failing to add a few floors. Before our work, no known algorithm could even discover motifs of size 9, let alone 15, 20, or more. We surpassed the state of the art, introduced new methods and key ideas (reversing the search, symmetry breaking) that enabled multiple new algorithms since our work, but we did not claim any guarantees about finding the largest motifs, nor the most significant motifs. We can exhaustively search all 8-node motifs, but we use sampling to find larger motifs and anti-motifs, and thus it should be clear that we do not guarantee discovery of all larger motifs.

Criticism 6 (subjective): Pachter claims our solutions are uninteresting and that we should retract our work.
Pachter compares our algorithm to the infamous arsenic paper and states “When you discover your work is flawed, the correct response is to retract it”. However, despite his technical criticisms, our work is neither flawed nor trivial, as we discussed above. Moreover, in contrast to the community uproar caused by the arsenic-based-life-form paper, our algorithm has stood the test of time, has been replicated and extended, and forms the foundation of much of the subsequent work on discovery of large network motifs.
Since the conference was 7 years ago, we can look back at how influential each of the papers has been, using a simple metric of citations (we recognize this is an imperfect metric, but it is usually an okay gauge at a high level). From all 38 papers published in RECOMB 2007, the typical paper has 14 citations, while ours has 75 citations, and is the 2nd most cited paper of the conference (after the work of Bonnie Berger’s lab who received the RECOMB test-of-time award in 2010). This indicates significant interest in our work that continues to this date (the latest citation was ironically on February 14, two days after Pachter’s blog post), and a great impact on the community, which would have been lost if the positive reviews of our paper had not outweighed the negative score given by Pachter.

In conclusion, we have demonstrated that, in contrast to Pachter’s criticisms, our paper is correct, non-trivial, impactful, and has stood the test of time.

February 17, 2014 at 1:35 am

Lior Pachter

From your rebuttal, I have learned that I was wrong when I stated that the material in the section titled “the role of combinatorial effects” was derived from my review. I am sorry for that. It simply did not occur to me that you could have written the paper I reviewed if you had thought about these issues before. Your own results clearly demonstrate that at least when they have a large number of nodes, the network motifs are meaningless. You yourselves acknowledge this fact when, in that section, you say, “The magnitude of this combinatorial clustering effect brings into question the current definition of network motif, when applied to larger structures.”

I see now that these issues were discussed in detail in Grochow’s thesis. How could they not have been mentioned in your RECOMB submission? How could this not have been the subject of your submission?

February 17, 2014 at 6:12 am

Postdoc

Dear Lior,

Let me get this straight. Seven years ago, you rejected a conference paper of a master student. The paper ends up being accepted because the other reviewers give positive scores. The fact that your peers (the other reviewers and the conference chair) disagree with you bothers you so much, that you:
(1) boycott the entire conference from that day on
(2) attack the paper vigorously in your blog 7 years later, insisting that it should be retracted
(3) call into question the student’s thesis award, without even having read the thesis

And all of this despite the *fact* that the paper has clearly stood the test of time.

Lior, you have zero respect for your peers (reviewers, conference chairs, thesis committee members) and for the students who do the work. Even if that paper was flawed (it is not), that would not justify your behavior.

Disclosure: I’m a former postdoc of Manolis group. I post this comment anonymously because I fear that my own work may become the target of personally motivated attacks and public ridicule on Lior’s blog. I think that’s very sad, but the truth. This blog is dividing our community instead of bringing us together, we are *fighting* over papers instead of *working together* to understand which methods work best.

February 17, 2014 at 8:49 am

Lior Pachter

Dear former postdoc of Manolis’ group.

I appreciate your comment, and would like to clarify some things in response. It was not the fact that my peers disagreed with me that upset me. It was that the paper was accepted despite my critique pointing out that the main point of the paper, namely the ability to find larger motifs, revealed a fundamental combinatorial artifact that renders it meaningless to look for larger motifs, a fact that Grochow and Kellis themselves acknowledged in their revision. There was an argument about this point among the reviewers before the paper was accepted, and I felt that the final decision to accept the paper was not based on its scientific merits. It is true that this motivated my boycott of the conference; the fact that I’ve maintained that boycott for seven years also has to do with many other problems with the conference.

Regarding your point (3), I admitted in my post that I was wrong to have assumed that Grochow and Kellis learned about the combinatorial artifact of their method from me. But that does not mean I did not read Grochow’s thesis. I did look at it before blogging about it, but unfortunately failed to perform a careful diff against the submission. I was fooled. I just assumed, as I had done with Supplementary Figure S4, honesty on the part of the authors. I will obviously never make that mistake again. As for the paper “standing the test of time”, the fact that it has been cited over the past few years unfortunately proves nothing other than that the field is in a sorry state. For example, in a previous blog post, I critiqued the paper “Controllability of complex networks” by Liu et al. That paper has 424 citations but as Carl Bergstrom pointed out in his PloS One paper, the result is nonsense. Amazingly, the first author Liu et al. contacted me just this week in response to a notification of the blog post I sent him at the end of September. He wrote, and I quote:

“Dear Professor Pachter

Thanks for your email. And sorry for not getting back to you reasonably fast. I have been working on a paper related to the impact of intrinsic nodal dynamics on network controllability. And I am wondering if you are interested in reading it, which might help us resolve the embarrassing Oops issue, at least to some extent.

Thanks.

Cheers

yang”

I greatly commend Yang Liu for coming clean, so to speak, an act that requires true courage, and I wish that some of the authors of the Kellis papers that I’ve critiqued would do the same. The point is that “working together to understand which methods work best” is exactly the wrong thing to do in this case. The right thing is to ask what the methods should be looking for in the first place. The irony of the Grochow and Kellis paper is that I suspect that if it had been written honestly, namely if its point had been that their method, which enabled searching for larger motifs than previous methods, had discovered meaningless motifs due to the fact that a combinatorial artifact appears when searching for large ones, then it may have received many more citations and truly persisted in the long run.

Regarding your final point: if one is an honest person doing honest work why fear my blog? In fact, I know for a fact, and firsthand in some cases, that most of Manolis’ current and former students and postdocs are good, hardworking, honest scientists. Yes, my blog is personal, and that is because science is a human activity. Manolis’ actions hurt real people. On the matter of public mockery, I think people should judge for themselves if that is what I am really doing here.

February 17, 2014 at 11:18 am

Claudiu Bandea

Again, how can we resist the call for ‘civility’ and ‘working together’ by Postdoc?

One way would be to recognize that, by encouraging or implementing a policy of ‘civility’ and ‘working together’ (as in most peer-reviewed publications), the conventional system has allowed (encouraged?) some people to flourish by abusing its weaknesses and loopholes, without the *fear* of open and explicit exposure.

Indeed, as relevant as this exposure [whether associated with individual effort (e.g. Lior, Graur et al, Samanta) or with dedicated platforms (e.g. Retraction Watch, PubPeer, PubMed Commons)] might be in correcting existing flaws, abuses, and fraud, its true power and value is in preventing future cases: very few people, if any, would be ‘robbing banks’ knowing that their moves are fully monitored and exposed.

February 17, 2014 at 3:17 am

homolog.us

“Before our work, no known algorithm could even discover motifs of size 9, let alone 15, 20, or more.”

Looks like our ‘unknown algorithm’ did that, and in 2003.

Why did we not apply it to yeast protein-interaction network like you? That is another story, to be revealed in our blog soon.

http://www.worldscientific.com/doi/abs/10.1142/S0219720004000466

cWINNOWER ALGORITHM FOR FINDING FUZZY DNA MOTIFS
An extended abstract of this paper has been published in CSB’03, Liang, S. “cWINNOWER algorithm for finding fuzzy DNA motifs”, Proceedings of the 2003 IEEE Computational Systems Bioinformatics Conference (CSB2003), (2003) 260–265.

The cWINNOWER algorithm detects fuzzy motifs in DNA sequences rich in proteinbinding signals. A signal is defined as any short nucleotide pattern having up to d mutations differing from a motif of length l. The algorithm finds such motifs if a clique consisting of a suffciently large number of mutated copies of the motif (i.e., the signals) is present in the DNA sequence. The cWINNOWER algorithm substantially improves the sensitivity of the winnower method of Pevzner and Sze by imposing a consensus constraint, enabling it to detect much weaker signals. We studied the minimum detectable clique size qc as a function of sequence length N for random sequences. We found that qc increases linearly with N for a fast version of the algorithm based on counting threemember sub-cliques. Imposing consensus constraints reduces qc by a factor of three in this case, which makes the algorithm dramatically more sensitive. Our most sensitive algorithm, which counts four-member sub-cliques, needs a minimum of only 13 signals to detect motifs in a sequence of length N=12,000 for (l,d)=(15,4).

February 17, 2014 at 4:29 pm

Anon

You are talking about sequence motifs. They are talking about *network* motifs – totally different. The difference is even discussed in Section 1.2 of their paper.

February 17, 2014 at 6:32 pm

homolog.us

Thanks Anon. I discussed that in the blog post linked below (http://www.homolog.us/blogs/blog/2014/02/17/network-nonsense-lior-pachter-blog-escalates/).

February 17, 2014 at 7:40 pm

Anon

This is in response to homolog.us’s 2/17 6:32pm response to my previous comment of 2/17 4:29pm, but the site wouldn’t let me nest responses deep enough.

I read your blog post and your paper (ahem – you explicitly say in your blog post that you didn’t bother to read Grochow-Kellis and decide for yourself, your just taking Lior’s word for it). Your paper was only finding cliques, which is fine, that’s all you needed for your application, and Grochow-Kellis may not be the best method for that. But the methods of your paper aren’t for finding general network motifs of size 9+ (or really of any size – you really are just finding (impressively) large cliques). Despite this blog post, it’s pretty clear if you read their paper for yourself that the Grochow-Kellis algorithm is much more than just a “fancy clique-finding method” as you say in your blog post. The two problems – say, finding cliques of size 50, and finding general network motifs of size 10 or 15 – are incomparable in that neither can be reduced to the other, so you can’t really compare the two to say which is “better.” The Grochow-Kellis algorithm may not be the best at finding cliques, but it was way better than the previous algorithms at finding general motifs. If you still can’t be bothered to read their paper, then at least read their response to Criticism 3 in the comments above.

February 17, 2014 at 9:58 pm

homolog.us

Dear Anon,

I do not have time to check the validity of your comment right now, but to make sure our readers get balanced view, I included your comment in my blog post. I hope you give me permission to do so.

http://www.homolog.us/blogs/blog/2014/02/17/network-nonsense-lior-pachter-blog-escalates/

February 18, 2014 at 6:52 am

Anon

@homolog.us: I not only give you permission, but I appreciate you posting my previous comment on your blog. It would be nice if you changed some of the references so that, as they appear on your blog, they are a bit clearer: where I said “Despite this blog post”, you could change that to “Despite [Lior’s] blog post” (or something like that) so it’s clear what I was referring to. And where I say “in the comments above” you could change that to “in the comments [on Lior’s post]”. Thanks!

February 17, 2014 at 11:02 am

homolog.us

‘Network Nonsense’ at Lior Pachter Blog Escalates

http://www.homolog.us/blogs/blog/2014/02/17/network-nonsense-lior-pachter-blog-escalates/

February 17, 2014 at 12:32 pm

homolog.us

What is the definition of being honest in science? I added a thought-provoking comment by Feynman in the above link –

“I would like to add something that’s not essential to the science, but something I kind of believe, which is that you should not fool the layman when you’re talking as a scientist. I am not trying to tell you what to do about cheating on your wife, or fooling your girlfriend, or something like that, when you’re not trying to be a scientist, but just trying to be an ordinary human being. We’ll leave those problems up to you and your rabbi. I’m talking about a specific, extra type of integrity that is not lying, but bending over backwards to show how you are maybe wrong, that you ought to have when acting as a scientist. And this is our responsibility as scientists, certainly to other scientists, and I think to laymen.

For example, I was a little surprised when I was talking to a friend who was going to go on the radio. He does work on cosmology and astronomy, and he wondered how he would explain what the applications of this work were. “Well,” I said, “there aren’t any.” He said, “Yes, but then we won’t get support for more research of this kind.” I think that’s kind of dishonest. If you’re representing yourself as a scientist, then you should explain to the layman what you’re doing–and if they don’t want to support you under those circumstances, then that’s their decision.”

February 20, 2014 at 9:36 pm

Anon

Lior,

I couldn’t agree more with your math (in this post, and the others). The mathematical observations you made were a pretty damning critique. But I feel like you have completely undermined the points you wanted to make by adding in so many personal jabs, insinuations and ad-hominem attacks. It is easy to dismiss this as a personal vendetta, and people are not forced to understand your mathematical arguments in order to dismiss what you say.

I can’t help but think, if for example for this paper, you had just stated it as: “You claim to have found one subgraph of 15 nodes that occurs 27720 times in a biological network, but you really only found one (not much larger) subgraph that happens to contain that one particular subgraph 27720 times by itself. This is not a network motif as much as a network curiosity, and I pointed this out to you in a review but you published it anyways by obscuring this fact”. But instead this simple point is lost in a word soup.

–A casual observer

February 21, 2014 at 8:44 am

Lior Pachter

Dear casual observer,

I appreciate the fact that you agree with my math. It would be helpful, I think, to focus on some of the implications of that math. For example, the implication of its use to deliberately mislead readers and journals? Not once.. but in three separate publications I have identified.

As for the accusations: if you are going to post (anonymously) on my blog claims that I have made “personal jabs, insinuations, and ad-hominem attacks”, then it is your responsibility to quote the precise words/passages where I do so. Reading carefully over my posts I cannot find any such attacks. Accusing someone of fraud I have, and that is a very serious matter, but I made clear (a) what I meant by that accusation and (b) the grounds for doing so. This is personal but not a jab. It is not an insinuation but direct, and it is not ad hominem but ad rem.

Regarding the specific matter of the subgraph occurring 27720 times, which appears to be a case where you disapprove of my writing style (rather than an example of an unfair attack), I’d just like to reiterate that the problem was not that I pointed out an artifact and the authors published it anyway. They not only did not obscure the fact, they spun it into a positive discovery in their revision. And furthermore, it turns out knew about it all along and omitted to disclose it in their submission. None of this can qualify as a personal jab, an insinuation or an ad-hominem attack. Its just the truth. I hope we can agree on that. Hopefully the truth still matters in science.

Sincerely,
Lior

June 18, 2014 at 11:26 pm

sandra b

*sigh*I love your blog. I love this post. I love that you are providing meaningful mathematical feedback as part of your reviews for papers. Often reviewers do not invest the time to give subtle, deep, carefully reasoned feedback to papers and I, for one, would have been really grateful to have received the feedback that you offered to Grochow and Kellis. I can’t see how more scientists wouldn’t **love*** this kind of exchange even about their own work. (I guess in my training I wasn’t made to be so fragile about taking criticism of my work.)

I don’t think anything you wrote was an attack ad hominem.
However, the authors’ reactions speaks volumes about their scientific integrity and personal character. There is little way to misinterpret that and that may be why it appears personal.

Now as to the “mystery” of how the paper got accepted to RECOMB and how the related thesis won a major award at MIT, I think we, as a scientific community, need to be more honest about the elephants in the room: politics and funding. (Are we scientists or salespeople?) We are sacrificing the quality of our work, work that could deeply benefit many, unless we really address these issues.

I wish you would continue to be involved in RECOMB and continue to be so vocal in your constructive criticism. Every society needs a gadfly and you are so good at it.

February 22, 2014 at 1:55 pm

Claudiu Bandea

In my previous comments (see above), I suggested that the traditional, closed peer-review system and the conventional ‘civility’ associated with the science enterprise, have allowed, if not encouraged, people to prosper by misrepresenting facts and overhyping their work (at the expense of science and their colleagues), without the *fear* of open and explicit exposure.

And, unfortunately, in order to be able to compete in such a corrupt environment, many of their peers had little choice but to lower their ethical and scientific standards and join this unproductive and reckless competition; and, by doing so, all of them (even the most reckless ones, who often display embarrassing CVs inflated with pompous titles and rewards) have become victims of the system.

So, the problem is primarily with the system, not with the people; we know that people have the potential of doing wonderful or despicable things, depending of the system in which they operate: just look at history.

Fortunately, the solution to this growing problem in science is relatively simple: establish a comprehensive and open peer-review system at all levels of scientific enterprise (including science funding and communication) that would allow and encourage full peer participation. As I previously mentioned, very few people, if any, will be robbing banks knowing that their actions are fully monitored. Also, very few people, who do not deserve it, will be rewarded, if the decisions about the rewards are made by the entire community of peers, not by a restricted group of people linked in a network of corrupted influence that is protected by the conventional ‘civility’ they promote.

Fortunately, the scientific and scholarly publishing is highly compatible with an open evaluation approach, as the whole reason for writing manuscripts is to have them read and evaluated by peers. To be specific, in the Pachter/Kellis case, my suggestion would be that they fully evaluate each other’s publications, as well those of their colleagues. And, although, this or other blogs are great venues for extended evaluations, I would suggest presenting them, at least in summary, on other platforms, such as PubMed Commons, which would increase their corrective and preventive power.

February 22, 2014 at 2:02 pm

Lior Pachter

I completely agree with your vision for open community peer review. I also intend to annotate the relevant papers on PubMed Commons and PubPeer as you and others have suggested- I simply have not gotten around to it yet, but plan to do so this week.

February 22, 2014 at 4:13 pm

Andy Jerkins

I agree with Caludiu’s assessment of the situation, but would offer an alternate explanation for why results and CVs are constantly hyped: the abysmal funding situation. Older investigators in my department often reminisce about how just having a semi-decent idea was enough to get funded by the NIH/NSF 20-30 years ago. Nowadays, incredible ideas backed by highly qualified investigators/loads of preliminary data are summarily rejected. Just check out how high your RO1 scores need to be to ensure funding (https://pbs.twimg.com/media/BgT7RBaCcAAT92V.jpg:large). And this graph only documents changes in the past four years!!!

This kind of situation leads to a horrific forcing function in which investigators constantly have to show high levels of productivity and “splashiness” of their research to simply stay in business. In this kind of environment, is it any wonder why people engage in this kind of behavior?

While I appreciate the solution laid out by Caludiu and many others, I’m not sure exactly how post-publication peer review will by itself solve this problem. The battleground will simply shift from published papers to the comments section at PubMed Commons and PubPeer. These forums are largely unmoderated and allow anonymous comments. What’s to prevent competitors from unleashing a barrage of negative comments (that may or may not have any merit) on your work? A common retort to this argument is that scientists are smart! Surely, they’ll be able to cut through the BS. But, given how easy it is to send out a tweet, write a blog post, or post a comment on these forums, do you not think that if post-publication peer review takes off, the number of comments will be an order of magnitude more than the number of papers being published? How will we keep up with this?

I don’t mean to belittle PPPR at all. I think it’s important. But given that the funding situation is unlikely to get better (and would it be unreasonable to say that the future only holds more cuts??), perhaps we should also have a serious discussion on limiting the number of Ph.Ds that are minted in this country. If we can’t ensure that there is adequate funding for all of them to start an independent research group, then we’re only increasing the incentive to inflate research accomplishments.

February 23, 2014 at 7:36 pm

Marnie Dunsmore

” perhaps we should also have a serious discussion on limiting the number of Ph.Ds that are minted in this country.”

It is true that the funding situation is terrible. That’s a problem across the board, not just in genetics or bioinformatics or anthropology. At the same time, there are clearly too many PhDs being minted.

In this case though, what is truly mysterious is that so many, who should know better, have remained silent or have even been complicit with shoddy work.

I’ve been watching developments in genetic anthropology since I inadvertently stumbled on Dienekes’ Anthropology blog back in 2010.

At first, I was just puzzled that there were so many papers published with obvious weaknesses such as over-reliance on two dimensional PCA plots or blind use of ADMIXTURE. Increasingly, I realized that there was a deeper problem.

It’s true that research dollars are lacking. However, it’s also clear that research dollars in genetics and genetic anthropology also are often poorly spent. There seems to be a tremendous lack of courage in the field.

In IC design (my field), there’s a common expression: “The silicon doesn’t give a damn what you think.” The silicon is the decider.

In genetics, the process of deciding seems to be much more difficult. There also appears to be a culture that values manners over technical precision and scientific honesty.

March 14, 2014 at 7:38 pm

Andy Jerkins

Hey Lior,

I’m working through your critique regarding the network motifs, but I was curious of whether this combinatorial problem would apply to the original network motif paper as well:

http://www.ncbi.nlm.nih.gov/pubmed/12399590

Thanks!

March 14, 2014 at 10:07 pm

Lior Pachter

There is a difference between the Milo et al. and the Grochow-Kellis paper. In the former, directed graphs are considered, and therefore directed motifs. While the same issue may arise, i.e. a large tournament (directed complete) graph may give rise to multiple copies of a small directed network motif, it will be much less of an issue than in the undirected setting of Grochow-Kellis. Having said that, there is an interesting Ramsey-theoretic literature on unavoidable subgraphs of tournaments, and it has implications for the Milo et al. paper. See, for example, http://onlinelibrary.wiley.com/doi/10.1002/(SICI)1097-0118(199608)22:4%3C335::AID-JGT7%3E3.0.CO;2-M/abstract

July 29, 2014 at 7:12 am

green coffee tone pills from complete nutrition

Excellent article. I will be dealing with a few of
these issues as well..

September 27, 2014 at 3:27 pm

Ito Calculus

I think the reason that network science rubbish papers exist is the continuous funding from DoD. Some program managers decided that they should create “Network Science”. Injection of cash motivated a lot of nonsense by money diggers

April 7, 2015 at 2:18 pm

CompBioGuy

In another twist to the saga surrounding the network deconvolution & network link prediction papers, Nature Biotech published the following today:

http://www.nature.com/nbt/journal/v33/n4/full/nbt0415-424.html

http://www.nature.com/nbt/journal/v33/n4/full/nbt.3185.html

http://www.nature.com/nbt/journal/v33/n4/full/nbt.3184.html

Nature Biotech also published an editorial that outlines how they are re-evaluating their policies regarding peer review of comp bio papers:

http://www.nature.com/nbt/journal/v33/n4/full/nbt.3202.html

I would like to thank Lior for taking the time to carefully go through the two papers and bring to our attention several issues regarding the studies. And although it is really unfortunate that his efforts are not formally recognized in any journal publication, my guess is that the Nature Biotech editorial is at least partly due to his vigilance/raising numerous concerns with the two papers.

June 19, 2015 at 12:00 pm

Steven Salzberg

I just noticed this – CompBioGuy, thanks for pointing out the “Corrigendum” (a fancy word for Erratum or Correction) by Manolis Kellis et al., and the entire paper by Bastiaens et al. showing that the Barabasi paper was a variant of a previously published method that Barabasi apparently didn’t even know about. (In Barzel and Barabasi’s reply, they “regret that we did not cite the relevant literature, which we were unaware of at the time of publication.”

Alas, though, this just gives the authors of the two original, flawed papers two MORE papers in Nature Biotech – both of these responses will no doubt show up as new pubs on the authors’ CVs. This shows once again that there is little or no penalty for getting things wrong, as long as you get them published.

Kudos to Lior for getting the authors to report corrections, but anti-kudos to Nature Biotech for publishing the corrections in the format of papers.

June 19, 2015 at 2:02 pm

CompBioGuy

I agree. Corrections should definitely not be published as papers as they show up as new pubs on Pubmed and could show up as additional entries on someone’s CV (although I am not sure how ethical it is to list an error as a new paper).

Broader picture – I am impressed at how a single individual’s blogs are shaping post-publication peer-review, eliciting responses from many people (including a number of senior PIs who may not have been as responsive to other forms of communication), and forcing journals to change their policies.

May 26, 2015 at 10:00 am

Harry Hab

The point about exploiting the confusion is a germane one. Biologists don’t do maths and mathematicians are intellectually vain individuals often with very little knowledge of how nature actually operates. Try to do justice to both sides and both will hate you. Bamboozle both sides and you become a prominent mathematical biologist.

June 15, 2015 at 6:47 am

cbouyio

Reblogged this on CBouyio.

August 25, 2015 at 10:44 am

Travis Gibson

Network reconstruction is as hard as identifying all of the parameters in the equations that govern the dynamics or flow over that network (a very tall task). For instance, if the dynamics are $\dot x(t) = Ax(t)$ , where $A$ is the weighted adjacency matrix of a graph, inferring the unweighted adjacency matrix (determining the topology of the graph) is as hard as determining the weighted adjacency matrix. So there is this huge “equivalence class” of network properties for which obtaining simple information about the network is as hard as determining all information regarding that network (as difficult in terms of needing sufficient richness of the measured signals, i.e. richness in x(t) in our example).

This arXiv paper http://arxiv.org/abs/1508.03559 gets this correct and illustrates why the two nature biotech papers have so many issues.

March 9, 2018 at 3:49 pm

rishabh

Hi Lior, this is very interesting. What do you think about multitask learning for network inference? I found online a few examples: https://www.biorxiv.org/content/biorxiv/early/2018/03/08/279224.full.pdf

November 25, 2019 at 5:12 pm

Liang Fu

I landed on this post by accident, and what an interesting piece!
I can see that Kellis and his associates tried to make the water murky. I am not a biologist but fortunately the flaw can be explained with simple math that many people can understand: Kellis claimed that the number 27,720 has biological significance, but Pachter showed clearly and I understood perfectly it did not.
If you guys want to support Kellis please explain to the others the biological significance of 27,720 AS CLEAR AND AS CONVICING as Pachter did the opposite.

	Peter Ellis (@pjie2) on A decade of molecular cell…
	Lior Pachter on Ting Guo, sexual assault alleg…
	Schlafly on Ting Guo, sexual assault alleg…
	Audrey Fu on Ting Guo, sexual assault alleg…
	BigBacaloa on I was wrong (part 3)
	≫ Caso de acoso sexu… on Sexual harassment case number…
	Mitchell on A decade of molecular cell…
	petrick zag on Estimating number of transcrip…
	let's find out on The perfect human is Puerto…
	A Dunbar on James Watson in his own w…

Why I read the network nonsense papers

Recent Comments

Top Posts & Pages

Recent posts

Archives

Biology

Computational Biology

Computer Science

Ideas

Math

Medicine

Statistics

Blog Stats

78 comments

Leave a Reply Cancel reply

Why I read the network nonsense papers

Share this:

Like this:

Related

Recent Comments

Top Posts & Pages

Recent posts

Archives

Biology

Computational Biology

Computer Science

Ideas

Math

Medicine

Statistics

Blog Stats

78 comments

Leave a Reply Cancel reply