This Wired article talks about the NetFlix data mining competition. They have offered a $1 million prize to whomever can beat their movie-recommendation algorithm by 10%. That seems like it should be completely feasible. Yet complex math hasn’t yet been the solution. They have reached 8+% improvements, but so far nobody has cracked that 10% barrier and claimed the prize.

Common sense tells me it should not be so hard to attain such an improvement, but of course that’s easy for me to say…

A “psychologist” has done quite well so far by taking into account “human factors” in addition to math. The story is dramatized for effect, but it illustrates the value in thinking about the context when trying to solve difficult quantitative problems. Otherwise, you may be shooting too much in the dark, and the high dimensionality of the data gets in the way. No matter how complex your model, pure math won’t always cut it; and as the author suggests, you may be prone to overfitting the model.

There are parallels in mining genetic data sets.  There have to be betters ways to look at these data sets and take the biological context into account. Everyone is excited about pathway analysis, and I can see some logic in that. But I think that’s only the beginning. I’d tell you my other ideas, but that might give away my strategic advantage in getting published in Nature or Science.

I read a recent opinion piece in the New York Times from a researcher at the Sloan-Kettering Cancer Center in New York. He states that researchers around the world are trying to find better ways to prevent and treat cancer yet are often not willing to work together or share their data. He points out that the patients who authorized this data to be collected gave it freely, so the data should be available publicly for validation and additional research. He suggests this indicates many researchers care more about their own resume than about the public interest.

There is a fine line…researchers need some motivation to collect data, process it, and analyze it. Publishing is a great motivation because it can mean more funding, prestige, and an avenue to obtain feedback. If they had to give up their data immediately, they would have to compete with other researchers who hadn’t made these efforts. On the other hand, as the author points out, the data should not be owned completely by the researchers. Sharing data should advance the cause of science.

For many types of Informatics research, authors are required to publish their data in publicly available repositories as their papers are published. These create a happy medium…findings can be published and recognition rewarded, yet the data are made available for others to validate their findings or do secondary research.

This topic is important to me because I will rely on such data repositories to do my PhD research without having to secure funding and find study subjects, etc. I am getting “recycled” data that have been used by others; but the beauty is that I think I have some new and interesting ways to look at the data that were not considered by the original authors.

GEO and CGEMS are examples of such data-sharing repositories. caBIG is an infrastructure being developed to help cancer researchers share data (my PhD advisor is involved in this effort).

Andy Grove, former CEO of Intel and fabled leader, trashed the biomedical research community in the US, stating that the current system discourages innovation. He has some interesting points.

It seems he’s simplifying quite a bit. I’m sure there are problems in the system and that innovation is reduced by near-sighted focusing on smaller ideas that can get funded rather than bigger questions that can win Nobel Prizes. However, I think those smaller pieces will continue coming together to address big problems over the coming years, despite some inefficiencies in the system (which should be addressed).

In a way, this reminds me of the Bill and Melinda Gates Foundation that aims to run their non-profit like a business to maintain high efficiency and wise use of resources. Maybe Grove can use his assets and leadership to show that something similar can be accomplished in biomedical research.

http://www.newsweek.com/id/68221/page/2