Hi, is this contest open to people outside US? Say New Zealand :-)

Eric –
Mike Linacre <orwik@winsteps.com>, was part of the team that was second in the Netlix Prize. He wants to know how to download the OMOP datasets. What is the link to download them?
Could you please send me this information? Borya has then asked me pass the info on to him and to invite him to both challenges. Thanks a lot. Sharmila

I had our engineer look up the file and it appears to be stored on amazon and that too has the 3920 0’s. Maybe something is up with the file, since both Orwik and amazon have the same number of 0’s. Can you send me the file directly and I will try to see if we can figure out the problem.

Mike: That is the correct number of lines. There is a comment in the FAQ discussion about this. The original file accidentally contained 80 lines with a condition id larger than 4519. These were removed from the file. I apologize for not updating the pdf accordingly.

Anand: No, this is not directly modeled in this version of the simulator.  The focus is on drug side effects, not drug-drug interactions.  There can certainly be times where the adverse event is more prevalent in the presence of a second drug, but not where two drugs are required for the event to occur.  In other words, I wouldn't go as far as to say they should be ignored, but they should not be reported.  The problem of drug-drug interactions is an important one, which didn't make it into this simulator for technical reasons but is planned for future simulators.  Considering all interactions makes the problem even more challenging, as now there are now D * D * C possible relationships (113 billion for the present simulation).  You can read more about the simulator at http://omop.fnih.org/osim.

Robin:  The BCPNN benchmarks were run from the Disproportionality Analysis package available at http://omop.fnih.org/methodslibrary, which you are free to compare to your method.  Another factor in some of the benchmarks that may not have been clearly mentioned is the use of drug and condition eras.  Eras are constructed from overlapping drug exposures or condition occurrences.  For example, a patient who received a 31 day dose of a drug the first of every month for a year would have a single drug era 1 year long.  Depending on the method, this may or may not make a difference, but it can be substantially different in disproportionality methods.  More details on eras can be found in the CDM specifications at http://omop.fnih.org/CDMandTerminologies.

I was member of the second-place team in the Netflix challenge ("The Ensemble"), along with with Mike Linacre (see his earlier posts here).  I haven't competed in any other contests besides OMOP.  Unfortunately, I didn't find out about OMOP until earlier this month.

Jeff Howbert

Jeff, Mike: 

I thought your names looked familiar. I also did Netflix, though way down the leaderboard.  Looks like data-mining is a small world :)

Mike - Not sure that reverse-engineering is the key here - more like disentangling confounding drug effects to suppress false positives. That would explain why a regression-based approach (BLR) seems to do pretty well - and I suspect that a lot of the players in that narrow band above the BLR benchmark are using some version of it.

I have been experimenting with modifications to the proportional-reporting methods - they are pretty simple to implement even if you don't have SAS (I am using VB.NET) and run quickly once you get the data into memory from disk. Since I don't seem to be in any danger of winning at this point, here is what I have been doing.

1) Pre-process data so that I can access patients one at a time and each patient has associated symptoms and drug prescriptions.

2) Run through a phase 'A'  where you count the number of days of patient days, drug treatments, symptom counts, and the count of where symptoms occur simultaneous with drugs (coincidences). I am not using any kind of extended windows on the drugs.

3) Determine a 'relative likelihood' that a drug causes a symptom, based on the actual coincidences (Na) vs expected (Ne).  L =(Na+0.5)/(Ne+0.5). The '0.5' factor is there so single events of low expected occurence don't become excessively dominant.

4) Run a phase 'B' where you run through each patient day (about 6 billion total) and tally symptom-drug coincidences. If a symptom associates with more than one drug, you can either tally it to the drug with the highest 'L' for that symptom (winner-take-all), or divide the count proportionally according to 'L'.  I have found that the two approaches, and a fair number of similar ad-hoc weighting rules yield substantially similar results.

5) Now that you have tallys, calculate actual (Pa) and expected (Pe) rates (probabilities) of symptoms. N = total # of days a drug was prescribed in whole dataset.  Pa=# of drug-symptom conincidences/N.  Pe = # of symptoms (over whole dataset)/# of patient days(whole dataset).

6) Now instead of just dividing Pa/Pe for a proportional metric, I calculated a 'pseudo-Z' metric (loosely) based on chi^2 for one degree of freedom.  Z = (Pa-Pe)*sqrt(N/Pe).  This can be roughly interpreted as the 'significance' of the relationship between the drug and the symptom, with positive values indicating a positive relation (drug causes symptom), negative values a negative relation (drug prevents symptom), and zero no relation.  Becuase values for this 'Z' range from ~ -20 to ~+4000 on the 'true relations dataset',  the extremes seem pretty meaningless in terms of traditional normal-distribution-Z. The metric does seem to be a useful way of ranking the relations, however, in that it can highlight a weak relation occuring in many cases over a strong one occuring in a very few cases - something the proportional methods don't seem set up to do.

ER

Thank you for your response, Eric.

Please don't take my comment as a negative. If the simulator matches the real world (as we hope), then reverse-engineering the simulator is also reverse-engineering the real world, something the physicists are trying to do right now at the Large Hadron Collider. So the scientific process becomes exactly what we are doing:

1. Construct a theory about the real world

2. Simulate data to match generating parameter values which accord with the theory

3. Construct a method which recovers the generating parameter values (the OMOP Cup)

4. Apply the method to real world data to obtain estimated parameter values

5. Compare the real world data with the data obtained by using the estimated parameter values as generators.

6. Use the results of the comparison to adjust the theory about the real world.

7. Loop back to 2.

I don't have access to SAS. Will it be possible for someone to post the 2-dimensional contingency table's used by the BCPNN-M benchmark method? It is ok to restrict the drug-condition pairs in the ground truth data.

Ed: That'll have to wait a few days.  The leaderboard is officially unofficial until we verify the results, which will happen soon.  I really want to thank everyone for participating, and for all your hard work.  I also hope the competition has raised interest in the types of problems that OMOP is looking at and we can get more great minds working on them. 

May I second Ed on that one! Thanks for getting this organized and seeing it through. I actually used the competition as the practical part of my data mining class this semester and, while we had some organizational issues of our own, this was a very good experience for all of us. Thanks again!

Christophe