Q: There are some relationships in the OMOP_TRUE_RELATIONSHIPS.txt file that occur rarely (if ever) in the 10 million patients. How are any methods supposed to detect this?

A: There will be some relationships in the data that are rare and/or weak enough that they will not show up even in a 10 million patient sample. Therefore it will likely not be possible for any method to achieve the highest possible score. However, we feel that this situation exists in real clinical data, and also gives us an indication beyond the OMOP Cup of the number of patients required to detect rare adverse events.

Joran, there is no guarantee that all of the drugs and conditions are used in the 10 million patient, 10 year sample. Since the list of drugs and conditions is generated before the patients, some rare drugs and conditions will never appear.

Can I assume Challenge 2 Year 10 is an exact copy of Challenge 1 data? This will save me downloading 3.7GB.

Robin: It is almost identical. The only difference is that in Challenge 2 Year 10 the Drug Exposure table is truncated so that prescriptions that would terminate in 2010 are changed to December 31, 2009. This was done for consistency with the other yearly data. You could make this change to the Challenge 1 data if you like. Depending on your model, this change may or may not make a difference.

Hans: I’m sorry if we were unclear. You can submit any real number as a predicted value in both Challenge 1 and Challenge 2. Reporting the rank of each prediction would be fine too, since that would be a special case. We had used binary predictions and 0-9 predictions in the examples for the sake of simplification, but it is not a limitation.

Joran: Yes, the benchmark submissions are submitted by the contest organizers. They would not be eligible to win the competition. OMOP is working on methods in parallel with the competition, and we will post some of those here. Random refers to a submission where every drug/condition pair is assigned a random value between 0 and 1. PRR stands for Proportional Reporting Ratio, a member of a class of methods called Disproportionality Analysis. I will be posting a few more benchmarks as well as more thorough explanations in the coming week, but you can read more about these methods at http://omop.fnih.org/methodslibrary

Steve: Thank you for discovering this issue with the true relationships file. There was typo when that file was produced that allowed the condition id to range up to 4619 instead of 4519. This resulted in 80 drug/condition pairs (associations are absent for all of them) in that file where the condition id is greater than 4519. Please ignore these entries in the file. I’ve uploaded a new version without these 80 lines. There was a similar typo in the competition rules document that has also been fixed.

Timothy: Submissions should show up on the leaderboard within an hour of submission. In your case, I have a record of a submission last night, but there is no file attached. Could you please try resubmitting the file, perhaps using a different browser. If it doesn’t work again, please send me an email at omopcup@gmail.com with OS/Browser info and any error messages, and we’ll try to resolve the problem.

Robin: That is one way of viewing it. Specifically, it says that the patient had 2 overlapping exposure periods of the same drug. This is something that happens in real medical data, though some of the particulars might be specific to this simulated data. The simulated data also does not have ‘dose’ information per se (additional fields in the CDM which are not used), so it is up to you how to use this information.

Apparently the time is reported as Greenwich Mean Time.

Robin: Yes, the website is running under GMT, but the time is converted to Eastern time when it is reported on the leaderboard.

And I would recommend removing old submissions from the upload list, to ensure the correct one gets scored.

Lisa: I’ll check on that.

Hi – I’m looking for clarification on how you define “a relationship exists” in the answer key. Does a relationship consist of at least one case of a drug-condition association, or is there some threshold that has to be crossed?

Hi,
Unless I misunderstood the rules for Challenge 2, I believe there is possibly a design flaw in the Challenge. There is nothing that stops us from using year 10’s data to predict year 1-9 (and get a better score? yes.. back to the future :) ). If you don’t want this to happen then you should state this clearly in the rules, although I don’t know how you can enforce it during the competition…

Harris: This was an issue we anticipated for Challenge 2. The only way to avoid this would have been to generate 10 additional independent 10 million patient histories, which was not computationally feasible to complete by the time the competition went live. This does make it difficult for us to detect if someone is just using the whole dataset for all timesteps (though scores that don’t change from one timestep to another are a red flag).

This problem is why we require potential winners to submit code to us so that we can verify that the score was produced by code that only utilized the correct subset of data. It is mentioned in the rules at the end of section 7: “For Challenge 2, Sponsor must be able to determine that the yearly results datasets were produced using only the appropriate yearly input file, without use of data from later time-points.” However, we were in error in not making that much more prominent. I will add it to the challenge 2 description file and the challenge 2 submission page.

We are aware that using challenge 1 results to game challenge 2 is possible, and that we can’t know for sure that a submission for challenge 2 is legal until the end of the competition. But we will verify that only the proper data was used before awarding any prizes. This uncertainty in the scoring is not ideal for us or the participants, but we decided it was the best option at the time.

I apologize for the confusion. If you made a submission that you think may not be eligible for the challenge 2 prize, you can send me an email at omopcup@gmail.com and I will remove it from the leaderboard.

Ed: To your question about the score changing, that does look worrisome. I will re-score all of the submissions and try to figure out what happened.

Eric: while we are at this, are we allowed to use OMOP_TRUE_RELATIONSHIPS.txt in producing our results?

Martijn, thanks for the confirmation – for a while I thought I may have inadvertently swapped some filenames. – ER

Looks like the scoring oracle does consider the relations in the ‘true relations’ file when it scores. By rolling the true relations file into a previous entry I boosted the score from 0.21590 to 0.22775, an increase of about 5%!.

Whether or not to consider these relations poses an interesting problem. These priors are a legitimate test for statistical techniques that don’t depend on training, but possibly not for NN-based techniques that use them as part of a training set. In either case, using the known relations to artificially boost one’s score contributes nothing to the development of better algorithms, and really isn’t in the spirit of the contest.

That being said, now that it is clear that the scoring oracle does base its score on the known relations, the public knowledge of that information makes the playing field a bit more level :)

ER

Rob, Ed, and Martijn: Thanks for uncovering these issues. We’re working right now to fix both of these problems with the scoring system. Expect a more thorough resolution tomorrow.

Harris: You are allowed, though not required, to use OMOP_TRUE_RELATIONSHIPS.txt in your results for both challenges.

Harris, I may have been a bit unclear in trying to make my point. I would certainly agree with you that the known relations are valuable – whether for evaluating statistical techniques or as training sets for machine learning methods.

The problem arises when it becomes possible to use them to game the system. The rules implied that these points would not be considered in determining scores. The fact that they currently are considered makes it possible for a contestant to achieve a signficant advantage not through better algorithms, but by taking advantage of this (formerly) not-generally known bug. If there were huge differences in scores near the top of this list, this might not be a big deal. From what I have seen of data-mining, competitions, however, near the end it comes down to a few people slugging it out over fractions of a percentage point. This particular bug provides about a 5% increase – pretty enormous – think a 5 meter headstart in a 100 meter dash :)

The important point isn’t necessarily eliminating all the idiosyncracies of the scoring system – it is sure to always have some – but in making sure everyone knows how they are being evaluated so everyone is really playing the same game.

ER

Am having trouble with corrupted data files, could someone confirm these counts, or correct them:
Range of person_ids: 1- 10,000,000
Number of persons with drug exposures: 9,562,432
Number of drug exposures: 92,803,110
Number of condition occurrences: 316,686,137
Thanks .. :-)

Thank you, Christophe. Much appreciated.

Here is another oddity. Can you (or someone) confirm or correct it?

There are 7,921 OMOP TRUE RELATIONSHIPS. But, for 188 of these relationships, either the drug or the condition or both are not present in the DRUG and CONDITION data files.

This prompts the question: Is the “M = DC” in the scoring formula based on all possible D (5000) and all possible C (4519), or on the observed D (4775) and observed C (4494 ) ?

Thank you, Eric.
Then, since the divisor is 5000*4519, inspection of the Announcements so far suggests that there are 0.0159207 * 5000 * 4519 = 360,000 drug-condition causations to be found.

In light of some issues discussed in this thread, we’ve updated the scoring algorithm for the competition. See the Feb 19 entry on the Announcements page for more info. Feel free to email me with any questions related to the change. Thanks to those of you that helped us find the issues.

Oops! Didn’t read far enough …
double prob = 0.0+Double.parseDouble(tokens[tokens.length-1]);

All is OK!

I have two general questions for the score system. First, will the demographic data affect the correlation between a certain drug D and a certain condition C? For example, C and D are correlated for male but not for female. If it is true, will your score system treat C and D correlated or not? Second, it seems the correlation between drug and condition could be strong, moderate, or small in your simulation procedure. If it is true, then your score system treats them the same. It seems the way you evaluate is discouraging the method which can rank the strong correlation higher. It is just a thought unrelated to the competition itself because everyone follows the same rule. But I think it is an important issue for the real problem itself.