This file contains a description of how to run the scoring scripts for
the NCI-DREAM sub-challenge 1.

The scripts have been created by Jim Costello for the NCI-DREAM
challenge.  Please contact me at jccostel@bu.edu if there are any
questions.

The data has been generated by the lab of Joe Gray.  Laura Heiser is
the contact person for any additional questions regarding the genomics
datasets or drug sensitivity heiserl@ohsu.edu

#####################################################################
##############           Scoring Scripts             ################
#####################################################################

Running the scoring scripts requires 3 steps.  You must first create
the set of random predictions.  These random predictions will be used
to calculate the significance of the predictions being tested. Second,
you will create an input file for the scoring script that calculates
the z-score of the gold standard data to the null distribution.  These
values are used for the weighted scoring of the concordanceindex.
Third, you can score your predictions that assume the format specified
on the NCI-DREAM website.  An example template file can be found at
files/dream7_drugsensitivity1_predictions_test.csv

The NCI-DREAM challenge website can be found at:
http://www.the-dream-project.org/challenges/nci-dream-drug-sensitivity-prediction-challenge

####################################################################
~~ Step 1

run the following:
> perl create_random_predictions.pl

This step needs to be run only once.  Rerunning the script will create
a new set of random predictions.


~~ Step 2

run the following:
> perl calculate_zscore_weights.pl

This step needs to be run only once.  It will create the an input file
for the final script, "files/DREAM7_DrugSensitivity1_drug_zscores.txt"
Note that you will overwrite the current version of this file which is
the one used to score the dream challenge.

~~ Step 3

The scoring script has one perl module dependency which needs to be
installed, namely Games::Go::Erf.  This perl module can be installed
using command line cpan or it can be downloaded at:
http://search.cpan.org/~reid/Games-Go-GoPair-1.001/

run the following:
> perl weighted_concordance_index.pl  files/dream7_drugsensitivity1_predictions_test.csv

The output of the script will be the score for each individual drug
using the probabilistic concordance index (c-index).  You will also
see the weighted probablistic c-index, where the weight is calculate
for each drug based on the statistical measure of the gold standard
test data compared to the empirical null distribution of ranodmly
generated predictions.  Finally, you will see the overall score wich
is the weighted average of all 31 drugs along with the significance
compared to the empirical null distribution of randomly generated
predictions.   Further details of the scoring can be found at:
http://www.the-dream-project.org/result/nci-dream-drug-sensitivity-prediction-challenge



#####################################################################
##############               Data Files              ################
#####################################################################

./files/DREAM7_DrugSensitivity1_Drug_Response_Training.txt
Contains the training data supplied to all participants
at the begining of the challenge

./files/DREAM7_DrugSensitivity1_drug_zscores.txt
Contains the weighting information for each drug.  The
overall score is calculated as the weighted average of
the probabilistic c-index and the weights reflect the 
signal present for each drug.

./files/DREAM7_DrugSensitivity1_Pooled_SD.txt
Contains the pooled standared deviation values for each drug.
As there are variable number of replicates per drug and cell
line, the pooled standard deviation for each drug was calculated
to account for a "noisy" gold standard.

./files/dream7_drugsensitivity1_predictions_test.csv
To be used as an example test file for the script 
"weighted_average_concordance_index.pl"  The rankings are 
random.

./files/DREAM7_DrugSensitivity1_test_data.txt
Contains the 18 cell lines and GI50 values for the 31
tested drugs.  


./files/DREAM7_DrugSensitivity1_Train_Test_Ranked.csv
Contains the full set of 53 cell lines and 31 drugs ranked based
on their GI50 values.

