Competition Class
=================

After deploying a Model Playground, users can create a model competition. Creating a competition allows you to:

    1. Verify the Model Playground performance metrics on aimodelshare.org
    2. Submit models to a leaderboard
    3. Grant access to other users to submit models to the leaderboard
    4. Easily compare model performance and structure


.. _submit_model:

submit_model
------------

Submits model/preprocessor to machine learning competition using live prediction API url generated by AI Modelshare library. The submitted model gets evaluated and compared with all existing models and a leaderboard can be generated. 

.. py:function:: Competition.submit_model(model_filepath, preprocessor_filepath, prediction_submission, sample_data=None, reproducibility_env_filepath=None, custom_metadata=None)

   :param model_filepath: Value - Absolute path to model file [REQUIRED] to be set by the user. .onnx is the only accepted model file extension. "example_model.onnx" filename for file in directory. "/User/xyz/model/example_model.onnx" absolute path to model file from local directory.
   :type model_filepath: string -ends with '.onnx'
   :param preprocessor_filepath: value - absolute path to preprocessor file. [REQUIRED] to be set by the user. "./preprocessor.zip". Searches for an exported zip preprocessor file in the current directory. File is generated from preprocessor module using export_preprocessor function from the AI Modelshare library.
   :type preprocessor_filepath: string
   :param prediction_submission: Values - predictions for test data. [REQUIRED] for evaluation metrics of the submitted model.
   :type prediction_submission: One-hot encoded prediction data for classification. List of values for regression. 
   :param sample_data: 
   :type sample_data: 
   :param reproducibility_env_filepath: [OPTIONAL] to be set by the user- absolute path to environment environment json file. Example:  "./reproducibility.json". File is generated using export_reproducibility_env function from the AI Modelshare library
   :type reproducibility_env_filepath: string
   :param custom_metadata: Dictionary of custom metadata metrics (keys) and values for the model to be submitted. 
   :type custom_metadata: Dictionary

   :return: Model version if the model is submitted sucessfully.

Example: :: 

	#-- Generate predicted values (sklearn)
	prediction_labels = model.predict(preprocessor(X_test))
	
	#-- Generate predicted values (keras)
	prediction_column_index=model.predict(preprocessor(X_test)).argmax(axis=1)
	# Extract correct prediction labels 
	prediction_labels = [y_train.columns[i] for i in prediction_column_index]

	# Submit Model to Competition Leaderboard
	mycompetition.submit_model(model_filepath = "model.onnx",
                                 preprocessor_filepath="preprocessor.zip",
                                 prediction_submission=prediction_labels)

.. _instantiate_model:

instantiate_model
-----------------

Import a model previously submitted to the competition leaderboard to use in your session.

.. py:function:: Competition.instantiate_model(version=None, trained=False, reproduce=False)

   :param version: Model version number from competition leaderboard.
   :type version: integer
   :param trained: If True, a trained model is instantiated, if False, the untrained model is instantiated
   :type trained: bool, default=False
   :param reproduce: Set to True to instantiate a model with reproducibility environment setup
   :type reproduce: bool, default=False

   :return: Model chosen from leaderboard

Example: :: 

	# Instantiate Model 1 from the leaderboard, pre-trained
	mymodel = mycompetition.instantiate_model(version=1, trained=True, reproduce=False)

.. note::
    If ``reproduce = True``, an untrained model will be instantiated, regardless of the ``trained`` parameter value.

.. _inspect_model:

inspect_model
-------------

Examine structure of model submitted to a competition leaderboard.

.. py:function:: Competition.inspect_model(version=None, naming_convention=None)

   :param version: Model version number from competition leaderboard.
   :type version: integer
   :param naming_convention: Either "keras" or "pytorch" depending on which kinds of layer names should be displayed
   :type naming_convention: string - either "keras" or "pytorch"

   :return: inspect_pd : dictionary of model summary & metadata


.. _compare_models:

compare_models
--------------

Compare the structure of two or more models submitted to a competition leaderboard. Use in conjunction with stylize_compare to visualize data. 

.. py:function:: Competition.compare_models(version_list="None",  verbose=1, naming_convention=None)

   :param version_list: list of model version numbers to compare (previously submitted to competition leaderboard).
   :type version_list: list of integers
   :param verbose: Controls the verbosity: the higher, the more detail 
   :type verbose: integer
   :param naming_convention: Either "keras" or "pytorch" depending on which kinds of layer names should be displayed
   :type naming_convention: string - either "keras" or "pytorch"

   :return: data : dictionary of model comparison information.

Example :: 

	# Compare two or more models
	data=mycompetition.compare_models([7,8], verbose=1)
	mycompetition.stylize_compare(data)

.. _stylize_compare: 

stylize_compare
---------------

Stylizes data received from compare_models to highlight similarities & differences.

.. py:function:: Competition.stylize_compare(compare_dict, naming_convention=None)

   :param compare_dict: Model data from compare_models()
   :type compare_dict: dictionary
   :param naming_convention: Either "keras" or "pytorch" depending on which kinds of layer names should be displayed
   :type naming_convention: string - either "keras" or "pytorch"

   :return: Formatted table of model comparisons. 

Example :: 

	# Compare two or more models
	data=mycompetition.compare_models([7,8], verbose=1)
	mycompetition.stylize_compare(data)

.. _inspect_y_test: 

inspect_y_test
--------------

Examines structure of y-test data to hep users understand how to submit models to the competition leaderboard.

.. py:function:: Competition.inspect_y_test()

   :param none:
   
   :return: Dictionary of a competition's y-test metadata.

Example: :: 

	mycompetition.inspect_y_test()

.. _get_leaderboard:

get_leaderboard
---------------

Get current competition leaderboard to rank all submitted models. Use in conjunction with stylize_leaderboard to visualize data. 

.. py:function:: Competition.get_leaderboard(verbose=3, columns=None)

   :param verbose: (Optional) controls the verbosity: the higher, the more detail.
   :type verbose: integer
   :param columns: (Optional) List of specific column names to include in the leaderboard, all else will be excluded. Performance metrics will always be displayed.
   :type columns: list of strings

   :return: Dictionary of leaderboard data.

Example: :: 

	data = mycompetition.get_leaderboard()
	mycompetition.stylize_leaderboard(data)

.. _stylize_leaderboard: 

stylize_leaderboard
-------------------

Stylizes data received from get_leaderbord.

.. py:function:: Competition.stylize_leaderboard(leaderboard, naming_convention="keras"

   :param leaderboard: Data dictionary object returned from get_leaderboard
   :type leaderboard: dictionary

   :return: Formatted competition leaderboard

Example: :: 

	data = mycompetition.get_leaderboard()
	mycompetition.stylize_leaderboard(data)

.. _update_access_list:

update_access_list
------------------

Updates list of authenticated participants who can submit new models to a competition.

.. py:function:: Competition.update_access_list(email_list=[],update_type="Replace_list")

   :param email_list: [REQUIRED] list of comma separated emails for users who are allowed to submit models to competition.
   :type email_list: list of strings
   :param update_type:[REQUIRED] options: ``string``: 'Add', 'Remove','Replace_list','Get. Add appends user emails to original list, Remove deletes users from list, 'Replace_list' overwrites the original list with the new list provided, and Get returns the current list.    
   :type update_type: string


   :return: "Success" upon successful request

Example :: 

	# Add, remove, or completely update authorized participants for competition later
	emaillist=["newemailaddress@gmail.com"]
	mycompetition.update_access_list(email_list=emaillist,update_type="Add")
