We'll be starting to add a hyperopt/automl feature to ramp-workflow. The goal is to make it easy to convert a submission into a template with hyperparameter placeholders, and to add a hyper-config file that defines the values to try for each hyperparameter. Then rampwf will interface with various hyperopt engines (implementations of grid search, random, bayesopt), run the optimization, and output 1) a table on score(s) vs hyperparameter value combinations and 2) a valid submission where the placeholders will be replaced by the best hyperparameter. We are planning to use jinja, which means that, for example, the python code

class Classifier(BaseEstimator):
    def __init__(self):
        self.clf = Pipeline([
            ('imputer', Imputer(strategy='median')),
            ('classifier', LogisticRegression(C=1.0))
        ])

will be replaced by

class Classifier(BaseEstimator):
    def __init__(self):
        self.clf = Pipeline([
            ('imputer', Imputer(strategy='{{ impute_strategy }}')),
            ('classifier', LogisticRegression(C={{ logistic_C }}))
        ])

and the config json file will specify values [median, mean] for strategy and [0.001, 0.01, 0.1, 0.5, 0.9, 0.99, 1.0] for logistic_C. The user will then call e.g. ramp_hyperopt --submission ... --strategy random. In addition, for each placeholder we will specify a default, so if ramp_test --submission is called on a templetized submission, it will use these default values.