Selecting basis by cross-validation with scikit-learn#

In this demo, we will demonstrate how to select an appropriate basis and its hyperparameters using cross-validation. In particular, we will learn:

What a scikit-learn pipeline is.
Why pipelines are useful.
How to combine NeMoS Basis and GLM objects in a pipeline.
How to select the number of bases and the basis type through cross-validation (or any other hyperparameter in the pipeline).
How to use a custom scoring metric to quantify the performance of each configuration.

What is a scikit-learn pipeline#

Pipeline illustration. — Schematic of a scikit-learn pipeline.

A pipeline is a sequence of data transformations leading up to a model. Each step before the final one transforms the input data into a different representation, and then the final model step fits, predicts, or scores based on the previous step’s output and some observations. Setting up such machinery can be simplified using the Pipeline class from scikit-learn.

To set up a scikit-learn Pipeline, ensure that:

Each intermediate step is a scikit-learn transformer object with a transform and/or fit_transform method.
The final step is an estimator object with a fit method, or a model with fit, predict, and score methods.

Each transformation step takes a 2D array X of shape (num_samples, num_original_features) as input and outputs another 2D array of shape (num_samples, num_transformed_features). The final step takes a pair (X, y), where X is as before, and y is a 1D array of shape (n_samples,) containing the observations to be modeled.

You can define a pipeline as follows:

from sklearn.pipeline import Pipeline

# Assume transformer_i/predictor is a transformer/model object
pipe = Pipeline(
    [
        ("label_1", transformer_1), 
        ("label_2", transformer_2),
        ...,
        ("label_n", transformer_n),
        ("label_model", model)
    ]
)

Note that you have to assign a label to each step of the pipeline.

Tip

Here we used a placeholder "label_i" for demonstration; you should choose a more descriptive name depending on the type of transformation step.

Calling pipe.fit(X, y) will perform the following computations:

# Chain of transformations
X1 = transformer_1.fit_transform(X)
X2 = transformer_2.fit_transform(X1)
# ...
Xn = transformer_n.fit_transform(Xn_1)

# Fit step
model.fit(Xn, y)

And the same holds for pipe.score and pipe.predict.

Why pipelines are useful#

Pipelines not only streamline and simplify your code but also offer several other advantages. The real power of pipelines becomes evident when combined with the scikit-learn model_selection module, which includes cross-validation and similar methods. This combination allows you to tune hyperparameters at each step of the pipeline in a straightforward manner.

In the following sections, we will showcase this approach with a concrete example: selecting the appropriate basis type and number of bases for a GLM regression in NeMoS.

Create a custom scorer#

By default, the GLM score method returns the model log-likelihood. If you want to try a different metric, such as the pseudo-R2, you can create a custom scorer and pass it to the cross-validation object:

from sklearn.metrics import make_scorer

pseudo_r2 = make_scorer(
    nmo.observation_models.PoissonObservations().pseudo_r2
)

We can now run the grid search providing the custom scorer

gridsearch = GridSearchCV(
    pipeline,
    param_grid=param_grid,
    cv=5,
    scoring=pseudo_r2,
)

# Run the 5-fold cross-validation grid search
gridsearch.fit(X, y)

GridSearchCV(cv=5,
             estimator=Pipeline(steps=[('transformerbasis',
                                        Transformer(RaisedCosineLinearEval(n_basis_funcs=6, width=2.0))),
                                       ('glm',
                                        GLM(
    observation_model=PoissonObservations(inverse_link_function=exp),
    regularizer=Ridge(),
    regularizer_strength=0.5,
    solver_name='GradientDescent'
))]),
             param_grid={'glm__regularizer_strength': (0.1, 0.01, 0.001, 1e-06),
                         'tra...
                                                     RaisedCosineLinearEval(n_basis_funcs=10, width=2.0),
                                                     RaisedCosineLogEval(n_basis_funcs=5, width=2.0, time_scaling=50.0, enforce_decay_to_zero=True),
                                                     RaisedCosineLogEval(n_basis_funcs=10, width=2.0, time_scaling=50.0, enforce_decay_to_zero=True),
                                                     MSplineEval(n_basis_funcs=5, order=4),
                                                     MSplineEval(n_basis_funcs=10, order=4))},
             scoring=make_scorer(pseudo_r2, response_method='predict'))

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

And finally, we can plot each model’s score.

Plot the pseudo-R2 scores

cvdf = pd.DataFrame(gridsearch.cv_results_)

# Read out the number of basis functions
cvdf["transformerbasis_config"] = [
    f"{b.__class__.__name__} - {b.n_basis_funcs}"
    for b in cvdf["param_transformerbasis__basis"]
]

cvdf_wide = cvdf.pivot(
    index="transformerbasis_config",
    columns="param_glm__regularizer_strength",
    values="mean_test_score",
)

doc_plots.plot_heatmap_cv_results(cvdf_wide, label="pseudo-R2")

../_images/44a0382c466cd5e22f7ed762b517a32aec9cb25b0cef2bd8c3e6d474b3255a2e.png

As you can see, the results with pseudo-R2 agree with those of the negative log-likelihood. Note that this new metric is normalized between 0 and 1, with a higher score indicating better performance.

Selecting basis by cross-validation with scikit-learn#

What is a scikit-learn pipeline#

Why pipelines are useful#

Combining basis transformations and GLM in a pipeline#

Converting NeMoS `Basis` to a transformer#

Creating and fitting a pipeline#

Select the number of basis by cross-validation#

Define the parameter grid#

Run the grid search#

Visualize the scores#

Visualize the predicted rate#

Evaluating different bases directly#

Create a custom scorer#

This Page

Selecting basis by cross-validation with scikit-learn#

What is a scikit-learn pipeline#

Why pipelines are useful#

Combining basis transformations and GLM in a pipeline#

Converting NeMoS Basis to a transformer#

Creating and fitting a pipeline#

Select the number of basis by cross-validation#

Define the parameter grid#

Run the grid search#

Visualize the scores#

Visualize the predicted rate#

Evaluating different bases directly#

Create a custom scorer#

This Page

Converting NeMoS `Basis` to a transformer#