--- title: Core keywords: fastai sidebar: home_sidebar summary: "The basics for building and training models are contained in this module." description: "The basics for building and training models are contained in this module." ---
{% raw %}

    /Users/hmamin/incendio/incendio/callbacks.py:26: UserWarning: Accio not available.
      warnings.warn('Accio not available.')

{% endraw %} {% raw %}

    %load_ext autoreload
    %autoreload 2
    %matplotlib inline

{% endraw %} {% raw %}

    /Users/hmamin/anaconda3/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

{% endraw %} {% raw %}

    # Used in notebook but not needed in package.
    import torch.nn.functional as F
    from torch.utils.data import Dataset, DataLoader

    from htools import assert_raises

    /Users/hmamin/anaconda3/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

{% endraw %}

## Model

`BaseModel` allows models to freeze/unfreeze layers and provides several methods for weight diagnostics. It should not be instantiated directly, but used as a parent class for a model. Like all PyTorch models, its children will still need to call `super().__init__()` and implement a `forward()` method.

{% raw %}
{% endraw %} {% raw %}

## `class` `BaseModel`[source]

> `BaseModel`() :: `Module`

    Base class for all neural network modules.

    Your models should also subclass this class.

    Modules can also contain other Modules, allowing to nest them in
    a tree structure. You can assign the submodules as regular attributes::

        import torch.nn as nn
        import torch.nn.functional as F

        class Model(nn.Module):
            def __init__(self):
                super(Model, self).__init__()
                self.conv1 = nn.Conv2d(1, 20, 5)
                self.conv2 = nn.Conv2d(20, 20, 5)

            def forward(self, x):
                x = F.relu(self.conv1(x))
                return F.relu(self.conv2(x))

    Submodules assigned in this way will be registered, and will have their
    parameters converted too when you call :meth:`to`, etc.

    :ivar training: Boolean represents whether this module is in training or
                    evaluation mode.
    :vartype training: bool

{% endraw %} {% raw %}

    class SimpleModel(BaseModel):

        def __init__(self, dim):
            super().__init__()
            self.fc1 = nn.Linear(dim, 2)
            self.fc2 = nn.Linear(2, 1)

        def forward(self, x):
            x = F.leaky_relu(self.fc1(x))
            return self.fc2(x)

{% endraw %} {% raw %}

    class GroupedModel(BaseModel):

        def __init__(self, dim):
            super().__init__()
            g1 = nn.Sequential(
                 nn.Linear(dim, 8),
                 nn.LeakyReLU(),
                 nn.Linear(8, 4),
                 nn.LeakyReLU()
            )
            g2 = nn.Linear(4, 1)
            self.groups = nn.ModuleList([g1, g2])

        def forward(self, x):
            for group in self.groups:
                x = group(x)
            return x

{% endraw %} {% raw %}

    snet = SimpleModel(2)
    snet.freeze()
    for n in range(5):
        snet.unfreeze(n_layers=n)
        unfrozen = [x[1] for x in snet.trainable()]
        print('Unfrozen', unfrozen)
        assert sum(unfrozen) == n
        assert not any(unfrozen[:-n])

    Unfrozen [False, False, False, False]
    Unfrozen [False, False, False, True]
    Unfrozen [False, False, True, True]
    Unfrozen [False, True, True, True]
    Unfrozen [True, True, True, True]

{% endraw %} {% raw %}

    snet.freeze()
    with assert_raises(AttributeError) as ar:
        for n in range(3):
            snet.unfreeze(n_groups=n)

    As expected, got AttributeError('SimpleModel' object has no attribute 'groups').

{% endraw %} {% raw %}

    gnet = GroupedModel(2)
    gnet.freeze()
    n_unfrozen = [0, 2, 6]
    for n, nu in zip(range(3), n_unfrozen):
        gnet.unfreeze(n_groups=n)
        unfrozen = [x[1] for x in gnet.trainable()]
        print('Unfrozen', unfrozen)
        assert sum(unfrozen) == nu

    Unfrozen [False, False, False, False, False, False]
    Unfrozen [False, False, False, False, True, True]
    Unfrozen [True, True, True, True, True, True]

{% endraw %} {% raw %}

    gnet.freeze()
    for n in range(7):
        gnet.unfreeze(n_layers=n)
        unfrozen = [x[1] for x in gnet.trainable()]
        print('Unfrozen', unfrozen)
        assert sum(unfrozen) == n
        assert not any(unfrozen[:-n])

    Unfrozen [False, False, False, False, False, False]
    Unfrozen [False, False, False, False, False, True]
    Unfrozen [False, False, False, False, True, True]
    Unfrozen [False, False, False, True, True, True]
    Unfrozen [False, False, True, True, True, True]
    Unfrozen [False, True, True, True, True, True]
    Unfrozen [True, True, True, True, True, True]

{% endraw %} {% raw %}

    optim = variable_lr_optimizer(snet, 2e-3)
    print(optim)

    with assert_raises(ValueError) as ar:
        optim = variable_lr_optimizer(snet, [3e-3, 1e-1])
        optim

    Adam (
    Parameter Group 0
        amsgrad: False
        betas: (0.9, 0.999)
        eps: 0.001
        lr: 0.002
        weight_decay: 0
    )
    As expected, got ValueError(Received more learning rates than layer groups.).

{% endraw %} {% raw %}

    update_optimizer(optim, 1e-3, 0.5)
    assert len(optim.param_groups) == 1
    assert optim.param_groups[0]['lr'] == 1e-3

{% endraw %} {% raw %}

    lrs = [1e-3, 3e-3]
    optim = variable_lr_optimizer(gnet, lrs)
    print(optim)
    assert [group['lr'] for group in optim.param_groups] == lrs

    update_optimizer(optim, 2e-3, lr_mult=1/3)
    print([group['lr'] for group in optim.param_groups])
    assert np.isclose(optim.param_groups[1]['lr'], optim.param_groups[0]['lr'] * 3)

    Adam (
    Parameter Group 0
        amsgrad: False
        betas: (0.9, 0.999)
        eps: 0.001
        lr: 0.001
        weight_decay: 0

    Parameter Group 1
        amsgrad: False
        betas: (0.9, 0.999)
        eps: 0.001
        lr: 0.003
        weight_decay: 0
    )
    [0.0006666666666666666, 0.002]

{% endraw %} {% raw %}

    optim = variable_lr_optimizer(gnet, 1e-3, lr_mult=0.5)
    print([group['lr'] for group in optim.param_groups])
    assert np.isclose(optim.param_groups[1]['lr'], optim.param_groups[0]['lr'] * 2)

    [0.0005, 0.001]

{% endraw %} {% raw %}

    optim = variable_lr_optimizer(snet, 2e-3)
    print(optim)

    with assert_raises(ValueError) as ar:
        optim = variable_lr_optimizer(snet, [3e-3, 1e-1])
        optim

    Adam (
    Parameter Group 0
        amsgrad: False
        betas: (0.9, 0.999)
        eps: 0.001
        lr: 0.002
        weight_decay: 0
    )
    As expected, got ValueError(Received more learning rates than layer groups.).

{% endraw %} {% raw %}

    update_optimizer(optim, 1e-3, 0.5)
    assert len(optim.param_groups) == 1
    assert optim.param_groups[0]['lr'] == 1e-3

{% endraw %} {% raw %}

    lrs = [1e-3, 3e-3]
    optim = variable_lr_optimizer(gnet, lrs)
    print(optim)
    assert [group['lr'] for group in optim.param_groups] == lrs

    update_optimizer(optim, 2e-3, lr_mult=1/3)
    print([group['lr'] for group in optim.param_groups])
    assert np.isclose(optim.param_groups[1]['lr'], optim.param_groups[0]['lr'] * 3)

    Adam (
    Parameter Group 0
        amsgrad: False
        betas: (0.9, 0.999)
        eps: 0.001
        lr: 0.001
        weight_decay: 0

    Parameter Group 1
        amsgrad: False
        betas: (0.9, 0.999)
        eps: 0.001
        lr: 0.003
        weight_decay: 0
    )
    [0.0006666666666666666, 0.002]

{% endraw %} {% raw %}

    optim = variable_lr_optimizer(gnet, 1e-3, lr_mult=0.5)
    print([group['lr'] for group in optim.param_groups])
    assert np.isclose(optim.param_groups[1]['lr'], optim.param_groups[0]['lr'] * 2)

    [0.0005, 0.001]

{% endraw %} {% raw %}
{% endraw %}

# Trainer

{% raw %}
{% endraw %} {% raw %}

#### `handle_interrupt`[source]

> `handle_interrupt`(`meth`)

    Decorator for Trainer.fit() method that allows the user to
    interrupt training with Ctrl+c while still running the
    `on_train_end` method for each of its callbacks. Without this,
    cutting training short would lose that functionality, so we
    couldn't do things like uploading to S3.

    Arguments
    ---------
    meth: callable
        The method to decorate.

    Returns
    -------
    callable: The wrapped method.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

## `class` `Trainer`[source]

> `Trainer`(`net`, `dl_train`, `dl_val`, `criterion`, `mode`:`('binary', 'multiclass', 'regression')`, `out_dir`, `optim`=`None`, `optim_type`=`'Adam'`, `eps`=`0.001`, `last_act`=`None`, `threshold`=`0.5`, `metrics`=`None`, `callbacks`=`None`, `device`=`device(type='cpu')`) :: `LoggerMixin`

    Mixin class that configures and returns a logger.

    Examples
    --------
    class Foo(LoggerMixin):

        def __init__(self, a, log_file):
            self.a = a
            self.log_file = log_file
            self.logger = self.get_logger(log_file)

        def walk(self, location):
            self.logger.info(f'walk received argument {location}')
            return f'walking to {location}'

{% endraw %} {% raw %}
{% endraw %} {% raw %}

## `class` `PredictionExaminer`[source]

> `PredictionExaminer`(`trainer`)

    Examine model predictions. This lets us view rows where the model was
    very confidently wrong, barely wrong, confidently right, barely right, or
    just random rows.

{% endraw %}