Fine tune with additional elements

[turbosonics]

I have a Seven Net model that I trained for silica structure AIMDs. Using this model, I tried to fine tune a new model using a new training set that contains water and silica interaction AIMDs. Target is to check if I can fine-tune train a model for bulk and surface-water interaction.

Then I see
ValueError: Chemical species in checkpoint is not compatible

And the last line of log.sevenn is
Ignore chemical species in yaml, use checkpoint

It looks like the issue is that the new training set for the fine-tune training contains H atoms which were not included in prior training set. Any way to resolve this situation? I didn't see any options regarding elements (or chemical species) in the input file example...

Following is the script that I used for the fine tuning. The input file contains the same hyperparameters with input file of from-scratch training of silica model.

model:
    chemical_species: 'Auto'
    cutoff: 6.0
    channel: 64
    is_parity: True
    lmax: 1
    num_convolution_layer: 3
    irreps_manual:
        - "64x0e"
        - "64x0e+64x1e"
        - "64x0e+64x1e"
        - "64x0e"

    weight_nn_hidden_neurons: [64, 64]
    radial_basis:
        radial_basis_name: 'bessel'
        bessel_basis_num: 10
    cutoff_function:
        cutoff_function_name: 'XPLOR'
        cutoff_on: 4.5

    act_gate: {'e': 'silu', 'o': 'tanh'}
    act_scalar: {'e': 'silu', 'o': 'tanh'}

    conv_denominator: "avg_num_neigh"
    train_shift_scale: True
    train_denominator: True
    self_connection_type: 'nequip'

train:
    train_shuffle: True
    random_seed: 123
    is_train_stress: True
    epoch: 2000

    loss: 'Huber'
    loss_param:
        delta: 0.01

    optimizer: 'adam'
    optim_param:
        lr: 0.002
    scheduler: 'ReduceLROnPlateau'
    scheduler_param:
        factor: 0.5
        patience: 50
    force_loss_weight: 1.0
    stress_loss_weight: 0.01

    per_epoch: 10
    error_record:
        - ['Energy', 'RMSE']
        - ['Force', 'RMSE']
        - ['Stress', 'RMSE']
        - ['Energy', 'MAE']
        - ['Force', 'MAE']
        - ['Stress', 'MAE']
        - ['Energy', 'Loss']
        - ['Force', 'Loss']
        - ['Stress', 'Loss']
        - ['TotalLoss', 'None']

    continue:
        reset_optimizer: True
        reset_scheduler: True
        reset_epoch: True
        checkpoint: '${PTMODEL_DIR}/${PTMODEL_NAME}'
        # Set True to use shift, scale, and avg_num_neigh from checkpoint (highly recommended)
        use_statistic_values_of_checkpoint: True

data:
    data_shuffle: True
    batch_size: 12
    data_divide_ratio: 0.2
    data_format: 'ase'
    load_dataset_path: ['${GEOMETRY_DIR}/${GEOMETRY_TRAIN_NAME}']

[YutackPark]

Try below and restart from scratch. (The silica training you mentioned)

model:
    chemical_species: 'Si, H, O'

With the chemical_species: 'auto', sevennet will seek element in given training set, and initialize model parameters (element dependent ones) based on it. Sadly, once the kind of element is initialized, it can not be changed.

The error you encountered is therefore, 1) model is initialized with Si, O as dataset has only Si and O. 2) The trained model can not inference or fine-tuned for water + silica, as it doesn't know what 'H' is.

In current state, where universal GNN models are popular and there is no performance drop caused by introducing extra elements, this behavior of SevenNet should be changed. Sorry for the inconvenience.

• If not intended, I recommend to use is_parity: False as your irreps_manual contains only even parity features. You are also safe to remove irreps_manual from the input.yaml, as the number of features (=64) are same regardless of 'L' value. The irreps_manual is useful when you want to use different number of channels w.r.t the 'L' values. You can see the irreps of edge features and node features after each message passing layers from log. If these notations are not familiar, check here: https://docs.e3nn.org/en/stable/guide/irreps.html#irreps-guide

[turbosonics]

Thanks.

I also tried a fine-tune training of silica-only training set based on MP pretrained model. However, I see a lot of sharp spikes from EFS loss and error curves, even from 0.002 LR (which IMO already small enough LR but I might be wrong). That's why I gave up fine tuning and begin from-scratch training for silica-only training set. Do you know any reason behind this behavior (fine-tune shows spikes or weird response while from-scratch with same hyperparameters perform better) and how to escape this?

In current state, where universal GNN models are popular and there is no performance drop caused by introducing extra elements, this behavior of SevenNet should be changed. Sorry for the inconvenience.

Thanks for this comment and hope this feature introduced to SevenNet soon. Because it's common for people to want to expand their models for more elements and more reactions and etc. Also, while doing this, it would be better to have some measures do deal with catastrophic forgetting. What I do to prevent catastrophic forgetting is, I include a bit of AIMD frames from previous training set to the next fine-tune training set. But I don't know if it can perfectly prevent catastrophic forgetting.

And thank you for recommendations on parity and irreps_manual. Infact, I was testing things around those, and I found a training case without irreps_manual converged better than the example case I shared. I found that case without manual irreps runs smooth as well.

But one question, may I ask what are pros and cons of using different number of channels irreps_manual? I find the model trained with too high channel (like 128) and L=2 increases memory requirement for MD simulation, so I'm trying to be careful about this. But I wonder what is merit of mixing differnt feature numbers (32, 64, 128) and L for each lines of irreps_manual (L=0 for the first and last lines, while L=1 for other lines)

[turbosonics]

Also, I found the error with best_metric:
scheduler_param: best_metric should be one of ['mode', 'factor', 'patience', 'threshold', 'threshold_mode', 'cooldown', 'min_lr', 'eps', 'last_epoch', 'verbose']

[YutackPark]

Also, I found the error with best_metric:
You should put 'best_metric' under 'train'. Not under 'train' and 'scheduler_param'.

However, I see a lot of sharp spikes from EFS loss and error curves, even from 0.002 LR (which IMO already small enough LR but I might be wrong).
I'm not currently experimenting with fine-tuning. But loss spike itself was also common when pre-training stage. We also found that loss spike doesn't mean poor accuracy of the trained potential. However I agree with that it is still a problem and make me unpleasant. How about using amsgrad=True under optim_param? It was effective for removing those spikes in my previous experience.

Do you know any reason behind this behavior (fine-tune shows spikes or weird response while from-scratch with same hyperparameters perform better) and how to escape this?
One common agreement, also can be seen from a very few literatures, is that as the number of fine-tuning data increases, the gain of using pre-trained model decreases. However, the statement is about the accuracy within the fine-tuning domain and if the task requires a broad knowledge of chemical space, using pre-trained model is still beneficial. I'm not sure about loss spikes, but we didn't dig into the problem deeply as trained potential was usable even with those spikes.

The catastrophic forgetting is critical. Sadly, we don't have enough data to suggest you the strategies to prevent forgetting or measuring the forgetting. I haven't seen any papers regarding this issue of pre-trained potentials yet. As I'm very interested in the forgetting, I'm open to discussion. I think it is communities next goal to fully leverage pre-trained potentials.

But one question, may I ask what are pros and cons of using different number of channels irreps_manual?
It's about accuracy speed trade-off. The parameters firstly used in GNoME. From our internal test we also found that it is sweet spot that gives a good balance between accuracy and speed. If you increase the number of channels for high L value, the model becomes more accurate but its cost can be painful.