Re-using fine-tuned models

Save and re-use fine-tuned models across all of our endpoints.

1. Import packages

First, we import the required packages and initialize the Nixtla client

import pandas as pd
from nixtla import NixtlaClient
from utilsforecast.losses import rmse
from utilsforecast.evaluation import evaluate

nixtla_client = NixtlaClient(
    # defaults to os.environ["NIXTLA_API_KEY"]
    api_key = 'my_api_key_provided_by_nixtla'
)

2. Load data

df = pd.read_parquet('https://datasets-nixtla.s3.amazonaws.com/m4-hourly.parquet')

h = 48
valid = df.groupby('unique_id', observed=True).tail(h)
train = df.drop(valid.index)
train.head()

3. Zero-shot forecast

We can try forecasting without any finetuning to see how well TimeGPT does.

fcst_kwargs = {'df': train, 'freq': 1, 'model': 'timegpt-1-long-horizon'}
fcst = nixtla_client.forecast(h=h, **fcst_kwargs)
zero_shot_eval = evaluate(fcst.merge(valid), metrics=[rmse], agg_fn='mean')
zero_shot_eval

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Querying model metadata...
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon, this may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Restricting input...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...

4. Fine-tune

We can now fine-tune TimeGPT a little and save our model for later use. We can define the ID that we want that model to have by providing it through output_model_id.

first_model_id = 'my-first-finetuned-model'
nixtla_client.finetune(output_model_id=first_model_id, **fcst_kwargs)

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Calling Fine-tune Endpoint...

'my-first-finetuned-model'

We can now forecast using this fine-tuned model by providing its ID through the finetuned_model_id argument.

first_finetune_fcst = nixtla_client.forecast(h=h, finetuned_model_id=first_model_id, **fcst_kwargs)
first_finetune_eval = evaluate(first_finetune_fcst.merge(valid), metrics=[rmse], agg_fn='mean')
zero_shot_eval.merge(first_finetune_eval, on=['metric'], suffixes=('_zero_shot', '_first_finetune'))

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon, this may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Restricting input...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...

We can see the error was reduced.

5. Further fine-tune

We can now take this model and fine-tune it a bit further by using the NixtlaClient.finetune method but providing our already fine-tuned model as finetuned_model_id, which will take that model and fine-tune it a bit more. We can also change the fine-tuning settings, like using finetune_depth=3, for example.

second_model_id = nixtla_client.finetune(finetuned_model_id=first_model_id, finetune_depth=3, **fcst_kwargs)
second_model_id

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Calling Fine-tune Endpoint...

'468b13fb-4b26-447a-bd87-87a64b50d913'

Since we didn’t provide output_model_id this time, it got assigned an UUID.

We can now use this model to forecast.

second_finetune_fcst = nixtla_client.forecast(h=h, finetuned_model_id=second_model_id, **fcst_kwargs)
second_finetune_eval = evaluate(second_finetune_fcst.merge(valid), metrics=[rmse], agg_fn='mean')
first_finetune_eval.merge(second_finetune_eval, on=['metric'], suffixes=('_first_finetune', '_second_finetune'))

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon, this may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Restricting input...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...

We can see the error was reduced a bit more.

6. Listing fine-tuned models

We can list our fine-tuned models with the NixtlaClient.finetuned_models method.

finetuned_models = nixtla_client.finetuned_models()
finetuned_models

[FinetunedModel(id='468b13fb-4b26-447a-bd87-87a64b50d913', created_at=datetime.datetime(2024, 12, 30, 17, 57, 31, 241455, tzinfo=TzInfo(UTC)), created_by='user', base_model_id='my-first-finetuned-model', steps=10, depth=3, loss='default', model='timegpt-1-long-horizon', freq='MS'),
 FinetunedModel(id='my-first-finetuned-model', created_at=datetime.datetime(2024, 12, 30, 17, 57, 16, 978907, tzinfo=TzInfo(UTC)), created_by='user', base_model_id='None', steps=10, depth=1, loss='default', model='timegpt-1-long-horizon', freq='MS')]

While that representation may be useful for programmatic use, in this exploratory setting it’s nicer to see them as a dataframe, which we can get by providing as_df=True.

nixtla_client.finetuned_models(as_df=True)

We can seee that the base_model_id of our second model is our first model, along with other metadata.

7. Deleting fine-tuned models

In order to keep things organized, and since there’s a limit of 50 fine-tuned models, you can delete models that weren’t so promising to make room for more experiments. For example, we can delete our first finetuned model. Note that even though it was used as the base for our second model, they’re saved independently so removing it won’t affect our second model, except for the dangling metadata.

nixtla_client.delete_finetuned_model(first_model_id)

True

We can verify that our first model model doesn’t show up anymore in our available models.

nixtla_client.finetuned_models(as_df=True)