Controlling the level of fine-tuning

It is possible to control the depth of fine-tuning with the finetune_depth parameter.

finetune_depth takes values among [1, 2, 3, 4, 5]. By default, it is set to 1, which means that a small set of the model’s parameters are being adjusted, whereas a value of 5 fine-tunes the maximum amount of parameters.

Increasing finetune_depth also increases the time to generate predictions. While it can generate better results, we must be careful to not overfit the model, in which case the predictions may not be as accurate.

Let’s run a small experiment to see how finetune_depth impacts the performance.

1. Import packages

First, we import the required packages and initialize the Nixtla client

import pandas as pd
from nixtla import NixtlaClient
from utilsforecast.losses import mae, mse
from utilsforecast.evaluation import evaluate

nixtla_client = NixtlaClient(
    # defaults to os.environ.get("NIXTLA_API_KEY")
    api_key = 'my_api_key_provided_by_nixtla'
)

👍

Use an Azure AI endpoint

To use an Azure AI endpoint, remember to set also the base_url argument:

nixtla_client = NixtlaClient(base_url="you azure ai endpoint", api_key="your api_key")

2. Load data

df = pd.read_csv('https://raw.githubusercontent.com/Nixtla/transfer-learning-time-series/main/datasets/air_passengers.csv')
df.head()

Now, we split the data into a training and test set so that we can measure the performance of the model as we vary finetune_depth.

train = df[:-24]
test = df[-24:]

Next, we fine-tune TimeGPT and vary finetune_depth to measure the impact on performance.

3. Fine-tuning with finetune_depth

📘

Available models in Azure AI

If you are using an Azure AI endpoint, please be sure to set model="azureai":

nixtla_client.forecast(..., model="azureai")

For the public API, we support two models: timegpt-1 and timegpt-1-long-horizon.

By default, timegpt-1 is used. Please see this tutorial on how and when to use timegpt-1-long-horizon.

As mentioned above, finetune_depth controls how many parameters from TimeGPT are fine-tuned on your particular dataset. If the value is set to 1, only a few parameters are fine-tuned. Setting it to 5 means that all parameters of the model will be fine-tuned.

Using a large value for finetune_depth can lead to better performances for large datasets with complex patterns. However, it can also lead to overfitting, in which case the accuracy of the forecasts may degrade, as we will see from the small experiment below.

depths = [1, 2, 3, 4, 5]

test = test.copy()

for depth in depths:
    preds_df = nixtla_client.forecast(
    df=train, 
    h=24, 
    finetune_steps=5,
    finetune_depth=depth,
    time_col='timestamp', 
    target_col='value')

    preds = preds_df['TimeGPT'].values

    test.loc[:,f'TimeGPT_depth{depth}'] = preds

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Inferred freq: MS
INFO:nixtla.nixtla_client:Querying model metadata...
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon. This may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...
INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Inferred freq: MS
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon. This may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...
INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Inferred freq: MS
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon. This may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...
INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Inferred freq: MS
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon. This may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...
INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Inferred freq: MS
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon. This may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...

test['unique_id'] = 0

evaluation = evaluate(test, metrics=[mae, mse], time_col="timestamp", target_col="value")
evaluation

From the result above, we can see that a finetune_depth of 2 achieves the best results since it has the lowest MAE and MSE.

Also notice that with a finetune_depth of 4 and 5, the performance degrades, which is a clear sign of overfitting.

Thus, keep in mind that fine-tuning can be a bit of trial and error. You might need to adjust the number of finetune_steps and the level of finetune_depth based on your specific needs and the complexity of your data. Usually, a higher finetune_depth works better for large datasets. In this specific tutorial, since we were forecasting a single series with a very short dataset, increasing the depth led to overfitting.

It’s recommended to monitor the model’s performance during fine-tuning and adjust as needed. Be aware that more finetune_steps and a larger value of finetune_depth may lead to longer training times and could potentially lead to overfitting if not managed properly.