Add TimesFM Time Series Forecasting Model #34082

jinan-zhou · 2024-10-11T00:19:45Z

What does this PR do?

This PR adds a new model, TimesFM, to HuggingFace. TimesFM is a time series forecasting model based on the transformer architecture. It was proposed in A decoder-only foundation model for time-series forecasting.

Code Example

import numpy as np
from transformers import TimesFMModel
from transformers import TimesFMConfig

config = TimesFMConfig()
model = TimesFMModel(config)

forecast_input = [
    np.sin(np.linspace(0, 20, 100)),
    np.sin(np.linspace(0, 20, 200)),
    np.sin(np.linspace(0, 20, 400)),
]
frequency_input = [0, 1, 2]

results = model(
    inputs=forecast_input,
    freq=frequency_input,
)

TODOs

Unit Tests
Documentations and doc strings
Pretrained weights
ONNX definition
Model parallelism

rajatsen91

Awesome work.

One general question: are there plans to support decoder only training and associated loss functions ?

src/transformers/models/timesfm/modeling_timesfm.py

rajatsen91 · 2024-10-24T20:39:57Z

Great work! I left some minor comments.

rvt123 · 2024-11-07T18:52:40Z

Would this PR add finetuning support for the model?

Rocketknight1 · 2024-11-25T14:59:11Z

Marking @kashif as the HF point of contact for this!

This reverts commit def36c4.

Cyrilvallez

Alright, super super nice! 🤗 Love that you already integrated the new init standard!! Left some final comments but we can merge right afterwards!
Thanks a lot, and sorry again for the delay in the reviews!

docs/source/en/model_doc/timesfm.md

src/transformers/models/timesfm/modular_timesfm.py

tests/models/timesfm/test_modeling_timesfm.py

Co-authored-by: Cyril Vallez <[email protected]>

Cyrilvallez

Alright, last nits then merging!! 🤗

docs/source/en/model_doc/timesfm.md

src/transformers/models/timesfm/modular_timesfm.py

Co-authored-by: Cyril Vallez <[email protected]>

Cyrilvallez

All right, LGTM!! Merging, thank you for being patient and bearing with me along the way!! 🤗

Thanks for the very nice addition!

* initial documentation * rename mask to attention_mask * smaller tests * fixup * fix copies * move to time series section * sort docs * isort fix * batch_size is not a configuration * rename to TimesFMModelForPrediction * initial script * add check_outputs * remove dropout_rate * works with torch.Tensor inputs * rename script * fix docstrings * fix freq when window_size is given * add loss * fix _quantile_loss * formatting * fix isort * add weight init * add support for sdpa and flash_attention_2 * fixes for flash_attention * formatting * remove flash_attention * fix tests * fix file name * fix quantile loss * added initial TimesFMModelIntegrationTests * fix formatting * fix import order * fix _quantile_loss * add doc for SDPA * use timesfm 2.0 * bug fix in timesfm decode function. * compare mean forecasts * refactor type hints, use CamelCase * consolidate decode func * more readable code for weight conversion * fix-copies * simpler init * renaem TimesFmMLP * use T5LayerNorm * fix tests * use initializer_range * TimesFmModel instead of TimesFmDecoder * TimesFmPositionalEmbedding takes config for its init * 2.0-500m-pytorch default configs * use TimesFmModel * fix formatting * ignore TimesFmModel for testing * fix docstring * override generate as its not needed * add doc strings * fix logging * add docstrings to output data classes * initial copy from t5 * added config and attention layers * add TimesFMPositionalEmbedding * calcuate scale_factor once * add more configs and TimesFMResidualBlock * fix input_dims * standardize code format with black * remove unneeded modules * TimesFM Model * order of imports * copy from Google official implementation * remove covariate forecasting * Adapting TimesFM to HF format * restructing in progress * adapted to HF convention * timesfm test * the model runs * fixing unit tests * fixing unit tests in progress * add post_init * do not change TimesFMOutput * fixing unit tests * all unit tests passed * remove timesfm_layers * add intermediate_size and initialize with config * initial documentation * rename mask to attention_mask * smaller tests * fixup * fix copies * move to time series section * sort docs * isort fix * batch_size is not a configuration * rename to TimesFMModelForPrediction * initial script * add check_outputs * remove dropout_rate * works with torch.Tensor inputs * rename script * fix docstrings * fix freq when window_size is given * add loss * fix _quantile_loss * formatting * fix isort * add weight init * add support for sdpa and flash_attention_2 * fixes for flash_attention * formatting * remove flash_attention * fix tests * fix file name * fix quantile loss * added initial TimesFMModelIntegrationTests * fix formatting * fix import order * fix _quantile_loss * add doc for SDPA * use timesfm 2.0 * bug fix in timesfm decode function. * compare mean forecasts * refactor type hints, use CamelCase * consolidate decode func * more readable code for weight conversion * fix-copies * simpler init * renaem TimesFmMLP * use T5LayerNorm * fix tests * use initializer_range * TimesFmModel instead of TimesFmDecoder * TimesFmPositionalEmbedding takes config for its init * 2.0-500m-pytorch default configs * use TimesFmModel * fix formatting * ignore TimesFmModel for testing * fix docstring * override generate as its not needed * add doc strings * fix logging * add docstrings to output data classes * add _CHECKPOINT_FOR_DOC * fix comments * Revert "fix comments" This reverts commit 8deeb3e. * add _prepare_4d_attention_mask * we do not have generative model classes * use Cache * return past_key_values * modules initialized with config only * update year * Update docs/source/en/model_doc/timesfm.md Co-authored-by: Steven Liu <[email protected]> * add layer_idx to cache * modular timesfm * fix test * unwrap sequential class * fix toctree * remove TimesFmOnnxConfig * fix modular * remove TimesFmStackedDecoder * split qkv layer into individual layers * rename projection layers * use ALL_ATTENTION_FUNCTIONS * is_causal is True * rename config * does not support flash_attn_2 * formatting * fix typo in docsstring * rename inputs * add time series mapping * Update src/transformers/models/olmo2/modeling_olmo2.py * Update src/transformers/models/moonshine/modeling_moonshine.py * use updated arguments * fix class name * add MODEL_FOR_TIME_SERIES_PREDICTION_MAPPING * isort * consolidate _preprocess into forward * fix a typo * fix a typo * fix toc * fix modular * remove aaserts * use self.config._attn_implementation * move to _postprocess_output * remove timesfm_get_large_negative_number * use view unstead of multiple unsqueeze * make helpers static methods of the Model * use to_tuple * use to_tuple if not return_dict * remove unused intitialization block as its incorporated in nn.Linear * remove unused num_key_value_groups * use the same convention as the masking method * update modular * do not use unsqueeze * use view instead of unsqueeze * use buffer for inv_timescales * formatting * modular conversion * remove unneeded intialization * add missing docstrings * remove cache * use simple_eager_attention_forward * support tp_plan * support for flex and flash attention masks * Revert "support for flex and flash attention masks" This reverts commit def36c4. * fix device * fix tests on gpu * remove unsued large model test * removed unneeded comments * add example usage * fix style * add import * Update docs/source/en/model_doc/timesfm.md Co-authored-by: Cyril Vallez <[email protected]> * inherit from LlamaRMSNorm * use can_return_tuple decorator * remvoe return_dict * fix year * Update docs/source/en/model_doc/timesfm.md Co-authored-by: Cyril Vallez <[email protected]> * pretrained does not inherit from GenerationMixin * use model for integration test --------- Co-authored-by: Kashif Rasul <[email protected]> Co-authored-by: Rajat Sen <[email protected]> Co-authored-by: Steven Liu <[email protected]> Co-authored-by: Cyril Vallez <[email protected]> Co-authored-by: Cyril Vallez <[email protected]>

This was referenced Oct 24, 2024

TimesFM in Hugging Face is coming google-research/timesfm#168

Open

PyTorch Implementation Coming google-research/timesfm#119

Open

rajatsen91 approved these changes Oct 24, 2024

View reviewed changes

Rocketknight1 assigned kashif Nov 25, 2024

Rocketknight1 added the New model label Nov 25, 2024

Rocketknight1 requested a review from kashif November 25, 2024 14:58

kashif added 21 commits November 28, 2024 20:37

initial documentation

f539965

rename mask to attention_mask

f95d6ee

smaller tests

d475529

fixup

f7c1fe0

fix copies

5a808be

move to time series section

6dbbd80

sort docs

bfa9302

isort fix

eb5807e

batch_size is not a configuration

be32365

rename to TimesFMModelForPrediction

c4a3610

initial script

56a5606

add check_outputs

01756ae

remove dropout_rate

942c23c

works with torch.Tensor inputs

e7650bd

rename script

c523f64

fix docstrings

e64f562

fix freq when window_size is given

f5dbab9

add loss

a8dcfa9

fix _quantile_loss

5fb1fe0

formatting

1445fe5

Merge branch 'main' into timesfm

b84c188

kashif added 12 commits March 31, 2025 13:30

support for flex and flash attention masks

def36c4

Revert "support for flex and flash attention masks"

5cc47cd

This reverts commit def36c4.

fix device

5e3a5e2

fix tests on gpu

7da546f

remove unsued large model test

debb032

removed unneeded comments

2a0c209

Merge branch 'main' into timesfm

b1c3c49

add example usage

87e8b12

Merge branch 'main' into timesfm

0493f61

fix style

70c3cb5

add import

76f72fb

Merge branch 'main' into timesfm

aa721d4

Cyrilvallez reviewed Apr 15, 2025

View reviewed changes

kashif and others added 7 commits April 15, 2025 19:28

Update docs/source/en/model_doc/timesfm.md

e7882d7

Co-authored-by: Cyril Vallez <[email protected]>

inherit from LlamaRMSNorm

a86136d

use can_return_tuple decorator

60e7e65

remvoe return_dict

a5b9010

fix year

ca86584

Merge branch 'main' into timesfm

0b711f1

Merge branch 'main' into timesfm

531f8e3

Cyrilvallez reviewed Apr 16, 2025

View reviewed changes

docs/source/en/model_doc/timesfm.md Outdated Show resolved Hide resolved

src/transformers/models/timesfm/modular_timesfm.py Outdated Show resolved Hide resolved

kashif and others added 3 commits April 16, 2025 15:15

Update docs/source/en/model_doc/timesfm.md

9b76c0b

Co-authored-by: Cyril Vallez <[email protected]>

pretrained does not inherit from GenerationMixin

0161ca9

use model for integration test

fa53c52

Cyrilvallez approved these changes Apr 16, 2025

View reviewed changes

Cyrilvallez merged commit a91020a into huggingface:main Apr 16, 2025
18 checks passed

kashif deleted the timesfm branch April 16, 2025 13:01

jinan-zhou mentioned this pull request May 22, 2025

Expose AutoModelForTimeSeriesPrediction for import #38307

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add TimesFM Time Series Forecasting Model #34082

Add TimesFM Time Series Forecasting Model #34082

Uh oh!

jinan-zhou commented Oct 11, 2024 •

edited by kashif

Loading

Uh oh!

rajatsen91 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rajatsen91 commented Oct 24, 2024

Uh oh!

rvt123 commented Nov 7, 2024

Uh oh!

Rocketknight1 commented Nov 25, 2024

Uh oh!

Cyrilvallez left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Cyrilvallez left a comment

Uh oh!

Uh oh!

Uh oh!

Cyrilvallez left a comment

Uh oh!

Uh oh!

Uh oh!

Add TimesFM Time Series Forecasting Model #34082

Add TimesFM Time Series Forecasting Model #34082

Uh oh!

Conversation

jinan-zhou commented Oct 11, 2024 • edited by kashif Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Example

TODOs

Uh oh!

rajatsen91 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rajatsen91 commented Oct 24, 2024

Uh oh!

rvt123 commented Nov 7, 2024

Uh oh!

Rocketknight1 commented Nov 25, 2024

Uh oh!

Cyrilvallez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Cyrilvallez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Cyrilvallez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jinan-zhou commented Oct 11, 2024 •

edited by kashif

Loading