ImportError: cannot import name 'top_k_top_p_filtering' from 'transformers'

[dmurphree]

Hello,

When trying to run cell 5 in tutorial-1.ipynb I get an import error:

ImportError: cannot import name 'top_k_top_p_filtering' from 'transformers'

It comes from the line:

from histogpt.helpers.inference import generate

I have transformers.version = 4.40.1

Any thoughts on why I'm getting this error?

Thanks!

[manuel-tran]

Hello,

Thank you for trying out HistoGPT. This seems to be related to a recent change in the 'transformers' library (huggingface/trl#1409) Could you try downgrading to version 4.38.2? We were using version 4.37.2 ourselves.

Hope this solves it!

[mariozupan]

Transformer version in colab is 4.40.2. I tried to install all what I need in colab. Unfortunately still have the same issue.

[mariozupan]

Colab:
!python --version
!pip show torch | grep Version
!pip show torchvision | grep Version
!pip show triton | grep Version
!pip show xformers | grep Version
!pip show transformers | grep Version

Python 3.10.12
Version: 2.2.1+cu121
Version: 0.17.1+cu121
Version: 2.2.0
Version: 0.0.25.post1
Version: 4.40.2

#Local, arch venv environment
Python 3.10.14
Version: 2.2.1+cu121
Version: 0.17.1+cu121
Version: 2.2.0
Version: 0.0.25.post1
Version: 4.40.2

[mariozupan]

Maybe this is solution for the error from topic:

pip install git+https://github.com/huggingface/trl.git@7630f877f91c556d9e5a3baa4b6e2894d90ff84c

[manuel-tran]

Hi, did it finally work for you? You said you also tried transformers version 4.38.2 or 4.37.2 in Google Colab and it did not work? I have retrieved the original code if you want to include it manually:

def top_k_top_p_filtering(
    logits: Tensor,
    top_k: int = 0,
    top_p: float = 1.0,
    filter_value: float = -float("Inf"),
    min_tokens_to_keep: int = 1,
) -> Tensor:
    """Filter a distribution of logits using top-k and/or nucleus (top-p) filtering
    Args:
        logits: logits distribution shape (batch size, vocabulary size)
        if top_k > 0: keep only top k tokens with highest probability (top-k filtering).
        if top_p < 1.0: keep the top tokens with cumulative probability >= top_p (nucleus filtering).
            Nucleus filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751)
        Make sure we keep at least min_tokens_to_keep per batch example in the output
    From: https://gist.github.com/thomwolf/1a5a29f6962089e871b94cbd09daf317
    """
    if top_k > 0:
        top_k = min(max(top_k, min_tokens_to_keep), logits.size(-1))  # Safety check
        # Remove all tokens with a probability less than the last token of the top-k
        indices_to_remove = logits < torch.topk(logits, top_k)[0][..., -1, None]
        logits[indices_to_remove] = filter_value

    if top_p < 1.0:
        sorted_logits, sorted_indices = torch.sort(logits, descending=True)
        cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)

        # Remove tokens with cumulative probability above the threshold (token with 0 are kept)
        sorted_indices_to_remove = cumulative_probs > top_p
        if min_tokens_to_keep > 1:
            # Keep at least min_tokens_to_keep (set to min_tokens_to_keep-1 because we add the first one below)
            sorted_indices_to_remove[..., :min_tokens_to_keep] = 0
        # Shift the indices to the right to keep also the first token above the threshold
        sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
        sorted_indices_to_remove[..., 0] = 0

        # scatter sorted tensors to original indexing
        indices_to_remove = sorted_indices_to_remove.scatter(1, sorted_indices, sorted_indices_to_remove)
        logits[indices_to_remove] = filter_value
    return logits