LM Classifiers¶

The following classes are the core elements of this library. MaskedLM4Classification is based on masked language models, while CausalLM4Classification is based on causal language models. The idea behind this concept is described in Background.

The tutorials Tutorial - Causal-Language-Model-Based Classifiers and Tutorial - Masked-Language-Model-Based Classifiers describe how these classes are applied.

class promptzl.modules.CausalLM4Classification(*args: Any, **kwargs: Any)¶

Bases: LLM4ClassificationBase, Module

Causal-LM-Based Classification

CausalLM4Classification works with all models that can be loaded through AutoModelForCausalLM.from_pretrained(model_id) with valid causal model_ids from the Hugging Face hub.

Parameters:

model_id (str) – Valid model identifier for the Hugging Face hub.
prompt (Prompt) – A prompt object. Example usage: Txt("This text ") + Key('text') + Txt(" is ") + Vbz([['good'], ['bad']]) or FnVbzPair(lambda e: f"{e['text']} It was ", Vbz([["bad", "horrible"], ["good"]])) More about the prompt object in Prompt Classes.
device (Optional[str]) – The device to be used. Defaults to None.
lower_verbalizer (bool) – A flag to determine if the verbalizer should be enhanced with lowercased words. Defaults to False.
model_args (Optional[Dict[str, Any]]) – Additional arguments for initializing the underlying huggingface-model.
tokenizer_args (Optional[Dict[str, Any]]) – Additional arguments for initializing the underlying huggingface-model.

model¶: Hugging Face model.

tokenizer¶: Hugging Face tokenizer.

forward(batch: Dict[str, torch.Tensor], return_model_output: bool = False, **kwargs: Any) → torch.Tensor | Tuple[torch.Tensor, transformers.generation.utils.ModelOutput]¶

Perform forward pass and return logits.

Parameters:

batch (Dict[str, torch.Tensor]) – The input batch.
return_model_output (bool) – A flag to determine if the model output should be returned.
kwargs – Additional arguments for the model.

Returns:

Output logits and model output (if return_model_output is set) from model.

Return type:

Union[torch.Tensor, Tuple[torch.Tensor, Any]]

class promptzl.modules.LLM4ClassificationBase(*args: Any, **kwargs: Any)¶

Bases: Module

Base Class for LM-Classifiers

Checks correctness of input and initializes the class.

Parameters:

model (PreTrainedModel) – The model to be used.
tokenizer (PreTrainedTokenizerBase) – The tokenizer to be used.
prompt (Prompt) – The prompt to be used. Example usage: Txt("This text ") + Key('text') + Txt(" is ") + Vbz([['good'], ['bad']]) or FnVbzPair(lambda e: f"{e['text']} It was ", Vbz([["bad", "horrible"], ["good"]])) More about the prompt object in Prompt Classes.
generate (bool) – A flag to determine if the model should be able to generate.
device (Optional[str], optional) – The device to be used. Defaults to None.
lower_verbalizer (bool, optional) – A flag to determine if the verbalizer should be lowercased. Defaults to False.

model¶: Hugging Face model.

tokenizer¶: Hugging Face tokenizer.

Raises:

AssertionError – If model is not of type PreTrainedModel.
AssertionError – If tokenizer is not of type PreTrainedTokenizerBase.
AssertionError – If prompt is not of type Prompt.
AssertionError – If generate is not of type bool.
AssertionError – If device is not of type str or None.
AssertionError – If lower_verbalizer is not of type bool.

calibrate_output(output: LLM4ClassificationOutput) → LLM4ClassificationOutput¶

Wrapper for the promptzl.utils.calibrate() method that retains the types (e.g., ‘torch’, ‘pandas’ etc.) and returns an updated promptzl.utils.LLM4ClassificationOutput object with calibrated probabilities. More about calibration is available in Calibration.

Parameters:

output (LLM4ClassificationOutput) – A promptzl.utils.LLM4ClassificationOutput object with predictions and probabilites.

Returns:

A promptzl.utils.LLM4ClassificationOutput: with calibrated probabilities and predictions. Logits are kept not altered if available.

Return type:

LLM4ClassificationOutput

classify(data: datasets.Dataset | datasets.DatasetDict, batch_size: int = 64, show_progress_bar: bool = False, return_logits: bool = False, return_type: str = 'torch', temperature: float = 1.0, **kwargs: Any) → LLM4ClassificationOutput | Dict[str, LLM4ClassificationOutput]¶

Classifies the data and returns the results in the requested format. For the prediction loop, smart-batching is used, where the data is sorted according to the lengths of the instances and then predicted as the longest first. After the prediction, the data is reordered into its initial order.

Parameters:

data (Union[Dataset, Any]) – The data to be classified.
batch_size (int) – The batch size to be used. Defaults to 64.
show_progress_bar (bool) – A flag to determine if the progress bar should be shown. Defaults to False.
return_logits (bool) – A flag to determine if the logits should be returned. Defaults to False. If the logits are returned and a label in the verbalizer contains more than one word, the logits are averaged for the label group. E.g. Vbz([['good', 'great'], ['bad']]) mean logits are computed for ['good', 'great']
return_type (str) – The return type. Defaults to “torch”. Supported types are “list”, “torch”, “numpy”, “pandas” and “polars”.
temperature (float) – The temperature for the softmax function. Defaults to 1.0.
kwargs – Additional arguments for the model’s forward function.

Returns:

The classified data (predictions, distribution and logits if return_logits is set).

Return type:

Union[LLM4ClassificationOutput, Dict[str, LLM4ClassificationOutput]]

forward(batch: Dict[str, torch.Tensor], return_model_output: bool = False, **kwargs: Any) → torch.Tensor | Tuple[torch.Tensor, transformers.generation.utils.ModelOutput]¶

Forward Function.

Perform the forward pass of the model and return the logits for each class. This method must be implemented in a child class.

Parameters:

batch (Dict[str, torch.Tensor]) – The input batch.
return_model_output (bool) – A flag to determine if the model output should be returned.
kwargs – Additional arguments for the model.

Raises:

NotImplementedError – If the forward function is not implemented.

Returns:

Output logits and model output (if return_model_output is set) from model.

Return type:

Union[torch.Tensor, Tuple[torch.Tensor, Any]]

static group_logits(logits: torch.Tensor, grouped_indices: List[List[int]]) → torch.Tensor¶

Combines the logits if different label words are used by taking the arithmetic mean of the logits for each class.

Parameters:: logits (torch.Tensor) – The logits to be combined.
Returns:: The combined logits.
Return type:: torch.Tensor

set_prompt(prompt: Prompt, lower_verbalizer: bool = False) → None¶

Can be used for initialization or updating the object.

Parameters:

prompt (Prompt) – The prompt to be set.
lower_verbalizer (bool, optional) – A flag to determine if the verbalizer should be lowercased. Defaults to False.

class promptzl.modules.MaskedLM4Classification(*args: Any, **kwargs: Any)¶

Bases: LLM4ClassificationBase, Module

Masked-Language-Modeling-Based Classification

MaskedLM4Classification works with all models that can be loaded through AutoModelForMaskedLM.from_pretrained(model_id) with valid masked model_ids from the Hugging Face hub.

Parameters:

model_id (str) – Valid model identifier for the Hugging Face hub.
prompt (Prompt) – A prompt object. Example usage: Txt("This text ") + Key('text') + Txt(" is ") + Vbz([['good'], ['bad']]) or FnVbzPair(lambda e: f"{e['text']} It was ", Vbz([["bad", "horrible"], ["good"]])) More about the prompt object in Prompt Classes.
device (Optional[str]) – The device to be used. Defaults to None.
lower_verbalizer (bool) – A flag to determine if the verbalizer should be enhanced with lowercased words. Defaults to False.
model_args (Optional[Dict[str, Any]]) – Additional arguments for initializing the underlying huggingface-model.
tokenizer_args (Optional[Dict[str, Any]]) – Additional arguments for initializing the underlying huggingface-model.

model¶: Hugging Face model.

tokenizer¶: Hugging Face tokenizer.

forward(batch: Dict[str, torch.tensor], return_model_output: bool = False, **kwargs: Any) → torch.Tensor | Tuple[torch.Tensor, transformers.generation.utils.ModelOutput]¶

Perform forward pass and return logits.

Parameters:

batch (Dict[str, torch.Tensor]) – The input batch.
return_model_output (bool) – A flag to determine if the model output should be returned.
kwargs – Additional arguments for the model.

Returns:

Output logits and model output (if return_model_output is set) from model.

Return type:

Union[torch.Tensor, Tuple[torch.Tensor, Any]]