TextGenerationMetrics Module¶
Provides metrics for evaluating text generation models and outputs.
- class HMB.TextGenerationMetrics.TextGenerationMetrics(tokenizer=None)[source]¶
Bases:
objectEncapsulates a comprehensive suite of text generation evaluation metrics for NLP tasks. Includes BLEU, ROUGE, METEOR, Edit Distance, Jaccard, Perplexity, F1, CHRF, and more.
Examples
from HMB.TextGenerationMetrics import TextGenerationMetrics # Initialize the metrics object. metrics = TextGenerationMetrics() # Example usage for BLEU score. generatedText = "The cat sat on the mat." referenceText = "The cat is sitting on the mat." bleuScore = metrics.CalculateBLEU(generatedText, referenceText) print(f"BLEU Score: {bleuScore:.4f}")
Initializes the metrics class with an optional tokenizer.
- Parameters:
tokenizer (Optional) – A tokenizer object for text preprocessing. If not provided, default tokenization methods will be used.
- __init__(tokenizer=None)[source]¶
Initializes the metrics class with an optional tokenizer.
- Parameters:
tokenizer (Optional) – A tokenizer object for text preprocessing. If not provided, default tokenization methods will be used.
- CalculateBLEU(generatedText, referenceText, weights=(0.25, 0.25, 0.25, 0.25))[source]¶
Calculates BLEU score for generated text against reference text. BLEU measures n-gram precision with optional smoothing.
\[BLEU = BP \times \exp\left(\sum_{n=1}^N w_n \times \log (p_n)\right)\]- where:
\(BP\) is the brevity penalty.
\(p_n\) is the n-gram precision for n-grams of size n.
\(w_n\) are the weights for each n-gram precision.
- CalculateROUGE(generatedText, referenceText)[source]¶
Calculates ROUGE scores (ROUGE-1, ROUGE-2, ROUGE-L) for generated text against reference text. ROUGE measures n-gram recall and longest common subsequence.
\[ROUGE_N = \frac{\sum_{gram_n \in ref} \min(count_{gen}(gram_n), count_{ref}(gram_n))}{\sum_{gram_n \in ref} count_{ref}(gram_n)}\]- where:
\(gram_n\) is an n-gram of size n.
- CalculateMETEOR(generatedText, referenceText)[source]¶
Calculates METEOR score for generated text against reference text. METEOR is based on unigram precision, recall, and F1.
\[METEOR = \frac{1}{N} \times \sum_{i=1}^N \max(0, \frac{2 \cdot P_i \cdot R_i}{P_i + R_i})\]- where:
\(P_i\) is precision for unigram i.
\(R_i\) is recall for unigram i.
- CalculateEditDistance(generatedText, referenceText)[source]¶
Calculates normalized edit distance similarity between generated and reference text. Edit distance is the minimum number of operations to transform one text into another.
\[Sim = 1 - \frac{D_{lev}(gen, ref)}{\max(|gen|, |ref|)}\]- where:
\(D_{lev}(gen, ref)\) is the Levenshtein distance between generated and reference text.
\(|gen|\) and \(|ref|\) are the lengths of generated and reference text.
- CalculateJaccardSimilarity(generatedText, referenceText)[source]¶
Calculates Jaccard similarity based on word overlap between generated and reference text.
This function employs a purely statistical, lexical approach. It treats text as a “bag of words” and calculates the ratio of shared unique words to the total unique words across both texts. It is highly sensitive to exact word matches and completely ignores word order, syntax, and semantic meaning. Consequently, it will assign a low score to texts that use different synonyms to express the exact same meaning, unlike embedding-based approaches.
- Architectural Correlation Analysis:
CalculateJaccardSimilarity (Lexical): High score only if exact words match. Fails on synonyms/paraphrases.
CalculateSemanticSimilarity (Embedding): High score if meaning is preserved, regardless of word choice.
Conclusion: Use Jaccard for exact duplication checks; use Semantic for meaning preservation.
\[Jaccard = \frac{|gen \cap ref|}{|gen \cup ref|}\]- where:
\(|gen \cap ref|\) is the size of the intersection of words.
\(|gen \cup ref|\) is the size of the union of words.
- Parameters:
- Returns:
The Jaccard similarity score between 0.0 and 1.0.
- Return type:
Examples
from HMB.TextGenerationMetrics import TextGenerationMetrics # Initialize the metrics object. obj = TextGenerationMetrics() # Calculate Jaccard similarity between two texts. text1 = "The study was conducted on January 15, 2023, and included 150 participants." text2 = "The research took place on the fifteenth of January, 2023, with a total of one hundred fifty subjects." similarity = obj.CalculateJaccardSimilarity(text1, text2) # Expected output: Jaccard Similarity: 0.1200 (example value, actual may vary). print(f"Jaccard Similarity: {similarity:.4f}")
- CalculateSemanticSimilarity(semanticModel, text1, text2)[source]¶
Calculates the cosine similarity between two text embeddings using a sentence transformer model.
This function utilizes a deep learning approach, mapping texts into a high-dimensional semantic space where geometric proximity indicates meaning equivalence. Unlike lexical overlap methods, it is highly context-aware and robust to paraphrasing, capturing syntactic structures and long-range dependencies. It will assign a high score to texts that use completely different words but convey the same underlying meaning, making it ideal for evaluating humanized text where vocabulary diversity is intentionally increased.
- Architectural Correlation Analysis:
CalculateJaccardSimilarity (Lexical): High score only if exact words match. Fails on synonyms/paraphrases.
CalculateSemanticSimilarity (Embedding): High score if meaning is preserved, regardless of word choice.
Conclusion: Use Jaccard for exact duplication checks; use Semantic for meaning preservation.
- Parameters:
- Returns:
The cosine similarity score between 0.0 and 1.0. Returns 0.05 on failure.
- Return type:
Examples
from sentence_transformers import SentenceTransformer from HMB.TextGenerationMetrics import TextGenerationMetrics # Load a pre-trained sentence transformer model. model = SentenceTransformer("all-MiniLM-L6-v2") # Initialize the metrics object. obj = TextGenerationMetrics() # Calculate semantic similarity between two texts. text1 = "The study was conducted on January 15, 2023, and included 150 participants." text2 = "The research took place on the fifteenth of January, 2023, with a total of one hundred fifty subjects." similarity = obj.CalculateSemanticSimilarity(model, text1, text2) # Expected output: Semantic Similarity: 0.7649 (example value, actual may vary). print(f"Semantic Similarity: {similarity:.4f}")
- CalculateLengthRatio(generatedText, referenceText)[source]¶
Calculates the ratio of generated text length to reference text length.
\[LengthRatio = \frac{|gen|}{|ref|}\]- where:
\(|gen|\) is the length of the generated text.
\(|ref|\) is the length of the reference text.
- CalculatePerplexity(generatedTokens, referenceTokens)[source]¶
Calculates the perplexity of generated tokens against reference tokens using a statistical approach.
Architectural Paradigm & Differences: This function employs a statistical, frequency-based approach, relying on empirical token counts from a provided reference corpus. It is context-agnostic; it treats tokens as independent events (a bag-of-words model) and calculates probabilities based solely on global frequency distributions, ignoring word order entirely.
Perplexity measures how well the generated text predicts the reference text.
\[Perplexity = \exp\left(-\frac{1}{N} \times \sum_{i=1}^N \log P(token_i)\right)\]- where:
\(N\) is the number of generated tokens.
\(P(token_i)\) is the probability of token i in the reference text.
- Parameters:
- Returns:
Perplexity score.
- Return type:
Examples
from HMB.TextGenerationMetrics import TextGenerationMetrics # Example reference and generated tokens. referenceTokens = "The cat sat on the mat".split() generatedTokens = "The cat sat on the mat".split() # Initialize the metrics object. obj = TextGenerationMetrics() # Calculate perplexity. perplexity = obj.CalculatePerplexity(generatedTokens, referenceTokens) print(f"Statistical Perplexity: {perplexity:.4f}")
- CalculateNeuralPerplexity(critTokenizer, critModel, text, context='')[source]¶
Calculates the perplexity score of the provided text, conditioned on an optional context, using a deep learning approach.
Architectural Paradigm & Differences: This function utilizes a deep learning approach, specifically a Transformer-based causal language model (such as GPT-2), to compute perplexity. It is highly context-aware. By processing the entire sequence through a neural network, it captures syntactic structures, word order, and long-range dependencies.
Perplexity is defined as the exponent of the average negative log-likelihood of the tokens. In this implementation, the loss is computed exclusively over the target text portion by masking the context tokens. This ensures the score strictly reflects the predictability of the new text given the preceding context, rather than rewarding predictable context tokens.
- Heuristic Interpretation:
Lower scores (e.g., < 20.0) often indicate AI-generated or highly predictable text.
Moderate scores (e.g., 20.0 to 50.0) suggest natural human-like variability.
Extremely high scores (e.g., > 150.0) may indicate incoherent, fragmented, or gibberish text.
Note: Perplexity is inherently model-dependent and architecture-specific. It should be evaluated alongside complementary metrics such as semantic similarity, fluency scoring, and stylistic analysis to ensure robust text quality detection.
- Parameters:
critTokenizer (transformers.PreTrainedTokenizer) – A pre-initialized tokenizer aligned with the critic model for consistent tokenization.
critModel (transformers.PreTrainedModel) – A pre-initialized causal language model used to compute the cross-entropy loss and subsequent perplexity.
text (str) – The target input text for which to calculate the perplexity score.
context (str) – An optional preceding context string to condition the calculation.
- Returns:
The calculated perplexity score. Returns 0.0 for empty inputs or invalid states.
- Return type:
Examples
from transformers import GPT2LMHeadModel, GPT2Tokenizer from HMB.TextGenerationMetrics import TextGenerationMetrics # Define the critic model name (e.g., GPT-2). critModelName = "gpt2" # Initialize the tokenizer from the pretrained model. critTokenizer = GPT2Tokenizer.from_pretrained(critModelName) # Initialize the model from the pretrained weights. critModel = GPT2LMHeadModel.from_pretrained(critModelName) # Ensure the tokenizer has a pad token to avoid errors. if (critTokenizer.pad_token is None): # Set the pad token to the end of sequence token. critTokenizer.pad_token = critTokenizer.eos_token text ="The cat sat on the mat." context = "The cat sat on the mat. The cat is happy." # Execute the context-aware neural function with the current strings. obj = TextGenerationMetrics() neuralPPL = obj.CalculateNeuralPerplexity(critTokenizer, critModel, text, context) print(f"Neural Context-Aware Perplexity: {neuralPPL:.4f}")
- CalculateFluencyScore(genTokenizer, genModel, text, context='')[source]¶
Calculates a normalized fluency score for the provided text using model-based perplexity.
Architectural Paradigm & Heuristics: This function converts raw perplexity into a bounded fluency score between 0.0 and 1.0 by applying a multi-stage heuristic pipeline. Unlike raw perplexity, which is unbounded and model-dependent, this score provides an interpretable, normalized metric suitable for ranking candidate text variations during the humanization process.
- Heuristics applied:
Hard Length Rejection: Texts with fewer than 3 words receive a minimal score.
Linear Perplexity Inversion: Maps typical perplexity ranges to a 0.0–1.0 range.
Aggressive Short-Text Penalty: Caps the maximum score for texts under 8 words.
Valid Token Scaling: Requires approximately 15 tokens for full score confidence.
Repetition Penalty: Detects and penalizes texts with excessive word repetition.
Punctuation Penalty: Reduces the score for texts lacking terminal punctuation.
Capitalization Penalty: Reduces the score for texts not starting with a capital letter.
- Parameters:
genTokenizer (transformers.PreTrainedTokenizer) – A pre-initialized tokenizer aligned with the model for consistent tokenization.
genModel (transformers.PreTrainedModel) – A pre-initialized language model used to compute the cross-entropy loss for perplexity estimation.
text (str) – The candidate text to evaluate for fluency.
context (str) – Optional preceding context to condition the loss calculation.
- Returns:
A normalized fluency score between 0.0 (incoherent) and 1.0 (perfectly fluent).
- Return type:
Examples
from transformers import GPT2LMHeadModel, GPT2Tokenizer from HMB.TextGenerationMetrics import TextGenerationMetrics # Load a pre-trained GPT-2 model and tokenizer. modelName = "gpt2" tokenizer = GPT2Tokenizer.from_pretrained(modelName) model = GPT2LMHeadModel.from_pretrained(modelName) # Ensure the tokenizer has a pad token to avoid errors. if (tokenizer.pad_token is None): tokenizer.pad_token = tokenizer.eos_token # Initialize the metrics object. metrics = TextGenerationMetrics() # Calculate fluency score for a candidate text with optional context. candidateText = "The cat sat on the mat." contextText = "In the living room, the cat was very comfortable." fluencyScore = metrics.CalculateFluencyScore(tokenizer, model, candidateText, contextText) print(f"Fluency Score: {fluencyScore:.4f}")
- CalculateAccuracy(generatedText, referenceText)[source]¶
Calculates accuracy of generated text against reference text. Accuracy is the proportion of matching tokens.
\[Accuracy = \frac{|\{gen \cap ref\}|}{|ref|}\]- where:
\(|\{gen \cap ref\}|\) is the number of matching tokens.
\(|ref|\) is the total number of tokens in the reference text.
- CalculateF1Score(generatedText, referenceText)[source]¶
Calculates F1 score of generated text against reference text. F1 is the harmonic mean of precision and recall.
\[F1 = \frac{2 \times P \times R}{P+R}\]- where:
\(P\) is precision (proportion of generated tokens in reference).
\(R\) is recall (proportion of reference tokens in generated).
- CalculateCHRF(generatedText, referenceText)[source]¶
Calculates CHRF score for generated text against reference text. CHRF is the character n-gram F-score.
\[CHRF = \frac{2 \cdot P \cdot R}{P + R}\]- where:
\(P\) is precision (proportion of n-grams in generated text).
\(R\) is recall (proportion of n-grams in reference text).
- CalculateRepetitionRate(generatedText, n=3)[source]¶
Calculates the repetition rate of n-grams in the generated text. Repetition rate is the proportion of n-grams that are repeated.
\[RepetitionRate = 1 - \frac{|unique\ ngrams|}{|total\ ngrams|}\]- where:
\(|unique\ ngrams|\) is the number of unique n-grams.
\(|total\ ngrams|\) is the total number of n-grams.
- CalculateLexicalDiversity(generatedText)[source]¶
Calculates lexical diversity of the generated text. Lexical diversity is the ratio of unique words to total words.
\[LexicalDiversity = \frac{|unique\ words|}{|total\ words|}\]- where:
\(|unique\ words|\) is the number of unique words.
\(|total\ words|\) is the total number of words.
- CalculateReadabilityScore(generatedText)[source]¶
Calculates the Flesch-Kincaid grade level of the generated text. Lower score = easier readability.
\[FleschKincaid = 0.39 \cdot \frac{total\ words}{total\ sentences} + 11.8 \cdot \frac{total\ syllables}{total\ words} - 15.59\]- where:
\(total\ words\) is the number of words.
\(total\ sentences\) is the number of sentences.
\(total\ syllables\) is the number of syllables.
- CalculateInformationDensity(generatedText)[source]¶
Calculates information density of the generated text. Information density is the ratio of content words to total words.
\[InformationDensity = \frac{|content\ words|}{|tokens|}\]- where:
\(|content\ words|\) is the number of content words (nouns, verbs, adjectives, adverbs).
\(|tokens|\) is the total number of tokens.
- CalculateHallucinationRate(generatedText, referenceText)[source]¶
Calculates hallucination rate of generated text against reference text. Hallucination rate is the proportion of words in the generated text not in the reference text.
\[HallucinationRate = \frac{|gen \setminus ref|}{|gen|}\]- where:
\(|gen \setminus ref|\) is the number of words in the generated text not in the reference text.
\(|gen|\) is the total number of words in the generated text (not counting hallucinations).
- CalculateOmissionRate(generatedText, referenceText)[source]¶
Calculates omission rate of generated text against reference text. Omission rate is the proportion of words in the reference text not in the generated text.
\[OmissionRate = \frac{|ref \setminus gen|}{|ref|}\]- where:
\(|ref \setminus gen|\) is the number of words in the reference text not in the generated text.
\(|ref|\) is the total number of words in the reference text.
- CalculateFactualityScore(generatedText, referenceText)[source]¶
Calculates factuality score based on hallucination and omission rates. Factuality combines precision and recall of content overlap. Factuality score is 1.0 for perfect factuality (no hallucinations or omissions).
\[F1 = \frac{2 \times P \times R}{P+R}\]- where:
\(P\) is precision (1.0 - hallucination rate).
\(R\) is recall (1.0 - omission rate).