TextHelper Module¶
Provides helper functions for text processing, cleaning, and analysis.
- class HMB.TextHelper.HuggingFaceTextSummarizer(modelName='facebook/bart-large-cnn', maxLength=130, minLength=30, maxInputLength=1024)[source]¶
Bases:
objectFlexible text summarization using Hugging Face Transformers.
This class provides a convenient interface for abstractive text summarization using pre-trained transformer models (e.g., BART, T5) via the Hugging Face pipeline. It automatically handles device selection (CPU/GPU), input chunking for long texts, and dynamic adjustment of summary length to avoid common warnings. The class is suitable for both short and long documents, and can be reused for multiple summarization tasks without reloading the model.
- Features:
Supports any Hugging Face summarization model (default: facebook/bart-large-cnn).
Automatically uses GPU if available, otherwise falls back to CPU.
Handles long texts by splitting into manageable chunks and summarizing each chunk.
Dynamically sets max_length and min_length based on tokenized input length to avoid warnings.
Returns a single concatenated summary for the entire input.
Easy to customize summary length and chunk size.
- Parameters:
modelName (str) – Name of the Hugging Face model to use for summarization. Default is “facebook/bart-large-cnn”.
maxLength (int) – Maximum length of the summary (in tokens). Default is 130.
minLength (int) – Minimum length of the summary (in tokens). Default is 30.
maxInputLength (int) – Maximum input length (in characters) before chunking. Default is 1024.
Notes
For best results, choose a model appropriate for your language and domain.
If your input text is very long, the class will split it into chunks and summarize each chunk separately.
The final summary is a concatenation of all chunk summaries.
maxLength and minLength are automatically adjusted to avoid warnings about input length.
You can customize chunk size by changing maxInputLength.
The class is thread-safe for repeated use.
Examples
from HMB.TextHelper import HuggingFaceTextSummarizer text = "Your long text to summarize goes here..." summarizer = HuggingFaceTextSummarizer( modelName="facebook/bart-large-cnn", # Use a pre-trained summarization model from Hugging Face. maxLength=130, # Maximum summary length in tokens. minLength=30, # Minimum summary length in tokens. maxInputLength=1024 # Maximum input length in characters before chunking. ) summary = summarizer.Summarize(text) print(summary)
Initialize the Summarizer with the specified model and parameters.
- Parameters:
modelName (str) – Hugging Face model name for summarization (e.g., “facebook/bart-large-cnn”, “t5-base”).
maxLength (int) – Maximum summary length in tokens (will be capped by input length).
minLength (int) – Minimum summary length in tokens (will be capped by input length).
maxInputLength (int) – Maximum input length in characters before chunking. Longer texts are split into chunks.
Notes
The model is loaded once and reused for all summarization calls.
Device selection is automatic: uses GPU if available, otherwise CPU.
- __init__(modelName='facebook/bart-large-cnn', maxLength=130, minLength=30, maxInputLength=1024)[source]¶
Initialize the Summarizer with the specified model and parameters.
- Parameters:
modelName (str) – Hugging Face model name for summarization (e.g., “facebook/bart-large-cnn”, “t5-base”).
maxLength (int) – Maximum summary length in tokens (will be capped by input length).
minLength (int) – Minimum summary length in tokens (will be capped by input length).
maxInputLength (int) – Maximum input length in characters before chunking. Longer texts are split into chunks.
Notes
The model is loaded once and reused for all summarization calls.
Device selection is automatic: uses GPU if available, otherwise CPU.
- Summarize(text)[source]¶
Summarize the input text using the loaded transformer model.
- Parameters:
text (str) – The text to summarize. Can be short or long (long texts are chunked automatically).
- Returns:
The summarized text. For long inputs, returns a concatenation of chunk summaries.
- Return type:
- Behavior:
If the input text exceeds maxInputLength, it is split into chunks of 1000 characters.
Each chunk is summarized separately, with maxLength and min_length set based on tokenized input length.
Warnings about maxLength/inputLength mismatch are avoided by dynamic adjustment.
If a chunk is too short, it is returned as-is without summarization.
The final output is a single string containing all chunk summaries.
Example:¶
summarizer = HuggingFaceTextSummarizer() summary = summarizer.Summarize("Very long text ...") print(summary)
Notes
For best results, clean your input text before summarization.
You can customize summary length by changing maxLength and minLength.
The method is robust to empty or very short inputs.
- class HMB.TextHelper.TextHelper[source]¶
Bases:
objectTextHelper: Small collection of common text-processing utilities.
This class provides thin wrappers around NLTK, gensim and scikit-learn helpers for tokenization, stopword removal, stemming, lemmatization, simple POS/chunking helpers, and common vectorization examples.
Notes
Methods are convenience wrappers and try to preserve original behavior.
Many methods assume NLTK corpora (punkt, stopwords) are available.
You need to pip install nltk, gensim, wordcloud, gtts, and scikit-learn to use this class.
- WordTokenize(document)[source]¶
Split a document into sentences using NLTK’s sent_tokenize.
Note: method name is historical; it returns sentence-level tokens.
- RemoveStopWords(document)[source]¶
Remove English stopwords from an iterable of tokens.
- Parameters:
document (Iterable) – Iterable of token strings.
- Returns:
Tokens with stopwords removed.
- Return type:
- Chunking(document)[source]¶
Named-entity chunking helper using NLTK’s ne_chunk.
- Parameters:
document (Iterable[str]) – Iterable of token strings (POS-tagged preferred).
- Returns:
Chunked parse tree produced by ne_chunk.
- Return type:
nltk.Tree
- BagOfWords(document)[source]¶
Vectorize documents using CountVectorizer and return the vectorizer and dense matrix.
- Parameters:
- Returns:
The fitted CountVectorizer instance and the dense array.
- Return type:
(vectorizer, numpy.ndarray)
- TFIDF(document)[source]¶
Vectorize documents using TfidfVectorizer and return the vectorizer and dense matrix.
- Parameters:
- Returns:
The fitted TfidfVectorizer instance and the dense array.
- Return type:
(vectorizer, numpy.ndarray)
- Doc2Vec(document, saveModel=False, savePath='d2v.model', epochs=100, vecSize=20, alpha=0.025)[source]¶
Train a Doc2Vec model on provided documents and return the model.
- LDA(document)[source]¶
Fit a simple LDA model using gensim’s LdaModel and return the model and topics.
- TextSummarization(document)[source]¶
Extractive summarization wrapper using gensim.summarization.summarize.
- CosineSimilarity(text1, text2)[source]¶
Demonstrate cosine similarity between two example texts and return the matrix.
- Parameters:
- Returns:
Cosine similarity matrix.
- Return type:
- JaccardSimilarity(text1, text2)[source]¶
Demonstrate Jaccard similarity between two example texts and return the score.
- EuclideanDistance(text1, text2)[source]¶
Demonstrate Euclidean distance between two example texts and return the matrix.
- Parameters:
- Returns:
Euclidean distance matrix.
- Return type:
- ManhattanDistance(text1, text2)[source]¶
Demonstrate Manhattan (L1) distance between two example texts and return the matrix.
- Parameters:
- Returns:
Manhattan distance matrix.
- Return type:
- WordCloud(document)[source]¶
Generate a word cloud object for the given text and return it.
- Parameters:
document (str) – Input text from which to build the word cloud.
- Returns:
The generated WordCloud instance (from wordcloud package).
- Return type:
WordCloud
- TextToSpeech(document, outFile='good.mp3')[source]¶
Convert text to speech using gTTS, save to an MP3 file, and return filename.
- SentimentAnalysis(document)[source]¶
Perform sentiment analysis using TextBlob and return the sentiment object.
- SpellingCorrection(document)[source]¶
Perform spelling correction using TextBlob and return corrected string.
- LanguageTranslation(document, to='es')[source]¶
Translate text using TextBlob’s translation wrapper and return the result.
- class HMB.TextHelper.TextSemanticShield(nlpModelName='en_core_web_sm')[source]¶
Bases:
objectExtracts named entities, domain-specific terms, and noun chunks via spaCy, then replaces them with high-entropy cryptographic tokens («K_<uuid>») so downstream transformations cannot corrupt technical terminology.
Examples
from HMB.TextHelper import TextSemanticShield text = "Alice works at OpenAI in San Francisco." shield = TextSemanticShield("en_core_web_sm") maskedText, entityMap = shield.ApplyCryptographicMask(text) print(maskedText) # e.g., "«K_1a2b3c4d» works at «K_5e6f7g8h» in «K_9i0j1k2l»." restoredText = shield.RestoreEntities(maskedText, entityMap) print(restoredText) # "Alice works at OpenAI in San Francisco."
Initialize the SemanticShield with a spaCy language model for entity extraction and masking.
- Parameters:
nlpModelName (-) – The name of the spaCy language model to load. Defaults to “en_core_web_sm”.
- __init__(nlpModelName='en_core_web_sm')[source]¶
Initialize the SemanticShield with a spaCy language model for entity extraction and masking.
- Parameters:
nlpModelName (-) – The name of the spaCy language model to load. Defaults to “en_core_web_sm”.
- ApplyCryptographicMask(text)[source]¶
Apply cryptographic masking to named entities, technical terms, and noun chunks in the input text. This method replaces detected entities with unique high-entropy tokens to prevent downstream transformations from altering critical terminology. It returns the masked text along with a mapping of tokens to original entities for restoration.
- Parameters:
text (-) – The input text to be processed and masked.
- Returns:
A tuple containing the masked text and a dictionary mapping cryptographic tokens to their corresponding original entities.
- Return type:
Tuple[str, Dict[str, str]]
- RestoreEntities(text, entityMap)[source]¶
Restore the original entities in the text by replacing cryptographic placeholders with their corresponding original entity texts. This method uses the mapping of cryptographic tokens to original entities to reconstruct the original text from the masked version.
- Parameters:
text (-) – The masked text containing cryptographic placeholders.
entityMap (-) – A dictionary mapping cryptographic tokens to their corresponding original entity texts.
- Returns:
The fully restored text with all original entities intact.
- Return type:
str
- HMB.TextHelper.CleanText(text, removeNonAscii=True, lowercase=True, removeSpecialChars=True, normalizeWhitespace=True, handleContractions=True, lemmatize=False, removeStopwords=False, removeCommonWords=False, numOfCommonWords=10, removeNonEnglishWords=False)[source]¶
Cleans the input text based on specified options. It applies multiple text normalization techniques including (1) removing non-ASCII characters, (2) converting to lowercase, (3) removing special characters and punctuation, (4) normalizing whitespace, (5) expanding contractions, (6) lemmatizing words, (7) removing stopwords, (8) removing common words, and (9) removing non-English words.
- Parameters:
text (str) – The raw input text to be cleaned.
removeNonAscii (bool) – Whether to remove non-ASCII characters. Default is True.
lowercase (bool) – Whether to convert text to lowercase. Default is True.
removeSpecialChars (bool) – Whether to remove special characters and punctuation. Default is True.
normalizeWhitespace (bool) – Whether to replace multiple spaces with a single space. Default is True.
handleContractions (bool) – Whether to expand contractions (e.g., “don’t” → “do not”). Default is True.
lemmatize (bool) – Whether to lemmatize words (reduce to base form). Default is False.
removeStopwords (bool) – Whether to remove common stop words. Default is False.
removeCommonWords (bool) – Whether to remove common words. Default is False.
numOfCommonWords (int) – Number of common words to remove if removeCommonWords is True. Default is 10.
removeNonEnglishWords (bool) – Whether to remove non-English words. Default is False.
- Returns:
The cleaned and normalized text.
- Return type:
Examples
import HMB.TextHelper as th raw = "I can't believe it's not butter! " cleaned = th.CleanText( raw, removeNonAscii=True, lowercase=True, removeSpecialChars=True, normalizeWhitespace=True, handleContractions=True, lemmatize=True, removeStopwords=True, removeCommonWords=False, numOfCommonWords=10, removeNonEnglishWords=False, ) print(cleaned)
- HMB.TextHelper.IsSyntacticallyValid(nlpModel, text)[source]¶
Perform a robust heuristic check to determine the syntactic validity of a given text.
This function evaluates whether a string resembles a grammatically correct English sentence by applying a series of structural and lexical heuristics. It is designed to filter out gibberish, repetitive loops, and nonsensical word salads that might otherwise pass basic length checks. The validation pipeline includes checks for proper capitalization, terminal punctuation, the presence of core grammatical components (verbs and subjects), structural anomaly detection, excessive word repetition, and a baseline stop word ratio to ensure natural language flow.
- Heuristics applied:
Length Check: The text must contain at least three words.
Capitalization: The text must start with a capital letter.
Terminal Punctuation: The text must end with a single standard punctuation mark.
Verb Presence: The text must contain at least one verb or auxiliary verb.
Subject Presence: The text must contain a nominal subject (active, passive, or clausal).
Word Salad Detection: Checks for nonsensical sequences like noun-determiner-preposition.
Repetition Check: Flags text where a single word dominates (>40% of tokens).
Stop Word Ratio: Ensures a reasonable proportion of common stop words in longer texts.
- Parameters:
nlpModel (spacy.Language) – A loaded spaCy language model instance.
text (str) – The input text string to evaluate.
- Returns:
True if the text passes all checks, False otherwise.
- Return type:
- Raises:
ValueError – If the provided nlpModel is None.
Examples
import spacy from HMB.TextHelper import IsSyntacticallyValid # Load a spaCy English model (e.g., "en_core_web_sm"). nlp = spacy.load("en_core_web_sm") # Example text to validate. text = "The quick brown fox jumps over the lazy dog." # Check if the text is syntactically valid. isValid = IsSyntacticallyValid(nlp, text) print(f"Is the text syntactically valid? {isValid}") # Expected output: True. isValid = IsSyntacticallyValid(nlp, "Mat the on sat cat the.") print(f"Is the text syntactically valid? {isValid}") # Expected output: False. isValid = IsSyntacticallyValid(nlp, "The the the the the cat.") print(f"Is the text syntactically valid? {isValid}") # Expected output: False. isValid = IsSyntacticallyValid(nlp, "sat on the mat.") print(f"Is the text syntactically valid? {isValid}") # Expected output: False. isValid = IsSyntacticallyValid(nlp, "The cat sat on the mat..") print(f"Is the text syntactically valid? {isValid}") # Expected output: False.
- HMB.TextHelper.CalculateDynamicTemperature(nlpModel, text, baseTemp=1.0)[source]¶
Analyzes sentence content and returns the optimal dynamic temperature for generation.
This function evaluates the text to determine if it is factual, mixed, or descriptive, and adjusts the base temperature accordingly to prevent hallucinations in factual text while encouraging creativity in descriptive text.
- Heuristics applied:
Numeric Presence: Detects numbers and quantities.
Temporal Entities: Detects dates and times.
Proper Noun Density: Measures the ratio of proper nouns to total tokens.
Lexical Density: Measures the ratio of nouns and adjectives to total tokens.
Passive Voice: Detects the presence of passive voice constructions.
Average Word Length: Technical texts typically use longer, more complex words.
Punctuation Density: Formal texts often contain more complex punctuation marks.
- Parameters:
- Returns:
The dynamically adjusted temperature based on the content analysis.
- Return type:
- Raises:
ValueError – If the provided nlpModel is None.
Examples
import spacy from HMB.TextHelper import CalculateDynamicTemperature # Load a spaCy English model (e.g., "en_core_web_sm"). nlp = spacy.load("en_core_web_sm") # Example text to analyze. text = "The study was conducted on January 15, 2023, and included 150 participants." # Calculate the dynamic temperature for the text. dynamicTemp = CalculateDynamicTemperature(nlp, text, baseTemp=1.0) print(f"Dynamic temperature: {dynamicTemp:.2f}") # Expected output: < 1.0 for factual content. text = "Once upon a time, in a land far away, there lived a brave knight." dynamicTemp = CalculateDynamicTemperature(nlp, text, baseTemp=1.0) print(f"Dynamic temperature: {dynamicTemp:.2f}") # Expected output: > 1.0 for descriptive content.
- HMB.TextHelper.CalculateContextualSimilarity(semanticModel, originalToken, synonym, context)[source]¶
Calculates how semantically similar a synonym is to the surrounding context.
This function uses a sentence transformer model to encode the context with the original token and with the synonym, and then computes the cosine similarity between the two embeddings. The context should contain a placeholder “_____” where the token is replaced, allowing for a direct comparison of how well the original token and the synonym fit within the same context. If the placeholder is not found, the function will construct simple sentences for comparison, but this may lead to less accurate similarity scores.
- Heuristics applied:
Identity Check: Returns 1.0 immediately if the original token and synonym are identical.
Input Validation: Returns a neutral score for empty inputs or missing models.
Placeholder Replacement: Uses the provided placeholder to maintain exact syntactic structure.
Fallback Construction: Constructs a neutral sentence if the placeholder is missing.
Embedding Normalization: Ensures tensors are 2D before computing cosine similarity.
- Parameters:
semanticModel (sentence_transformers.SentenceTransformer) – A pre-initialized sentence transformer model.
originalToken (str) – The original word or phrase that is being replaced.
synonym (str) – The candidate synonym that is being evaluated for contextual fit.
context (str) – The surrounding text containing a placeholder “_____” where the original token is located. This allows the method to compare the semantic fit of the original token and the synonym within the same context. If the placeholder is not present, the method will attempt a fallback comparison, but for best results, the context should include “_____” to indicate where the token is being replaced.
- Returns:
A similarity score between 0.0 and 1.0 indicating how well the synonym fits semantically within the context compared to the original token. Higher scores suggest that the synonym is more consistent with the surrounding text, while lower scores indicate that the synonym may not fit as well within the context. This score can be used to filter or rank synonym replacements to ensure they maintain the intended meaning and coherence of the text.
- Return type:
Examples
from sentence_transformers import SentenceTransformer from HMB.TextHelper import CalculateContextualSimilarity # Load a pre-trained sentence transformer model. model = SentenceTransformer("all-MiniLM-L6-v2") # Define the original token, synonym, and context with a placeholder. originalToken = "happy" synonym = "joyful" context = "She felt very _____ after receiving the good news." # Calculate the contextual similarity score. similarityScore = CalculateContextualSimilarity(model, originalToken, synonym, context) # Expected output: A float value between 0.0 and 1.0 indicating the semantic similarity. print(f"Contextual similarity score: {similarityScore:.4f}")
- HMB.TextHelper.GetContextualSynonym(nlpModel, maskPipeline, stemmer, word, context)[source]¶
Retrieves a contextual synonym for a specified word based on the provided surrounding text.
- This function employs a multi-stage strategy to ensure high-quality synonym replacement:
Validation: Verifies the target word (or its morphological variant) exists within the context.
Inference: Utilizes a Masked Language Model (e.g., RoBERTa) to predict semantically appropriate substitutions.
Filtering: Excludes exact matches and morphological variants using lemmatization and stemming.
Fallback: If the model fails to identify a distinct synonym, a lexical synonym is retrieved from WordNet.
- Parameters:
nlpModel (spacy.Language) – A loaded spaCy language model instance for tokenization and lemmatization.
maskPipeline (transformers.Pipeline) – A pre-initialized masked language model pipeline for inference.
stemmer (nltk.stem.PorterStemmer) – A pre-initialized stemmer for morphological filtering.
word (str) – The target word for which a synonym is required.
context (str) – The sentence or phrase containing the target word.
- Returns:
A string representing the top predicted contextual synonym.
- Return type:
Examples
import spacy from transformers import pipeline from nltk.stem import PorterStemmer from HMB.TextHelper import GetContextualSynonym # Load a spaCy English model (e.g., "en_core_web_sm"). nlp = spacy.load("en_core_web_sm") # Initialize a masked language model pipeline (e.g., RoBERTa). maskPipeline = pipeline("fill-mask", model="roberta-base") # Initialize a Porter stemmer. stemmer = PorterStemmer() # Define the target word and context. targetWord = "happy" contextSentence = "She felt very happy after receiving the good news." # Retrieve a contextual synonym for the target word. synonym = GetContextualSynonym(nlp, maskPipeline, stemmer, targetWord, contextSentence) print(f"Contextual synonym for '{targetWord}': {synonym}")
- HMB.TextHelper.BackTranslateDeepRestructure(sourceToPivotTokenizer, sourceToPivotModel, pivotToSourceTokenizer, pivotToSourceModel, text, device='cpu')[source]¶
Performs deep restructuring via back-translation through an intermediate pivot language.
This function takes the input text, translates it from the source language to a pivot language using a provided translation model, and then translates it back to the source language using a second translation model. This process can create more significant changes in sentence structure and word choice compared to simple paraphrasing, as the translation models may rephrase sentences in ways that are more natural in the pivot language and then re-translate them back with a different structure. This is a powerful technique for generating more human-like variations of the original text while still preserving the core meaning.
- Parameters:
sourceToPivotTokenizer (transformers.PreTrainedTokenizer) – Tokenizer for the source to pivot model.
sourceToPivotModel (transformers.PreTrainedModel) – Model for translating source to pivot language.
pivotToSourceTokenizer (transformers.PreTrainedTokenizer) – Tokenizer for the pivot to source model.
pivotToSourceModel (transformers.PreTrainedModel) – Model for translating pivot to source language.
text (str) – The input text to be back-translated.
device (str) – The compute device to use for model inference.
- Returns:
The back-translated text, which has been processed through the pivot language.
- Return type:
Examples
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM from HMB.TextHelper import BackTranslateDeepRestructure # Load the source to pivot translation model and tokenizer (e.g., English to French). sourceToPivotTokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-fr") sourceToPivotModel = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-en-fr") # Load the pivot to source translation model and tokenizer (e.g., French to English). pivotToSourceTokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-fr-en") pivotToSourceModel = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-fr-en") # Define the input text to be back-translated. inputText = "The quick brown fox jumps over the lazy dog." # Perform back-translation through the pivot language. backTranslatedText = BackTranslateDeepRestructure( sourceToPivotTokenizer, sourceToPivotModel, pivotToSourceTokenizer, pivotToSourceModel, inputText, device="cpu" ) # Print the back-translated text, which may have a different structure and word choice. # Example output: Back-translated text: The fast brown fox jumps on the lazy dog. print("Back-translated text:", backTranslatedText)
- HMB.TextHelper.CalculateMaskedLikelihoodDelta(maskPipeline, context, originalWord, replacementWord, offset, errorLength)[source]¶
Calculates the probability delta between an original word and a replacement word at a specific masked position in the context using a Masked Language Model.
- Parameters:
maskPipeline (transformers.Pipeline) – The initialized fill-mask pipeline.
context (str) – The full sentence containing the error.
originalWord (str) – The original erroneous word.
replacementWord (str) – The suggested replacement word.
offset (int) – The character index where the error starts.
errorLength (int) – The character length of the error span.
- Returns:
The probability delta (P(Replacement) - P(Original)). A positive value indicates the replacement is more natural.
- Return type:
Examples
from transformers import pipeline from HMB.TextHelper import CalculateMaskedLikelihoodDelta # Initialize a masked language model pipeline (e.g., RoBERTa). maskPipeline = pipeline("fill-mask", model="roberta-base") # Define the context, original word, replacement word, and error position. context = "She felt very happy after receiving the good news." originalWord = "happy" replacementWord = "joyful" offset = context.index(originalWord) errorLength = len(originalWord) # Calculate the likelihood delta for the replacement. delta = CalculateMaskedLikelihoodDelta(maskPipeline, context, originalWord, replacementWord, offset, errorLength) print(f"Likelihood delta for replacement: {delta:.6f}")
- HMB.TextHelper.GetInflectedForm(token, targetTag)[source]¶
Retrieves a dynamically inflected morphological form of a given spaCy token.
- HMB.TextHelper.GetPastParticipleForm(token)[source]¶
Retrieves the past participle form (VBN) of a given spaCy token.
- Parameters:
token (spacy.tokens.Token) – The spaCy token to be inflected.
- Returns:
The past participle form of the word.
- Return type:
- HMB.TextHelper.GetPastTenseForm(token)[source]¶
Retrieves the simple past tense form (VBD) of a given spaCy token.
- Parameters:
token (spacy.tokens.Token) – The spaCy token to be inflected.
- Returns:
The simple past tense form of the word.
- Return type:
- HMB.TextHelper.CalculateEnglishCorrectnessScore(match, currentText, maskPipeline)[source]¶
Calculates a robust correctness score for a specific grammar match.
The score is derived from LanguageTool metadata (category, error length, confidence) and contextual checks (plural/singular agreement, masked likelihood delta).
- Parameters:
match (language_tool_python.Match) – The grammar match object.
currentText (str) – The current state of the text being evaluated.
maskPipeline (transformers.Pipeline) – The masked language model pipeline.
- Returns:
The calculated correctness score.
- Return type:
- HMB.TextHelper.ApplyLanguageToolCorrections(inputText, languageToolInstance, maskPipeline)[source]¶
Iteratively applies the highest-scoring grammar corrections to the input text.
- Parameters:
inputText (str) – The text to be corrected.
languageToolInstance (language_tool_python.LanguageTool) – The initialized tool.
maskPipeline (transformers.Pipeline) – The masked language model pipeline.
- Returns:
The fully corrected input text.
- Return type:
Examples
import language_tool_python from transformers import pipeline from HMB.TextHelper import ApplyLanguageToolCorrections # Initialize the LanguageTool instance for English. languageToolInstance = language_tool_python.LanguageTool("en-US") # Initialize a masked language model pipeline (e.g., RoBERTa). maskPipeline = pipeline("fill-mask", model="roberta-base") # Define the input text with grammatical errors. inputText = "She go to the market yesterday and buy some fruits." # Apply the grammar corrections iteratively. correctedText = ApplyLanguageToolCorrections(inputText, languageToolInstance, maskPipeline) # Print the fully corrected text. print("Corrected text:", correctedText)
- HMB.TextHelper.EnforceTemporalConsistency(inputText, nlpModel)[source]¶
Dynamically enforces past tense for verbs when past time markers are detected.
- HMB.TextHelper.CorrectGrammar(text, maskPipeline=None)[source]¶
Applies comprehensive grammar correction using LanguageTool with advanced contextual correctness scoring and dynamic temporal consistency enforcement.
- Parameters:
text (str) – The input text to be grammar-checked.
maskPipeline (transformers.Pipeline, optional) – A fill-mask pipeline for MLM scoring.
- Returns:
The grammar-corrected text.
- Return type:
Examples
from transformers import pipeline from HMB.TextHelper import CorrectGrammar # Initialize a masked language model pipeline (e.g., RoBERTa). maskPipeline = pipeline("fill-mask", model="roberta-base") # Define the text to be corrected. text = "He go to the store yesterday and buyed some milk." # Execute the correct grammar function to fix the text. correction = CorrectGrammar(text, maskPipeline) print(f"Corrected text: {correction}")