ArabicTextHelper Module

Helpers for Arabic text processing.

class HMB.ArabicTextHelper.ArabicTextHelper[source]

Bases: TextHelper

ArabicTextHelper: Utilities for Arabic text preprocessing.

This helper provides common Arabic-specific text preprocessing routines such as regex-based normalization and cleaning, ISRI stemming, Qalsadi lemmatization, and stopword removal. The methods are thin wrappers around NLTK and qalsadi functionality where applicable and try to preserve the original behavior while providing clear documentation.

Notes

  • Some methods require external packages (nltk, qalsadi) to be installed.

  • Methods accept and return lists of strings for batch processing.

  • Can be installed via pip install emoji nltk qalsadi.

ArabicRegexPreprocessing(data, normalizeChars=True, removeHashes=True, removeUsernames=True, removeEmojis=True, removeLinks=True, removeDiacritics=True)[source]

Apply regex-based normalization and cleaning to Arabic text documents.

Parameters:

  • data (list-like) – Iterable of strings to preprocess.

  • normalizeChars (bool) – Normalize common Arabic character variants.

  • removeHashes (bool) – Remove or normalize hashtags (#tag).

  • removeUsernames (bool) – Remove @user mentions.

  • removeEmojis (bool) – Remove emoji characters.

  • removeLinks (bool) – Remove URLs and web-like tokens.

  • removeDiacritics (bool) – Remove Arabic diacritics (tashkeel).

Returns:

Preprocessed documents as strings.

Return type:

list

Note

This function performs character normalization, optional diacritics stripping, removal of usernames/hashtags/links/emojis, and token cleaning (removing non-Arabic letters and digits).

ArabicISRIStemmerPreprocessing(data)[source]

Apply ISRI stemming to each document using NLTK’s ISRIStemmer.

Parameters:

data (list-like) – Iterable of strings to stem. Each string is tokenized using the helper’s SentenceTokeize method before stemming.

Returns:

Stemmed documents as joined strings.

Return type:

list

ArabicQalsadiLemmatizerPreprocessing(data)[source]

Lemmatize Arabic text using the Qalsadi lemmatizer.

Parameters:

data (list-like) – Iterable of strings to lemmatize.

Returns:

Lemmatized documents as joined strings.

Return type:

list

Note

Requires the qalsadi package to be installed.

ArabicStopwordsRemovalPreprocessing(data)[source]

Remove Arabic stopwords from tokenized documents.

Parameters:

data (list-like) – Iterable of strings to remove stopwords from. Each string is tokenized using the helper’s SentenceTokeize method.

Returns:

Documents with stopwords removed and joined back into strings.

Return type:

list

RemovePunctuations(text, extra_punct=None)[source]

Remove punctuation characters from a text string, including Arabic punctuation.

Parameters:

  • text (str) – Input string.

  • extra_punct (str or None) – Additional characters to treat as punctuation.

Returns:

Text with punctuation replaced by spaces.

Return type:

str

RemoveDiacritics(text)[source]

Remove Arabic diacritics (tashkeel) from a string.

Parameters:

text (str) – Input Arabic text.

Returns:

Text without Arabic diacritics.

Return type:

str

NormalizeArabic(text, removeElongation=True, normalizeHamza=True)[source]

Normalize common Arabic characters and optionally remove elongation.

Parameters:

  • text (str) – Input Arabic text.

  • removeElongation (bool) – If True, collapse repeated characters (e.g., “جممميل” → “جميل”).

  • normalizeHamza (bool) – If True, normalize hamza variants to bare alef/hamza forms.

Returns:

Normalized text.

Return type:

str

ArabicToEnglishNumbers(text)[source]

Convert Arabic-Indic and Persian digits to Western digits (0-9).

Parameters:

text (str) – Input string containing digits.

Returns:

String where Arabic/Persian digits are mapped to ASCII digits.

Return type:

str

IsArabic(text)[source]

Heuristic check whether a string contains Arabic characters.

Parameters:

text (str) – Input string.

Returns:

True if any Arabic-range character is present, False otherwise.

Return type:

bool

TokenizeArabic(text)[source]

Simple Arabic tokenizer: extract contiguous Arabic letter sequences as tokens.

Parameters:

text (str) – Input Arabic text.

Returns:

List of Arabic word tokens.

Return type:

list

Note

This is an intentionally lightweight tokenizer. For advanced tokenization consider using language-specific tokenizers (Farasa, Camel Tools, etc.).

CleanAndNormalize(text, removeDiacritics=True, removePunct=True, normalizeHamza=True, removeElongation=True, convertNumbers=True)[source]

Run a compact cleaning pipeline on a single text string.

This helper composes commonly used operations: normalize characters, optionally remove diacritics, strip punctuation, convert Arabic numbers to Western digits, collapse whitespace, and trim the result.

Parameters:

  • text (str) – Input text.

  • removeDiacritics (bool) – Remove Arabic diacritics if True.

  • removePunct (bool) – Remove punctuation if True.

  • normalizeHamza (bool) – Normalize hamza/alef variants if True.

  • removeElongation (bool) – Remove repeated character elongation if True.

  • convertNumbers (bool) – Convert Arabic/Persian digits to ASCII digits if True.

Returns:

Cleaned and normalized text.

Return type:

str

StripNonArabic(text, keepSpaces=True)[source]

Remove characters that are not in the Arabic Unicode block.

Parameters:

  • text (str) – Input string.

  • keepSpaces (bool) – If True, preserve spaces between words; otherwise remove them.

Returns:

String containing only Arabic letters (and spaces if requested).

Return type:

str

ArabicCharRatio(text)[source]

Compute the ratio of Arabic characters to all characters in the string.

Parameters:

text (str) – Input string.

Returns:

Fraction in [0,1] of characters that are Arabic. Returns 0.0 for empty input.

Return type:

float

ArabicWordCount(text)[source]

Count Arabic words/tokens in a string using the lightweight tokenizer.

Parameters:

text (str) – Input string.

Returns:

Number of Arabic word tokens found.

Return type:

int

GetArabicCharNGrams(text, n=3)[source]

Return character n-grams from the Arabic-only content of the text.

Parameters:

  • text (str) – Input string.

  • n (int) – Size of the n-grams (default 3).

Returns:

List of character n-gram strings. Returns empty list if no Arabic text found.

Return type:

list

Note

This operates on the concatenated Arabic tokens (spaces removed) to produce contiguous character n-grams.

RemoveEmojis(text)[source]

Remove emoji characters from a string.

Parameters:

text (str) – Input string.

Returns:

String with emojis removed.

Return type:

str