DatasetsHelper Module¶

Utilities to detect and prepare image classification datasets and to validate prepared dataset layouts (train/val/test). The module exposes the GenericImagesDatasetHandler class.

class HMB.DatasetsHelper.TabularPreprocessor(ignoreCategorical=False, numericScaler='Standard')[source]¶

Bases: object

Generic tabular data preprocessor for PyTorch classification pipelines.

Handles numeric and categorical features, missing value imputation, feature scaling, ordinal encoding, and label encoding. Artifacts can be saved/loaded via joblib for reproducibility across training and inference.

Parameters:

ignoreCategorical (bool) – If True, categorical/object columns are completely ignored during fitting and transformation. Only numeric features are processed. Defaults to False.
numericScaler (str|object|None) – Specifies the scaler to apply to numeric features.

Variables:

ignoreCategorical (bool) – Flag indicating whether categorical columns are skipped.
numericColumns (list) – List of detected numeric column names.
categoricalColumns (list) – List of detected categorical column names (empty if ignored).
numericImputer (SimpleImputer) – Imputer for numeric columns (fitted on training data).
categoricalImputer (SimpleImputer) – Imputer for categorical columns (fitted on training data).
numericScaler (sklearn-like estimator or None) – Scaler for numeric columns (fitted on training data). Can be None to disable scaling. The scaler may be a StandardScaler, MinMaxScaler, RobustScaler or any object implementing fit/transform.
categoricalEncoder (OrdinalEncoder) – Encoder for categorical columns (fitted on training data).
labelEncoder (LabelEncoder) – Encoder for target labels (fitted on training data).
_featureNames (list) – Cached ordered list of feature names used during transformation.
_isLoaded (bool) – Internal flag indicating whether artifacts were loaded or preprocessor was fitted.

Notes

The Fit method detects numeric and categorical columns, fits imputers, scalers, and encoders on the provided DataFrame.
The Transform method applies the fitted transformations to a new DataFrame and returns feature arrays and label vectors suitable for PyTorch models.
The Save and Load methods handle persistence of all artifacts to/from disk using joblib.
The IsLoaded method allows checking whether the preprocessor is ready for transformation (either fitted or loaded).
The GetFeatureNames method provides the ordered list of feature names corresponding to the transformed feature matrix columns.
The ValidateSchema method checks that a new DataFrame contains the expected columns before transformation, raising an error if there are mismatches.

Examples

# Initialize the preprocessor and ignore categorical columns.
preprocessor = TabularPreprocessor(ignoreCategorical=True, numericScaler="Standard")
preprocessor.Fit(trainDf, labelColumn="Label")

# Transform training, validation, and test DataFrames.
xTrain, yTrain = preprocessor.Transform(trainDf, labelColumn="Label")
xVal, yVal = preprocessor.Transform(valDf, labelColumn="Label")
xTest, yTest = preprocessor.Transform(testDf, labelColumn="Label")

# Save the preprocessor artifacts for later use.
preprocessor.Save("PreprocessorArtifacts")

Fit(df, labelColumn=None)[source]¶

Fit the preprocessor on a training DataFrame.

Parameters:

df (pandas.DataFrame) – The training DataFrame containing features and optional label column.
labelColumn (str|None) – Optional name of the label column to encode. If None, no label encoding is performed.

Transform(df, labelColumn=None)[source]¶

Parameters:

df (DataFrame)
labelColumn (str | None)

Save(path)[source]¶

Parameters:: path (str)

Load(path)[source]¶

Parameters:: path (str)

IsLoaded()[source]¶

Return type:: bool

GetFeatureNames()[source]¶

ValidateSchema(df)[source]¶

Return type:: bool
Parameters:: df (DataFrame)

class HMB.DatasetsHelper.RawImageFolder(root, rootType='str', categories=None)[source]¶

Bases: object

Lightweight loader to get image paths and labels without transforms.

Variables:

samples (list) – List of tuples (imagePath, labelIndex) for all images in the dataset.
classToIdx (dict) – Mapping of class names to integer indices.
classes (list) – Sorted list of class names detected in the dataset.

Notes

The constructor scans the provided root directory for subdirectories (each representing a class) and collects image file paths along with their corresponding class labels.
Only files with extensions matching those in IMAGE_SUFFIXES are considered as valid images.
The __getitem__ method returns the file path and label index for a given sample index, allowing for lazy loading of images.

Initialize the RawImageFolder by scanning the root directory for class subfolders and image files.

Parameters:

root (str|dict) – If rootType is “str”, this should be a string path to the dataset root directory containing class subfolders. If rootType is “dict”, this should be a dictionary mapping split names (“train”, “val”, “test”) to their respective root directories.
rootType (str) – Type of the root parameter, either “str” for a single directory or “dict” for split-based directories. Default is “str”.
categories (list|None) – Optional list of category names to include. If None, all subdirectories in the root will be considered as classes.

__init__(root, rootType='str', categories=None)[source]¶

Initialize the RawImageFolder by scanning the root directory for class subfolders and image files.

Parameters:

root (str|dict) – If rootType is “str”, this should be a string path to the dataset root directory containing class subfolders. If rootType is “dict”, this should be a dictionary mapping split names (“train”, “val”, “test”) to their respective root directories.
rootType (str) – Type of the root parameter, either “str” for a single directory or “dict” for split-based directories. Default is “str”.
categories (list|None) – Optional list of category names to include. If None, all subdirectories in the root will be considered as classes.

__len__()[source]¶

Calculate the total number of samples (images) in the dataset.

Returns:: Total number of samples (images) in the dataset.
Return type:: int

__getitem__(idx)[source]¶

Retrieve the image path and label index for a given sample index.

Parameters:: idx (int) – Index of the sample to retrieve.
Returns:: A tuple containing the image file path (str) and the corresponding label index (int).
Return type:: tuple

GetPaths()[source]¶

Get the list of image file paths in the dataset.

Returns:: A list of image file paths (str) in the dataset.
Return type:: list

GetLabels()[source]¶

Get the list of labels corresponding to the image paths in the dataset.

Returns:: A list of labels (str) corresponding to the image paths in the dataset.
Return type:: list

ToDataFrame()[source]¶

Convert the dataset samples into a pandas DataFrame for easier manipulation.

Returns:: A DataFrame with columns “image_path”, “label”, and “split” (if available).
Return type:: pandas.DataFrame

GetLabelEncoder()[source]¶

Get the fitted LabelEncoder instance for encoding and decoding labels.

Returns:: The fitted LabelEncoder instance.
Return type:: sklearn.preprocessing.LabelEncoder

EncodeLabel(label)[source]¶

Encode a string label into its corresponding integer index using the fitted LabelEncoder.

Parameters:: label (str) – The string label to encode.
Returns:: The integer index corresponding to the provided string label.
Return type:: int

DecodeLabel(index)[source]¶

Decode an integer label index back into its corresponding string label using the fitted LabelEncoder.

Parameters:: index (int) – The integer index to decode.
Returns:: The string label corresponding to the provided integer index.
Return type:: str

class HMB.DatasetsHelper.GenericImagesDatasetHandler(sourceDir, configPath=None, imageExtensions=None, autoDetect=True, requiredSplits={'test', 'train', 'val'})[source]¶

Bases: object

Generic handler for image classification datasets.

The handler can auto-detect common dataset layouts (nested class folders or flat filenames with class prefix) or load a provided configuration JSON. It exposes utilities to create a standard train/val/test layout (folders per class), generate a lightweight dataset YAML for training tools, and write a small DatasetManifest.json with provenance and file checksums.

Variables:

imageExtensions (set) – Allowed image file extensions used when scanning.

Parameters:

sourceDir (Path)
configPath (Path)
imageExtensions (set)
autoDetect (bool)
requiredSplits (set)

Notes

Auto-detection supports three strategies: pre-split (train/val/test), nested class folders, and flat filenames with a class prefix.
Prepared datasets are written with train/, val/ and test/ folders under the requested output directory. A Dataset.yaml and DatasetManifest.json are written alongside the splits.

Key methods:

Prepare(outputDir, valSplit, testSplit, balance, balanceMethod, balanceTarget, randomSeed) Prepare the dataset in a standard layout. Balancing (optional) supports duplication and augmentation methods and applies only to the train split.
CreateConfigFile(outputPath, splits, minSamplesPerClass, description) Generate and write a DatasetConfig.json describing classes, counts and metadata.
ValidateDatasetStructure(datasetPath, minSamplesPerClass=1, returnStructured=False) Validate a pre-split dataset; when returnStructured=True returns a dict with per-split/per-class counts, sample file paths and readability flags.
CreateYAML(outputDir) Create a small Dataset.yaml suitable for training tools (also includes YOLO-compatible keys nc and names).
BuildManifest(outputDir, splitMapping) Write DatasetManifest.json with basic provenance and per-file SHA256 where possible.

Examples

from pathlib import Path
from HMB.DatasetsHelper import GenericImagesDatasetHandler

# Initialize handler for a dataset folder and auto-detect structure.
datasetPath = Path("/path/to/dataset")
handler = GenericImagesDatasetHandler(datasetPath)

# Print detected configuration summary.
handler.PrintSummary()

# Prepare dataset in standard train/val/test layout under output folder with balancing.
outputPath = Path("/path/to/output_dataset")
# Enable balancing, choose method and target, and set a reproducible seed.
handler.Prepare(
  outputPath,
  valSplit=0.1,
  testSplit=0.1,
  balance=True,
  balanceMethod="augmentation",
  balanceTarget="max",
  randomSeed=42
)

# Create a dataset configuration JSON file (saved in the output dataset folder).
configPath = handler.CreateConfigFile(
  outputPath=outputPath / "DatasetConfig.json",
  splits={"train": 0.8, "val": 0.1, "test": 0.1},
  minSamplesPerClass=20,
  description="My Custom Dataset"
)

# Validate an existing dataset structure and report issues (legacy list form).
issues = handler.ValidateDatasetStructure(outputPath, minSamplesPerClass=5)
if (isinstance(issues, list) and len(issues) > 0):
  print("Dataset validation issues found:")
  for issue in issues:
    print(f" - {issue}")
else:
  print("Dataset structure is valid (no issues found in legacy list form).")

# If you want a structured report with per-split/per-class details, request it explicitly:
structured = handler.ValidateDatasetStructure(outputPath, minSamplesPerClass=5, returnStructured=True)
# `structured` is a dict: {"splits": {..}, "issues": [...]}
if (structured.get("issues")):
  print("Structured validation issues:")
  for issue in structured["issues"]:
    print(f" - {issue}")
else:
  print("Structured validation passed.")

Initialize the dataset handler.

Parameters:

sourceDir (Path) – Path to the dataset root folder to analyze.
configPath (Path|None) – Optional path to a dataset configuration JSON.
imageExtensions (set|None) – Optional iterable of image file extensions to consider (for example {“.jpg”, “.png”}). If None, uses the class default set.
autoDetect (bool) – If True (default) auto-detect dataset structure during init. If False, skip auto-detection so the instance can be configured manually (useful in tests and programmatic workflows).
requiredSplits (set) – Set of required split folder names to check for when detecting pre-split structure. Defaults to {“train”, “val”, “test”}.

Raises:

ValueError – If the provided sourceDir does not exist.

__init__(sourceDir, configPath=None, imageExtensions=None, autoDetect=True, requiredSplits={'test', 'train', 'val'})[source]¶

Initialize the dataset handler.

Parameters:

sourceDir (Path) – Path to the dataset root folder to analyze.
configPath (Path|None) – Optional path to a dataset configuration JSON.
imageExtensions (set|None) – Optional iterable of image file extensions to consider (for example {“.jpg”, “.png”}). If None, uses the class default set.
autoDetect (bool) – If True (default) auto-detect dataset structure during init. If False, skip auto-detection so the instance can be configured manually (useful in tests and programmatic workflows).
requiredSplits (set) – Set of required split folder names to check for when detecting pre-split structure. Defaults to {“train”, “val”, “test”}.

Raises:

ValueError – If the provided sourceDir does not exist.

AutoDetectStructure()[source]¶

Auto-detect dataset structure and populate the internal config.

The method attempts three strategies:: Strategy 1: Pre-split layout (highest priority). Strategy 2: Look for nested class folders (generic). Strategy 3: Look for flat structure with labels in filename (generic).

Raises:: ValueError – If neither strategy detects a valid multi-class layout.

DetectPreSplitStructure()[source]¶

Check if the source directory already contains train/val/test subdirectories with at least one class folder each.

Returns:: True if a valid pre-split structure is detected.
Return type:: bool

DetectNestedStructure()[source]¶

Detect nested class folder structure dynamically.

Returns:: Mapping of folderName->className when more than one class folder is found, otherwise an empty dict.
Return type:: dict

DetectFlatStructure()[source]¶

Detect flat filename structure where class is encoded as a prefix in the filename (for example: class_imagename.jpg).

Returns:: Mapping of detected prefix->className or empty dict if detection fails.
Return type:: dict

GetImages(dirPath)[source]¶

Get all image files in the given directory (non-recursive) using the handler’s allowed extensions.

Parameters:: dirPath (Path) – Directory to scan for images.
Returns:: List of Path objects for detected image files.
Return type:: list[Path]

LoadDetectConfig()[source]¶

Load configuration from a JSON file if provided, otherwise auto-detect layout.

Side effects:: Sets self.config and self.classMapping accordingly.

CreateConfigFile(outputPath=None, splits=None, minSamplesPerClass=None, description=None)[source]¶

Auto-generate a complete dataset configuration JSON based on detected mapping and source data. The file contains class mappings, splits defaults and basic metadata including per-class sample counts.

Parameters:

outputPath (Path|None) – Destination path for the generated JSON. Defaults to <sourceDir>/DatasetConfig.json when None.
splits (dict|None) – Optional custom split fractions to override defaults.
minSamplesPerClass (int|None) – Minimum suggested samples per class.
description (str|None) – Optional human-readable dataset description.

Returns:

Path to the written configuration JSON.

Return type:

Path

GetConfig()[source]¶

Return a compact representation of the detected configuration.

Returns:: Configuration dictionary.
Return type:: dict

PlotClassDistribution(fileName='ClassDistribution.pdf', title='Class Distribution', save=False, display=True, fontSize=12, dpi=720)[source]¶

Plot a bar chart of the class distribution in the dataset.

This method prefers to use the centralized PlotClassDistribution utility from PlotsHelper when available, which creates a more polished and informative plot. If that utility is not available or fails, it falls back to a simpler bar chart implementation.

Parameters:

fileName (str) – Path to save the plot PDF. Default: “ClassDistribution.pdf”.
title (str) – Title of the plot. Default: “Class Distribution”.
save (bool) – If True, save the plot to fileName. Default: False.
display (bool) – If True, display the plot interactively. Default: True.
returnPreds (bool) – If True, return the matplotlib figure and axis objects. Default: False.
fontSize (int) – Font size for labels and title. Default: 12.
dpi (int) – DPI for saved figure. Default: 720.

PrintSummary()[source]¶: Print a short summary of the dataset: source path, format and per-class counts.

ValidateDatasetStructure(datasetPath, minSamplesPerClass=1, returnStructured=False)[source]¶

Quick validation helper for datasets organized with “train”, “val”, “test” top-level folders and class subdirectories.

Parameters:

datasetPath (str|Path) – Path to dataset root containing train/val/test.
minSamplesPerClass (int) – Minimal images expected per class in a split.
returnStructured (bool) – If True, return a structured dict with per-split and per-class details (counts, sample path, readable) in addition to collected issues. If False, return only the list of issue strings (legacy behavior).

Returns:

If returnStructured is False returns list[str] of issue messages.

If True returns dict with keys:

”splits”: {split: {className: {“count”: int, “sample”: str|None, “readable”: bool}}}
”issues”: list[str]

Return type:

list[str] | dict

BuildManifest(outputDir, splitMapping)[source]¶

Build a lightweight JSON manifest describing the produced dataset and including SHA256 checksums for files when possible.

Parameters:

outputDir (Path) – Output dataset directory used as reference for relative paths.
splitMapping (dict) – Mapping of splitName->[file Paths] produced by Prepare*.

Returns:

Path to the written DatasetManifest.json file.

Return type:

Path

Prepare(outputDir, valSplit=0.1, testSplit=0.1, balance=None, balanceMethod='duplication', balanceTarget='max', randomSeed=42)[source]¶

Prepare dataset in a standard train/val/test layout by creating train/val/test folders per class and copying files according to the requested splits.

Parameters:

outputDir (Path) – Destination folder where the train/val/test layout will be created.
valSplit (float) – Fraction of data to use for validation (default 0.1).
testSplit (float) – Fraction of data to use for testing (default 0.1).
balance (bool|None) – If True, apply class balancing after creating splits. Balancing is applied only to the “train” split.
balanceMethod (str) – One of {“duplication”, “augmentation”}. Default: “duplication”.
balanceTarget (str|int) – Target per-class count. If “max” (default) expand all classes up to the current maximum class size. If an int, expand all classes up to that number.
randomSeed (int) – Random seed for reproducibility.

Returns:

Path to the created dataset YAML file.

Return type:

Path

PreparePreSplit(outputDir)[source]¶

Copy an existing train/val/test layout from sourceDir to outputDir. Assumes sourceDir/train/, sourceDir/val/, sourceDir/test/ exist and contain class folders.

Parameters:: outputDir (Path) – Destination base path.
Returns:: Mapping of splitName -> list[Path] of copied files.
Return type:: dict

PrepareNested(outputDir, trainSplit, valSplit, testSplit, randomSeed=42)[source]¶

Prepare dataset from nested class folder structure by splitting each class folder into train/val/test and copying files accordingly.

Parameters:

outputDir (Path) – Destination base path.
trainSplit (float) – Fraction of non-test data to use for training.
valSplit (float) – Fraction of non-test data to use for validation.
testSplit (float) – Fraction to reserve for test.
randomSeed (int) – Random seed for reproducibility.

Returns:

Mapping of splitName->list[Path] of copied files.

Return type:

dict

PrepareFlat(outputDir, trainSplit, valSplit, testSplit, randomSeed=42)[source]¶

Prepare dataset from a flat filename layout where class is encoded as the filename prefix. Groups files by class, splits them and copies to the train/val/test layout.

Parameters:

outputDir (Path) – Destination base path.
trainSplit (float) – Fraction of non-test data to use for training.
valSplit (float) – Fraction of non-test data to use for validation.
testSplit (float) – Fraction to reserve for test.
randomSeed (int) – Random seed for reproducibility.

Returns:

Mapping of splitName->list[Path] of copied files.

Return type:

dict

CreateYAML(outputDir)[source]¶

Create a simple dataset YAML usable by common training tools. The file includes generic keys (num_classes, class_names) and retains YOLO-compatible keys (nc, names) for convenience and backward compatibility.

Parameters:: outputDir (Path) – Base path where train/val/test folders reside.
Returns:: Path to the written Dataset.yaml file.
Return type:: Path

class HMB.DatasetsHelper.PyTorchCustomDataset(dataDir, transform=None, allowedExtensions=('.bmp', '.TIF', '.PNG', '.jpeg', '.JPEG', '.webp', '.gif', '.tif', '.tiff', '.TIFF', '.png', '.jpg', '.JPG', '.BMP', '.WEBP', '.GIF'))[source]¶

Bases: Dataset

PyTorch dataset for image classification tasks, loading images from a directory structure where each class has its own subfolder.

Parameters:

dataDir (str) – Path to the root directory containing class subfolders with images.
transform (callable, optional) – Optional transform to be applied on a sample.
allowedExtensions (tuple, optional) – Tuple of allowed image file extensions. Defaults to IMAGE_SUFFIXES.

Initialize the custom dataset for image classification tasks.

Parameters:

dataDir (str) – Path to the root directory containing class subfolders with images.
transform (callable, optional) – Optional transform to be applied on a sample.
allowedExtensions (tuple, optional) – Tuple of allowed image file extensions. Defaults to IMAGE_SUFFIXES.

__init__(dataDir, transform=None, allowedExtensions=('.bmp', '.TIF', '.PNG', '.jpeg', '.JPEG', '.webp', '.gif', '.tif', '.tiff', '.TIFF', '.png', '.jpg', '.JPG', '.BMP', '.WEBP', '.GIF'))[source]¶

Initialize the custom dataset for image classification tasks.

Parameters:

dataDir (str) – Path to the root directory containing class subfolders with images.
transform (callable, optional) – Optional transform to be applied on a sample.
allowedExtensions (tuple, optional) – Tuple of allowed image file extensions. Defaults to IMAGE_SUFFIXES.

__len__()[source]¶

Get the total number of samples in the dataset.

Returns:: Number of samples in the dataset.
Return type:: int

__getitem__(idx)[source]¶

Retrieve an image and its label by index.

Parameters:: idx (int) – Index of the sample to retrieve.
Returns:: (image, label) where image is a PIL Image or transformed tensor, and label is an int class index.
Return type:: tuple

GetClassMapping()[source]¶

Get the mapping of class names to their corresponding indices.

Returns:: A dictionary mapping class names to integer indices.
Return type:: dict

GetClasses()[source]¶

Get the list of class names in the dataset.

Returns:: A list of class names.
Return type:: list

GetSamples()[source]¶

Get the list of samples in the dataset, where each sample is a tuple of (image path, class index).

Returns:: A list of tuples, each containing the image path and its corresponding class index.
Return type:: list

class HMB.DatasetsHelper.PyTorchSegmentationDataset(imagePaths, maskPaths, transforms=None, imageSize=256, numClasses=1)[source]¶

Bases: Dataset

PyTorch Dataset class for image segmentation tasks. Loads images and corresponding masks from provided file paths, applies optional transformations, encodes masks into class indices, and converts data to tensors.

Parameters:

imagePaths (List[str]) – List of file paths to input images.
maskPaths (List[str]) – List of file paths to corresponding segmentation masks.
transforms (Callable|None) – Optional callable for data augmentation/transforms.
imageSize (int) – Target size to resize images and masks (square). Default is 256.
numClasses (int) – Number of segmentation classes. Default is 2.

Notes

This dataset implements the standard PyTorch dataset protocol with the usual methods such as __len__ and __getitem__ provided on the instance. Detailed behaviour and return types for these methods are documented in their respective method docstrings below.

Initialize the segmentation dataset instance.

Parameters:

imagePaths (List[str]) – List of image file paths.
maskPaths (List[str]) – List of corresponding mask file paths.
transforms (Callable|None) – Optional augmentation/transform callable that accepts and returns a dict with keys “image” and “mask”.
imageSize (int) – Target square size (height,width) to resize images and masks.
numClasses (int) – Number of segmentation classes used for encoding masks.

__init__(imagePaths, maskPaths, transforms=None, imageSize=256, numClasses=1)[source]¶

Initialize the segmentation dataset instance.

Parameters:

imagePaths (List[str]) – List of image file paths.
maskPaths (List[str]) – List of corresponding mask file paths.
transforms (Callable|None) – Optional augmentation/transform callable that accepts and returns a dict with keys “image” and “mask”.
imageSize (int) – Target square size (height,width) to resize images and masks.
numClasses (int) – Number of segmentation classes used for encoding masks.

__len__()[source]¶

Return the number of samples available in the dataset.

Returns:: Number of image-mask pairs.
Return type:: int

LoadImage(path)[source]¶

Load an image from disk and return an RGB numpy array.

The function first attempts to use OpenCV for performance and falls back to PIL if OpenCV fails or is unavailable.

Parameters:: path (str|Path) – Filesystem path to the image file.
Returns:: HxWx3 uint8 RGB image array.
Return type:: numpy.ndarray

LoadMask(path)[source]¶

Load a segmentation mask from disk as a 2D numpy array.

Attempts to use OpenCV to read the mask in grayscale. If OpenCV is not available, falls back to PIL.

Parameters:: path (str|Path) – Filesystem path to the mask image.
Returns:: HxW integer mask array (single-channel).
Return type:: numpy.ndarray

EncodeMask(mask)[source]¶

Encode a grayscale mask into integer class indices. - For binary (numClasses == 1): threshold at 128 → {0, 1} - For multi-class (numClasses >= 2): assume pixel values are class IDs; clip to [0, numClasses-1]

Parameters:: mask (numpy.ndarray) – HxW grayscale mask array.
Returns:: (H, W) of dtype int64
Return type:: numpy.ndarray

ToTensors(image, mask)[source]¶

Convert numpy image and mask arrays to PyTorch tensors.

The image is normalized to [0,1] float32 and transposed to channel-first format (C,H,W). The mask is returned as a torch tensor of shape (H,W).

Parameters:

image (numpy.ndarray) – HxWx3 image array (uint8 or float).
mask (numpy.ndarray) – HxWx1 integer mask array.

Returns:

(imageTensor, maskTensor) where imageTensor is torch.FloatTensor shape (C,H,W) and maskTensor is torch tensor shape (H,W).

Return type:

tuple

__getitem__(index)[source]¶

Retrieve the image and mask tensors for a given index.

The method loads image and mask files, resizes them to the configured imageSize, applies optional augmentation/transforms (expects a callable that accepts and returns a dict with “image” and “mask”), encodes the mask into class indices and converts both to torch tensors.

Parameters:: index (int) – Index of the sample to retrieve.
Returns:: (imageTensor, maskTensor) as returned by ToTensors.
Return type:: tuple

GetPixelStats()[source]¶

Compute per-channel min, max, mean, and standard deviation across the dataset for both images and masks. This can be useful for normalization and analysis.

Returns:

Dictionary with keys “image” and “mask”, each containing a dict with keys: ”min”, “max”, “mean”, “std” mapping to numpy arrays of shape (3,) for images and (1,) for masks.

Return type:

dict

class HMB.DatasetsHelper.PyTorchVideoClassificationDataset(rootDir, split, transform=None, numFrames=None, sampleRate=None, classNames=None, fileExtensions=None)[source]¶

Bases: Dataset

Generic dataset class for loading video files with class labels.

Expected directory structure:

rootDir/

split/ # e.g., “train”, “val”, “test”

class1/: video1.mp4 video2.mp4
class2/: video3.mp4 video4.mp4

Each video file is read and processed into a tensor, and the corresponding class label is returned as an integer. The dataset supports configurable frame sampling and optional transformations.

Key features:

Configurable root directory and dataset split.
Automatic discovery of video files based on class subdirectories.
Mapping of class names to numeric labels.
Robust video reading with PyAV and safe handling of edge cases (e.g., empty videos).
Optional transformation pipeline for data augmentation or preprocessing.

Initialize the dataset.

Parameters:

rootDir (str) – Root directory containing train/val/test splits.
split (str) – Dataset split name.
transform (Optional[PyTorchVideoTransforms]) – Optional video transformation pipeline.
numFrames (Optional[int]) – Number of frames to sample per video.
sampleRate (Optional[int]) – Frame sampling rate.
classNames (Optional[List[str]]) – List of expected class directory names.
fileExtensions (Optional[Tuple[str, ...]]) – Tuple of supported video file extensions.

__init__(rootDir, split, transform=None, numFrames=None, sampleRate=None, classNames=None, fileExtensions=None)[source]¶

Initialize the dataset.

Parameters:

rootDir (str) – Root directory containing train/val/test splits.
split (str) – Dataset split name.
transform (Optional[PyTorchVideoTransforms]) – Optional video transformation pipeline.
numFrames (Optional[int]) – Number of frames to sample per video.
sampleRate (Optional[int]) – Frame sampling rate.
classNames (Optional[List[str]]) – List of expected class directory names.
fileExtensions (Optional[Tuple[str, ...]]) – Tuple of supported video file extensions.

Return type:

None

__len__()[source]¶

Return total number of video samples. This method simply returns the length of the internal list that tracks all discovered video file paths, which corresponds to the total number of valid samples available for loading and processing. It relies on the assumption that the _LoadVideoList method has been executed successfully during initialization to populate this list with valid entries. If no valid videos were found, this method would return zero, which is handled by the error checking logic in the constructor.

Return type:: int

__getitem__(sampleIndex)[source]¶

Load and preprocess a single video sample. This method retrieves the file path and corresponding label for the requested sample index, decodes the video into a sequence of frames, applies any specified transformations, and constructs a standardized output dictionary containing the processed video tensor, numeric label, and original file path. It ensures that the video data is returned in a format suitable for model ingestion, with pixel values normalized and dimensions ordered correctly. The method also handles the case where no transformations are provided by applying a default conversion and normalization process to the raw video array.

Parameters:: sampleIndex (int) – Integer index of the video sample to retrieve.
Return type:: dict

HMB.DatasetsHelper.CreateSegmentationDataLoaders(dataDir, imageSize=256, batchSize=8, numWorkers=4, numClasses=1, pinMemory=False, imagesFolderName='Images', masksFolderName='Masks')[source]¶

Create PyTorch DataLoader objects for segmentation tasks from a directory that contains either an Images/ and Masks/ folder pair or a looser tree where image/mask pairs are discovered recursively.

The function will look for Images/ and Masks/ under dataDir. When the standard layout is missing it will attempt to gather image-mask pairs using GatherSegmentationPairs (non-recursive pairing by basename matching).

Parameters:

dataDir (str) – Root path to the dataset containing images and masks.
imageSize (int) – Target square size to resize images and masks. Default 256.
batchSize (int) – Batch size for the returned DataLoaders. Default 8.
numWorkers (int) – Number of worker processes for the DataLoader. Default 4.
numClasses (int) – Number of segmentation classes (used by SegmentationDataset).
pinMemory (bool) – Whether to use pinned memory in DataLoaders for faster GPU transfer.
imagesFolderName (str) – Name of the folder containing input images (default “Images”).
masksFolderName (str) – Name of the folder containing segmentation masks (default “Masks”).

Returns:

(trainLoader, valLoader) PyTorch DataLoader instances for training and validation.

Return type:

tuple

HMB.DatasetsHelper.GatherSegmentationPairs(rootDir)[source]¶

Discover image/mask file pairs by walking a directory tree.

The function treats directories whose name contains the substring “mask” (case-insensitive) as mask folders and everything else as image folders. It matches images and masks by basename (ignoring extension) using a best-match heuristic (prefix containment) and returns two lists: paired images and paired masks in identical order.

Parameters:: rootDir (str) – Root directory to recursively scan for images and masks.
Returns:: (pairedImagePaths, pairedMaskPaths) where each is a list of filesystem paths (strings).
Return type:: tuple

HMB.DatasetsHelper.DiscoverCsvFiles(dataDir)[source]¶

Recursively discover CSV files in the given directory.

Parameters:: dataDir (str) – The root directory to search for CSV files.
Returns:: A list of file paths to discovered CSV files.
Return type:: List[str]

HMB.DatasetsHelper.SanitizeArray(inputData)[source]¶

Replace infinite values and fill NaNs in array-like inputs.

If inputData is a pandas DataFrame, replace +/-inf with NaN and fill numeric columns with the column mean (or 0 if mean is not finite). Non-numeric columns are forward/back filled where possible.
If inputData is a numpy array or other sequence, convert to float64, replace non-finite entries with NaN and fill each column with the column mean (or 0 if no finite vals).

Returns the same type as input where practical (DataFrame -> DataFrame, ndarray -> ndarray). Note: List inputs will be converted to numpy.ndarray.

Parameters:: inputData (array-like) – Input data to sanitize. It can be a pandas DataFrame, a numpy array, or a list of lists.
Returns:: Sanitized data with infinities replaced and NaNs filled.
Return type:: array-like

HMB.DatasetsHelper.ReadAndConcatCsv(files, nrows=None)[source]¶

Read multiple CSV files into pandas DataFrames, add a “SourceFile” column to each, and concatenate them.

Parameters:

files (List[str]) – List of file paths to CSV files to read and concatenate.
nrows (int, optional) – If provided, limits the number of rows read from each CSV file for testing purposes. Defaults to None (read all rows).

Returns:

A single DataFrame containing the concatenated data from all CSV files, with an additional “SourceFile” column indicating the origin of each row.

Return type:

pd.DataFrame

HMB.DatasetsHelper.LoadAllCsvs(dataDir, nrows=None)[source]¶

Discover all CSV files in the specified directory, read them into pandas DataFrames, and concatenate them into a single DataFrame.

Parameters:

dataDir (str) – The directory to search for CSV files.
nrows (int, optional) – If provided, limits the number of rows read from each CSV file for testing purposes. Defaults to None (read all rows).

Returns:

A single DataFrame containing the concatenated data from all discovered CSV files, with an additional “SourceFile” column indicating the origin of each row.

Return type:

pd.DataFrame

class HMB.DatasetsHelper.TFFolderBasedDataPipeline(dataDir, batchSize=32, imageSize=224, ratioTuple=(0.8, 0.1, 0.1))[source]¶

Bases: object

A unified pipeline to handle folder scanning, auto-splitting, and tf.data.Dataset creation.

This class is designed to simplify the process of creating TensorFlow data pipelines for image classification tasks. It automatically checks for the presence of “train”, “val”, and “test” subdirectories, and if they are not found, it attempts to split the dataset based on class subdirectories. It then builds tf.data.Dataset pipelines for each split, applying appropriate transformations and caching strategies for training and evaluation. The class also calculates and stores the number of samples in each split for easy access.

Note

The expected directory structure for the dataset is as follows:

Dataset/
├── Class_A/
│   ├── image1.jpg
│   └── image2.png
├── Class_B/
│   ├── image3.jpg
│   └── image4.bmp
└── Class_C/
    └── image5.tiff

If “train”, “val”, and “test” folders are not found, it will create them by splitting the class folders using an 80/10/10 ratio. The resulting structure will be:

SplitDataset/
  ├── train/
  │   ├── Class_A/
  │   │   ├── image1.jpg
  │   │   └── image2.png
  │   ├── Class_B/
  │   │   ├── image3.jpg
  │   │   └── image4.bmp
  │   └── Class_C/
  │       └── image5.tiff
  ├── val/
  │   ├── Class_A/
  │   │   ├── image6.jpg
  │   │   └── image7.png
  │   ├── Class_B/
  │   │   ├── image8.jpg
  │   │   └── image9.bmp
  │   └── Class_C/
  │       └── image10.tiff
  └── test/
      ├── Class_A/
      │   ├── image11.jpg
      │   └── image12.png
      ├── Class_B/
      │   ├── image13.jpg
      │   └── image14.bmp
      └── Class_C/
          └── image15.tiff

Note

This class requires the splitfolders library for auto-splitting functionality. If the library is not installed, the auto-splitting feature will not work, and the class will raise an error if it cannot find the expected “train”, “val”, and “test” directories or class subdirectories to split. To install splitfolders, you can use pip: pip install splitfolders

Warning

The auto-splitting process will create a new directory named “SplitDataset” in the parent directory of the original dataset. Ensure that you have write permissions to the parent directory and that you do not have an existing “SplitDataset” directory that you do not want to be overwritten, as the splitting process will create this directory if it does not already exist. Always back up your data before performing operations that modify the filesystem.

Examples

import matplotlib.pyplot as plt
from HMB.DatasetsHelper import TFFolderBasedDataPipeline

# Define the root directory containing the dataset.
dataDir = "/path/to/your/dataset"
# Define the batch size for the data pipeline.
batchSize = 32
# Define the target image size for resizing.
imageSize = 224

# Instantiate the unified data pipeline.
pipeline = TFFolderBasedDataPipeline(
  dataDir=dataDir,
  batchSize=batchSize,
  imageSize=imageSize
)

# Access the training, validation, and test datasets.
trainDataset = pipeline.train
valDataset = pipeline.val
testDataset = pipeline.test

# Print the dataset lengths and class names for verification.
print("Dataset Lengths:", pipeline.lengths)
# Print the discovered class names.
print("Class Names:", pipeline.classNames)

# Create a dictionary of available loaders for easy iteration.
availableLoaders = {
  "Train": pipeline.train,
  "Val"  : pipeline.val,
  "Test" : pipeline.test,
}
# Filter out None values for missing splits.
availableLoaders = {k: v for k, v in availableLoaders.items() if (v is not None)}

# Print the available loader keys.
print("Available Loaders:", list(availableLoaders.keys()))
# Print the sample batch shapes header.
print("Sample batch shapes:")

numOfClasses = len(pipeline.classNames)
print(f"Number of classes: {numOfClasses}")

# Iterate through the available loaders to print batch shapes.
for split, loader in availableLoaders.items():
  # Take one batch from the loader.
  for batch in loader.take(1):
    # Unpack the batch into images, labels, and filenames.
    images, labels, filenames = batch
    # Print the shapes for the current split.
    print(f"{split} - Images shape: {images.shape}, Labels shape: {labels.shape}, Filenames shape: {filenames.shape}")

# Visualize a few samples from the training set to verify augmentations.
if (pipeline.train is not None):
  # Create a matplotlib figure for the visualization.
  plt.figure(figsize=(12, 6))
  # Take one batch from the training loader.
  for i, batch in enumerate(pipeline.train.take(1)):
    # Unpack the batch.
    images, labels, filenames = batch
    # Iterate through the first 8 images in the batch.
    for j in range(min(8, images.shape[0])):
      # Add a subplot for the current image.
      plt.subplot(2, 4, j + 1)
      # Display the image.
      plt.imshow(images[j].numpy())
      # Set the title with the label and class name.
      plt.title(f"Label: {labels[j].numpy()} - {pipeline.classNames[labels[j].numpy()]}")
      # Turn off the axis.
      plt.axis("off")
  # Set the main title for the figure.
  plt.suptitle("Sample Augmented Training Images")
  # Display the plot.
  plt.show()

Initialize the data pipeline. This method sets up the internal state, checks for directory structure, prepares splits if necessary, and builds the tf.data.Dataset pipelines for training, validation, and testing.

The method performs the following steps:

Stores the provided parameters as instance variables.
Calls the _prepareSplits method to check for the presence of “train”, “val”, and “test” directories and to perform auto-splitting if they are not found.
Calls the _buildIndex method to scan the directories, discover class names, and calculate the number of samples in each split.
Calls the _buildPipeline method three times to create tf.data.Dataset pipelines for the “train”, “val”, and “test” splits, applying appropriate transformations and caching strategies for each.

Parameters:

dataDir (str) – The root directory containing the dataset. It should contain “train”, “val”, and “test” subdirectories, or class subdirectories for auto-splitting.
batchSize (int) – The batch size to use for the tf.data.Dataset pipelines. Default is 32.
imageSize (int) – The target size to which images will be resized. Default is 224 (for 224x224 images).
ratioTuple (tuple) – A tuple specifying the train/val/test split ratios for auto-splitting. Default is (0.8, 0.1, 0.1).

__init__(dataDir, batchSize=32, imageSize=224, ratioTuple=(0.8, 0.1, 0.1))[source]¶

The method performs the following steps:

Stores the provided parameters as instance variables.
Calls the _prepareSplits method to check for the presence of “train”, “val”, and “test” directories and to perform auto-splitting if they are not found.
Calls the _buildIndex method to scan the directories, discover class names, and calculate the number of samples in each split.
Calls the _buildPipeline method three times to create tf.data.Dataset pipelines for the “train”, “val”, and “test” splits, applying appropriate transformations and caching strategies for each.

Parameters:

dataDir (str) – The root directory containing the dataset. It should contain “train”, “val”, and “test” subdirectories, or class subdirectories for auto-splitting.
batchSize (int) – The batch size to use for the tf.data.Dataset pipelines. Default is 32.
imageSize (int) – The target size to which images will be resized. Default is 224 (for 224x224 images).
ratioTuple (tuple) – A tuple specifying the train/val/test split ratios for auto-splitting. Default is (0.8, 0.1, 0.1).

Return type:

None