PerformanceMetrics Module¶

Implements performance metrics for evaluating models and algorithms.

HMB.PerformanceMetrics.CalculatePerformanceMetrics(confMatrix, eps=1e-10, addWeightedAverage=False, addPerClass=False)[source]¶

Calculate performance metrics from a confusion matrix.

\[\begin{split}\text{Precision} = \frac{TP}{TP + FP}\\ \text{Recall} = \frac{TP}{TP + FN}\\ F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}\\ \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}\\ \text{Specificity} = \frac{TN}{TN + FP}\\ \text{MCC} = \frac{TP\cdot TN - FP\cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}\end{split}\]

Parameters:

confMatrix (list or numpy.ndarray) – Confusion matrix representing the classification results.
eps (float) – Small value to avoid division by zero. Default is 1e-10.
addWeightedAverage (bool) – Whether to include weighted averages in the output. Default is False.
addPerClass (bool) – Whether to include per-class metrics in the output. Default is False.

Returns:

A dictionary containing performance metrics including TP, FP, FN, TN and various: macro/micro/weighted averages.

Return type:

dict

Examples

import numpy as np
import HMB.PerformanceMetrics as pm

confMatrix = [[50, 2, 1], [5, 45, 0], [0, 3, 47]]
metrics = pm.CalculatePerformanceMetrics(confMatrix, addWeightedAverage=True)
for key, value in metrics.items():
  print(f"{key}: {np.round(value, 4)}")

Another example (using sklearn’s confusion_matrix):

import numpy as np
from sklearn.metrics import confusion_matrix
import HMB.PerformanceMetrics as pm

yTrue = [0, 1, 2, 0, 1, 2]
yPred = [0, 2, 1, 0, 0, 1]
confMatrix = confusion_matrix(yTrue, yPred)
metrics = pm.CalculatePerformanceMetrics(confMatrix, addWeightedAverage=True)
for key, value in metrics.items():
  print(f"{key}: {np.round(value, 4)}")

HMB.PerformanceMetrics.PlotConfusionMatrix(cm, classes, normalize=False, roundDigits=3, title='Confusion Matrix', cmap=<matplotlib.colors.LinearSegmentedColormap object>, display=True, save=False, fileName='ConfusionMatrix.pdf', fontSize=15, annotate=True, figSize=(8, 8), colorbar=True, returnFig=False, dpi=720)[source]¶

Plot a confusion matrix with options for normalization, annotation, saving, and display.

Parameters:

cm (list or numpy.ndarray) – Confusion matrix representing the classification results.
classes (list) – List of class labels to display on axes.
normalize (bool) – Whether to normalize the confusion matrix by row sums. Default is False.
roundDigits (int) – Number of decimal places to round normalized values. Default is 3.
title (str) – Title of the plot. Default is “Confusion Matrix”.
cmap (matplotlib.colors.Colormap or None) – Colormap for the plot. Default is plt.cm.Blues.
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot to fileName. Default is False.
fileName (str) – File name to save the plot. Default is “ConfusionMatrix.pdf”.
fontSize (int) – Font size for labels and annotations. Default is 15.
annotate (bool) – Whether to annotate cells with values. Default is True.
figSize (tuple) – Figure size in inches. Default is (8, 8).
colorbar (bool) – Whether to show colorbar. Default is True.
returnFig (bool) – Whether to return the matplotlib figure object. Default is False.
dpi (int) – DPI for saving the figure. Default is 720.

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

The confusion matrix can be normalized by row sums for better interpretability.
Annotated values in each cell help visualize the distribution of predictions.
Saving and displaying the plot are optional and controlled by parameters.

\[\text{Normalized CM}_{i,j} = \frac{CM_{i,j}}{\sum_j CM_{i,j}}\]

\[\text{Precision} = \frac{TP}{TP + FP}\]

\[\text{Recall} = \frac{TP}{TP + FN}\]

\[\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}\]

Examples

import numpy as np
import HMB.PerformanceMetrics as pm

confMatrix = [[50, 2, 1], [5, 45, 0], [0, 3, 47]]
classLabels = ["Class 0", "Class 1", "Class 2"]
pm.PlotConfusionMatrix(
  confMatrix,
  classes=classLabels,
  normalize=False,
  title="Confusion Matrix",
  annotate=True,
  fontSize=15,
  figSize=(6, 6),
  colorbar=True,
  display=True,
  save=False
)

HMB.PerformanceMetrics.PlotRegressionResults(yTrue, yPred, title='Regression Results', fontSize=14, figsize=(10, 5), display=True, save=False, fileName='RegressionResults.pdf', dpi=720, returnFig=False)[source]¶

Plot regression results: predicted vs. true values and residuals.

Parameters:

yTrue (array-like) – True target values.
yPred (array-like) – Predicted target values.
title (str) – Plot title.
fontSize (int) – Font size for labels and title.
figsize (tuple) – Figure size.
display (bool) – Whether to display the plot.
save (bool) – Whether to save the plot.
fileName (str) – File name to save the plot.
dpi (int) – DPI for saving the figure.
returnFig (bool) – Whether to return the figure object.

Returns:

The matplotlib figure object if returnFig is True, else None.

Return type:

fig

HMB.PerformanceMetrics.PlotROCAUCCurve(yTrue, yPred, classes, areProbabilities=False, title='ROC Curve & AUC', figSize=(5, 5), cmap=None, display=True, save=False, fileName='ROC_AUC.pdf', fontSize=15, plotDiagonal=True, annotateAUC=True, showLegend=True, returnFig=False, dpi=720)[source]¶

Plot ROC curves and calculate AUC for each class, with options for annotation, saving, and display.

Parameters:

yTrue (array-like or numpy.ndarray) – True labels (one-hot encoded or binary).
yPred (array-like or numpy.ndarray) – Predicted labels or probabilities (one-hot encoded, binary, or probabilities).
classes (list) – List of class names for labeling curves.
areProbabilities (bool) – Whether yPred contains probabilities. Default is False.
title (str) – Title of the plot. Default is “ROC Curve & AUC”.
figSize (tuple) – Figure size in inches. Default is (5, 5).
cmap (matplotlib.colors.Colormap or None) – Colormap for ROC curves. Default is None.
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot to fileName. Default is False.
fileName (str) – File name to save the plot. Default is “ROC_AUC.pdf”.
fontSize (int) – Font size for labels and annotations. Default is 15.
plotDiagonal (bool) – Whether to plot the diagonal reference line. Default is True.
annotateAUC (bool) – Whether to annotate AUC value on each curve. Default is True.
showLegend (bool) – Whether to show legend. Default is True.
returnFig (bool) – Whether to return the matplotlib figure object. Default is False.
dpi (int) – DPI for saving the figure. Default is 720.

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

ROC curves visualize the trade-off between true positive rate and false positive rate for each class.
AUC (Area Under Curve) quantifies the overall ability of the classifier to distinguish between classes.
Saving and displaying the plot are optional and controlled by parameters.

\[\text{TPR} = \frac{TP}{TP + FN} \qquad \text{FPR} = \frac{FP}{FP + TN}\]

\[\text{AUC} = \int_0^1 \text{TPR}(\text{FPR}) \, d\text{FPR}\]

Examples

import numpy as np
import HMB.PerformanceMetrics as pm

yTrue = [0, 1, 2, 0, 1, 2]
yPred = [
  [0.8, 0.1, 0.1],
  [0.2, 0.7, 0.1],
  [0.1, 0.2, 0.7],
  [0.9, 0.05, 0.05],
  [0.1, 0.8, 0.1],
  [0.05, 0.1, 0.85]
]
classLabels = ["Class 0", "Class 1", "Class 2"]
pm.PlotROCAUCCurve(
  np.array(yTrue),
  np.array(yPred),
  classes=classLabels,
  areProbabilities=True,
  title="ROC Curve & AUC",
  display=True,
  save=False
)

HMB.PerformanceMetrics.PlotPRCCurve(yTrue, yPred, classes, areProbabilities=False, title='PRC Curve', figSize=(5, 5), cmap=None, display=True, save=False, fileName='PRC.pdf', fontSize=15, annotateAvg=True, showLegend=True, returnFig=False, dpi=720)[source]¶

Plot Precision-Recall curves (PRC) and calculate average precision for each class, with options for annotation, saving, and display.

Parameters:

yTrue (array-like or numpy.ndarray) – True labels (one-hot encoded or binary).
yPred (array-like or numpy.ndarray) – Predicted labels or probabilities (one-hot encoded, binary, or probabilities).
classes (list) – List of class names for labeling curves.
areProbabilities (bool) – Whether yPred contains probabilities. Default is False.
title (str) – Title of the plot. Default is “PRC Curve”.
figSize (tuple) – Figure size in inches. Default is (5, 5).
cmap (matplotlib.colors.Colormap or None) – Colormap for PRC curves. Default is None.
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot to fileName. Default is False.
fileName (str) – File name to save the plot. Default is “PRC.pdf”.
fontSize (int) – Font size for labels and annotations. Default is 15.
annotateAvg (bool) – Whether to annotate average precision value on each curve. Default is True.
showLegend (bool) – Whether to show legend. Default is True.
returnFig (bool) – Whether to return the matplotlib figure object. Default is False.
dpi (int) – DPI for saving the figure. Default is 720.

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

Precision-Recall curves visualize the trade-off between precision (PPV) and recall (TPR) for each class.
Average precision quantifies the overall ability of the classifier to balance precision and recall.
Saving and displaying the plot are optional and controlled by parameters.

\[\text{Precision} = \frac{TP}{TP + FP}\]

\[\text{Recall} = \frac{TP}{TP + FN}\]

\[\text{Average Precision} = \int_0^1 \text{Precision}(\text{Recall}) \, d\text{Recall}\]

Examples

import numpy as np
import HMB.PerformanceMetrics as pm

yTrue = [0, 1, 2, 0, 1, 2]
yPred = [
  [0.8, 0.1, 0.1],
  [0.2, 0.7, 0.1],
  [0.1, 0.2, 0.7],
  [0.9, 0.05, 0.05],
  [0.1, 0.8, 0.1],
  [0.05, 0.1, 0.85]
]
classLabels = ["Class 0", "Class 1", "Class 2"]
pm.PlotPRCCurve(
  np.array(yTrue),
  np.array(yPred),
  classes=classLabels,
  areProbabilities=True,
  title="PRC Curve",
  display=True,
  save=False
)

HMB.PerformanceMetrics.PlotMultiTrialROCAUC(allYTrue, allYPred, classes, confidenceLevel=0.95, which='CI', title='Multi-Trial ROC Curve with Confidence Intervals', figSize=(8, 8), cmap=None, display=True, save=False, fileName='MultiTrial_ROC_AUC.pdf', fontSize=15, plotDiagonal=True, showLegend=True, returnFig=False, dpi=720, addZoomedInset=True)[source]¶

Plot averaged ROC curves across multiple trials with confidence intervals.

\[\text{TPR} = \frac{TP}{TP + FN}\]

\[\text{FPR} = \frac{FP}{FP + TN}\]

\[\text{AUC} = \int_0^1 \text{TPR}(\text{FPR}) \, d\text{FPR}\]

Parameters:

allYTrue (list) – List of ground truth arrays from all trials. Each element shape: (nSamples, nClasses) or (nSamples,).
allYPred (list) – List of prediction probability arrays from all trials. Each element shape: (nSamples, nClasses).
classes (list) – List of class names.
confidenceLevel (float) – Confidence level for intervals (default 0.95).
which (str) – Method for confidence intervals: “CI” for confidence intervals, “SD” for standard deviation.
title (str) – Plot title.
figSize (tuple) – Figure size in inches.
cmap (colormap) – Matplotlib colormap for different classes.
display (bool) – Whether to display the plot.
save (bool) – Whether to save the plot.
fileName (str) – File name for saving.
fontSize (int) – Font size for labels.
plotDiagonal (bool) – Whether to plot the diagonal reference line.
showLegend (bool) – Whether to show legend.
returnFig (bool) – Whether to return figure object.
dpi (int) – DPI for saving the figure.
addZoomedInset (bool) – Whether to add a zoomed inset for the top-left corner of the ROC plot.

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

This function computes and plots the mean ROC curve across multiple trials for each class.
Confidence intervals are calculated using the normal approximation method.
The plot includes options for saving, displaying, and customizing appearance.

Examples

import numpy as np
import HMB.PerformanceMetrics as pm

# Simulated data for 3 trials and 3 classes.
allYTrue = [
  np.array([0, 1, 2, 0, 1, 2]),
  np.array([0, 1, 2, 0, 1, 2]),
  np.array([0, 1, 2, 0, 1, 2])
]
allYPred = [
  np.array([
    [0.8, 0.1, 0.1],
    [0.2, 0.7, 0.1],
    [0.1, 0.2, 0.7],
    [0.9, 0.05, 0.05],
    [0.1, 0.8, 0.1],
    [0.05, 0.1, 0.85]
  ]),
  np.array([
    [0.7, 0.2, 0.1],
    [0.3, 0.6, 0.1],
    [0.2, 0.3, 0.5],
    [0.85, 0.1, 0.05],
    [0.15, 0.75, 0.1],
    [0.1, 0.2, 0.7]
  ]),
  np.array([
    [0.75, 0.15, 0.1],
    [0.25, 0.65, 0.1],
    [0.15, 0.25, 0.6],
    [0.88, 0.07, 0.05],
    [0.12, 0.78, 0.1],
    [0.08, 0.15, 0.77]
  ])
]
classLabels = ["Class 0", "Class 1", "Class 2"]
pm.PlotMultiTrialROCAUC(
  allYTrue,
  allYPred,
  classes=classLabels,
  confidenceLevel=0.95,
  which="CI",
  title="Multi-Trial ROC Curve with Confidence Intervals",
  display=True,
  save=False
)

HMB.PerformanceMetrics.PlotMultiTrialPRCurve(allYTrue, allYPred, classes, confidenceLevel=0.95, which='CI', title='Multi-Trial Precision-Recall Curve with Confidence Intervals', figSize=(8, 8), cmap=None, display=True, save=False, fileName='MultiTrial_PRC.pdf', fontSize=15, showLegend=True, returnFig=False, dpi=720, addZoomedInset=True)[source]¶

Plot averaged Precision-Recall curves across multiple trials with confidence intervals.

\[\text{Precision} = \frac{TP}{TP + FP}\]

\[\text{Recall} = \frac{TP}{TP + FN}\]

\[\text{Average Precision} = \int_0^1 \text{Precision}(\text{Recall}) \, d\text{Recall}\]

Parameters:

allYTrue (list) – List of ground truth arrays from all trials. Each element shape: (nSamples, nClasses) or (nSamples,).
allYPred (list) – List of prediction probability arrays from all trials. Each element shape: (nSamples, nClasses).
classes (list) – List of class names.
confidenceLevel (float) – Confidence level for intervals (default 0.95).
title (str) – Plot title.
figSize (tuple) – Figure size in inches.
cmap (colormap) – Matplotlib colormap for different classes.
display (bool) – Whether to display the plot.
save (bool) – Whether to save the plot.
fileName (str) – File name for saving.
fontSize (int) – Font size for labels.
showLegend (bool) – Whether to show legend.
returnFig (bool) – Whether to return figure object.
dpi (int) – DPI for saving.
addZoomedInset (bool) – Whether to add a zoomed inset for the top-right corner of the PRC plot.

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

This function computes and plots the mean Precision-Recall curve across multiple trials for each class.
Confidence intervals are calculated using the normal approximation method.
The plot includes options for saving, displaying, and customizing appearance.

Examples

import numpy as np
import HMB.PerformanceMetrics as pm

# Simulated data for 3 trials and 3 classes.
allYTrue = [
  np.array([0, 1, 2, 0, 1, 2]),
  np.array([0, 1, 2, 0, 1, 2]),
  np.array([0, 1, 2, 0, 1, 2])
]
allYPred = [
  np.array([
    [0.8, 0.1, 0.1],
    [0.2, 0.7, 0.1],
    [0.1, 0.2, 0.7],
    [0.9, 0.05, 0.05],
    [0.1, 0.8, 0.1],
    [0.05, 0.1, 0.85]
  ]),
  np.array([
    [0.7, 0.2, 0.1],
    [0.3, 0.6, 0.1],
    [0.2, 0.3, 0.5],
    [0.85, 0.1, 0.05],
    [0.15, 0.75, 0.1],
    [0.1, 0.2, 0.7]
  ]),
  np.array([
    [0.75, 0.15, 0.1],
    [0.25, 0.65, 0.1],
    [0.15, 0.25, 0.6],
    [0.88, 0.07, 0.05],
    [0.12, 0.78, 0.1],
    [0.08, 0.15, 0.77]
  ])
]
classLabels = ["Class 0", "Class 1", "Class 2"]
pm.PlotMultiTrialPRCurve(
  allYTrue,
  allYPred,
  classes=classLabels,
  confidenceLevel=0.95,
  title="Multi-Trial Precision-Recall Curve with Confidence Intervals",
  display=True,
  save=False
)

HMB.PerformanceMetrics.PlotCounterfactualOutcomes(X, classifier, treatmentCol=None, lowVal=None, highVal=None, classNames=None, title='Counterfactual Outcome Distributions', save=False, fileName='CounterfactualOutcomePlot.pdf', display=True, returnPreds=False, fontSize=12, dpi=720)[source]¶

Plot counterfactual outcome distributions for two treatment scenarios using a histogram.

Parameters:

X (pandas.DataFrame) – Feature matrix.
classifier – Trained classifier with predict method.
treatmentCol (str or None) – Name of the treatment column. If None, prints available columns and returns.
lowVal (numeric or None) – Value representing “no/low” treatment. If None, uses min value in column.
highVal (numeric or None) – Value representing “high” treatment. If None, uses max value in column.
classNames (list or None) – List of class names for x-axis ticks. If None, inferred from classifier or predictions.
title (str) – Title of the plot. Default is “Counterfactual Outcome Distributions”.
save (bool) – Whether to save the plot to fileName. Default is False.
fileName (str) – File name to save the plot. Default is “CounterfactualOutcomePlot.pdf”.
display (bool) – Whether to display the plot. Default is True.
returnPreds (bool) – If True, returns (yPredLow, yPredHigh). Default is False.
fontSize (int) – Font size for labels and title. Default is 12.
dpi (int) – DPI for saving the figure. Default is 720.

Returns:

None if returnPreds is False, otherwise (yPredLow, yPredHigh).

Return type:

None or tuple

Notes

The function creates two copies of X, sets the treatment column to lowVal and highVal, and predicts outcomes for both scenarios.
The results are plotted as overlaid histograms, with x-axis ticks labeled by classNames if provided.
The plot is saved to fileName and optionally displayed.

Example

import HMB.PerformanceMetrics as pm

X = ...  # Load or create your feature DataFrame.
classifier = ...  # Load or train your classifier.
classNames = []  # Specify class names.

pm.PlotCounterfactualOutcomes(
  X, classifier,
  treatmentCol="Inflight Wi-Fi service",
  classNames=classNames,
  title="Counterfactual Outcome Distributions",
  save=True,
  fileName="CounterfactualOutcomePlot.pdf",
  display=True,
  fontSize=14
)

HMB.PerformanceMetrics.PlotInteractionEffect(X, classifier, feature1, feature2, gridSize=30, title='Interaction Effect Plot', save=False, fileName='InteractionEffectPlot.pdf', display=True, fontSize=12, plotType='surface', backend='matplotlib', dpi=720)[source]¶

Plot the interaction effect between two features on the predicted outcome using a 3D surface or contour plot.

Parameters:

X (pandas.DataFrame) – Feature matrix.
classifier – Trained classifier with predict or predict_proba method.
feature1 (str) – Name of the first feature (x-axis).
feature2 (str) – Name of the second feature (y-axis).
gridSize (int) – Number of grid points for each feature. Default is 30.
title (str) – Title of the plot. Default is “Interaction Effect Plot”.
save (bool) – Whether to save the plot to fileName. Default is False.
fileName (str) – File name to save the plot. Default is “InteractionEffectPlot.pdf”.
display (bool) – Whether to display the plot. Default is True.
fontSize (int) – Font size for labels and title. Default is 12.
plotType (str) – “surface” for 3D surface plot, “contour” for contour plot. Default is “surface”.
backend (str) – “matplotlib” for static plots, “plotly” for interactive HTML. Default is “matplotlib”.
dpi (int) – DPI for saving the figure. Default is 720.

Returns:

Figure object (matplotlib or plotly) or None.

Notes

For classifiers with predict_proba, the positive class probability is plotted if binary, otherwise the max probability.
For classifiers without predict_proba, the predicted class is plotted.
The plot is saved to fileName and optionally displayed.

Example

import HMB.PerformanceMetrics as pm

X = ...  # Load or create your feature DataFrame.
classifier = ...  # Load or train your classifier.

pm.PlotInteractionEffect(
  X, classifier,
  feature1="Flight Distance",
  feature2="Seat Comfort",
  gridSize=30,
  title="Interaction Effect Plot",
  save=True,
  fileName="InteractionEffectPlot.pdf",
  display=True,
  fontSize=14,
  plotType="surface",
  backend="matplotlib"
)

HMB.PerformanceMetrics.PlotCalibrationCurveFromModel(classifier, X, y, classNames=None, nBins=10, strategy='uniform', title='Calibration Curve', save=False, fileName='CalibrationCurve.pdf', display=True, fontSize=12, plotHistogram=False, returnFig=False, cmap=None, dpi=720)[source]¶

Plot calibration curves to evaluate how well predicted probabilities align with observed outcomes.

Parameters:

classifier – Trained classifier with predict_proba method.
X (pandas.DataFrame or numpy.ndarray) – Feature matrix.
y (array-like) – True labels.
classNames (list or None) – List of class names. If None, inferred from classifier.
nBins (int) – Number of bins to discretize the [0, 1] interval. Default is 10.
strategy (str) – Binning strategy (“uniform” or “quantile”). Default is “uniform”.
title (str) – Title of the plot. Default is “Calibration Curve”.
save (bool) – Whether to save the plot to fileName. Default is False.
fileName (str) – File name to save the plot. Default is “CalibrationCurve.pdf”.
display (bool) – Whether to display the plot. Default is True.
fontSize (int) – Font size for labels and title. Default is 12.
plotHistogram (bool) – Whether to plot a histogram of predicted probabilities below the calibration curve. Default is False.
returnFig (bool) – Whether to return the matplotlib figure object. Default is False.
cmap (str or Colormap or None) – Colormap to use for the curves. If None, uses “tab10”. Default is None.
dpi (int) – DPI for saving the figure. Default is 720.

Returns:

The figure object if returnFig is True, otherwise None.

Return type:

None or matplotlib.figure.Figure

Notes

For binary classification, plots the calibration curve for the positive class.
For multiclass, plots one-vs-rest calibration curves for each class.
The plot is saved to fileName and optionally displayed.
If plotHistogram is True, a histogram of predicted probabilities is shown below the calibration curve.
For best results, use with probabilistic models and sufficient sample size per bin.

Example

import HMB.PerformanceMetrics as pm

X = ...  # Load or create your feature DataFrame.
y = ...  # Load or create your true labels.
classifier = ...  # Load or train your classifier.
classNames = []  # Specify class names.

pm.PlotCalibrationCurveFromModel(
  classifier, X, y,
  classNames=classNames,
  nBins=10,
  strategy="uniform",
  title="Calibration Curve",
  save=True,
  fileName="CalibrationCurve.pdf",
  display=True,
  fontSize=14,
  plotHistogram=True,
  returnFig=False,
  cmap="tab20"
)

HMB.PerformanceMetrics.PlotCalibrationCurve(probs, labels, nBins=10, title='Calibration Curve', fontSize=14, figSize=(6, 6), display=True, save=False, fileName='CalibrationCurve.pdf', dpi=720, returnFig=False, color='blue')[source]¶

Plot calibration curve given predicted probabilities and true labels.

Parameters:

probs (numpy.ndarray) – Predicted probabilities of shape (nSamples, nClasses).
labels (array-like) – True labels of shape (nSamples,).
nBins (int) – Number of bins to discretize the [0, 1] interval. Default is 10.
title (str) – Title of the plot. Default is “Calibration Curve”.
fontSize (int) – Font size for labels and title. Default is 14.
figSize (tuple) – Figure size in inches. Default is (6, 6).
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot to fileName. Default is False.
fileName (str) – File name to save the plot. Default is “CalibrationCurve.pdf”.
dpi (int) – DPI for saving the figure. Default is 720.
returnFig (bool) – Whether to return the matplotlib figure object. Default is False.
color (str) – Color of the calibration curve. Default is “blue”.

Returns:

(binCenters, accuracy, confidence, support) where:

binCenters (numpy.ndarray): Centers of the bins.
accuracy (numpy.ndarray): Accuracy in each bin.
confidence (numpy.ndarray): Average confidence in each bin.
support (numpy.ndarray): Number of samples in each bin.

Return type:

tuple

Notes

The function computes the calibration data and plots the calibration curve.
The plot is saved to fileName and optionally displayed.

Example

import numpy as np
import HMB.PerformanceMetrics as pm

# Example predicted probabilities and true labels.
probs = np.array([[0.1, 0.9], [0.8, 0.2], [0.4, 0.6], [0.9, 0.1]])
labels = np.array([1, 0, 1, 0])

pm.PlotCalibrationCurve(
  probs, labels,
  nBins=5,
  title="Calibration Curve Example",
  fontSize=12,
  figSize=(5, 5),
  display=True,
  save=True,
  fileName="CalibrationCurveExample.pdf",
  dpi=300,
  returnFig=False,
  color="green"
)

HMB.PerformanceMetrics.PlotTopKAccuracyCurve(probs, labels, maxK=10, title='Top-k Accuracy Curve', figSize=(6, 6), save=False, fileName='TopKAccuracyCurve.pdf', display=True, fontSize=12, returnFig=False, dpi=720, color='blue')[source]¶

Plot Top-k accuracy curve given predicted probabilities and true labels.

Parameters:

probs (numpy.ndarray) – Predicted probabilities of shape (nSamples, nClasses).
labels (array-like) – True labels of shape (nSamples,).
maxK (int) – Maximum value of k for Top-k accuracy. Default is 10.
title (str) – Title of the plot. Default is “Top-k Accuracy Curve”.
figSize (tuple) – Figure size in inches. Default is (6, 6).
save (bool) – Whether to save the plot to fileName. Default is False.
fileName (str) – File name to save the plot. Default is “TopKAccuracyCurve.pdf”.
display (bool) – Whether to display the plot. Default is True.
fontSize (int) – Font size for labels and title. Default is 12.
returnFig (bool) – Whether to return the matplotlib figure object. Default is False.
dpi (int) – DPI for saving the figure. Default is 720.
color (str) – Color of the Top-k accuracy curve. Default is “blue”.

Returns:

The figure object if returnFig is True, otherwise None.

Return type:

None or matplotlib.figure.Figure

Notes

The function computes Top-k accuracy for k=1 to maxK and plots the curve.
The plot is saved to fileName and optionally displayed.

Example

import numpy as np
import HMB.PerformanceMetrics as pm

# Example predicted probabilities and true labels.
probs = np.array([[0.1, 0.9], [0.8, 0.2], [0.4, 0.6], [0.9, 0.1]])
labels = np.array([1, 0, 1, 0])

pm.PlotTopKAccuracyCurve(
  probs, labels,
  maxK=2,
  title="Top-k Accuracy Curve Example",
  figSize=(5, 5),
  save=True,
  fileName="TopKAccuracyCurveExample.pdf",
  display=True,
  fontSize=12,
  returnFig=False,
  dpi=300,
  color="green"
)

HMB.PerformanceMetrics.HistoryPlotter(history, title, metrics=('loss',), xLabel='Epochs', fontSize=14, save=False, savePath=None, dpi=720, colors=None, labels=None, display=True, figSize=(10, 5), returnFig=False, smooth=True, smoothFactor=0.6)[source]¶

Plot training history metrics (e.g., loss, accuracy) for train and validation sets.

Parameters:

history (dict) – Dictionary containing training history with keys like “train_loss”, “val_loss”, etc.
title (str) – Title of the plot.
metrics (tuple or list) – Metrics to plot (e.g., (“loss”, “accuracy”)). Default is (“loss”,).
xLabel (str) – Label for x-axis. Default is “Epochs”.
fontSize (int) – Font size for labels and title. Default is 14.
save (bool) – Whether to save the plot. Default is False.
savePath (str or None) – Path to save the plot. Default is None.
dpi (int) – DPI for saving the figure. Default is 720.
colors (dict or None) – Optional dict mapping metric names to colors.
labels (dict or None) – Optional dict mapping metric names to custom labels.
display (bool) – Whether to display the plot. Default is True.
figSize (tuple) – Figure size in inches. Default is (10, 5).
returnFig (bool) – Whether to return the matplotlib figure object. Default is False.
smooth (bool) – Whether to apply smoothing to the curves. Default is True.
smoothFactor (float) – Smoothing factor for curves (0 to 1). Default is 0.6.

Returns:

The axes object, figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

Supports plotting multiple metrics for both training and validation.
Custom colors and labels can be provided for each metric.
Saving and displaying the plot are optional and controlled by parameters.

Examples

import HMB.PerformanceMetrics as pm

history = {
  "train_loss": [0.8, 0.6, 0.4],
  "val_loss": [0.9, 0.7, 0.5],
  "train_accuracy": [0.5, 0.7, 0.9],
  "val_accuracy": [0.4, 0.6, 0.8]
}
pm.HistoryPlotter(
  history,
  title="Training History",
  metrics=("loss", "accuracy"),
  doSave=True,
  savePath="history_plot.pdf"
)

HMB.PerformanceMetrics.PlotMetricCurve(dataToPlot, xData=None, title='Metric Curve', xLabel='X', yLabel='Y', fontSize=15, xTicks=None, yTicks=None, xTicksRotation=0, yTicksRotation=0, save=False, savePath=None, dpi=720, display=True, figSize=(5, 5), returnFig=False)[source]¶

Plot a metric curve given data and optional x-axis values.

Parameters:

dataToPlot (array-like) – Data to plot on the y-axis.
xData (array-like or None) – Data for the x-axis. If None, uses indices of dataToPlot. Default is None.
title (str) – Title of the plot. Default is “Metric Curve”.
xLabel (str) – Label for the x-axis. Default is “X”.
yLabel (str) – Label for the y-axis. Default is “Y”.
fontSize (int) – Font size for labels and title. Default is 15.
xTicks (array-like or None) – Ticks for the x-axis. Default is None.
yTicks (array-like or None) – Ticks for the y-axis. Default is None.
xTicksRotation (int) – Rotation angle for x-axis ticks. Default is 0.
yTicksRotation (int) – Rotation angle for y-axis ticks. Default is 0.
save (bool) – Whether to save the plot. Default is False.
savePath (str or None) – Path to save the plot. Default is None.
dpi (int) – DPI for saving the figure. Default is 720.
display (bool) – Whether to display the plot. Default is True.
figSize (tuple) – Figure size in inches. Default is (5, 5).
returnFig (bool) – Whether to return the matplotlib figure object. Default is False.

Returns:

The figure object if returnFig is True, otherwise None

Return type:

matplotlib.figure.Figure or None

HMB.PerformanceMetrics.PlotCumulativeGainLiftChart(yTrue, yScores, posLabel=1, title='Cumulative Gain & Lift Chart', classNames=None, figsize=(10, 5), fontSize=14, display=True, save=False, fileName='CumulativeGainLiftChart.pdf', dpi=720, returnFig=False)[source]¶

Plot Cumulative Gain and Lift charts to visualize model effectiveness for ranking tasks.

Parameters:

yTrue (array-like) – True binary labels.
yScores (array-like) – Target scores/probabilities for the positive class.
posLabel (int or str) – The label of the positive class. Default is 1.
title (str) – Plot title. Default is “Cumulative Gain & Lift Chart”.
classNames (list or None) – Optional class names for legend. Default is None.
figsize (tuple) – Figure size. Default is (10, 5).
fontSize (int) – Font size for labels and title. Default is 14.
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot. Default is False.
fileName (str) – File name to save the plot. Default is “CumulativeGainLiftChart.pdf”.
dpi (int) – DPI for saving the figure. Default is 720.
returnFig (bool) – Whether to return the figure object. Default is False.

Returns:

The matplotlib figure object(s) if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

The Cumulative Gain chart shows the proportion of positives captured as more samples are included, sorted by model score.
The Lift chart shows the improvement over random selection at each proportion of the sample.
Saving and displaying the plot are optional and controlled by parameters.

Example

import HMB.PerformanceMetrics as pm

yTrue = [0, 1, 1, 0, 1, 0, 1, 0, 1, 0]
yScores = [0.1, 0.8, 0.7, 0.2, 0.9, 0.3, 0.6, 0.4, 0.5, 0.05]
pm.PlotCumulativeGainLiftChart(
  yTrue, yScores,
  title="Model Ranking Effectiveness",
  display=True,
  save=False
)

HMB.PerformanceMetrics.PlotErrorAnalysis(yTrue, yPred, X=None, classNames=None, maxExamples=5, fontSize=12, figsize=(14, 10), display=True, save=False, fileName='ErrorAnalysis.pdf', dpi=720, returnFig=False)[source]¶

Plot error analysis showing examples of false positives, false negatives, true positives, and true negatives.

Parameters:

yTrue (array-like) – True labels.
yPred (array-like) – Predicted labels.
X (array-like, DataFrame, or None) – Optional input samples to display. Default is None.
classNames (list or None) – Optional class names for display. Default is None.
maxExamples (int) – Max examples per error type to show. Default is 5.
fontSize (int) – Font size for text. Default is 12.
figsize (tuple) – Figure size. Default is (14, 10).
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot. Default is False.
fileName (str) – File name to save the plot. Default is “ErrorAnalysis.pdf”.
dpi (int) – DPI for saving the figure. Default is 720.
returnFig (bool) – Whether to return the figure object. Default is False.

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

Shows up to maxExamples for each of FP, FN, TP, TN, with sample indices and optionally sample data.
Column widths are dynamically calculated based on actual content (class names, indices, samples).
Useful for qualitative error analysis and debugging.
Saving and displaying the plot are optional and controlled by parameters.

Example

import HMB.PerformanceMetrics as pm

yTrue = [0, 1, 1, 0, 1, 0, 1, 0, 1, 0]
yPred = [0, 1, 0, 0, 1, 1, 1, 0, 0, 0]
PlotErrorAnalysis(
  yTrue, yPred,
  maxExamples=5,
  display=True,
  save=False
)

HMB.PerformanceMetrics.PlotClasswisePRFBar(cm, classNames=None, title='Classwise Performance Metrics', fontSize=14, figsize=(8, 5), display=True, save=False, fileName='ClasswisePRFBar.pdf', dpi=720, returnFig=False, xAxisRotation=45)[source]¶

Plot classwise Precision, Recall, and F1-score as a grouped bar chart.

Parameters:

cm (array-like) – Confusion matrix (2D array).
classNames (list or None) – List of class names. If None, uses class indices.
title (str) – Plot title. Default is “Classwise Performance Metrics”.
fontSize (int) – Font size for labels and title. Default is 14.
figsize (tuple) – Figure size. Default is (8, 5).
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot. Default is False.
fileName (str) – File name to save the plot. Default is “ClasswisePRFBar.pdf”.
dpi (int) – DPI for saving the figure. Default is 720.
returnFig (bool) – Whether to return the figure object. Default is False.
xAxisRotation (int) – Rotation angle for x-axis labels. Default is 45.

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

Computes precision, recall, and F1-score from the confusion matrix.
Displays a grouped bar chart for each class.
Saving and displaying the plot are optional and controlled by parameters.

Example

import HMB.PerformanceMetrics as pm

cm = [[50, 2, 1],
      [10, 45, 5],
      [0, 3, 47]]
classNames = ["Class A", "Class B", "Class C"]
pm.PlotClasswisePRFBar(
  cm, classNames=classNames,
  fontSize=12,
  figsize=(9, 6),
  display=True,
  save=True,
  fileName="ClasswisePRFBar.pdf",
  dpi=300,
  returnFig=False
)

HMB.PerformanceMetrics.PlotErrorMatrix(cm, classNames=None, fontSize=14, figsize=(7, 6), display=True, save=False, fileName='ErrorMatrix.pdf', dpi=720, returnFig=False)[source]¶

Plot confusion matrix (error matrix) with highlighted most common errors.

Parameters:

cm (array-like) – Confusion matrix (2D array).
classNames (list or None) – List of class names. If None, uses class indices.
fontSize (int) – Font size for labels and title. Default is 14.
figsize (tuple) – Figure size. Default is (7, 6).
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot. Default is False.
fileName (str) – File name to save the plot. Default is “ErrorMatrix.pdf”.
dpi (int) – DPI for saving the figure. Default is 720.
returnFig (bool) – Whether to return the figure object. Default is False.

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

Highlights the most common misclassifications in red.
Saving and displaying the plot are optional and controlled by parameters.

Example

import HMB.PerformanceMetrics as pm
import numpy as np

cm = np.array([[50, 2, 1],
               [10, 45, 5],
               [0, 3, 47]])
classNames = ["Class A", "Class B", "Class C"]
pm.PlotErrorMatrix(
  cm, classNames=classNames,
  fontSize=12,
  figsize=(8, 7),
  display=True,
  save=True,
  fileName="ErrorMatrix.pdf",
  dpi=300,
  returnFig=False
)

HMB.PerformanceMetrics.PlotMisclassificationExamples(yTrue, yPred, X=None, maxExamples=5, fontSize=12, figsize=(10, 5), display=True, save=False, fileName='MisclassificationExamples.pdf', dpi=720, returnFig=False)[source]¶

Plot most frequent misclassifications with example indices and optional sample data.

Parameters:

yTrue (array-like) – True labels.
yPred (array-like) – Predicted labels.
X (array-like, DataFrame, or None) – Optional input samples to display. Default is None.
maxExamples (int) – Max misclassification types to show. Default is 5.
fontSize (int) – Font size for text. Default is 12.
figsize (tuple) – Figure size. Default is (10, 5).
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot. Default is False.
fileName (str) – File name to save the plot. Default is “MisclassificationExamples.pdf”.
dpi (int) – DPI for saving the figure. Default is 720.
returnFig (bool) – Whether to return the figure object. Default is False.

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

Shows the most frequent misclassification pairs (true, predicted) with counts and optional samples.
Useful for qualitative error analysis and debugging.
Saving and displaying the plot are optional and controlled by parameters.

Example

import HMB.PerformanceMetrics as pm
import numpy as np

yTrue = np.array([0, 1, 1, 0, 1, 0, 1])
yPred = np.array([0, 1, 0, 0, 1, 1, 1])
X = np.array(["sample1", "sample2", "sample3", "sample4", "sample5", "sample6", "sample7"])
pm.PlotMisclassificationExamples(
  yTrue, yPred, X=X,
  maxExamples=3,
  fontSize=12,
  figsize=(9, 6),
  display=True,
  save=True,
  fileName="MisclassificationExamples.pdf",
  dpi=300,
  returnFig=False
)

HMB.PerformanceMetrics.PlotPredictionConfidenceHistogram(yPredProba, yPred=None, fontSize=14, figsize=(8, 5), bins=20, display=True, save=False, fileName='PredictionConfidenceHistogram.pdf', dpi=720, returnFig=False)[source]¶

Plot histogram of prediction confidences (predicted probabilities).

Parameters:

yPredProba (array-like) – Predicted probabilities (2D array for multi-class).
yPred (array-like or None) – Optional predicted classes. If None, uses argmax of yPredProba.
fontSize (int) – Font size for labels and title. Default is 14.
figsize (tuple) – Figure size. Default is (8, 5).
bins (int) – Number of bins for histogram. Default is 20.
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot. Default is False.
fileName (str) – File name to save the plot. Default is “PredictionConfidenceHistogram.pdf”.
dpi (int) – DPI for saving the figure. Default is 720.
returnFig (bool) – Whether to return the figure object. Default is False.

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

Shows histogram of predicted probabilities for the predicted class.
Useful for assessing model confidence and calibration.
Saving and displaying the plot are optional and controlled by parameters.

Example

import HMB.PerformanceMetrics as pm
import numpy as np

yPredProba = np.array([[0.1, 0.7, 0.2],
                      [0.8, 0.1, 0.1],
                      [0.3, 0.4, 0.3],
                      [0.2, 0.2, 0.6]])
yPred = np.array([1, 0, 1, 2])
pm.PlotPredictionConfidenceHistogram(
  yPredProba, yPred=yPred,
  fontSize=12,
  figsize=(9, 6),
  bins=10,
  display=True,
  save=True,
  fileName="PredictionConfidenceHistogram.pdf",
  dpi=300,
  returnFig=False
)

HMB.PerformanceMetrics.PlotClassificationResiduals(yTrue, yPred, fontSize=14, figsize=(8, 5), display=True, save=False, fileName='ClassificationResiduals.pdf', dpi=720, returnFig=False)[source]¶

Plot residuals for classification tasks (true - predicted).

Parameters:

yTrue (array-like) – True labels.
yPred (array-like) – Predicted labels.
fontSize (int) – Font size for labels and title. Default is 14.
figsize (tuple) – Figure size. Default is (8, 5).
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot. Default is False.
fileName (str) – File name to save the plot. Default is “ClassificationResiduals.pdf”.
dpi (int) – DPI for saving the figure. Default is 720.
returnFig (bool) – Whether to return the figure object. Default is False.

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Example

import HMB.PerformanceMetrics as pm

yTrue = [0, 1, 1, 0, 1, 0, 1]
yPred = [0, 1, 0, 0, 1, 1, 1]
pm.PlotClassificationResiduals(
  yTrue, yPred,
  fontSize=12,
  figsize=(9, 6),
  display=True,
  save=True,
  fileName="ClassificationResiduals.pdf",
  dpi=300,
  returnFig=False
)

HMB.PerformanceMetrics.PlotFeatureImportance(model, featureNames, title='Feature Importance', fontSize=14, figsize=(8, 5), display=True, save=False, fileName='FeatureImportance.pdf', dpi=720, returnFig=False, topN=None)[source]¶

Plot feature importance from a trained model.

Parameters:

model – Trained model with feature_importances_ or coef_ attribute.
featureNames (list) – List of feature names.
title (str) – Title of the plot. Default is “Feature Importance”.
fontSize (int) – Font size for labels and title. Default is 14.
figsize (tuple) – Figure size. Default is (8, 5).
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot. Default is False.
fileName (str) – File name to save the plot. Default is “FeatureImportance.pdf”.
dpi (int) – DPI for saving the figure. Default is 720.
returnFig (bool) – Whether to return the figure object. Default is False.
topN (int or None) – If specified, show only the top N features. Default is None (show all).

Returns:

The matplotlib figure object if returnFig is True, otherwise None.

Return type:

matplotlib.figure.Figure or None

Notes

Supports models with feature_importances_ (e.g., tree-based) or coef_ (e.g., linear models).
Displays a bar chart of feature importances.
Saving and displaying the plot are optional and controlled by parameters.

Example

import HMB.PerformanceMetrics as pm
from sklearn.ensemble import RandomForestClassifier
import numpy as np
from sklearn.datasets import load_iris

data = load_iris()
X = data.data
y = data.target
featureNames = data.feature_names

model = RandomForestClassifier()
model.fit(X, y)
pm.PlotFeatureImportance(
  model, featureNames,
  title="Iris Feature Importance",
  fontSize=12,
  figsize=(9, 6),
  display=True,
  save=True,
  fileName="IrisFeatureImportance.pdf",
  dpi=300,
  returnFig=False,
  topN=3
)

HMB.PerformanceMetrics.SampleMonteCarloDirichletFromProbs(probs, T=100, concentration=50.0, rng=None)[source]¶

Create T Monte Carlo softmax probability samples for each row in probs by sampling from a Dirichlet distribution with concentration proportional to the provided probs.

Parameters:

probs (numpy.ndarray) – 2D array of shape (N, C) with base probabilities.
T (int) – Number of Monte Carlo samples to draw per instance. Default is 100.
concentration (float) – Concentration parameter for the Dirichlet distribution. Higher values lead to samples closer to the base probs.
rng (numpy.random.Generator or None) – Optional random number generator for reproducibility. If None, a new generator is created.

Returns:

3D array of shape (T, N, C) with Monte Carlo probability samples.

Return type:

numpy.ndarray

Example

import numpy as np
import HMB.PerformanceMetrics as pm

probs = np.array([[0.7, 0.2, 0.1],
                  [0.1, 0.8, 0.1]])
T = 500
samples = pm.SampleMonteCarloDirichletFromProbs(probs, T=T, concentration=30.0)
print(samples.shape)  # Should be (500, 2, 3).

HMB.PerformanceMetrics.ComputeMonteCarloUncertaintyMeasures(probsMC, eps=1e-12)[source]¶

Given Monte Carlo probability samples, compute useful uncertainty measures.

Parameters:

probsMC (numpy.ndarray) – 3D array of shape (T, N, C) with Monte Carlo probability samples.
eps (float) – Small value to avoid log(0). Default is 1e-12.

Returns:

Dictionary containing the following keys:

”predictiveMean”: (N, C) array of predictive mean probabilities.
”predictiveEntropy”: (N,) array of predictive entropy values.
”expectedEntropy”: (N,) array of expected entropy values.
”mutualInformation”: (N,) array of mutual information values (epistemic uncertainty).
”varTop”: (N,) array of variance of top class probabilities across T samples.
”predictedIdx”: (N,) array of predicted class indices from predictive mean.
”predictedConfidence”: (N,) array of predicted class confidences from predictive mean.

Return type:

dict

Example

import numpy as np
import HMB.PerformanceMetrics as pm

probs = np.array([[0.7, 0.2, 0.1],
                  [0.1, 0.8, 0.1]])
T = 500
probsMC = pm.SampleMonteCarloDirichletFromProbs(probs, T=T, concentration=30.0)
uncertaintyMeasures = pm.ComputeMonteCarloUncertaintyMeasures(probsMC)
print(uncertaintyMeasures["predictiveMean"]) # Should be close to original probs.

HMB.PerformanceMetrics.ComputeECE(probabilities, labels, binCount=15, nBins=None)[source]¶

Compute Expected Calibration Error (ECE).

This function accepts either:

a 2D array of per-class probabilities (N x C) with integer class labels.
a 1D array of confidences (N,) with labels as 0/1 correctness indicators or class labels.

The optional legacy keyword nBins is accepted for compatibility and overrides binCount when provided.

Parameters:

probabilities (list or numpy.ndarray) – List/array of predicted probabilities or confidences.
labels (list or numpy.ndarray) – List/array of true labels or correctness indicators.
binCount (int) – Number of bins to use. Default is 15.
nBins (int or None) – Legacy argument name for number of bins. If provided, overrides binCount.

Returns:

Expected Calibration Error (ECE) value.

Return type:

float

HMB.PerformanceMetrics.ComputeECEPlotReliability(confidences, predictions, labels, nBins=15, title='Expected Calibration Error (ECE)', fontSize=14, figSize=(6, 6), display=True, save=False, fileName='ECE.pdf', dpi=720, returnFig=False, cmap='Blues', applyXYLimits=True)[source]¶

Compute Expected Calibration Error (ECE) and plot reliability diagram.

Parameters:

confidences (list or numpy.ndarray) – List/array of predicted confidences (max class prob).
predictions (list or numpy.ndarray) – List/array of predicted labels.
labels (list or numpy.ndarray) – List/array of true labels.
nBins (int) – number of bins to use.
title (str) – Title of the plot. Default is “Expected Calibration Error (ECE)”.
fontSize (int) – Font size for labels and title. Default is 14.
figSize (tuple) – Figure size. Default is (6, 6).
display (bool) – Whether to display the plot. Default is True.
save (bool) – Whether to save the plot. Default is False.
fileName (str) – File name to save the plot. Default is “ECE.pdf”.
dpi (int) – DPI for saving the figure. Default is 720.
returnFig (bool) – Whether to return the figure object. Default is False.
cmap (str) – Colormap for the plot. Default is “Blues”.
applyXYLimits (bool) – Whether to apply x and y limits [0, 1]. Default is True.

Returns:

Expected calibration error. binAcc (list): List of accuracies per bin. binConf (list): List of average confidences per bin. binCounts (list): List of sample counts per bin. fig (matplotlib.figure.Figure, optional): The matplotlib figure object if returnFig is True.

Return type:

ece (float)

Notes

ECE quantifies the difference between predicted confidence and actual accuracy.
The reliability diagram visualizes calibration across confidence bins.
Saving and displaying the plot are optional and controlled by parameters.

Example

import numpy as np
import HMB.PerformanceMetrics as pm

# You would typically get confidences and correctness from model predictions.
probs = np.array([[0.7, 0.2, 0.1], [0.1, 0.8, 0.1]])
T = 500
probsMC = pm.SampleMonteCarloDirichletFromProbs(probs, T=T, concentration=30.0)
uncertaintyMeasures = pm.ComputeMonteCarloUncertaintyMeasures(probsMC)
confidences = uncertaintyMeasures["predictedConfidence"]
predictions = uncertaintyMeasures["predictedIdx"]
labels = np.array([0, 1])  # True labels for the examples.

# Sample data for demonstration:
# confidences = np.array([0.9, 0.8, 0.7, 0.6, 0.5])
# predictions = np.array([1, 0, 1, 1, 0])
# labels = np.array([1, 0, 0, 1, 0])

ece, binAcc, binConf, binCounts = pm.ComputeECEPlotReliability(
  confidences,
  predictions,
  labels,
  nBins=5,
  title="ECE Example",
  fontSize=14,
  figSize=(6, 6),
  display=True,
  save=False,
  fileName="ECE_Example.pdf",
  dpi=300,
  returnFig=False,
  cmap="Blues",
  applyXYLimits=True
)
print(f"ECE: {ece}")
print(f"Bin Accuracies: {binAcc}")
print(f"Bin Confidences: {binConf}")
print(f"Bin Counts: {binCounts}")

HMB.PerformanceMetrics.RiskCoverageCurve(confidences, correctness, title='Risk-Coverage (Accuracy vs Coverage)', fontSize=14, figSize=(6, 6), display=True, save=False, fileName='RiskCoverage.pdf', dpi=720, returnFig=False, color='blue')[source]¶

Compute and plot risk (error) vs coverage curve sorted by confidence. The risk-coverage curve shows accuracy as a function of coverage when rejecting low-confidence predictions. The more area under the curve (AUC), the better the selective prediction performance. For example, a model that is perfectly calibrated and accurate will have AUC=1.0, while a random model will have AUC close to the accuracy level.

Parameters:

confidences (numpy.ndarray) – 1D array of prediction confidences.
correctness (numpy.ndarray) – 1D boolean array of correctness (True=correct).
title (str) – Plot title.
fontSize (int) – Font size for plot.
figSize (tuple) – Figure size.
display (bool) – Whether to display the plot.
save (bool) – Whether to save the plot.
fileName (str) – File name to save the plot.
dpi (int) – DPI for saved figure.
returnFig (bool) – Whether to return the figure object.
color (str) – Color for the plot line.

Returns:

coverage levels.

Return type:

coverage (numpy.ndarray)

Example

import numpy as np
import HMB.PerformanceMetrics as pm

# You would typically get confidences and correctness from model predictions.
probs = np.array([[0.7, 0.2, 0.1], [0.1, 0.8, 0.1]])
T = 500
probsMC = pm.SampleMonteCarloDirichletFromProbs(probs, T=T, concentration=30.0)
uncertaintyMeasures = pm.ComputeMonteCarloUncertaintyMeasures(probsMC)
confidences = uncertaintyMeasures["predictedConfidence"]
predictions = uncertaintyMeasures["predictedIdx"]
labels = np.array([0, 1])  # True labels.
correctness = (predictions == labels).astype(int)

# Sample data for demonstration:
# confidences = np.array([0.9, 0.8, 0.7, 0.6, 0.5])
# correctness = np.array([1, 0, 1, 1, 0])  # 1=correct, 0=incorrect.

coverage, accuracy, aucVal, fig = pm.RiskCoverageCurve(
  confidences,
  correctness,
  title="Risk-Coverage (Accuracy vs Coverage)",
  fontSize=14,
  figSize=(6, 6),
  display=True,
  save=False,
  fileName="RiskCoverage.pdf",
  dpi=720,
  returnFig=False,
  color="blue"
)

HMB.PerformanceMetrics.ComputeBrierScore(confidences, correctness)[source]¶

Compute Brier Score given prediction confidences and correctness indicators. Compute Brier score = mean((conf - correct)^2).

Parameters:

confidences (list or numpy.ndarray) – List/array of predicted confidences (0.0 to 1.0).
correctness (list or numpy.ndarray) – List/array of correctness indicators (0.0 or 1.0).

Returns:

Brier score value, or None if inputs are invalid.

Return type:

float

Example

import numpy as np
import HMB.PerformanceMetrics as pm

confidences = [0.9, 0.8, 0.7, 0.6, 0.5]
correctness = [1, 0, 1, 1, 0]
brierScore = pm.ComputeBrierScore(confidences, correctness)
print(f"Brier Score: {brierScore}")