AgentsHelper Module¶

Helper functions for agent utilities and orchestration.

class HMB.AgentsHelper.RandomAgent(ActionSpaceSampleFunc)[source]¶

Bases: object

Random policy baseline that always samples from a provided sampler. Useful as a non-learning baseline or to perform pure exploration.

Initialize RandomAgent.

Parameters:: ActionSpaceSampleFunc (callable) – Callable returning a random action index when called.

__init__(ActionSpaceSampleFunc)[source]¶

Initialize RandomAgent.

Parameters:: ActionSpaceSampleFunc (callable) – Callable returning a random action index when called.

ChooseAction(state=None)[source]¶

Return a randomly sampled action.

Parameters:: state – Ignored by RandomAgent but kept for API compatibility.
Returns:: Random action index returned by ActionSpaceSampleFunc().
Return type:: int

class HMB.AgentsHelper.QAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1)[source]¶

Bases: object

Base tabular Q-agent with epsilon-greedy action selection.

This base class stores a Q-table and implements a simple epsilon-greedy action selection policy. Subclasses should implement UpdateParameters to perform learning updates (Q-learning, SARSA, Expected SARSA, …).

Parameters:

ActionSpaceSampleFunc (callable) – Callable returning a sampled action index from the environment’s action space; used for exploration.
alpha (float) – Learning rate (0 < alpha <= 1).
gamma (float) – Discount factor for future rewards (0 <= gamma <= 1).
noOfStates (int) – Number of discrete states (size of first Q dimension).
noOfActions (int) – Number of discrete actions (size of second Q dimension).
epsilon (float, optional) – Exploration probability for epsilon-greedy policy (default 0.1).

Variables:

qTable (numpy.ndarray) – Array of shape (noOfStates, noOfActions) storing Q-values.
epsilon (alpha, gamma,) – Learning and policy hyperparameters.
ActionSpaceSampleFunc (callable) – Stored reference to the sampler.

Initialize the QAgent and allocate the Q-table.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.

__init__(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1)[source]¶

Initialize the QAgent and allocate the Q-table.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.

ChooseAction(state)[source]¶

Select an action using an epsilon-greedy strategy.

Parameters:: state (int) – Current discrete state index.
Returns:: Chosen action index. If exploring, result of ActionSpaceSampleFunc; otherwise the greedy action (argmax over Q-values).
Return type:: int

GetAction(state)[source]¶

Return the greedy action for the given state using the Q-table.

Parameters:: state (int) – State index to query.
Returns:: Index of the action with maximal Q-value for the state. Ties are resolved by numpy.argmax (first occurrence).
Return type:: int

class HMB.AgentsHelper.QLearningAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1)[source]¶

Bases: QAgent

Tabular Q-learning agent implementing the standard off-policy update.

Implements the classical Q-learning update which bootstraps using the maximum action-value in the next state.

Initialize the QAgent and allocate the Q-table.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.

UpdateParameters(state, nextState, reward, action, nextAction)[source]¶

Perform a Q-learning update for a single transition.

Parameters:

state (int) – Previous state index.
nextState (int) – Next state index after taking the action.
reward (float) – Observed reward for the transition.
action (int) – Action index taken in state.
nextAction (int or None) – Present for API parity with on-policy agents but unused by Q-learning.

class HMB.AgentsHelper.SARSAAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1)[source]¶

Bases: QAgent

On-policy SARSA agent that updates using the Q-value of the next taken action.

Uses the (state, action, reward, nextState, nextAction) tuple to form the TD target for on-policy updates.

Initialize the QAgent and allocate the Q-table.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.

UpdateParameters(prevState, nextState, reward, prevAction, nextAction)[source]¶

Perform a SARSA update for the observed transition.

Parameters:

prevState (int) – The previous state index.
nextState (int) – The subsequent state index.
reward (float) – Observed reward for the transition.
prevAction (int) – Action taken in prevState.
nextAction (int) – Action taken in nextState (on-policy).

class HMB.AgentsHelper.ExpectedSARSAAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1)[source]¶

Bases: QAgent

Expected SARSA agent that uses the expectation under the epsilon-greedy policy as the bootstrap target.

The expected value is computed exactly by accounting for the epsilon mass that is shared between greedy and non-greedy actions and handling ties.

Initialize the QAgent and allocate the Q-table.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.

UpdateParameters(prevState, nextState, reward, prevAction, nextAction)[source]¶

Perform an Expected SARSA update using the expectation over the policy in the next state.

Parameters:

prevState (int) – Previous state index.
nextState (int) – Next state index.
reward (float) – Observed reward.
prevAction (int) – Action taken in previous state.
nextAction (int or None) – Present for API parity but unused here.

class HMB.AgentsHelper.GreedyAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions)[source]¶

Bases: QAgent

Deterministic greedy policy that always picks the current best action. This convenience subclass of QAgent enforces epsilon=0 for deterministic action selection.

Initialize GreedyAgent by calling QAgent with epsilon forced to 0.

Parameters:: 0. (See QAgent.__init__ for parameter descriptions; epsilon is forced to)

__init__(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions)[source]¶

Initialize GreedyAgent by calling QAgent with epsilon forced to 0.

Parameters:: 0. (See QAgent.__init__ for parameter descriptions; epsilon is forced to)

ChooseAction(state)[source]¶

Choose the greedy action deterministically.

Parameters:: state (int) – Current state index.
Returns:: Index of the greedy action according to the stored Q-table.
Return type:: int

class HMB.AgentsHelper.SoftmaxPolicyAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, temperature=1.0, epsilon=0.0)[source]¶

Bases: QAgent

Softmax (Boltzmann) policy over Q-values for stochastic action selection.

The temperature parameter controls exploration: lower values concentrate probability mass on higher-valued actions, higher values approach a uniform distribution. An optional small epsilon mixes the softmax probabilities with a uniform distribution.

Initialize the SoftmaxPolicyAgent.

Parameters:

temperature (float) – Softmax temperature (must be > 0). Lower favors greedy actions.
epsilon (float, optional) – Small probability to mix the softmax distribution with a uniform distribution (default 0.0).

__init__(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, temperature=1.0, epsilon=0.0)[source]¶

Initialize the SoftmaxPolicyAgent.

Parameters:

temperature (float) – Softmax temperature (must be > 0). Lower favors greedy actions.
epsilon (float, optional) – Small probability to mix the softmax distribution with a uniform distribution (default 0.0).

ChooseAction(state)[source]¶

Sample an action according to the softmax policy (optionally mixed with epsilon uniform).

Parameters:: state (int) – Current state index.
Returns:: Sampled action index.
Return type:: int

class HMB.AgentsHelper.DoubleQLearningAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1)[source]¶

Bases: QAgent

Double Q-learning agent implementing two Q-tables to reduce overestimation.

On each update a random coin flip decides which table is updated; the other table is used to evaluate the chosen action (the Double Q-learning scheme described by Van Hasselt et al.).

Initialize double-Q tables and parameters.

Parameters:: internally. (See QAgent.__init__ for parameter descriptions; an additional second Q-table is allocated)

__init__(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1)[source]¶

Initialize double-Q tables and parameters.

Parameters:: internally. (See QAgent.__init__ for parameter descriptions; an additional second Q-table is allocated)

ChooseAction(state)[source]¶

Choose action using the sum of both Q-tables for improved selection.

Parameters:: state (int) – Current state index.
Returns:: Action index chosen either randomly (with probability epsilon) or as the argmax of qTable + qTable2.
Return type:: int

UpdateParameters(state, nextState, reward, action, nextAction=None)[source]¶

Perform Double Q-learning update by randomly choosing which table to update.

If the chosen table is A then the argmax is computed on A and evaluated using B (and vice-versa). This reduces the maximization bias present in standard Q-learning.

Parameters:

state (int) – Current state index.
nextState (int) – Next state index.
reward (float) – Observed reward.
action (int) – Action taken in state.
nextAction (int or None) – Present for API parity; unused here.

class HMB.AgentsHelper.QLambdaAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, lambd=0.9, epsilon=0.1)[source]¶

Bases: QAgent

Q(u03BB) agent with accumulating eligibility traces (tabular).

Implements off-policy Q(u03BB) with accumulating traces. The agent maintains an eligibility matrix of the same shape as the Q-table and updates all Q-values proportionally to their eligibility on each step.

Initialize QLambdaAgent with eligibility traces.

Parameters:: lambd (float) – Eligibility trace decay parameter (u03BB), typically in [0, 1].

__init__(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, lambd=0.9, epsilon=0.1)[source]¶

Initialize QLambdaAgent with eligibility traces.

Parameters:: lambd (float) – Eligibility trace decay parameter (u03BB), typically in [0, 1].

ResetTraces()[source]¶

Reset eligibility traces to zero.

Call this at the start of each episode if episodic resets are used.

UpdateParameters(state, nextState, reward, action, nextAction)[source]¶

Perform Q(u03BB) update with accumulating traces for a single transition.

This implementation uses the off-policy max over next-state actions when forming the TD target (i.e., Q-based bootstrap).

Parameters:

state (int) – Current state index.
nextState (int) – Next state index after taking action.
reward (float) – Observed reward.
action (int) – Action taken in state.
nextAction (int or None) – Present for API parity but unused for the off-policy Q(u03BB) update.

class HMB.AgentsHelper.SARSALambdaAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, lambd=0.9, epsilon=0.1, traceType='accumulating')[source]¶

Bases: QAgent

On-policy SARSA(u03BB) agent with eligibility traces.

This agent implements accumulating or replacing eligibility traces for SARSA-style on-policy learning. Call ResetTraces() at episode start.

Parameters:

ActionSpaceSampleFunc (callable) – Action sampler used for exploration.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
lambd (float, optional) – Trace-decay parameter (default 0.9).
epsilon (float, optional) – Epsilon for epsilon-greedy policy (default 0.1).
trace_type (str, optional) – “accumulating” or “replacing” (default “accumulating”).

Initialize the SARSALambdaAgent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
lambd (float, optional) – Trace-decay parameter for eligibility traces.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.
traceType (str, optional) – Type of eligibility traces (“accumulating” or “replacing”).

__init__(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, lambd=0.9, epsilon=0.1, traceType='accumulating')[source]¶

Initialize the SARSALambdaAgent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
lambd (float, optional) – Trace-decay parameter for eligibility traces.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.
traceType (str, optional) – Type of eligibility traces (“accumulating” or “replacing”).

ResetTraces()[source]¶: Zero the eligibility traces (call at episode start).

UpdateParameters(prevState, nextState, reward, prevAction, nextAction)[source]¶

Perform SARSA(u03BB) update using the on-policy TD error.

Parameters follow the SARSA convention; nextAction is used for the on-policy bootstrap.

class HMB.AgentsHelper.MonteCarloAgent(ActionSpaceSampleFunc, gamma, noOfStates, noOfActions, epsilon=0.1, useFirstVisit=True)[source]¶

Bases: object

First-visit Monte Carlo control (episodic) with incremental averaging.

Stores an episode buffer and updates Q at episode end using returns.

Parameters:

ActionSpaceSampleFunc (callable) – Action sampler for exploration.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Epsilon for epsilon-greedy policy (default 0.1).
useFirstVisit (bool, optional) – If True use first-visit MC, otherwise every-visit.

Initialize the MonteCarloAgent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.
useFirstVisit (bool, optional) – If True use first-visit MC, otherwise every-visit.

__init__(ActionSpaceSampleFunc, gamma, noOfStates, noOfActions, epsilon=0.1, useFirstVisit=True)[source]¶

Initialize the MonteCarloAgent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.
useFirstVisit (bool, optional) – If True use first-visit MC, otherwise every-visit.

ChooseAction(state)[source]¶

Epsilon-greedy action selection based on current Q-table.

Parameters:: state (int) – Current discrete state index.
Returns:: Chosen action index. If exploring, result of ActionSpaceSampleFunc; otherwise the greedy action (argmax over Q-values).
Return type:: int

StoreTransition(state, action, reward)[source]¶

Append a transition to the current episode buffer.

Parameters:

state (int) – State index.
action (int) – Action index.
reward (float) – Reward value.

EndEpisodeAndUpdate()[source]¶: Process stored episode and perform first-visit (or every-visit) Monte Carlo updates, then clear the episode buffer.

class HMB.AgentsHelper.DynaQAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1, planningSteps=5)[source]¶

Bases: QAgent

Dyna-Q agent: model-based planning with a simple tabular model.

Learns a one-step model (last observed next state and reward per (s,a)) and performs planning updates by sampling previously observed pairs.

Parameters:

ActionSpaceSampleFunc (callable) – Action sampler for exploration.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Epsilon for epsilon-greedy policy (default 0.1).
planningSteps (int, optional) – Number of model-based planning updates per real step.

Initialize the DynaQAgent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.
planningSteps (int, optional) – Number of planning steps to perform.

__init__(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1, planningSteps=5)[source]¶

Initialize the DynaQAgent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.
planningSteps (int, optional) – Number of planning steps to perform.

UpdateParameters(state, nextState, reward, action, nextAction=None)[source]¶

Perform a real experience Q-learning update, update the model, and run planningSteps simulated updates sampled from observed (s,a) pairs.

Parameters:

state (int) – Current state index.
nextState (int) – Next state index.
reward (float) – Observed reward.
action (int) – Action taken in state.
nextAction (int or None) – Present for API parity; unused here.

class HMB.AgentsHelper.UCB1Agent(ActionSpaceSampleFunc, noOfActions, c=1.0)[source]¶

Bases: object

UCB1 multi-armed bandit agent (no states).

Uses the UCB1 formula to select actions in pure bandit problems.

Initialize the UCB1Agent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
noOfActions (int) – Number of available actions (arms).
c (float, optional) – Exploration parameter for UCB1 (default 1.0).

__init__(ActionSpaceSampleFunc, noOfActions, c=1.0)[source]¶

Initialize the UCB1Agent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
noOfActions (int) – Number of available actions (arms).
c (float, optional) – Exploration parameter for UCB1 (default 1.0).

ChooseAction(state=None)[source]¶

Choose an action using the UCB1 rule.

Parameters:: state – Ignored by UCB1Agent but kept for API compatibility.
Returns:: Action index selected by the UCB1 algorithm.
Return type:: int

UpdateParameters(action, reward)[source]¶

Update running average for the chosen arm.

Parameters:

action (int) – Chosen action index.
reward (float) – Observed reward.

class HMB.AgentsHelper.CountBonusQLAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1, bonus_coef=1.0, countType='state')[source]¶

Bases: QAgent

Q-learning augmented with a count-based intrinsic bonus to encourage exploration.

The bonus can be computed per-state or per state-action pair.

Initialize the CountBonusQLAgent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.
bonus_coef (float, optional) – Coefficient for the exploration bonus.
countType (str, optional) – Type of count-based bonus (“state” or “state_action”).

__init__(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1, bonus_coef=1.0, countType='state')[source]¶

Initialize the CountBonusQLAgent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.
bonus_coef (float, optional) – Coefficient for the exploration bonus.
countType (str, optional) – Type of count-based bonus (“state” or “state_action”).

UpdateParameters(state, nextState, reward, action, nextAction=None)[source]¶

Perform Q-learning update with an added exploration bonus computed from visitation counts.

Parameters:

state (int) – Current state index.
nextState (int) – Next state index.
reward (float) – Observed reward.
action (int) – Action taken in state.
nextAction (int or None) – Present for API parity; unused here.

class HMB.AgentsHelper.NStepTDAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, n=3, epsilon=0.1)[source]¶

Bases: object

n-step on-policy TD agent (n-step SARSA style).

Maintains a buffer of recent transitions and performs n-step updates.

Initialize the NStepTDAgent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
n (int, optional) – Number of steps for n-step TD updates (default 3).
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.

__init__(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, n=3, epsilon=0.1)[source]¶

Initialize the NStepTDAgent.

Parameters:

ActionSpaceSampleFunc (callable) – Function to sample random actions.
alpha (float) – Learning rate.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
n (int, optional) – Number of steps for n-step TD updates (default 3).
epsilon (float, optional) – Exploration probability for epsilon-greedy policy.

ChooseAction(state)[source]¶

Select an action using an epsilon-greedy strategy.

Parameters:: state (int) – Current discrete state index.
Returns:: Chosen action index. If exploring, result of ActionSpaceSampleFunc; otherwise the greedy action (argmax over Q-values).
Return type:: int

UpdateParameters(state, nextState, reward, action, nextAction, done=False)[source]¶

Step the agent with a (state, action, reward) sample and perform any ready n-step updates. If done==True, flush remaining updates.

Parameters:

state (int) – Current state index.
nextState (int) – Next state index.
reward (float) – Observed reward.
action (int) – Action taken in state.
nextAction (int) – Action taken in nextState (on-policy).
done (bool, optional) – Flag indicating episode termination (default False).

class HMB.AgentsHelper.PrioritizedSweepingAgent(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1, planningSteps=5, theta=0.0001)[source]¶

Bases: QAgent

Prioritized Sweeping agent (model-based planning with prioritized updates).

This agent maintains a one-step model (reward and next-state for each observed (s,a)) and a predecessor map for states. When a real transition is observed it computes a priority for the state and uses a priority queue to selectively perform planning updates on predecessor state-action pairs with largest expected TD error first. This is useful for sample-efficient planning in small tabular MDPs.

Parameters:

ActionSpaceSampleFunc (callable) – Action sampler used for exploration.
alpha (float) – Learning rate for Q-value updates.
gamma (float) – Discount factor.
noOfStates (int) – Number of discrete states.
noOfActions (int) – Number of discrete actions.
epsilon (float, optional) – Epsilon for epsilon-greedy action selection (default 0.1).
planningSteps (int, optional) – Number of prioritized-planning iterations to perform per real update (default 5).
theta (float, optional) – Priority threshold; only priorities greater than theta are pushed (default 1e-4).

Initialize PrioritizedSweepingAgent and allocate model/priorities.

__init__(ActionSpaceSampleFunc, alpha, gamma, noOfStates, noOfActions, epsilon=0.1, planningSteps=5, theta=0.0001)[source]¶: Initialize PrioritizedSweepingAgent and allocate model/priorities.

ResetModel()[source]¶: Clear the learned model and predecessor information.

ChooseAction(state)[source]¶: Choose an action using epsilon-greedy based on current Q-table.

UpdateParameters(state, nextState, reward, action, nextAction=None)[source]¶

Process a real experience (state, action, reward, nextState):

Update Q with a standard Q-learning step (real experience).
Update internal one-step model and predecessors.
Compute priority for the experienced state and push it if large.
Run up to planningSteps prioritized planning updates.

Parameters:

state (int) – Current state index.
nextState (int) – Next state index observed.
reward (float) – Observed reward.
action (int) – Action taken in state.