Recurrent PPO RND

Recurrent PPO RND is modified version of PPO RND to use recurrent network like LSTM or GRU. You can see why recurrent network is useful in Recurrent PPO.

Configuration

Network

class: RecurrentPPORNDNetwork

Since it uses the recurrent network, you must consider the hidden state which can achieve the action-observation history.

Note that since PPO uses the Actor-Critic architecure and the parameter sharing, the encoding layer must be shared between Actor and Critic. Be careful not to share parameters between PPO and RND networks.

RND uses extrinsic and intrinsic reward streams. Each stream can be different episodic or non-episodic, and can have different discount factors. RND constitutes of the predictor and target networks. Both of them should have the similar architectures (not must same) but their initial parameters should not be the same. The target network is deterministic, which means it will be never updated.

You need to implement below methods.

Forward Actor-Critic

@abstractmethod
def forward_actor_critic(
    self, 
    obs_seq: Observation, 
    hidden_state: Tensor
) -> tuple[PolicyDist, Tensor, Tensor, Tensor]

Parameters:

Returns:

Refer to the following explanation:

• seq_batch_size: the number of independent sequences
• seq_len: the length of each sequence
• num_layers: the number of recurrent layers
• D: 2 if bidirectional otherwise 1
• H: the value depends on the type of the recurrent network

When you use LSTM, H = H_cell + H_out. See details in https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html.
When you use GRU, H = H_out. See details in https://pytorch.org/docs/stable/generated/torch.nn.GRU.html.

Forward RND

@abstractmethod
def forward_rnd(
    self, 
    obs: Observation, 
    hidden_state: Tensor
) -> tuple[Tensor, Tensor]

The value of out_features depends on you.

Parameters:

Returns: