Flexibility for replay buffer storage

[JoseLuisC99]

For environments with large observation spaces and numerous agents (such as MAgent), storing the replay buffer in RAM may become infeasible due to hardware limitations. To address this issue, TorchRL offers an alternative storage option: LazyMemmapStorage.

As explained in the documentation, LazyMemmapStorage functions similarly to LazyTensorStorage but utilizes disk files instead of RAM. This approach allows for handling extremely large datasets while maintaining efficient, contiguous data access.

It would be highly beneficial if BenchMARL provided greater flexibility in choosing between storing the replay buffer in RAM or on the hard disk, depending on the use case and resource availability.

Relevant links: https://pytorch.org/rl/stable/reference/generated/torchrl.data.replay_buffers.LazyMemmapStorage.html

[JoseLuisC99]

I created a PR #155 with the solution that I implemented for myself, it can help you. With this solution, I only add an extra argument to my experiment when I create it.

experiment = Experiment(
  task=task_config,
  algorithm_config=VdnConfig.get_from_yaml(),
  model_config=model_config,
  critic_model_config=model_config,
  seed=seed,
  config=experiment_config,
  replay_buffer_storage=PhysicalStorage.DISK,
)

[matteobettini]

Thanks a mil for this! Yes, very useful! I'll work towards merging it.