DRL and open-ended trading universes where assets are added and removed

[miohtama]

I am testing out DRL for DEX cryptocurrency trading.

The trading universe is open-ended. New tokens are added in and removed out very frequently due to the short life cycle of memecoins and such.

What's the best way to deal with such open-ended trading pair data when training DRL models, and then in trading itself?

• Set price zero?
• No data rows before and after inclusion?
• etc.

For those interested, you can find more about our memecoin data here - I am setting up a script that allows import all this to FinRL.

[miohtama]

I am following this example.

Did my first stab on this: with trade data with unequal start and end dates for assets I got as far as here:

DRLAgent.train_model(model, episodes=5)

And getting:

ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 0 and the array at index 1 has size 50

With traceback:

Cell In[7], [line 1](vscode-notebook-cell:?execution_count=7&line=1)
----> [1](vscode-notebook-cell:?execution_count=7&line=1) DRLAgent.train_model(model, episodes=5)

File ~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/models.py:80, in DRLAgent.train_model(model, episodes)
     [69](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/models.py:69) @staticmethod
     [70](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/models.py:70) def train_model(model, episodes=100):
     [71](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/models.py:71)     """Trains portfolio optimization model.
     [72](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/models.py:72) 
     [73](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/models.py:73)     Args:
   (...)
     [78](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/models.py:78)         An instance of the trained model.
     [79](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/models.py:79)     """
---> [80](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/models.py:80)     model.train(episodes)
     [81](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/models.py:81)     return model

File ~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/algorithms.py:105, in PolicyGradient.train(self, episodes)
     [99](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/algorithms.py:99) """Training sequence.
    [100](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/algorithms.py:100) 
    [101](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/algorithms.py:101) Args:
    [102](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/algorithms.py:102)     episodes: Number of episodes to simulate.
    [103](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/algorithms.py:103) """
    [104](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/algorithms.py:104) for i in tqdm(range(1, episodes + 1)):
--> [105](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/algorithms.py:105)     obs = self.train_env.reset()  # observation
    [106](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/algorithms.py:106)     self.train_pvm.reset()  # reset portfolio vector memory
    [107](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/agents/portfolio_optimization/algorithms.py:107)     done = False

File ~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:393, in PortfolioOptimizationEnv.reset(self)
    [390](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:390) self._time_index = self._time_window - 1
    [391](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:391) self._reset_memory()
--> [393](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:393) self._state, self._info = self._get_state_and_info_from_time_index(
    [394](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:394)     self._time_index
    [395](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:395) )
    [396](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:396) self._portfolio_value = self._initial_amount
    [397](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:397) self._terminal = False

File ~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:460, in PortfolioOptimizationEnv._get_state_and_info_from_time_index(self, time_index)
    [458](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:458)     display(state)
    [459](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:459)     display(tic_data)
--> [460](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:460)     state = tic_data if state is None else np.append(state, tic_data, axis=2)
    [461](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:461) state = state.transpose((0, 2, 1))
    [462](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:462) info = {
    [463](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:463)     "tics": self._tic_list,
    [464](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:464)     "start_time": start_time,
   (...)
    [469](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:469)     "price_variation": self._price_variation,
    [470](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:470) }

File ~/code/getting-started/.venv/lib/python3.11/site-packages/numpy/lib/function_base.py:5618, in append(arr, values, axis)
   [5616](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/.venv/lib/python3.11/site-packages/numpy/lib/function_base.py:5616)     values = ravel(values)
   [5617](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/.venv/lib/python3.11/site-packages/numpy/lib/function_base.py:5617)     axis = arr.ndim-1
-> [5618](https://file+.vscode-resource.vscode-cdn.net/Users/moo/code/getting-started/scratchpad/finrl/~/code/getting-started/.venv/lib/python3.11/site-packages/numpy/lib/function_base.py:5618) return concatenate((arr, values), axis=axis)

[miohtama]

My second attempt setting missing price data to zero get all the way training, but looks like the portfolio goes to zero and DRLAgent.train_model() cannot operate on this data:

0%|          | 0/5 [00:00<?, ?it/s]/Users/moo/code/getting-started/deps/FinRL/finrl/meta/env_portfolio_optimization/env_portfolio_optimization.py:343: RuntimeWarning: invalid value encountered in divide
  weights = portfolio [/](https://file+.vscode-resource.vscode-cdn.net/) self._portfolio_value
 20%|██        | 1/5 [00:38<02:33, 38.27s/it]
=================================
Initial portfolio value:10000
Final portfolio value: nan
Final accumulative portfolio value: nan
Maximum DrawDown: -1.0
Sharpe ratio: -0.6392890260431509
=================================

[miohtama]

Same result trying to fill missing data with NaN

  0%|          | 0/5 [00:00<?, ?it/s]
=================================
Initial portfolio value:10000
Final portfolio value: nan
Final accumulative portfolio value: nan
Maximum DrawDown: -1.0
Sharpe ratio: nan
============================