[BUG]: Transform Dataset operator behaves anomalously - cannot generated code or access input dataframes

[liunelson]

• Scenario: Transform Dataset
• Step: 3.4
• Link to the project: https://app.staging.terarium.ai/projects/34e7b955-f408-4e8b-be27-9429480ad6aa/workflow/954cf33c-f207-449d-b069-5337b8fc24ba?operator=0e564ce4-afc6-460c-8feb-57c1934cccfe

Describe the issue

The Transform Dataset operator has a few bugs:

1. After just two code cells (printing what d2 is and asking the LLM for the join code), the session seems to have lost access to the input dataset d1.
   

2. Starting from a new Transform Dataset operator, now the generated LLM code won't run fully - the status is either stuck at Busy or Offline
   

3. Minor issue: unknown code output shown in the operator every time - not informative, nor should it be shown to users?

{'modules': {}, 'variables': {}, 'functions': {}}

[blanchco]

Asked Jataware about this, beaker seems to just hang with this command which may cause downstream issues https://unchartedsoftware.slack.com/archives/C049E6PA16V/p1728054249803609

[blanchco]

It seems this might be an issue with Jupyter itself, we run out of memory. It might be worth looking into a solution where if beaker starts to hang like this we can add a timeout and start a new session informing the user of the issue

[liunelson]

What's the plan for this issue?