MAESTRO, TRANSP and flux surfaces

Bug Fixes

• 🐛 Relaxed requirements of pip packages to avoid installation issues.
• 🐛 Corrupted TRANSP CDF files can now be read by removing bad time slices.
• 🐛 Improved interaction with TRANSP singularity to avoid trlook and trfinish issues, and to be able to run locally.
• 🐛 Improved logic for TRANSP cases that failed. Errors are now properly reported and process stopped.
• 🐛 areal elongation is now properly calculated in PROFILEStools.py, fixing H98 calculations. @josephb-hall

New Features

• 💥 Flux surfaces now handled always using their own class mitim_flux_surfaces. Parameterization, including Miller Extended Harmonics (MXH) now happens as a class method. @josephb-hall
• 💥 Capability to easily run "fixed-boundary" millerized equilibrium using the FreeGS code. The freegs pip package is now part of the requirements.
• 💥 Better check for missing or wrong flags in TGLF, TGYRO and PORTALS.
• 💥 Various conversion routines: profiles to transp, transp to profiles, freegs to transp, geqdsk to profiles, etc. All part of their own objects.
• 💥 TRANSP Helper object can now interact with namelist and prepare UFILES.
• 💥 Start of the common Neural Network surrogate class, and implementation of EPED NN. @josephb-hall
• 💥 First release of MAESTRO (Modular and Accelerated Engine for Simulation of Transport and Reactor Optimization) workflow. Implementation of TRANSP, PORTALS and EPED_NN beats. Ready for stress-testing. @AudreySaltzman @josephb-hall.
• 💥 Surrogates in PORTALS can now receive any other variable that are not included in gradients or profiles, such as q and shear. This will be useful when combining cases from different plasmas or scenarios.
• 💥 TRANSP reader mitim_plot_transp can now work with folder as input, and it will find the CDF
• 💥 if omfit_classes geqdsk reader cannot find the separatrix, now MITIM will increase resolution and go find it again. Seems to be working in some extreme cases.
• 💥 TRANSP runs do not need to use MRY anymore, now we can build RFS and ZFS files, which removes the need for scruncher.
• 💥 Further improvements to ASTRA class. @josephb-hall @AudreySaltzman @mmuraca
• 💥 TGLF output class has now a to_xarray() converter now. @aaronkho

Changes for developers (internal execution)

• 🔎 New contexts: timers and loggers. All part of IOtools.py
• 🔎 Removed deprecated functions and freegsu references to SPARC. Code moved to new locations.
• 🔎 Old "im" module moved out of MITIM-fusion. Too many changes to keep it up to date.
• 🔎 Naming changes to physicsBasedParams in PORTALS.

Back-compatibility considerations

• 🔮 It has been decided that PORTALS-TGLF runs by default use only the botorch optimizers, instead of root and ga that was used before. Only in expensive situations (e.g. CGYRO), one should try to refine solutions. This makes the convergence faster in wall-time.
• 🔮 config_user.json is now removed from the package folder. A template is provided in templates but the user can choose to place their config_user.json anywhere. Follow the read-the-docs for instructions: https://mitim-fusion.readthedocs.io/en/latest/installation.html#user-configuration @aaronkho @josephb-hall
• 🔮 machine-specific bashrc "recommendations" are now private and not part of the MITIM-fusion repository.

Improvements in efficiency, structure and physics

Bug Fixes

• 🐛 Optimization stopping criteria changed to follow maximization logic, not residual minimization.
• 🐛 Fixed bug that wrote surrogate data file with wrong nuei.
• 🐛 Removed references to pablorf folders.
• 🐛 Reading and plotting of geqdsk now more robust, fixed bug in removeCoils function.
• 🐛 Fixed bug that made waveform calculation in TGLF unstable.
• 🐛 PROFILES variable $\epsilon$ (inverse aspect ratio) now correctly defined as $r/R$.
• 🐛 Corrected betae for 100 times lower (before, it was a percentage)
• 🐛 Keep botorch==0.9.4 until fix is found.

New Features

• 💥 First release of basic ASTRA class (@AudreySaltzman, @mmuraca).
• 💥 Equilibrium and flux surfaces to MXH coefficients conversion routine available (@josephb-hall).
• 💥 Users will now be warned if PORTALS, TGLF or BO option they selected is not available.
• 💥 POWERSTATE class now receives custom methods for transport and target calculations, in view of SMARTSsolver future development (@cholland). These can be provided externally as long as they inherit from parent classes.
• 💥 POWERSTATE flux-matching class now works fine outside of surrogate-based optimization methods, for quick evaluation of steady-state profiles with analytical or surrogated transport coefficients. Future development will include direct TGLF call, resembling a pythonic TGYRO solver.
• 💥 BO and PORTALS results now written as csv files and handled as dataframes.
• 💥 Much faster POWERSTATE class (changed logic for detachment and unsqueezing of tensors) and much faster derivedQuantities calculation of PROFILES via parallelization of geometric variables calculations.
• 💥 Zeff in PROFILES can now be modified directly without the need to handle manually the individual ion densities.
• 💥 Maximization objective now added as part of the optimization_data results file.
• 💥 Direct PROFILES to TGLF in-memory converter ready to go (prep_direct_tglf() method). No need for "iteration=0" of TGYRO. In next release, this will be default.
• 💥 POPCON matching tools available (@josephb-hall).

Changes for developers (internal execution)

• 🔎 Significant renaming of internal methods, functions and classes to follow better standards.
• 🔎 Only POWERSTATE class is needed for PORTALS plotting and analysis, which eases implementation with non-TGYRO transport methods.
• 🔎 Generalized update_var function towards a channel-agnostic framework
• 🔎 Generally, better organization of: parameterization of POWERSTATE, targets calculations (including fine targets). Removal of deprecated options and methods.
• 🔎 Radiation coefficients for ADAS radiation now are part of external csv file instead of hardcoded in python.

Back-compatibility considerations

• 🔮 Changed name of options variables and dictionaries in TGYRO, PORTALS and BO.
• 🔮 Fine targets resolution in PORTALS is now enabled by default.
• 🔮 Changed TGLF settings default in solvers.
• 🔮 Changed PORTALS radial resolution by default.
• 🔮 POWERSTATE now must receive arrays without rho=0.0, which is added by default.
• 🔮 Folders aux and exe renamed to utils and scripts to avoid issues with Windows machines and github cloning.

Misc bug fixes and faster TGLF scans

Bug Fixes

• 🐛 High-resolution fluxes in standard PORTALS plotting now have the correct coordinate system instead of defaulting to rho(-).
• 🐛 Plotting of PORTALS initialization stage now shows next profile to evaluate.
• 🐛 Reading of geqdsk files does not crash that often: robust to empty lines and option for several time slices in same file.
• 🐛 Reading of input.gacode files more robust: allows time row and non scientific notation.
• 🐛 User is now warned if PORTALS is run with an ion specie not included in radiation calculations.
• 🐛 PORTALS does not crash if surrogate specified with no transformation.
• 🐛 Fixed routines for singularity execution of TRANSP.
• 🐛 removed scipy modules causing problems for certain numpy version.

New Features

• 💥 Generalization of TGLF scans class to allow for parallel execution of scans (including for analysis purposes and waveform generation).
• 💥 CGYRO class development has commenced! Linear runs possible, but still in development phase. Use at your own risk.
• 💥 CSV tables easily created from PROFILES class for quick communication of global parameter results (e.g. PORTALS predictions).

Changes for developers (internal execution)

• Changed use of (most) lambda functions in PORTALS to "def" functions.
• Generalization of handler of GACODE settings (TGLF, CGYRO, TGYRO).
• Capability to tell the execution farming to check files inside output folder when coming back from remote machine.

Refactored remote execution handler

Bug Fixes

• 🐛 Particle flux in GB units was multiplied by a factor of 3/2 when written into data tables and extracted surrogates. Now, that factor is removed. This does not change results of past profile predictions (output data is standardized anyway) but it changes the interpretation of output tables for the particle flux.
• 🐛 Fixed situation that made MITIM crash when added data points (from previous runs) to new surrogates have different input dimensions.
• 🐛 Input features are made equal when the changes are very close to machine precision, to solve problem with dimensions that should not be fitted (issue appeared only when PORTALS restarting from previous surrogates, from a different input.gacode).
• 🐛 Fixed issue with zero particle flux with powerstate targets when plotting PORTALS.
• 🐛 Solved issue that caused MITIM-TGYRO to crash if anchor point was chosen past rho>0.9, and catch case where numerical scheme cannot run with zero iterations.
• 🐛 Path canonicalization solved issues (encountered in some clusters) solved.

New Features

• 💥 More robust handling of external command execution, using paramiko and tqdm packages. Transferred files are now first included in a tarball to send and receive, and they do not stop at tunnel anymore.
• 💥 Slurm setup (account, partition, constraints, memory requirements) now are part of JSON. Code classes may change memory requirements, but a default can now be added.
• 💥 TGLF settings (defaults and TGLFsettings options) are now handled as part of JSON and control files in templates/. Added SAT3 as option 6.
• 💥 PORTALS plotter can now combine several runs in a single notebook.
• 💥 TRANSP plotter can now combine several runs in a single notebook, and plotting occurs automatically from alias mitim_plot_transp.
• 💥 General improvements to documentation, including new config file specifications, FAQ and clear instructions about the role of machine's bashrc files.
• 💥 Capability to extract PORTALS surrogates in an elegant way, together with the possibility to run standalone TGLF and TGYRO, extracted from specific evaluations of PORTALS during the BO workflow.
• 💥 Fluxes produced during a PORTALS-CGYRO run can now be easily (and automatically in PORTALS plotting) compared to TGLF.
• 💥 Notebooks improvements: No need to block terminal, more robust to segmentation faults, and added colors to tabs.
• 💥 User config file can now have argument of "modules" to send to each machine, to execute before any code.

Changes for developers (internal execution)

• New NEO class handles VGEN runs for calculation of Er and rotation from input.gacode files.
• PORTALS plotting separated from analysis, and much cleaner interface with general plotting routines.
• Default physics-based params are now part of the class, to retrieve again if eliminated before restarting.
• Start of the CGYRO class and NERSC perlmutter connection (in the works).

First complete version of MITIM-fusion, Hello World

This release signifies the first version of the MITIM-fusion repository to be released publicly. It is ready for prime time!