PreSync update for Cactus
Arrangement Authors     : Steve Brandt, Samuel Cupp
--------------------------------------------------------------------------------
This repository contains the thornlist presync.th, which is a list of thorns on
the presync branch which are believed to work and pass tests in the testsuite.
In addition, a docker image is provided which, when built, downloads and
compiles the presync branch of the toolkit and runs the testsuite in both
presync and backward compatible modes. It generates out.txt and pout.txt, which
are the testsuite outputs for the backward compatible and presync tests,
respectively. Below is a brief description of PreSync and some resources for
changing thorns to use PreSync.
--------------------------------------------------------------------------------

Presync now handles synchronization and boundary condition registration and
application from within Carpet. This is done by declaring reads/writes for each
function in the schedule.ccl. This document explains the process of updating an
existing thorn to use the new mechanisms in PreSync. This document consists of
the following sections:

  I. Basic Step-by-Step Update Procedure
 II. PreSync With Fortran
III. Internal Functions
 IV. Registering Boundary Functions
  V. Applying Boundary Conditions: Special Cases
 VI. Generation of New Macros: rdwr.pl

               ----------------------------------------
                I. Step-By-Step Thorn Update Procedure
               ----------------------------------------

1) Anything which has Boundary in the interface.ccl (via Requires, Inherits,
   Shares, etc.) should be changed to need Boundary2. The changes could be
   migrated to Boundary, but they are currently in a different thorn.

2) Previously, a BC was selected for a variable by calling Boundary_SelectGroupForBC
   or Boundary_SelectVarForBC. These functions were scheduled whenever boundary
   functions needed to be applied, along with ApplyBCs group from the Boundary
   thorn. To use PreSync, selection routines are instead scheduled in the GROUP
   Boundary2_Selection. The ApplyBCs group is not used in the PreSync and isn't
   scheduled. The old boundary scheduling should be encased in the conditional

       if(CCTK_ParameterValInt("use_psync","Carpet") == 0) {
       }

   and the new boundary scheduling should be encased in the conditional

       if(CCTK_ParameterValInt("use_psync","Carpet") == 1) {
       }

4) All scheduled functions have new macros. Change DECLARE_CCTK_ARGUMENTS to
   DECLARE_CCTK_ARGUMENTS_FunctionName. The header file cctk_Arguments_Checked.h
   also needs to be included. This will only allow access to those variables with
   read/write declarations. Any variables used but not declared will result in a
   compiler error. Please note that the macro will use the function name in the
   schedule.ccl. While Cactus is case-insensitive, the C preprocessor is not.
   Therefore, the macro will take the case of the name in the schedule.ccl in the
   event that it differs from the case in the actual code. For this reason, it is
   recommended that the two match in case to prevent an easily missable cause
   for compilation failure. For Fortran code, please see the next section.

5) Compile errors will appear file-by-file. Each function must have the correct
   read/write declarations before the compiler will go to the next file. Since
   the given line is the variable's first appearance, it is usually clear if
   the variable is read or written. In addition, read-only variables are declared
   const, so a variable which is read and written will give another error for
   the write. The opposite case is not true - a function which reads and writes
   a variable will not give an error if WRITES is present but READS is not.

               ----------------------------------------
                       II. PreSync with Fortran
               ----------------------------------------

Currently, the new macros are not usable with Fortran code. However, they can
still be used to check read/write declarations. As with C code, use the new macro
and header file. The argument list of the function must also be changed from
CCTK_ARGUMENTS to CCTK_ARGUMENTS_FunctionName. The argument list results in an
argument passing error at runtime, but the read/write errors at compile time are
still accessible.

Unfortunately, the compiler (combined with preprocessing) does not provide
the errors in a useful form. Many errors will appear which are caused by the many
undeclared variables. The only errors of interest are implicit variable errors.
The line of code must be found for each error, as it prints out the wrong line
with the error. Otherwise, the process is the same as with C code.

Once the code compiles successfully, revert the macro and argument list to the
original form and compile again. Eventually, full Fortran compatibility will
hopefully be added, but this method is functional in the interim.

               ----------------------------------------
                       III. Internal Functions
               ----------------------------------------

Internal functions can greatly complicate the read/write declarations, as
their usage of variables will not trigger a compiler error in the scheduled
function. This generally manifests in two ways. Either the function uses
variables not declared at all by the parent function, or it uses a variable
in such a way as to change the needed declarations. The second is less clear
without an example. Let Fun1 be a scheduled function, and Fun2 be an internal
function called at the beginning of Fun1. After following the usual procedure,
Fun1 reads and writes var1 in its schedule. However, Fun2 only writes var1.
Then, Fun2 writes var1 before Fun1 reads it. Therefore, the read is not needed.

The best way to handle internal functions is to include the macro of a scheduled
function. If the function is only called by one scheduled funciton, then simply
use that macro. If called by several, there are several options.

First, the macro of one can be included for debugging, and any read/write
declarations resulting from the internal function are added to all scheduled
functions which use it.To be certain that nothing was missed, this may require
recompiling using the macro of each scheduled function. While tedious, this
will work.

Alternatively, the function can be scheduled nowhere. That is, adding something
of the form
        schedule FunctionName
        {
        LANG: C
        }  ""
to the schedule.ccl will result in a macro being generated for this function.
After determining the read/write declarations for the internal function, they
can be copied into all the scheduled functions which call this internal function.

However, these methods will not catch the case where a read is not needed. For
that problem, the only solution currently is to manually verify the read/write
declarations are correct.

               ----------------------------------------
                  IV. Registering Boundary Functions
               ----------------------------------------

Physical boundary conditions (like those in the Boundary thorn) should be
registered with PreSync using the function Boundary_RegisterPhysicalBC.
This aliased function from Boundary2 is aliased identically to Boundary,
so any routines which already used Boundary to apply their boundary conditions
will not need significant changes. For those which did not use Boundary,
there are several options. The preferred method is to change the scheduled
function to register with PreSync. Then, PreSync can trigger BC application
as needed. Alternatively, the scheduling can simply have the WRITE region be
(boundary). This doesn't benefit from PreSync's features, but it should allow
for compatibility with PreSync.

Symmetry BCs (which are registered with SymBase) should also be registered
with PreSync. However, they are still registered with SymBase. PreSync simply
determines when these functions should be run. The Symmetry boundary conditions
are registered with the function Boundary_RegisterSymmetryBC. Before, these
functions were scheduled in BoundaryConditions GROUP from Boundary. Now, they
are called, and the argument list is different. The old version is copied as
Old_Name, while the original is still Name. The Old_Name is scheduled with
the (use_psync = 0) conditional for backward compatibility.

To see an example of how to use these features, see CactusNumerical/ReflectionSymmetry.

reflection.h includes PreSync.h and a prototype for the registered function.
Boundary2 thorn has a similar header file, but it includes a prototype for
every BC it contains.

register.c includes the registration function. For symmetry functions like
ReflectionSymmetry, the function is also registered with SymBase. In
ReflectionSymmetry_SymbaseRegister, Boundary_RegisterSymmetryBC is added to
the function. Note that it is also contained in a conditional to maintain
backward compatibility.

               ----------------------------------------
                 V. Boundary Conditions: Special Cases
               ----------------------------------------

In some circumstances, boundary conditions or synchronization may need to be
applied manually. This should be avoided and only used if absolutely necessary.
As an example, the variable cXt1 in ML_BSSN is only written, never read. It is
output as a check, but never used in any calculations. Because of this, it will
never trigger synchronization or application of boundary conditions. Thus, the
CarpetIOASCII thorn must be changed to handle it internally. This amounts to
calling ManualSyncGF, which is provided by Carpet. This function must be added
to the interface.ccl. The function checks the region of validity, so it will not
be synchronized if that is not needed.

               ----------------------------------------
                VI. Generation of New Macros: rdwr.pl
               ----------------------------------------

This section discusses the new Perl script which creates the macros accompanying
the Presync update. The file rdwr.pl in the flesh parses the interface.ccl and
schedule.ccl to create these macros. The top-level subroutine GenerateArguments
is called in the CST and handles all the internal subroutine calls.

For each thorn, the interface.ccl is parsed, and information on variables is
stored in a hash table. After all interface.ccl have been parsed, each thorn's
schedule.ccl is read. The READ/WRITE declarations are parsed for each scheduled
subroutine. In the event that a subroutine is scheduled multiple times with
different declarations, a union of all instances is taken for the macro. Once
the whole file has been parsed, a DECLARE_CCTK_ARGUMENTS_FunctionName macro
is generated for every scheduled subroutine. For Fortran subroutines, a
CCTK_ARGUMENTS_FunctionName macro is also generated. These macros are stored in
a header file called "cctk_Arguments_Checked.h".