diff --git a/docs/source/CaImAn_Tips.rst b/docs/source/CaImAn_Tips.rst index 0d4bd7269..1576c3591 100644 --- a/docs/source/CaImAn_Tips.rst +++ b/docs/source/CaImAn_Tips.rst @@ -1,5 +1,5 @@ -CaImAn Tips -=========== +General Tips +============ Motion Correction tips ---------------------- @@ -33,7 +33,7 @@ Motion Correction tips of chunks (``params.motion.splits_els`` and ``params.motion.split_rig``) with the length of your recording (e.g. ``int((number of total frames)/200)``). -CaImAn Online Processing tips +Caiman Online Processing tips ----------------------------- - Important parameters for online processing are the CNN threshold @@ -71,7 +71,7 @@ CaImAn Online Processing tips - When using the CNN for screening candidate components, the usage of a GPU can lead to significant computational gains. -CaImAn Batch processing tips +Caiman Batch processing tips ---------------------------- - In order to optimize memory consumption and parallelize computing, it diff --git a/docs/source/CaImAn_features_and_references.rst b/docs/source/CaImAn_features_and_references.rst index c23cfcb23..1e1d6e188 100644 --- a/docs/source/CaImAn_features_and_references.rst +++ b/docs/source/CaImAn_features_and_references.rst @@ -1,7 +1,7 @@ -Features --------- +Caiman Features +--------------- -CaImAn includes a variety of scalable methods for the analysis of +Caiman includes a variety of scalable methods for the analysis of calcium (and voltage) imaging data: | @@ -44,7 +44,7 @@ calcium (and voltage) imaging data: - Uses a Mask R-CNN to identify neurons in the FOV - Extracts spiking activity using adaptive template matching. - - Fully integrated with CaImAn, inherits all its capabilities. + - Fully integrated with Caiman, inherits all its capabilities. | - **Behavioral Analysis** [Behavior]_ @@ -59,8 +59,7 @@ calcium (and voltage) imaging data: - Noise parameters estimation under the Poisson-Gaussian noise model - Fast algorithm that scales to large datasets - - A basic demo can be found at - ``CaImAn/demos/notebooks/demo_VST.ipynb`` + - A basic demo can be found in the default demos as demo_VST.ipynb diff --git a/docs/source/GUI.rst b/docs/source/GUI.rst index a1a07cd81..c96015c69 100644 --- a/docs/source/GUI.rst +++ b/docs/source/GUI.rst @@ -1,14 +1,14 @@ Graphical interfaces ==================== -The CaImAn GUI ---------------- +The Caiman GUI +-------------- .. image:: ../img/GUI_img.png :width: 600px :align: center -CaImAn comes with an experimental visual interface. To see an example on +Caiman comes with an experimental visual interface. To see an example on how use it, first load and run either \* demo_OnACID_mesoscope.py \* demo_caiman_basic.py \* demo_pipeline.py @@ -37,7 +37,7 @@ More features will be added in the future. External interfaces ------------------- -A couple of very nice visualization tools that directly call CaImAn have been developed recently +A couple of very nice visualization tools that directly call Caiman have been developed recently by external researchers. For more information please check the following projects: - `mesmerize-core `_ - also simplifies grid search for parameter tuning diff --git a/docs/source/Getting_Started.rst b/docs/source/Getting_Started.rst index b63802f17..1294ecd7c 100644 --- a/docs/source/Getting_Started.rst +++ b/docs/source/Getting_Started.rst @@ -1,16 +1,15 @@ -Getting Started with CaImAn +Getting Started =========================== Demos ----- - Notebooks: The notebooks provide a simple and friendly way to get - into CaImAn and understand its main characteristics. They are located - in the ``demos/notebooks``. To launch one of the jupyter notebooks: + into Caiman and understand its main characteristics. They are located + in the ``demos/notebooks``. To launch one of the jupyter notebooks, activate your conda caiman environment, enter the caiman_data directory, and then: .. code:: bash - source activate CaImAn jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10 and select the notebook from within Jupyter’s browser. The argument @@ -21,9 +20,6 @@ Demos trying demo_pipeline.py first as it contains most of the tasks required by calcium imaging. For behavior use demo_behavior.py -- If you want to directly launch the python files, your python console - still must be in the CaImAn directory. - Basic Structure --------------- @@ -36,13 +32,13 @@ the following objects: easily and is passed into all the algorithms. - ``MotionCorrect``: An object for motion correction which can be used for both rigid and piece-wise rigid motion correction. -- ``cnmf``: An object for running the CaImAn batch algorithm either in +- ``cnmf``: An object for running the Caiman batch algorithm either in patches or not, suitable for both two-photon (CNMF) and one-photon (CNMF-E) data. -- ``online_cnmf``: An object for running the CaImAn online (OnACID) +- ``online_cnmf``: An object for running the Caiman online (OnACID) algorithm on two-photon data with or without motion correction. - ``estimates``: A single object that stores the results of the - algorithms (CaImAn batch, CaImAn online) in a unified way that also + algorithms (batch, online) in a unified way that also contains plotting methods. To see examples of how these methods are used, please consult the demos. @@ -51,7 +47,7 @@ To see examples of how these methods are used, please consult the demos. Parameters ----------- -CaImAn gives you access to a lot of parameters and lets you adapt the analysis to your data. Parameters are stored in +Caiman gives you access to a lot of parameters and lets you adapt the analysis to your data. Parameters are stored in the ``params`` object in a set of dictionaries, sorted by the part of the analysis they are used in: - ``data``: General params describing the dataset like dimensions, decay time, filename and framerate @@ -123,7 +119,7 @@ As mentioned above, the results of the analysis are stored within the Result variables for 2p batch analysis -------------------------------------- -The results of CaImAn are saved in an ``estimates`` object. This is +The results of Caiman are saved in an ``estimates`` object. This is stored inside the cnmf object, i.e. it can be accessed using ``cnmf.estimates``. The variables of interest are: @@ -305,11 +301,12 @@ The cluster is set up with Caiman's ``setup_cluster()`` function, which takes in n_processes=None, ignore_preexisting=False) -The ``backend`` parameter determines the type of cluster used. The default value, ``'multiprocessing'``, uses the -multiprocessing package, but ``ipyparallel`` is also available. More information on these choices can be -found `here `_. You can set the number of -processes (cpu cores) to use with the ``n_processes`` parameter: the default value ``None`` will lead to the function -selecting one *less* than the total number of logical cores available. +The **backend** parameter determines the type of cluster used. The default value, **'multiprocessing'**, uses the +multiprocessing package, but **ipyparallel** is also available. You can set the number of +processes (cpu cores) to use with the **n_processes** parameter: the default value **None** will lead to the function +selecting one *less* than the total number of logical cores available. + +More information on these choices can be found :doc:`in the cluster doc `. The parameter ``ignore_preexisting``, which defaults to ``False``, is a failsafe used to avoid overwhelming your resources. If you try to start another cluster when Caiman already has one running, you will get an error. However, sometimes @@ -317,7 +314,7 @@ on more powerful machines you may want to spin up multiple Caiman environments. set ``ignore_preexisting`` to ``True``. The output variable ``cluster`` is the multicore processing object that will be used in subsequent processing steps. It will -be passed as a parameter in subsequent stages and is the fulcrum for parallelization. The +be passed as a parameter in subsequent stages and sets policy for parallelization. The other output that can be useful to check is ``n_processes``, as it will tell you how many CPU cores you have set up in your cluster. @@ -347,4 +344,4 @@ will be read/written by column or by row, respectively. This is important becaus faster on C-order arrays vs F-order arrays. For motion correction, which needs to access contiguous sequences of frames (often in the middle of the movie), it is much more efficient to read and write in F order. On the other hand, when it comes to CNMF, you need to access individual pixels across the entire movie, so Caiman saves the -motion-corrected movie in C-order before running CNMF. \ No newline at end of file +motion-corrected movie in C-order before running CNMF. diff --git a/docs/source/Handling_Movies.rst b/docs/source/Handling_Movies.rst index e2c00badb..71d742f17 100644 --- a/docs/source/Handling_Movies.rst +++ b/docs/source/Handling_Movies.rst @@ -2,7 +2,7 @@ Handling movies ================ This page gives a basic intro on the movie object for loading, manipulating, -displaying and saving movies within CaImAn. +displaying and saving movies. Loading movies -------------- diff --git a/docs/source/Installation.rst b/docs/source/Installation.rst index 288f6372d..939d56120 100644 --- a/docs/source/Installation.rst +++ b/docs/source/Installation.rst @@ -1,7 +1,7 @@ Installation and Updating ========================= -This page will give more detailed setup instructions for Caiman than the `main readme <../../README.md>`_. +This page will give more detailed setup instructions for Caiman than the main readme in the source tree. This is the place you should visit first if you run into problems and need to troubleshoot. It includes info on initial setup, as well as updating with new releases. There is a Table of Contents on the left-hand side of this page, so if you need help with a particular task, that should help you get oriented. @@ -36,7 +36,7 @@ and make contributions to Caiman. Section 1A. Install with conda ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -These are basically the Quickstart instructions on the `main readme page <../../README.md>`_, so if you follow those, you +These are basically the Quickstart instructions on the main readme page at the repo: , so if you follow those, you should be good to go. .. raw:: html @@ -72,8 +72,7 @@ Section 1B. Development-mode install ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dev mode install allows you to modify the source files of Caiman and makes it easier -to contribute to the project, fix bugs etc. The general motivation for setting up -an editable development environment is described in more detail in our `contributors page <../../CONTRIBUTING.md>`_. +to contribute to the project, fix bugs etc. If you install in dev mode you will likely need to set some environment variables manually (it is done automatically when you do the conda install): this is discussed in Section 4C. @@ -107,7 +106,7 @@ At the conda prompt: .. code:: bash - git clone https://github.com/your-username/CaImAn + git clone git@github.com:flatironinstitute/CaImAn.git cd CaImAn mamba env create -f environment.yml -n caiman conda activate caiman @@ -140,7 +139,7 @@ Dev Mode Install on MacOS and Linux .. code:: bash - git clone https://github.com/your-username/CaImAn + git clone git@github.com:flatironinstitute/CaImAn.git cd CaImAn/ mamba env create -f environment.yml -n caiman source activate caiman @@ -235,7 +234,7 @@ below to update the code. However, if the environment file has changed since your last update this may lead to you not the latest version. None of this applies to the conda-forge route (for which instructions are given above). -From the conda environment you used to install CaImAn: +From the conda environment you used to install Caiman: 1. ``pip uninstall caiman`` @@ -257,7 +256,7 @@ From the conda environment you used to install CaImAn: variables have changed. - The same applies if you want to modify some internal function of - CaImAn. If you used the ``pip install -e .`` option then you can + Caiman. If you used the ``pip install -e .`` option then you can directly modify it (that's why it's the editable developer mode). If you used the ``pip install .`` option then you will need to ``pip uninstall caiman`` followed by ``pip install .`` for your @@ -278,7 +277,7 @@ Section 3C: Upgrade the demos with caimanmanager Upgrade the demos When you upgrade Caiman, sometimes the underlying APIs change. When this happens and it impacts a demo (or otherwise -requires changes to files in ``caiman_data``), we update the demo and data. This means that upgrading CaImAn works +requires changes to files in ``caiman_data``), we update the demo and data. This means that upgrading Caiman works best if you also replace the ``caiman_data`` directory with a new version. To check if the demos or datafiles have changed since your last install, you can run ``caimanmanager check``. If they have not, @@ -372,6 +371,8 @@ following the instructions Section 4D: Other topics ~~~~~~~~~~~~~~~~~~~~~~~~~ +See also: + +* :doc:`Our clustering doc ` +* :doc:`Caiman and GPUs ` -- `Running Caiman on a cluster <./CLUSTER.md>`_ -- `Setting up Caiman to use your GPUs <./README-GPU.md>`_ diff --git a/docs/source/On_file_types_and_sizes.rst b/docs/source/On_file_types_and_sizes.rst deleted file mode 100644 index 45c577327..000000000 --- a/docs/source/On_file_types_and_sizes.rst +++ /dev/null @@ -1,60 +0,0 @@ -On file types and sizes -======================= - -CaImAn is designed to perform analysis on datasets saved over a single -or multiple files. However maximum efficiency is achieved when each -dataset is saved as a sequence of files of medium size (1-2GBs). Please -note the following: - -- If you’re using TIFF files make sure that the files are saved in - multipage format. This is particularly important as multipage TIFF - files can be indexed and individual frames can be read without - loading the entire file in memory. On the contrary single page TIFFs - would load the entire file before reading an individual frame. This - can cause significant problems in CaImAn in terms of speed and memory - consumption, as a lot of the parallelization (e.g. during motion - correction) happens by passing the path to a file to multiple - processes each of which will only read and process a small part of - it. **Bear in mind that TIFF files of size 4GB or larger saved - through ImageJ/FIJI are automatically save in single page format and - should be avoided**. If you have such a file you can split into - multiple shorter files through ImageJ/FIJI or through CaImAn using - the following script - -:: - - import numpy as np - import caiman as cm - fname = '' # path to file - m = cm.load(fname) # load the file - T = m.shape(0) # total number of frames for the file - L = 1000 # length of each individual file - fileparts = fname.split(".") - for t in np.arange(0,T,L): - m[t:t+L].save((".").join(fileparts[:-1]) + '_' + str(t//L) + '.' + fileparts[-1]) - -HDF5/H5 files in general do not suffer from this problem. - -- Single frame files should be avoided. The reason is that several - functions, e.g. motion correction, memory mapping, are designed to - work on small sets of frames and in general assume that each file has - more than 1 frames. If your data is saved as a series of single frame - files, you should convert them in a single (or multiple) files. You - can do this by using the following script: - -:: - - import os - import glob - import caiman as cm - fld = '' # path to folder where the data is located - fls = glob.glob(os.path.join(fld,'*.tif')) # change tif to the extension you need - fls.sort() # make sure your files are sorted alphanumerically - m = cm.load_movie_chain(fls) - m.save(os.path.join(fld,'data.tif')) - -If the number of frames is too big, you can split into multiple files as -explained above. Make sure that your files are sorted alphanumerically -before combining them. This can be tricky if your files are initially -saved as ’file1.tif, file2.tif, …, file10.tif`. In this case you can -consult this `page `__. diff --git a/docs/source/Overview.rst b/docs/source/Overview.rst index 5813c935c..25c494569 100644 --- a/docs/source/Overview.rst +++ b/docs/source/Overview.rst @@ -1,13 +1,13 @@ Overview -========= +======== .. image:: ../LOGOS/Caiman_logo_FI.png :width: 300px :align: right -CaImAn is a Python toolbox for large scale **Ca**\ lcium **Im**\ aging data **An**\ alysis and behavioral analysis. +Caiman is a Python toolbox for large scale **Ca**\ lcium **Im**\ aging data **An**\ alysis and behavioral analysis. -CaImAn implements a set of essential methods required in the analysis pipeline of large scale calcium imaging data. Fast and scalable algorithms are implemented for motion correction, source extraction, spike deconvolution, and component registration across multiple days. It is suitable for both two-photon and one-photon fluorescence microscopy data, and can be run in both batch and online modes. CaImAn also contains some routines for the analysis of behavior from video cameras. A list of features as well as relevant references can be found `here +Caiman implements a set of essential methods required in the analysis pipeline of large scale calcium imaging data. Fast and scalable algorithms are implemented for motion correction, source extraction, spike deconvolution, and component registration across multiple days. It is suitable for both two-photon and one-photon fluorescence microscopy data, and can be run in both batch and online modes. It also contains some routines for the analysis of behavior from video cameras. A list of features as well as relevant references can be found `here `_. Companion paper diff --git a/docs/CLUSTER.md b/docs/source/cluster.md similarity index 85% rename from docs/CLUSTER.md rename to docs/source/cluster.md index 8ed0f007c..cd46a9457 100644 --- a/docs/CLUSTER.md +++ b/docs/source/cluster.md @@ -1,12 +1,12 @@ On Clustering ============= -Some parts of the CaImAn library (and demos) are best done by farming work out over multiple CPUs (or in some cases, multiple separate systems in a cluster). Python provides (at least) two popular frameworks for this, multiprocessing and ipyparallel. Both of them have several modes and parameters, each with their own advantages and disadvantages. Their behaviour differs on different operating systems (e.g. Linux versus OSX) and different Python environments (script versus Spyder versus Jupyter). CaImAn provides some support for some modes of both frameworks, to make it easier for people to find a setup that works on their platform. +Some parts of the Caiman library (and demos) are best done by farming work out over multiple CPUs (or in some cases, multiple separate systems in a cluster). Python provides (at least) two popular frameworks for this, multiprocessing and ipyparallel. Both of them have several modes and parameters, each with their own advantages and disadvantages. Their behaviour differs on different operating systems (e.g. Linux versus OSX) and different Python environments (script versus Spyder versus Jupyter). It provides some support for some modes of both frameworks, to make it easier for people to find a setup that works on their platform. This document covers some of the reasons you might choose one framework over another, how you can pick a clustering option with the codebase, and some of the issues we have seen with each. -Docs on both -============ +Ipyparallel vs multiprocessing +------------------------------ If you are already reading this, we encourage you to also do independent reading on both. * [Ipyparallel docs](https://ipyparallel.readthedocs.io/en/latest/) * [Multiprocessing docs](https://docs.python.org/3/library/multiprocessing.html) @@ -16,21 +16,20 @@ Multiprocessing has three modes, each of which behaves differently: * fork - The normal on all platforms but Windows. Efficient, but if any of the libraries Caiman uses are themselves multithreaded, can create problems * forkserver - Fast, but your code needs to either be running under Spyder or Jupyter, or needs a certain organisation to make it work. -Comparison -========== +In sum: * Multiprocessing is designed to run multiple processes on the same system, to efficiently use multiple machines. * Ipyparallel can do just that, but can also efficiently use multiple systems (and can integrate with cluster systems to do this automatically How to select one with the codebase -=================================== +----------------------------------- Some of the demos use cm.cluster.setup_cluster() to setup a clustered environment. The "backend" parameter selects which configuration will be used for the cluster. The parameter can take the following documented values: * multiprocessing (or local) - Uses the multiprocessing module with its default backend for the platform * ipyparallel - Use ipyparallel Issues -====== +------ * On OSX, some math libraries (blas, lapack) by default use a hardware-accelerated system framework that interacts badly with threads and might not work if run from a forked process. This causes some worker processes to hang forever if more than one of them tries to do some computations at the same time. You may be able to prevent this with "export VECLIB_MAXIMUM_THREADS=1" before running your code, or telling Conda to install a blas/lapack that is not built against Veclib. -* On Linux and OSX, some multithreaded builds of math libraries do their own multithreading and this can break if CaImAn forks to make new processes. This can result in hangs and/or core dumps. We have in our instructions other environment variables you should set to tell them not to multithread. +* On Linux and OSX, some multithreaded builds of math libraries do their own multithreading and this can break if Caiman forks to make new processes. This can result in hangs and/or core dumps. We have in our instructions other environment variables you should set to tell them not to multithread. * OpenCV is not easy to use safely with multithreaded or multi-process software. We take steps to disable multithreading in OpenCV, but some builds of it may not allow this. If you run into these issues let us know; we may ask you to switch the OpenCV version you use (Conda makes this easy). -* Some builds of OpenCV use OpenMP, which by default opens a thread for each of your CPUs for background work. This multiplies out to a potentially large number if you have already started many processes for CaImAn work. If this is both true and a problem for your environment, you should limit OpenMP to a single thread by doing the following before running CaImAn code: "export OMP_NUM_THREADS=1", or alternatively switch to a different build of OpenCV. +* Some builds of OpenCV use OpenMP, which by default opens a thread for each of your CPUs for background work. This multiplies out to a potentially large number if you have already started many processes for Caiman work. If this is both true and a problem for your environment, you should limit OpenMP to a single thread by doing the following before running Caiman code: "export OMP_NUM_THREADS=1", or alternatively switch to a different build of OpenCV. diff --git a/docs/source/conf.py b/docs/source/conf.py index 170874d7d..bf5a83bcf 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -55,7 +55,7 @@ # General information about the project. project = 'CaImAn' -copyright = '2019, Flatiron Institute, Simons Foundation, New York, NY' +copyright = '2024, Flatiron Institute, Simons Foundation, New York, NY' author = 'Eftychios Pnevmatikakis and Andrea Giovannucci' # The version info for the project you're documenting, acts as replacement for diff --git a/docs/file_formats.md b/docs/source/file_formats.md similarity index 88% rename from docs/file_formats.md rename to docs/source/file_formats.md index 3e283de22..6e1a94246 100644 --- a/docs/file_formats.md +++ b/docs/source/file_formats.md @@ -1,5 +1,5 @@ File Formats -====== +============ These notes cover imaging data you'll load into Caiman for analysis, typically 2d (occasionally 3d) data over time. There are several formats we support, some with caveats. If there are other formats you want, or caveats you find particularly burdensome, reach out and if it's easy we may be able to adjust the software, or alternatively offer ideas on how to convert your files for use with Caiman. @@ -12,39 +12,54 @@ If you are comfortable programming, you will find the code to load the movies in If you want to implement your own handlers with a fork of our codebase, or submit a PR to us, start there. Caiman converts files from these formats into its own internal formats during motion correction. Notes for specific formats follow: -### Matlab (\*.mat) + +Matlab (\*.mat) +--------------- Caiman may be able to handle Matlab's mat files, although it is not a desirable format. We rely on the scipy.io.matlab functions to handle files of this type, except for certain versions of the format which are actually hdf5 files -### TIFF (\*.tiff, \*.tif, \*.btf) +TIFF (\*.tiff, \*.tif, \*.btf) +------------------------------ Caiman can handle many TIFF files, but TIFF is a very flexible file format and we do not handle every variant. Because calcium imaging data has a time element, there are many ways that files might saved in this format (series, pages, some other things). We don't anticipate every variant. -### AVI/MKV (\*.avi, \*.mkv) +AVI/MKV (\*.avi, \*.mkv) +------------------------ Caiman can handle many AVI or MKV files, both of which are common video formats often generated by microscopes. AVI is a flexible container format that can use a number of different codecs for video encoding; we depend on available libraries in the environment or on the system to handle this; this is an area where Caiman has different capabilities depending on the operating system it runs on. We rely on OpenCV to handle this format, but try to automatically fall back to PIMS using its PyAv backend if OpenCV fails. You can force use of this fallback by setting the CAIMAN\_LOAD\_AVI\_FORCE\_FALLBACK environment variable -### Numpy arrays (\*.npy, \*.npz) +Numpy arrays (\*.npy, \*.npz) +----------------------------- Caiman can load files in numpy's native array file format. This will likely only have been produced by some other software (that you found or wrote) that preprocessed your original data and saved it, and that software was likely written in Python because few things not written in Python might've written this Python-specific format. It's fast and easy to work with this format -### HDF5/N5/zarr (\*.h5, \*.hdf5, \*.n5, \*.zarr) +HDF5/N5/zarr (\*.h5, \*.hdf5, \*.n5, \*.zarr) +--------------------------------------------- Caiman can easily work with files using these container formats for grid data. In many ways they're ideal - the format is regular, parsing them is easy, a variety of programming languages can work with them, and reads/writes are very fast. HDF5 files are really files; n5 and zarr files are directories with a hierarchy beneath, which can make moving them between hosts annoying (but their performance is even better). Because all these formats can have multiple datasets inside a single store, if there is more than one dataset present you'll need to tell Caiman what dataset you want to work with when you first start processing the data (using the var\_name\_hdf5 flag to cm.load() and related functions). If there is only one dataset present in the file, Caiman will default to using that one and doesn't need to be told. -### Memmapped Numpy Array (\*.mmap) +Memmapped Numpy Array (\*.mmap) +------------------------------- Caiman internally uses this data format, and you can use it too with Caiman, although it is really unlikely you'll come across this format elsewhere and we don't recommend you generate it outside of Caiman (we also may eventually change this format or remove it) -### Scanbox Format (\*.sbx) +Scanbox Format (\*.sbx) +----------------------- This format is a common format used by the Scanbox application (written in Matlab). It is a variant of the .mat file (above) with a particular internal structure and interpretation of values. If you need to use it, Caiman may be able to load it, but we don't recommend you use this format otherwise. The scipy.io.loadmat() functions (with a lot of supporting code) are used to support this. -### SIMA (\*.sima) +SIMA (\*.sima) +-------------- Support for this format was removed in an earlier version of Caiman. -### Zipfile full of images (\*.zip) +Zipfile full of images (\*.zip) +------------------------------- The default functions don't support this. Read the docs for movies.py:from\_zip\_file\_to\_movie() to use this support -## Conversion tools +File Formats: Addenda +===================== + +Conversion tools +---------------- - ImageJ - A powerful (but high learning curve) tool that can visualise and convert a number of formats - TiffIt - https://github.com/EricThomson/tiffit - Can convert incompatible tiff files into compatible tiff files - ImageMagick - A suite of commandline tools for image manipulation -## Adding new types +Adding new types +---------------- Qualifiers for other formats we might consider adding on request: - Grid data formats or movie formats are more likely than formats designed to store single images - Support for storing large enough datasets to be usable by caiman diff --git a/docs/source/index.rst b/docs/source/index.rst index 042e21a40..25e475cab 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -12,11 +12,14 @@ Contents: :maxdepth: 3 Overview + CaImAn_features_and_references Installation Getting_Started Handling_Movies + performance_guide CaImAn_Tips - On_file_types_and_sizes + file_formats GUI - CaImAn_features_and_references + readme-gpu + cluster core_functions diff --git a/docs/performance_guide.md b/docs/source/performance_guide.md similarity index 70% rename from docs/performance_guide.md rename to docs/source/performance_guide.md index 9943f94d4..cb604f94b 100644 --- a/docs/performance_guide.md +++ b/docs/source/performance_guide.md @@ -1,28 +1,28 @@ -CaImAn Performance Guide -======================= +Performance Guide +======================== Do not start with this guide. -* First, please get CaImAn working with your data, using the minimal set of changes to defaults needed to verify that it is working -* Second, become familiar with the CaImAn software; without that it will be hard for you to know what to change and why +* First, please get Caiman working with your data, using the minimal set of changes to defaults needed to verify that it is working +* Second, become familiar with the Caiman software; without that it will be hard for you to know what to change and why This guide will change according to feedback and common questions, and is designed to address both CPU optimisation and memory optimisation. As of this writing, the guide is minimal; more substantial advice will be filled in over time. Online versus offline -===================== -The first choice with CaImAn is whether to use it in online or offline mode. This is not purely a performance concern - the features available between the two modes are not identical, with the online mode having some newer features and having generally better performance. The difference is covered better elsewhere (including the CaImAn paper), but in brief: +--------------------- +The first choice with Caiman is whether to use it in online or offline mode. This is not purely a performance concern - the features available between the two modes are not identical, with the online mode having some newer features and having generally better performance. The difference is covered better elsewhere (including the Caiman paper), but in brief: * data passes through the offline mode in a series of steps that run to completion before the next * data passes through the online mode streaming through those steps, where a chunk of data may be entering an early stage (like motion correction) while other chunks may already be in a later stage (like signal extraction) Clustering options -================== -CaImAn supports running in linear mode (which is very slow and useful mainly for debugging), ipyparallel, and multiprocessing (aka local). You can also set the number of processes CaImAn will use (when not in linear mode). Clustering options are set in the call to `cm.cluster.setup_cluster()` and are one knob you will want to look at (although note that you may find bugs with unusual settings, or find that some settings require an unreasonable amount of RAM with your data). +------------------ +Caiman supports running in linear mode (which is very slow and useful mainly for debugging), ipyparallel, and multiprocessing. You can also set the number of processes Caiman will use (when not in linear mode). Clustering options are set in the call to `cm.cluster.setup_cluster()` and are one knob you will want to look at (although note that you may find bugs with unusual settings, or find that some settings require an unreasonable amount of RAM with your data). Patch size -========== -Patch size is the other side of clustering options, and specify the granularity of breaking down the data for processing. This setting must be set in ways that fit your clustering options; spinning up a number of worker processes will not be helpful if there are not sufficient patches to hand out to them for work. Patches are the unit of potential work in most stages in the CaImAn pipeline (online or offline). +---------- +Patch size is the other side of clustering options, and specify the granularity of breaking down the data for processing. This setting must be set in ways that fit your clustering options; spinning up a number of worker processes will not be helpful if there are not sufficient patches to hand out to them for work. Patches are the unit of potential work in most stages in the Caiman pipeline (online or offline). Alternate builds of dependencies and unsupported environment variables -====================================================================== -If you cannot get the performance (or memory usage) you need with CaImAn, you may be able to explore alternate versions of CaImAn dependencies in the conda environment, as well as some of the undocumented (such as GPU-based function) features in the CaImAn codebase. This is only for the adventurous, as we will not be able to support such features very readily, and our environment might not match yours. If you do this, please familiarise yourself with the codebase first. +---------------------------------------------------------------------- +If you cannot get the performance (or memory usage) you need with Caiman, you may be able to explore alternate versions of Caiman dependencies in the conda environment, as well as some of the undocumented (such as GPU-based function) features in the Caiman codebase. This is only for the adventurous, as we will not be able to support such features very readily, and our environment might not match yours. If you do this, please familiarise yourself with the codebase first. If you have suitable GPU hardware, look into enabling GPU support in tensorflow, by reading the README-GPU.md doc. diff --git a/docs/README-GPU.md b/docs/source/readme-gpu.md similarity index 59% rename from docs/README-GPU.md rename to docs/source/readme-gpu.md index c9d97903d..71d33ed80 100644 --- a/docs/README-GPU.md +++ b/docs/source/readme-gpu.md @@ -1,11 +1,11 @@ -CaImAn and Tensorflow -===================== -Some parts of CaImAn already use Tensorflow, and those parts can benefit from using a +CaImAn and GPUs +=============== +Some parts of Caiman already use Tensorflow, and those parts can benefit from using a hardware-accelerated version of Tensorflow. These versions can be installed using Conda (presently only on the Linux and Windows platforms). Note that this is experimental and our ability to support this environment is limited. To do this: -* First install the CaImAn environment as normal. Ideally test it. +* First install the Caiman environment as normal. Ideally test it. * Next, install tensorflow-gpu as follows: `conda install tensorflow-gpu` * Finally, reinstall opencv using the latest openblas build. `conda update opencv` @@ -13,39 +13,28 @@ The last step should get you a current openblas-based build, although in some ca build of opencv (which may not have graphical bindings). If this happens, you may be able to search for the most recent opencv with openblas and switch to it using existing conda tooling. -To do this, check after switching package versions that KERAS_BACKEND is set to tensorflow. If not, you can set it in your terminal by typing `KERAS_BACKEND=tensorflow`. On some platforms it defaults to theano, which will not be hardware-accelerated. This can affect the computational of the CaImAn online algorithm. +To do this, check after switching package versions that KERAS_BACKEND is set to tensorflow. If not, you can set it in your terminal by typing `KERAS_BACKEND=tensorflow`. On some platforms it defaults to theano, which will not be hardware-accelerated. This can affect the computational of the Caiman online algorithm. -CaImAn and CUDA -=============== +Caiman and CUDA +--------------- -CaImAn has experimental support for computing FFTs on your GPU, -using the CUDA libraries instead of OpenCV or numpy. This can be used during motion correction. +Caiman has experimental support for computing FFTs on your GPU, +using the pycuda libraries instead of OpenCV or numpy. This can be used during motion correction. -# Installation -We assume you have CaImAn generally working first. If you do not, +Installation +------------ +We assume you have Caiman generally working first. If you do not, follow the general instructions first and verify you have a working environment. -Next, you will want to install the CUDA libraries. One way to do this is -to go to the nVIDIA site here: -https://developer.nvidia.com/gpu-accelerated-libraries - -Follow the docs there (this can be involved), install the samples, -and make sure your videocard is found using the deviceQuery tool. -If it does not show up, do not go further; debug your setup until this works. - -When cuda is done being set up, enter your conda environment and do the following: -``` -conda install -c lukepfister pycuda -pip install scikit-cuda -``` +One (but less-tested with Caiman) route is to install pycuda from conda-forge; activate your +caiman environment and then do `conda install -c conda-forge pycuda`. This may be all you need. -The above installs pycuda from what (as of this writing) is the best available -distribution of pycuda in a conda repository, and it installs skcuda using pip -(as it must be compiled against the CUDA on your system). Better options may -become available for both of these as conda-forge evolves. +It is possible to instead install CUDA, then use pip to install pycuda, but this is involved enough +that it's better to try the above first. -# Use +Use +--- The CUDA codepaths will only be active if the needed libraries are installed on your system. Otherwise non-CUDA codepaths will be active (even if CUDA is requested in our code). The following functions have been extended with a