Revisions to paper text

paper/paper.bib

@@ -345,7 +345,7 @@ @misc{beniaguev_dendro_plexing_2024
 @inproceedings{
 hassani2023dilated,
 title={Dilated convolution with learnable spacings},
-author={Ismail Khalfaoui Hassani and Thomas Pellegrini and Timoth{\'e}e Masquelier},
+author={Ismail Khalfaoui-Hassani and Thomas Pellegrini and Timoth{\'e}e Masquelier},
 booktitle={The Eleventh International Conference on Learning Representations },
 year={2023},
 url={https://openreview.net/forum?id=Q3-1vRh3HOA}
@@ -530,3 +530,11 @@ @misc{kingma2017adammethodstochasticoptimization
       primaryClass={cs.LG},
       url={https://arxiv.org/abs/1412.6980}, 
 }
+
+@inproceedings{PeiYe2021,
+  title={Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity},
+  author={Felix Pei and Joel Ye and David M. Zoltowski and Anqi Wu and Raeed H. Chowdhury and Hansem Sohn and Joseph E. O’Doherty and Krishna V. Shenoy and Matthew T. Kaufman and Mark Churchland and Mehrdad Jazayeri and Lee E. Miller and Jonathan Pillow and Il Memming Park and Eva L. Dyer and Chethan Pandarinath},
+  booktitle={Advances in Neural Information Processing Systems (NeurIPS), Track on Datasets and Benchmarks},
+  year={2021},
+  url={https://arxiv.org/abs/2109.04463}
+}

paper/paper.md

@@ -113,7 +113,7 @@ downloads:
 
 
 +++ {"part": "abstract"}
-Neuroscientists are increasingly initiating large-scale collaborations which bring together tens to hundreds of researchers. However, while these projects represent a step-change in scale, they retain a traditional structure with centralised funding, participating laboratories and data sharing on publication. Inspired by an open-source project in pure mathematics, we set out to test the feasibility of an alternative structure by running a grassroots, massively collaborative project in computational neuroscience. To do so, we launched a public Git repository, with code for training spiking neural networks to solve a sound localisation task via surrogate gradient descent. We then invited anyone, anywhere to use this code as a springboard for exploring questions of interest to them, and encouraged participants to share their work both asynchronously through Git and synchronously at monthly online workshops. At a scientific level, our work investigated how a range of biologically-relevant parameters, from time delays to membrane time constants and levels of inhibition, could impact sound localisation in networks of spiking units. At a more macro-level, our project brought together 31 researchers from multiple countries, provided hands-on research experience to early career participants, and opportunities for supervision and teaching to later career participants. Looking ahead, our project provides a glimpse of what open, collaborative science could look like and provides a necessary, tentative step towards it.  
+Neuroscientists are increasingly initiating large-scale collaborations which bring together tens to hundreds of researchers. However, while these projects represent a step-change in scale, their workflow resembles many scientific endeavours. That is, there are participating laboratories who collaborate together and then make their data, methods and results available. Inspired by open-source projects in pure mathematics, we set out to test the feasibility of an alternative structure by running a grassroots, massively collaborative project in computational neuroscience. To do so, we launched a public Git repository, with code for training spiking neural networks to solve a sound localisation task via surrogate gradient descent. We then invited anyone, anywhere to use this code as a springboard for exploring questions of interest to them, and encouraged participants to share their work both asynchronously through Git and synchronously at monthly online workshops. At a scientific level, our work investigated how a range of biologically-relevant parameters, from time delays to membrane time constants and levels of inhibition, could impact sound localisation in networks of spiking units. At a more macro-level, our project brought together 31 researchers from multiple countries, provided hands-on research experience to early career participants, and opportunities for supervision and teaching to later career participants. Looking ahead, our project provides a glimpse of what open, collaborative science could look like and provides a necessary, tentative step towards it.  
 +++
 
 # Introduction

paper/sections/discussion.md

@@ -1,6 +1,6 @@
 ## What went well
 
-The decision to start from the code base of the [Cosyne tutorial](https://neural-reckoning.github.io/cosyne-tutorial-2022/) {cite:p}`10.5281/zenodo.7044500` was very helpful. It meant that users had a clear entry path for the project without needing prior expertise, and a code base that was designed to be easy to understand. In addition, the popularity of the tutorial (over 30k views on YouTube at the time of writing) meant that many people heard about this project and were interested in participating. In addition, the GitHub-based infrastructure allowed for asynchronous work and a website that was automatically updated each time anyone made a change to their code or to the text of the paper.
+The decision to start from the code base of the [Cosyne tutorial](https://neural-reckoning.github.io/cosyne-tutorial-2022/) {cite:p}`10.5281/zenodo.7044500` was very helpful. It meant that users had a clear entry path for the project without needing prior expertise, and a code base that was designed to be easy to understand. In addition, the popularity of the tutorial (over 30k views on YouTube at the time of writing) meant that many people heard about this project and were interested in participating. In addition, the GitHub-based infrastructure allowed for asynchronous work our website, that was automatically updated each time anyone made a change to their code or to the text of the paper, allowed for easy sharing of results. 
 
 By providing models which used spiking neurons to transform sensory inputs into behavioural outputs, participants were free to explore in virtually any direction they wished, much like an open-world or sandbox video game. Indeed over the course of the project we explored the full sensory-motor transformation from manipulating the nature of the input signals to perturbing unit activity and assessing network behaviour. Consequently, our code forms an excellent basis for teaching, as concepts from across neuroscience can be introduced and then implemented in class. In this direction, we integrated our project into two university courses and provide slides and a highly annotated python notebook, for those interested in teaching with these models.  
 
@@ -13,8 +13,8 @@ Our second challenge, was the project's exploratory nature. While this appealed
 
 A third challenge, which arose towards the end of the project, was how to fairly assign credit. We had initially - and perhaps somewhat idealistically - stated that anyone who contributed to the project, either by writing code or participating in one of the workshops, would be included on the author list. To the extent that it was possible, we have followed through with this, though we were simply unable to contact several of the participants and so could not include them as authors. Another issue with this system is that participants with unequal contributions, e.g. attending a workshop vs contributing an entire section of the paper, would be assigned similar credit, i.e. authorship. To resolve this, we experimented with using the number or size of GitHub commits to order authors, however we found that these metrics did not accurately reflect contributions. For example, it may be quicker to commit a large-amount of low quality text than a concise well written section, and similarly there is no good reason to distinguish between two authors who submit the same amount of work through a different number of commits. We attempted to address this challenge by providing a contributions table ([](#contributors)) and agreeing an author order. This order was agreed on unanimously, though could easily cause issues in other projects. Consequently, we recommend that a strategy for credit assignment be determined collaboratively at the start of the project, and made explicit so that participants can clearly understand how their contribution will translate to credit. Alternatively, such projects could publish under a pseudonym, e.g. COMOB.    
 
-Ultimately, while the project explored many interesting directions, which will form the basis for future work, we did not reach a point where we could draw strong scientific conclusions about sound localization. From group discussions we concluded that this is likely due to the free-form nature of our project, which would have benefited from a more coordinated approach. The question is, how to do this without compromising the ideals of a grass-roots project? Extending the voting idea above, one approach would be to make the proposer of the, democratically selected, project responsible for making sure that results are comparable and generally keeping the project on the right track. A role similar to a traditional supervisor, but with the critical difference that they are elected by their peers and only on a project by project basis. 
+Ultimately, while the project explored many interesting directions, which will form the basis for future work, we did not reach a point where we could draw strong scientific conclusions about sound localization. From group discussions we concluded that this is likely due to the free-form nature of our project, which would have benefited from a more coordinated approach. The question is, how to do this without compromising the ideals of a grass-roots project? Extending the voting idea above, one approach would be to make the proposer of the democratically selected project responsible for making sure that results are comparable and generally keeping the project on the right track. A role similar to a traditional supervisor, but with the critical difference that they are elected by their peers and only on a project by project basis. 
 
 ## Conclusions
 
-This paper does not present a scientific breakthrough. However, it does demonstrate the feasibility of open research projects which bring together large number of participants across countries and career stages to work together collaboratively on scientific projects. Moreover, several follow-up research projects are planned based on pilot data from our work and, building on our experience, we plan to launch a second COMOB project soon. 
+This paper does not present a scientific breakthrough. However, it does demonstrate the feasibility of open research projects which bring together large number of participants across countries and career stages to work together collaboratively on scientific projects. Looking ahead, we hope that the diversity of expertise and perspectives, such projects support, will allow for discoveries beyond what any single group could realise. 

paper/sections/intro.md

@@ -1,7 +1,7 @@
-Inspired by the success of endeavours like the [Human Genome Project](https://www.genome.gov/human-genome-project) and [CERN](https://home.cern/), neuroscientists are increasingly initiating large-scale collaborations. Though, how to best structure these projects remains an open-question {cite:p}`doi.org/10.1038/539159a`. The largest efforts, e.g. the [International Brain Laboratory](https://www.internationalbrainlab.com/) [@doi.org/10.1016/j.neuron.2017.12.013;@doi.org/10.1016/j.conb.2020.10.020], [The Blue Brain Project](https://www.epfl.ch/research/domains/bluebrain/) and [Human Brain Project](https://www.humanbrainproject.eu) bring together tens to hundreds of researchers across multiple laboratories. However, while these projects represent a step-change in scale, they retain a legacy structure which resembles a consortia grant. I.e. there are participating laboratories who collaborate together and then make their data, methods and results available upon publication. As such, interested participants face a high barrier to entry: joining a participating laboratory, initiating a collaboration with the project, or awaiting publications. So how could these projects be structured differently?   
+Inspired by the success of endeavours like the [Human Genome Project](https://www.genome.gov/human-genome-project) and [CERN](https://home.cern/), neuroscientists are increasingly initiating large-scale collaborations. The largest efforts, such as the [International Brain Laboratory](https://www.internationalbrainlab.com/) [@doi.org/10.1016/j.neuron.2017.12.013;@doi.org/10.1016/j.conb.2020.10.020], [The Blue Brain Project](https://www.epfl.ch/research/domains/bluebrain/) and [Human Brain Project](https://www.humanbrainproject.eu) bring together tens to hundreds of researchers across multiple laboratories. These projects have generated scientific insights, large-scale datasets, tools and educational materials. However, while they represent a step-change in scale, their workflow resembles many scientific endeavours. That is, there are participating laboratories who collaborate together and then make their data, methods and results available. As such, to participate, individuals must join a participating laboratory, initiate a collaboration with the project, or wait for the publication of data and resources. So how could these projects be structured differently{cite:p}`doi.org/10.1038/539159a`?
 
-One alternative is a bench marking contest, in which participants compete to obtain the best score on a specific task. Such contests have driven progress in fields from machine learning {cite:p}`10.1109/CVPR.2009.5206848` to [protein folding](https://predictioncenter.org/), and have begun to enter neuroscience. For example, in [Brain-Score](https://www.brain-score.org/) [@10.1101/407007;@10.1016/j.neuron.2020.07.040] participants submit models, capable of completing a visual processing task, which are then ranked according to a quantitative metric. As participants can compete both remotely and independently, these contests offer a significantly lower barrier to entry. Though, they emphasise competition over collaboration, and critically they require a well defined, quantifiable endpoint. In [Brain-Score](https://www.brain-score.org/), this endpoint is a composite metric which describes the model's similarity to experimental data in terms of both behaviour and unit activity. However, this metric's relevance is debatable {cite:p}`doi:10.1017/S0140525X22002813` and more broadly, defining clear endpoints for neuroscientific questions remains challenging.    
+One alternative are bench marking contests, in which participants compete to obtain the best score on a specific task. Bench marking contests have driven progress in fields from computer vision {cite:p}`10.1109/CVPR.2009.5206848` to [protein folding](https://predictioncenter.org/), and have begun to enter neuroscience. For example, in [Brain-Score](https://www.brain-score.org/) [@10.1101/407007;@10.1016/j.neuron.2020.07.040] participants submit models, capable of completing a visual processing task, which are then ranked according to a quantitative metric. As participants can compete both remotely and independently, these contests offer a low barrier to entry. However, defining quantifiable endpoints for neuroscientific questions remains challenging {cite:p}`doi:10.1017/S0140525X22002813`.  
 
-Another alternative is massively collaborative projects in which participants work together to solve a common goal. For example, in the [Polymath Project](https://polymathprojects.org/) unsolved mathematical problems are posed, and then participants share comments, ideas and equations online as they collectively work towards solutions. Similarly, the [Busy Beaver Challenge](https://bbchallenge.org/) recently [announced](https://discuss.bbchallenge.org/t/july-2nd-2024-we-have-proved-bb-5-47-176-870/237) a formal proof of a conjecture that was open for decades, [based mainly on contributions from amateur mathematicians, organised purely online](https://www.quantamagazine.org/amateur-mathematicians-find-fifth-busy-beaver-turing-machine-20240702/). Inspired by this approach, we founded [COMOB (Collaborative Modelling of the Brain)](https://comob-project.github.io/) - an open-source movement, which aims to tackle neuroscientific questions. Here, we share our experiences and results from our first project, in which we explored spiking neural network models of sound localization.
+Another alternative are massively collaborative projects, in which a problem is openly stated and contributions are welcomed from anyone willing to participate. For example, in the [Polymath Project](https://polymathprojects.org/) unsolved mathematical problems are posed, and then participants share comments, ideas and equations online as they collectively work towards solutions. Similarly, the [Busy Beaver Challenge](https://bbchallenge.org/) recently [announced](https://discuss.bbchallenge.org/t/july-2nd-2024-we-have-proved-bb-5-47-176-870/237) a formal proof of a conjecture that was open for decades, [based mainly on online contributions from amateur mathematicians](https://www.quantamagazine.org/amateur-mathematicians-find-fifth-busy-beaver-turing-machine-20240702/). Inspired by this approach, we founded [COMOB (Collaborative Modelling of the Brain)](https://comob-project.github.io/) - to explore if and how this collaborative model could be leveraged in neuroscience. 
 
-We start by detailing how we ran the project in terms of organisation and infrastructure in [](#metascience). We then briefly summarise our scientific results in [](#science). We conclude the main text with a [](#discussion) of what went well, what went wrong, and how we think future projects of this sort could learn from our experiences. Finally, in the [](#appendices) we provide longer, detailed write-ups of some of our scientific results.
+ Here, we share our experiences and results. We start by detailing how we ran the project in terms of organisation and infrastructure in [](#metascience). We then briefly summarise our scientific results in [](#science). We conclude the main text with a [](#discussion) of what went well, what went wrong, and how future projects like this could learn from our experiences. Finally, in the [](#appendices) we provide detailed write-ups of our scientific results.