Large scale inference docs #994

benjeffery · 2025-02-04T10:23:26Z

Fixes #840

benjeffery · 2025-02-04T10:23:42Z

@hyanwong Can I have a read-through here?

codecov · 2025-02-04T10:38:16Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.17%. Comparing base (e3b2155) to head (d6725ce).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #994      +/-   ##
==========================================
- Coverage   93.17%   93.17%   -0.01%     
==========================================
  Files          18       18              
  Lines        6374     6369       -5     
  Branches     1088     1088              
==========================================
- Hits         5939     5934       -5     
  Misses        296      296              
  Partials      139      139

Flag	Coverage Δ
C	`93.17% <100.00%> (-0.01%)`	⬇️
python	`95.52% <100.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jeromekelleher

LGTM

jeromekelleher · 2025-02-04T19:38:37Z

tsinfer/inference.py

+    :param int min_work_per_job: The minimum amount of work (as a count of genotypes) to
+        allocate to a single parallel job. If the amount of work in a group of ancestors
+        exceeds this level it will be broken up into parallel partitions, subject to
+        the constriant of `max_num_partitions`.


typo, constriant

Fixed in 6c02df1

hyanwong

Great works, thanks @benjeffery

It's all quite complicated, and I'm not sure I have a feel for how all the parts fit together, but the descriptions are detailed enough that I could follow them without problems.

I guess at some future point we might want a schematic, but that can wait for now. I reckon you can merge this and get someone (e.g. Duncan or Savita?) to try it out.

hyanwong · 2025-02-05T17:07:08Z

docs/large_scale.md

+entire genotype array for the contig being inferred needs to fit in RAM.
+This is the high-water mark for memory usage in tsinfer.
+Note the `genotype_encoding` argument, setting this to
+{class}`tsinfer.GenotypeEncoding.ONE_BIT` reduces the memory footprint of


Do we need to say that this can't be used if there is missing data?

Fixed in 6c02df1

hyanwong · 2025-02-05T17:07:28Z

docs/large_scale.md

+The plot below shows the number of ancestors matched in each group for a typical
+human data set:
+
+```{figure} _static/ancestor_grouping.png


May be worth indicating that the group number is ordered by time, so that group 0 represents the oldest ancestors?

Fixed in 6c02df1

hyanwong · 2025-02-05T17:07:33Z

docs/large_scale.md

+{meth}`match_ancestors_batch_group_finalise` will then insert the matches and
+output the tree sequence to `work_dir`.
+
+At anypoint the process can be resumed from the last successfully completed call to 


"anypoint" -> "any point"

hyanwong · 2025-02-05T17:07:55Z

docs/large_scale.md

+
+At anypoint the process can be resumed from the last successfully completed call to 
+{meth}`match_ancestors_batch_groups`. As the tree sequences in `work_dir` checkpoint the
+progress.


I'm not sure I understand / can parse this last sentence

Fixed in 6c02df1 hopefully makes more sense now.

benjeffery · 2025-02-06T13:04:45Z

Thanks @hyanwong

I guess at some future point we might want a schematic

I think I'll add a quick one now

benjeffery · 2025-02-06T13:54:59Z

I guess at some future point we might want a schematic

Added in d039461

Let me know if it makes more sense now!

hyanwong · 2025-02-07T10:17:10Z

Looks great, thanks.

agladstein

Here are some initial thoughts.

agladstein · 2025-02-07T21:18:48Z

docs/large_scale.md

+tsinfer scales well and has been successfully used with datasets up to half a
+million samples. Here we detail considerations and tips for each step of the
+inference process to help you scale up your analysis. A snakemake pipeline
+which implements this parallelisation scheme is available at https://github.com/benjeffery/tsinfer-snakemake.


Nit: It would be good to make this a link.

Suggested change

which implements this parallelisation scheme is available at https://github.com/benjeffery/tsinfer-snakemake.

which implements this parallelisation scheme is available at <https://github.com/benjeffery/tsinfer-snakemake>.

It would also be good to have a small README in that repo giving a little summary of what it is and how to run it, and possibly redirecting here for more info.

edcf3ff

Yes, documenting that repo is a big item on my todo list.

agladstein · 2025-02-07T21:21:47Z

docs/large_scale.md

+
+# Large Scale Inference
+
+tsinfer scales well and has been successfully used with datasets up to half a


Can you provide some example citations here? If I wanted to use tsinfer for large datasets, I would probably also want to see the pubs it was used in.
(I assume "has been successfully used" means it's been published)

Publications are in preparation

agladstein · 2025-02-10T19:15:32Z

docs/large_scale.md

+
+(sec_large_scale_ancestor_generation)=
+
+## Data preparation


This section is a little sparse. If I hadn't used VCF zarr format before, I'd be a little intimidated on how to use it. Can you include a short example?

I've added a TODO in abc170f as there will be general tutorial on this as it applies to small scale too.

agladstein · 2025-02-10T19:16:01Z

docs/large_scale.md

+## Data preparation
+
+For large scale inference the data must be in [VCF Zarr](https://github.com/sgkit-dev/vcf-zarr-spec)
+format, read by the {class}`VariantData` class. [bio2zarr](https://github.com/sgkit-dev/bio2zarr)


I don't see any docs on VariantData

They are todo for this release, I think they are in the code, but need autodocing.

agladstein · 2025-02-10T19:29:56Z

docs/large_scale.md

+which implements this parallelisation scheme is available at https://github.com/benjeffery/tsinfer-snakemake.
+
+(sec_large_scale_ancestor_generation)=
+


I think in this intro section, before going into more detail, it could be helpful to give an overview about what is "hard" about large scale inference. What counts has "large scale" and if you don't do the following suggestions, where might you see your inference be slow/get stuck/run out of memory...? And then give a brief overview of how you get around these problems.

Added more context in c6b5e1e

agladstein · 2025-02-10T19:41:07Z

docs/large_scale.md

+the genotype array by a factor of 8, for a surprisingly small increase in
+runtime. With this encoding, the RAM needed is roughly 
+`num_sites * num_samples * ploidy / 8 bytes.` However this encoding
+only supports biallelic sites, with no missingness.


Can you give a quick example? Something like - if you are generating ancestors for --- sites, --- samples, --- ploidy, and running on a 32G machine, using 16 threads with genotype_encoding = tsinfer.GenotypeEncoding.ONE_BIT, we can expect the genotype array to fit in memory. But we would expect ---- sites to hit a memory error.

The formula is exact, so people should be able to work it out?

agladstein · 2025-02-10T19:45:43Z

docs/large_scale.md

+groups, depending on job set up and queueing time on your cluster.
+
+Calls to {meth}`match_ancestors_batch_group_partition` will only use a single core, but 
+{meth}`match_ancestors_batch_groups` will use as many cores as `num_threads` is set to


Suggested change

{meth}`match_ancestors_batch_groups` will use as many cores as `num_threads` is set to

{meth}`match_ancestors_batch_groups` will use as many cores as `num_threads` is set to.

agladstein · 2025-02-10T19:59:58Z

docs/large_scale.md

+`num_sites * num_samples * ploidy / 8 bytes.` However this encoding
+only supports biallelic sites, with no missingness.
+
+## Ancestor matching


I read this section through once and don't feel like I fully understood enough to start implementing. I understood that the ancestor matching can be slow and can be parallelized by using partitions. After reading once through, my main question was is this parallezation on a multicore single node, or can this parallezation be done on single cores on multiple machines, and what is the max parallezation I can get out of it?

It's possible, with more careful multiple readings of this section, I'd figure out the answer to those questions. But, goal is probably for the reader (with my background) to be able to read it once through carefully and understand.

I've added a bit extra in d6725ce
It's a complicated scheme, unfortunately.

agladstein · 2025-02-10T20:01:23Z

docs/large_scale.md

+```
+
+There are five tsinfer API methods that can be used to parallelise ancestor
+matching. 


List those five tsinfer API methods here.

agladstein · 2025-02-10T20:18:49Z

docs/large_scale.md

+## Sample matching 
+
+Sample matching is far simpler than ancestor matching as it is essentially the same as a single group
+of ancestors. There are three API methods that work together to enable distributed sample matching.


List here what those three methods are.

benjeffery · 2025-02-11T15:30:18Z

Thanks for all the feedback @agladstein! Hopefully I have addressed it.

Large scale inference docs

37645ef

benjeffery force-pushed the large-scale-docs branch from 952dbb9 to 37645ef Compare February 4, 2025 12:28

jeromekelleher approved these changes Feb 4, 2025

View reviewed changes

benjeffery mentioned this pull request Feb 5, 2025

Document resume option #834

Closed

hyanwong approved these changes Feb 5, 2025

View reviewed changes

benjeffery added 2 commits February 6, 2025 13:49

Add example flow

d039461

Nits

6c02df1

agladstein reviewed Feb 10, 2025

View reviewed changes

benjeffery added 7 commits February 11, 2025 15:07

Add link

edcf3ff

Add tut todo

abc170f

Add more context

c6b5e1e

Explain out-of-core

587c110

Full stop

61292ce

List methods

6a65530

Reduce confusion?

d6725ce

	which implements this parallelisation scheme is available at https://github.com/benjeffery/tsinfer-snakemake.
	which implements this parallelisation scheme is available at <https://github.com/benjeffery/tsinfer-snakemake>.


		# Large Scale Inference

		tsinfer scales well and has been successfully used with datasets up to half a

		which implements this parallelisation scheme is available at https://github.com/benjeffery/tsinfer-snakemake.

		(sec_large_scale_ancestor_generation)=

	{meth}`match_ancestors_batch_groups` will use as many cores as `num_threads` is set to
	{meth}`match_ancestors_batch_groups` will use as many cores as `num_threads` is set to.

Large scale inference docs #994

Are you sure you want to change the base?

Large scale inference docs #994

Conversation

benjeffery commented Feb 4, 2025

benjeffery commented Feb 4, 2025

codecov bot commented Feb 4, 2025 • edited Loading

Codecov Report

jeromekelleher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hyanwong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjeffery commented Feb 6, 2025

benjeffery commented Feb 6, 2025 • edited Loading

hyanwong commented Feb 7, 2025

agladstein left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjeffery commented Feb 11, 2025

codecov bot commented Feb 4, 2025 •

edited

Loading

benjeffery commented Feb 6, 2025 •

edited

Loading