From a555c2264da9fb217b83324872c2d2307d707e68 Mon Sep 17 00:00:00 2001 From: Jaclyn Taroni <19534205+jaclyn-taroni@users.noreply.github.com> Date: Mon, 25 Nov 2024 09:56:10 -0500 Subject: [PATCH 1/7] Add renumbered version of GSEA notebook that uses pseudobulk DE results --- .../04-gene_set_enrichment_analysis.Rmd | 286 ++ .../04-gene_set_enrichment_analysis.nb.html | 3501 +++++++++++++++++ 2 files changed, 3787 insertions(+) create mode 100644 scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd create mode 100644 scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html diff --git a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd new file mode 100644 index 00000000..0f300979 --- /dev/null +++ b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd @@ -0,0 +1,286 @@ +--- +title: "Pathway analysis: Gene Set Enrichment Analysis (GSEA)" +output: + html_notebook: + toc: true + toc_float: true +author: CCDL for ALSF +date: 2024 +--- + +## Objectives + +This notebook will demonstrate how to: + +- Prepare tabular data of gene-level statistics for use with Gene Set Enrichment Analysis (GSEA) +- Access [Molecular Signatures Database gene set collections](https://www.gsea-msigdb.org/gsea/msigdb/collections.jsp) via the `msigdbr` package +- Perform GSEA with the `clusterProfiler` package +- Visualize GSEA results with the `enrichplot` package + +--- + +In this notebook, we'll analyze the differential expression results from the last notebook. + +GSEA is a functional class scoring (FCS) approach to pathway analysis that was first introduced in [Subramanian _et al._ (2005)](https://doi.org/10.1073/pnas.0506580102). +The rationale behind FCS approaches is that small changes in individual genes that participate in the same biological process or pathway can be significant and of biological interest. + +There are 3 general steps in FCS methods ([Khatri _et al._ 2012]( https://doi.org/10.1371/journal.pcbi.1002375)): + +1. Calculate a gene-level statistic (here, we'll use the summary log fold changes in our DESeq2 results) +2. Gene-level statistics are aggregated into a pathway-level statistic +3. Assess the statistical significance of the pathway-level statistic + +#### Other resources + +* For another example using `clusterProfiler` for GSEA, see [_Intro to DGE: Functional Analysis._ from Harvard Chan Bioinformatics Core Training.](https://hbctraining.github.io/DGE_workshop/lessons/09_functional_analysis.html) +* The way we'll use `clusterProfiler` here uses `fgsea` (Fast Gene Set Enrichment Analysis) under the hood. +You can read more about fgsea in [Korotkevich _et al._ (2021)](https://doi.org/10.1101/060012). +* [refine.bio examples Gene set enrichment analysis - RNA-seq](https://alexslemonade.github.io/refinebio-examples/03-rnaseq/pathway-analysis_rnaseq_02_gsea.html) from which this material has been adapted. + +## Set up + +### Libraries + +```{r libraries} +# Package to run GSEA +library(clusterProfiler) +# Package that contains the MSigDB gene sets in tidy format +library(msigdbr) +``` + +### Directories and Files + +#### Directories + +```{r create_dir, live = TRUE} +# We'll use the marker genes as GSEA input +rms_analysis_dir <- file.path("analysis", "rms") + +# We'll create a directory to specifically hold the pathway results if it doesn't +# exist yet +results_dir <- file.path(rms_analysis_dir, "pathway-analysis") +fs::dir_create(results_dir) +``` + +#### Input files + + +```{r input_files} +input_file <- file.path(rms_analysis_dir, + "deseq", + "rms_myoblast_deseq_results.tsv") +``` + +#### Output files + +We'll save our table of GSEA results as a TSV. + +```{r output_files} +output_file <- file.path(results_dir, + "rms_myoblast_gsea_results.tsv") +``` + +## Gene sets + +We will use gene sets from the [Molecular Signatures Database (MSigDB)](https://www.gsea-msigdb.org/gsea/msigdb/index.jsp) from the Broad Institute ([Subramanian, Tamayo *et al.* 2005](https://doi.org/10.1073/pnas.0506580102)). +The [`msigdbr`](https://cran.r-project.org/web/packages/msigdbr/index.html) package contains MSigDB datasets already in the tidy format required by `clusterProfiler` and supports multiple organisms. + +Let's take a look at what organisms the package supports. + +```{r show_species} +msigdbr_species() +``` + +MSigDB contains 8 different gene set collections. + + H: hallmark gene sets + C1: positional gene sets + C2: curated gene sets + C3: motif gene sets + C4: computational gene sets + C5: GO gene sets + C6: oncogenic signatures + C7: immunologic signatures + +We'll use the Hallmark collection for GSEA. +Here's an excerpt of the [collection description](https://www.gsea-msigdb.org/gsea/msigdb/collection_details.jsp#H): + +> Hallmark gene sets summarize and represent specific well-defined biological states or processes and display coherent expression. These gene sets were generated by a computational methodology based on identifying gene set overlaps and retaining genes that display coordinate expression. The hallmarks reduce noise and redundancy and provide a better delineated biological space for GSEA. + +Notably, there are only 50 gene sets included in this collection. +The fewer gene sets we test, the lower our multiple hypothesis testing burden. + +We can retrieve only the Hallmark gene sets by specifying `category = "H"` to the `msigdbr()` function. + +```{r immunologic_sets, live = TRUE} +hs_hallmarks_df <- msigdbr(species = "Homo sapiens", + category = "H") +``` + +## Gene Set Enrichment Analysis + +_Adapted from [refine.bio examples](https://github.com/AlexsLemonade/refinebio-examples/blob/33cdeff66d57f9fe8ee4fcb5156aea4ac2dce07f/03-rnaseq/pathway-analysis_rnaseq_02_gsea.Rmd)_ + +![](diagrams/subramanian_fig1.jpg) + +**Figure 1. [Subramanian _et al._ (2005)](https://doi.org/10.1073/pnas.0506580102).** + +GSEA calculates a pathway-level metric, called an enrichment score (sometimes abbreviated as ES), by ranking genes by a gene-level statistic. +This score reflects whether or not a gene set or pathway is over-represented at the top or bottom of the gene rankings ([Subramanian _et al._ 2005](https://doi.org/10.1073/pnas.0506580102); [Yu](http://yulab-smu.top/clusterProfiler-book/chapter2.html#gene-set-enrichment-analysis)) + +Specifically, all genes are ranked from most positive to most negative based on their statistic and a running sum is calculated: +Starting with the most highly ranked genes, the running sum increases for each gene in the pathway and decreases for each gene not in the pathway. +The enrichment score for a pathway is the running sum's maximum deviation from zero. +GSEA also assesses statistical significance of the scores for each pathway through permutation testing. +As a result, each input pathway will have a p-value associated with it that is then corrected for multiple hypothesis testing ([Subramanian _et al._ 2005](https://doi.org/10.1073/pnas.0506580102); [Yu](http://yulab-smu.top/clusterProfiler-book/chapter2.html#gene-set-enrichment-analysis)). + +The implementation of GSEA we use in here examples requires a gene list ordered by some statistic and input gene sets. +When you use previously computed gene-level statistics with GSEA, it is called GSEA pre-ranked. + +## DESeq2 results + +```{r read_in_markers, live = TRUE} +deseq_df <- readr::read_tsv(input_file) +``` + +```{r deseq_head} +head(deseq_df) +``` + +This data frame uses Ensembl gene identifiers. +We'll need to make sure our gene sets use the same identifiers. +Let's take a look at the first few rows of the data frame that contains the hallmark genesets. + +```{r hallmark_head, live = TRUE} +head(hs_hallmarks_df) +``` + +We can see that the gene sets from `msigdbr` have Ensembl gene identifiers associated with them, so we don't need to do any conversion. +However, we'll need to pass the correct column to the function that runs GSEA. + +One good thing about Ensembl gene identifiers is that they are less likely to be duplicated than, for example, gene symbols. +(Multiple Ensembl gene identifiers can map to the same symbol.) + +The GSEA approach requires on discriminating between genes that are in a gene set and those that are not. +Practically speaking, gene sets are just collections of gene identifiers! +When the function we use for GSEA pre-ranked gets a list with duplicated gene identifiers, it can produce unexpected results. +So, let's check for duplicates in the data frame of DESeq2 results. + +```{r check_duplicates, live = TRUE} +any(duplicated(deseq_df$ensembl_id)) +``` + +There are no duplicates for us to worry about! + +### Pre-ranked list + +The `GSEA()` function takes a pre-ranked (sorted) named vector of statistics, where the names in the vector are gene identifiers. +This is step 1 -- gene-level statistics. + +```{r lfc_vector} +lfc_vector <- deseq_df |> + # Extract a vector of `log2FoldChange` named by `ensembl_id` + dplyr::pull(log2FoldChange, name = ensembl_id) +lfc_vector <- sort(lfc_vector, decreasing = TRUE) +``` + +Let's look at the top ranked values. + +```{r head_lfc, live = TRUE} +# Look at first entries of the log fold change vector +head(lfc_vector) +``` + +And the bottom of the list. + +```{r tail_lfc, live = TRUE} +# Look at the last entries of the log fold change vector +tail(lfc_vector) +``` + +## Run GSEA + +Now for the analysis! + +We can use the `GSEA()` function to perform GSEA with any generic set of gene sets, but there are several functions for using specific, commonly used gene sets (e.g., `gseKEGG()`). + +```{r run_gsea} +gsea_results <- GSEA(geneList = lfc_vector, # ordered ranked gene list + minGSSize = 25, # minimum gene set size + maxGSSize = 500, # maximum gene set set + pvalueCutoff = 0.05, + pAdjustMethod = "BH", # correction for multiple hypothesis testing + TERM2GENE = dplyr::select(hs_hallmarks_df, + gs_name, + ensembl_gene)) # pass the correct identifier column +``` +Let's take a look at the GSEA results. + +```{r view_gsea, live = TRUE, eval = FALSE} +View(gsea_results@result |> + dplyr::arrange(dplyr::desc(NES)) +) +``` + +Normalized enrichment scores (NES) are enrichment scores that are scaled to make gene sets that contain different number of genes comparable. + +Pathways with significant, highly positive NES are enriched in ERMS myoblasts, whereas pathways with significant, highly negative NES are enriched in ARMS myoblasts. + +Let's write these results to file. + +```{r write_gsea} +gsea_results@result |> readr::write_tsv(output_file) +``` + +### Visualizing GSEA results + +We can visualize GSEA results for individual pathways or gene sets using `enrichplot::gseaplot()`. +Let's take a look at 3 different pathways -- one with a highly positive NES, one with a highly negative NES, and one that was not a significant result -- to get more insight into how ES are calculated. + +#### Highly Positive NES + +Let's take look at a pathway with a highly positive NES (`HALLMARK_MYC_TARGETS_V2`) using a GSEA plot. + +```{r highly_pos} +enrichplot::gseaplot(gsea_results, + geneSetID = "HALLMARK_MYC_TARGETS_V2", + title = "HALLMARK_MYC_TARGETS_V2", + color.line = "#0066FF") +``` + +Notice how the genes that are in the gene set, indicated by the black bars, tend to be on the left side of the graph indicating that they have positive gene-level scores. + +#### Highly Negative NES + +The gene set `HALLMARK_MYOGENESIS` had a highly negative NES. + +```{r highly_neg} +enrichplot::gseaplot(gsea_results, + geneSetID = "HALLMARK_MYOGENESIS", + title = "HALLMARK_MYOGENESIS", + color.line = "#0066FF") +``` + +This gene set shows the opposite pattern -- genes in the pathway tend to be on the right side of the graph. + +#### A non-significant result + +The `@results` slot will only show gene sets that pass the `pvalueCutoff` threshold we supplied to `GSEA()`, but we can plot any gene set so long as we know its name. +Let's look at `HALLMARK_P53_PATHWAY`, which was not in the results we viewed earlier. + +```{r p53, live = TRUE} +enrichplot::gseaplot(gsea_results, + geneSetID = "HALLMARK_P53_PATHWAY", + title = "HALLMARK_P53_PATHWAY", + color.line = "#0066FF") +``` + +Genes in the pathway are distributed more evenly throughout the ranked list, resulting in a more "middling" score. + +*Note: The plots returned by `enrichplot::gseaplot` are ggplots, so we could use `ggplot2::ggsave()` to save them to file if we wanted to.* + +## Session Info + +```{r session_info} +sessionInfo() +``` diff --git a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html new file mode 100644 index 00000000..618e4c60 --- /dev/null +++ b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html @@ -0,0 +1,3501 @@ + + + + + + + + + + + + + + + +Pathway analysis: Gene Set Enrichment Analysis (GSEA) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+
+
+
+
+ +
+ + + + + + + + +
+

Objectives

+

This notebook will demonstrate how to:

+
    +
  • Prepare tabular data of gene-level statistics for use with Gene Set +Enrichment Analysis (GSEA)
  • +
  • Access Molecular +Signatures Database gene set collections via the +msigdbr package
  • +
  • Perform GSEA with the clusterProfiler package
  • +
  • Visualize GSEA results with the enrichplot package
  • +
+
+

In this notebook, we’ll analyze the differential expression results +from the last notebook.

+

GSEA is a functional class scoring (FCS) approach to pathway analysis +that was first introduced in Subramanian et +al. (2005). The rationale behind FCS approaches is that small +changes in individual genes that participate in the same biological +process or pathway can be significant and of biological interest.

+

There are 3 general steps in FCS methods (Khatri et +al. 2012):

+
    +
  1. Calculate a gene-level statistic (here, we’ll use the summary log +fold changes in our DESeq2 results)
  2. +
  3. Gene-level statistics are aggregated into a pathway-level +statistic
  4. +
  5. Assess the statistical significance of the pathway-level +statistic
  6. +
+
+

Other resources

+ +
+
+
+

Set up

+
+

Libraries

+ + + +
# Package to run GSEA
+ + +
Warning message:
+replacing previous import ‘S4Arrays::makeNindexFromArrayViewport’ by ‘DelayedArray::makeNindexFromArrayViewport’ when loading ‘SummarizedExperiment’ 
+ + +
library(clusterProfiler)
+ + +
clusterProfiler v4.12.0  For help: https://yulab-smu.top/biomedical-knowledge-mining-book/
+
+If you use clusterProfiler in published research, please cite:
+T Wu, E Hu, S Xu, M Chen, P Guo, Z Dai, T Feng, L Zhou, W Tang, L Zhan, X Fu, S Liu, X Bo, and G Yu. clusterProfiler 4.0: A universal enrichment tool for interpreting omics data. The Innovation. 2021, 2(3):100141
+
+Attaching package: ‘clusterProfiler’
+
+The following object is masked from ‘package:stats’:
+
+    filter
+ + +
# Package that contains the MSigDB gene sets in tidy format
+library(msigdbr)
+ + + +
+
+

Directories and Files

+
+

Directories

+ + + +
# We'll use the marker genes as GSEA input
+rms_analysis_dir <- file.path("analysis", "rms")
+
+# We'll create a directory to specifically hold the pathway results if it doesn't
+# exist yet
+results_dir <- file.path(rms_analysis_dir, "pathway-analysis")
+fs::dir_create(results_dir)
+ + + +
+
+

Input files

+ + + +
input_file <- file.path(rms_analysis_dir,
+                        "deseq",
+                        "rms_myoblast_deseq_results.tsv")
+ + + +
+
+

Output files

+

We’ll save our table of GSEA results as a TSV.

+ + + +
output_file <- file.path(results_dir,
+                         "rms_myoblast_gsea_results.tsv")
+ + + +
+
+
+
+

Gene sets

+

We will use gene sets from the Molecular +Signatures Database (MSigDB) from the Broad Institute (Subramanian, Tamayo +et al. 2005). The msigdbr +package contains MSigDB datasets already in the tidy format required by +clusterProfiler and supports multiple organisms.

+

Let’s take a look at what organisms the package supports.

+ + + +
msigdbr_species()
+ + +
+ +
+ + + +

MSigDB contains 8 different gene set collections.

+
H: hallmark gene sets
+C1: positional gene sets
+C2: curated gene sets
+C3: motif gene sets
+C4: computational gene sets
+C5: GO gene sets
+C6: oncogenic signatures
+C7: immunologic signatures
+

We’ll use the Hallmark collection for GSEA. Here’s an excerpt of the +collection +description:

+
+

Hallmark gene sets summarize and represent specific well-defined +biological states or processes and display coherent expression. These +gene sets were generated by a computational methodology based on +identifying gene set overlaps and retaining genes that display +coordinate expression. The hallmarks reduce noise and redundancy and +provide a better delineated biological space for GSEA.

+
+

Notably, there are only 50 gene sets included in this collection. The +fewer gene sets we test, the lower our multiple hypothesis testing +burden.

+

We can retrieve only the Hallmark gene sets by specifying +category = "H" to the msigdbr() function.

+ + + +
hs_hallmarks_df <- msigdbr(species = "Homo sapiens",
+                           category = "H")
+ + + +
+
+

Gene Set Enrichment Analysis

+

Adapted from refine.bio +examples

+

+

Figure 1. Subramanian et +al. (2005).

+

GSEA calculates a pathway-level metric, called an enrichment score +(sometimes abbreviated as ES), by ranking genes by a gene-level +statistic. This score reflects whether or not a gene set or pathway is +over-represented at the top or bottom of the gene rankings (Subramanian et +al. 2005; Yu)

+

Specifically, all genes are ranked from most positive to most +negative based on their statistic and a running sum is calculated: +Starting with the most highly ranked genes, the running sum increases +for each gene in the pathway and decreases for each gene not in the +pathway. The enrichment score for a pathway is the running sum’s maximum +deviation from zero. GSEA also assesses statistical significance of the +scores for each pathway through permutation testing. As a result, each +input pathway will have a p-value associated with it that is then +corrected for multiple hypothesis testing (Subramanian et +al. 2005; Yu).

+

The implementation of GSEA we use in here examples requires a gene +list ordered by some statistic and input gene sets. When you use +previously computed gene-level statistics with GSEA, it is called GSEA +pre-ranked.

+
+
+

DESeq2 results

+ + + +
deseq_df <- readr::read_tsv(input_file)
+ + +
Rows: 60319 Columns: 27
+── Column specification ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
+Delimiter: "\t"
+chr  (2): ensembl_id, gene_symbol
+dbl (25): baseMean, log2FoldChange, lfcSE, pvalue, padj, SCPCL000478.mean, SCPCL000478.detected, SCPCL000479.mean, SCPCL000479.detected, SCPCL000480.mean, SCPCL000480.detected, SCPCL000481.m...
+
+ℹ Use `spec()` to retrieve the full column specification for this data.
+ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
+ + + + + + +
head(deseq_df)
+ + +
+ +
+ + + +

This data frame uses Ensembl gene identifiers. We’ll need to make +sure our gene sets use the same identifiers. Let’s take a look at the +first few rows of the data frame that contains the hallmark +genesets.

+ + + +
head(hs_hallmarks_df)
+ + +
+ +
+ + + +

We can see that the gene sets from msigdbr have Ensembl +gene identifiers associated with them, so we don’t need to do any +conversion. However, we’ll need to pass the correct column to the +function that runs GSEA.

+

One good thing about Ensembl gene identifiers is that they are less +likely to be duplicated than, for example, gene symbols. (Multiple +Ensembl gene identifiers can map to the same symbol.)

+

The GSEA approach requires on discriminating between genes that are +in a gene set and those that are not. Practically speaking, gene sets +are just collections of gene identifiers! When the function we use for +GSEA pre-ranked gets a list with duplicated gene identifiers, it can +produce unexpected results. So, let’s check for duplicates in the data +frame of DESeq2 results.

+ + + +
any(duplicated(deseq_df$ensembl_id))
+ + +
[1] FALSE
+ + + +

There are no duplicates for us to worry about!

+
+

Pre-ranked list

+

The GSEA() function takes a pre-ranked (sorted) named +vector of statistics, where the names in the vector are gene +identifiers. This is step 1 – gene-level statistics.

+ + + +
lfc_vector <- deseq_df |>
+  # Extract a vector of `log2FoldChange` named by `ensembl_id`
+  dplyr::pull(log2FoldChange, name = ensembl_id)
+lfc_vector <- sort(lfc_vector, decreasing = TRUE)
+ + + +

Let’s look at the top ranked values.

+ + + +
# Look at first entries of the log fold change vector
+head(lfc_vector)
+ + +
ENSG00000263366 ENSG00000223760 ENSG00000253377 ENSG00000265843 ENSG00000104722 ENSG00000228835 
+      11.364714       10.573752       10.476990       10.199449       10.019651        9.898818 
+ + + +

And the bottom of the list.

+ + + +
# Look at the last entries of the log fold change vector
+tail(lfc_vector)
+ + +
ENSG00000269186 ENSG00000268388 ENSG00000165606 ENSG00000285640 ENSG00000118432 ENSG00000184221 
+      -10.93216       -11.35119       -11.36925       -11.90034       -11.92082       -12.12577 
+ + + +
+
+
+

Run GSEA

+

Now for the analysis!

+

We can use the GSEA() function to perform GSEA with any +generic set of gene sets, but there are several functions for using +specific, commonly used gene sets (e.g., gseKEGG()).

+ + + +
gsea_results <- GSEA(geneList = lfc_vector,  # ordered ranked gene list
+                     minGSSize = 25,  # minimum gene set size
+                     maxGSSize = 500,  # maximum gene set set
+                     pvalueCutoff = 0.05,
+                     pAdjustMethod = "BH",  # correction for multiple hypothesis testing
+                     TERM2GENE = dplyr::select(hs_hallmarks_df,
+                                               gs_name,
+                                               ensembl_gene))  # pass the correct identifier column
+ + +
using 'fgsea' for GSEA analysis, please cite Korotkevich et al (2019).
+
+preparing geneSet collections...
+GSEA analysis...
+Warning in preparePathwaysAndStats(pathways, stats, minSize, maxSize, gseaParam,  :
+  There are ties in the preranked stats (20.32% of the list).
+The order of those tied genes will be arbitrary, which may produce unexpected results.
+leading edge analysis...
+done...
+ + + +

Let’s take a look at the GSEA results.

+ + + +
View(gsea_results@result |>
+       dplyr::arrange(dplyr::desc(NES))
+)
+ + + +

Normalized enrichment scores (NES) are enrichment scores that are +scaled to make gene sets that contain different number of genes +comparable.

+

Pathways with significant, highly positive NES are enriched in ERMS +myoblasts, whereas pathways with significant, highly negative NES are +enriched in ARMS myoblasts.

+

Let’s write these results to file.

+ + + +
gsea_results@result |> readr::write_tsv(output_file)
+ + + +
+

Visualizing GSEA results

+

We can visualize GSEA results for individual pathways or gene sets +using enrichplot::gseaplot(). Let’s take a look at 3 +different pathways – one with a highly positive NES, one with a highly +negative NES, and one that was not a significant result – to get more +insight into how ES are calculated.

+
+

Highly Positive NES

+

Let’s take look at a pathway with a highly positive NES +(HALLMARK_MYC_TARGETS_V2) using a GSEA plot.

+ + + +
enrichplot::gseaplot(gsea_results,
+                     geneSetID = "HALLMARK_MYC_TARGETS_V2",
+                     title = "HALLMARK_MYC_TARGETS_V2",
+                     color.line = "#0066FF")
+ + +

+ + + +

Notice how the genes that are in the gene set, indicated by the black +bars, tend to be on the left side of the graph indicating that they have +positive gene-level scores.

+
+
+

Highly Negative NES

+

The gene set HALLMARK_MYOGENESIS had a highly negative +NES.

+ + + +
enrichplot::gseaplot(gsea_results,
+                     geneSetID = "HALLMARK_MYOGENESIS",
+                     title = "HALLMARK_MYOGENESIS",
+                     color.line = "#0066FF")
+ + +

+ + + +

This gene set shows the opposite pattern – genes in the pathway tend +to be on the right side of the graph.

+
+
+

A non-significant result

+

The @results slot will only show gene sets that pass the +pvalueCutoff threshold we supplied to GSEA(), +but we can plot any gene set so long as we know its name. Let’s look at +HALLMARK_P53_PATHWAY, which was not in the results we +viewed earlier.

+ + + +
enrichplot::gseaplot(gsea_results,
+                     geneSetID = "HALLMARK_P53_PATHWAY",
+                     title = "HALLMARK_P53_PATHWAY",
+                     color.line = "#0066FF")
+ + +

+ + + +

Genes in the pathway are distributed more evenly throughout the +ranked list, resulting in a more “middling” score.

+

Note: The plots returned by enrichplot::gseaplot are +ggplots, so we could use ggplot2::ggsave() to save them to +file if we wanted to.

+
+
+
+
+

Session Info

+ + + +
sessionInfo()
+ + +
R version 4.4.0 (2024-04-24)
+Platform: x86_64-pc-linux-gnu
+Running under: Ubuntu 22.04.4 LTS
+
+Matrix products: default
+BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
+LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0
+
+locale:
+ [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C             
+ [9] LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
+
+time zone: Etc/UTC
+tzcode source: system (glibc)
+
+attached base packages:
+[1] stats     graphics  grDevices datasets  utils     methods   base     
+
+other attached packages:
+[1] msigdbr_7.5.1          clusterProfiler_4.12.0
+
+loaded via a namespace (and not attached):
+  [1] RColorBrewer_1.1-3          rstudioapi_0.16.0           jsonlite_1.8.8              magrittr_2.0.3              estimability_1.5            ggbeeswarm_0.7.2           
+  [7] farver_2.1.1                rmarkdown_2.26              fs_1.6.4                    zlibbioc_1.50.0             vctrs_0.6.5                 memoise_2.0.1              
+ [13] DelayedMatrixStats_1.26.0   ggtree_3.12.0               htmltools_0.5.8.1           S4Arrays_1.4.0              BiocNeighbors_1.22.0        SparseArray_1.4.0          
+ [19] gridGraphics_0.5-1          plyr_1.8.9                  emmeans_1.10.1              cachem_1.0.8                igraph_2.0.3                lifecycle_1.0.4            
+ [25] pkgconfig_2.0.3             gson_0.1.0                  rsvd_1.0.5                  Matrix_1.7-0                R6_2.5.1                    fastmap_1.1.1              
+ [31] GenomeInfoDbData_1.2.12     MatrixGenerics_1.16.0       digest_0.6.35               aplot_0.2.2                 enrichplot_1.24.0           colorspace_2.1-0           
+ [37] patchwork_1.2.0             AnnotationDbi_1.66.0        S4Vectors_0.42.0            scater_1.32.0               qusage_2.38.0               irlba_2.3.5.1              
+ [43] GenomicRanges_1.56.0        RSQLite_2.3.6               beachmat_2.20.0             labeling_0.4.3              fansi_1.0.6                 httr_1.4.7                 
+ [49] polyclip_1.10-6             abind_1.4-5                 compiler_4.4.0              bit64_4.0.5                 withr_3.0.0                 BiocParallel_1.38.0        
+ [55] viridis_0.6.5               DBI_1.2.2                   ggforce_0.4.2               MASS_7.3-60.2               DelayedArray_0.30.0         HDO.db_0.99.1              
+ [61] tools_4.4.0                 vipor_0.4.7                 beeswarm_0.4.0              ape_5.8                     scatterpie_0.2.2            fftw_1.0-8                 
+ [67] glue_1.7.0                  nlme_3.1-164                GOSemSim_2.30.0             grid_4.4.0                  shadowtext_0.1.3            reshape2_1.4.4             
+ [73] fgsea_1.30.0                generics_0.1.3              gtable_0.3.5                tzdb_0.4.0                  tidyr_1.3.1                 data.table_1.15.4          
+ [79] hms_1.1.3                   ScaledMatrix_1.12.0         BiocSingular_1.20.0         tidygraph_1.3.1             utf8_1.2.4                  XVector_0.44.0             
+ [85] BiocGenerics_0.50.0         ggrepel_0.9.5               pillar_1.9.0                stringr_1.5.1               vroom_1.6.5                 babelgene_22.9             
+ [91] limma_3.60.0                yulab.utils_0.1.4           splines_4.4.0               dplyr_1.1.4                 tweenr_2.0.3                treeio_1.28.0              
+ [97] lattice_0.22-6              renv_1.0.7                  bit_4.0.5                   tidyselect_1.2.1            GO.db_3.19.1                SingleCellExperiment_1.26.0
+[103] Biostrings_2.72.0           scuttle_1.14.0              knitr_1.46                  gridExtra_2.3               IRanges_2.38.0              SummarizedExperiment_1.34.0
+[109] stats4_4.4.0                xfun_0.43                   graphlayouts_1.1.1          Biobase_2.64.0              statmod_1.5.0               matrixStats_1.3.0          
+[115] stringi_1.8.3               UCSC.utils_1.0.0            lazyeval_0.2.2              ggfun_0.1.4                 yaml_2.3.8                  evaluate_0.23              
+[121] codetools_0.2-20            ggraph_2.2.1                tibble_3.2.1                qvalue_2.36.0               BiocManager_1.30.22         ggplotify_0.1.2            
+[127] cli_3.6.2                   xtable_1.8-4                munsell_0.5.1               Rcpp_1.0.12                 GenomeInfoDb_1.40.0         coda_0.19-4.1              
+[133] png_0.1-8                   parallel_4.4.0              ggplot2_3.5.1               readr_2.1.5                 blob_1.2.4                  DOSE_3.30.0                
+[139] sparseMatrixStats_1.16.0    mvtnorm_1.2-4               viridisLite_0.4.2           tidytree_0.4.6              scales_1.3.0                purrr_1.0.2                
+[145] crayon_1.5.2                rlang_1.1.3                 cowplot_1.1.3               fastmatch_1.1-4             KEGGREST_1.44.0            
+ + +
+ +
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcyAoR1NFQSkiCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjQKLS0tCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOgoKLSBQcmVwYXJlIHRhYnVsYXIgZGF0YSBvZiBnZW5lLWxldmVsIHN0YXRpc3RpY3MgZm9yIHVzZSB3aXRoIEdlbmUgU2V0IEVucmljaG1lbnQgQW5hbHlzaXMgKEdTRUEpCi0gQWNjZXNzIFtNb2xlY3VsYXIgU2lnbmF0dXJlcyBEYXRhYmFzZSBnZW5lIHNldCBjb2xsZWN0aW9uc10oaHR0cHM6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2NvbGxlY3Rpb25zLmpzcCkgdmlhIHRoZSBgbXNpZ2RicmAgcGFja2FnZQotIFBlcmZvcm0gR1NFQSB3aXRoIHRoZSBgY2x1c3RlclByb2ZpbGVyYCBwYWNrYWdlCi0gVmlzdWFsaXplIEdTRUEgcmVzdWx0cyB3aXRoIHRoZSBgZW5yaWNocGxvdGAgcGFja2FnZQoKLS0tCgpJbiB0aGlzIG5vdGVib29rLCB3ZSdsbCBhbmFseXplIHRoZSBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiByZXN1bHRzIGZyb20gdGhlIGxhc3Qgbm90ZWJvb2suCgpHU0VBIGlzIGEgZnVuY3Rpb25hbCBjbGFzcyBzY29yaW5nIChGQ1MpIGFwcHJvYWNoIHRvIHBhdGh3YXkgYW5hbHlzaXMgdGhhdCB3YXMgZmlyc3QgaW50cm9kdWNlZCBpbiBbU3VicmFtYW5pYW4gX2V0IGFsLl8gKDIwMDUpXShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpLgpUaGUgcmF0aW9uYWxlIGJlaGluZCBGQ1MgYXBwcm9hY2hlcyBpcyB0aGF0IHNtYWxsIGNoYW5nZXMgaW4gaW5kaXZpZHVhbCBnZW5lcyB0aGF0IHBhcnRpY2lwYXRlIGluIHRoZSBzYW1lIGJpb2xvZ2ljYWwgcHJvY2VzcyBvciBwYXRod2F5IGNhbiBiZSBzaWduaWZpY2FudCBhbmQgb2YgYmlvbG9naWNhbCBpbnRlcmVzdC4KClRoZXJlIGFyZSAzIGdlbmVyYWwgc3RlcHMgaW4gRkNTIG1ldGhvZHMgKFtLaGF0cmkgX2V0IGFsLl8gMjAxMl0oIGh0dHBzOi8vZG9pLm9yZy8xMC4xMzcxL2pvdXJuYWwucGNiaS4xMDAyMzc1KSk6CgoxLiBDYWxjdWxhdGUgYSBnZW5lLWxldmVsIHN0YXRpc3RpYyAoaGVyZSwgd2UnbGwgdXNlIHRoZSBzdW1tYXJ5IGxvZyBmb2xkIGNoYW5nZXMgaW4gb3VyIERFU2VxMiByZXN1bHRzKQoyLiBHZW5lLWxldmVsIHN0YXRpc3RpY3MgYXJlIGFnZ3JlZ2F0ZWQgaW50byBhIHBhdGh3YXktbGV2ZWwgc3RhdGlzdGljCjMuIEFzc2VzcyB0aGUgc3RhdGlzdGljYWwgc2lnbmlmaWNhbmNlIG9mIHRoZSBwYXRod2F5LWxldmVsIHN0YXRpc3RpYwoKIyMjIyBPdGhlciByZXNvdXJjZXMKCiogRm9yIGFub3RoZXIgZXhhbXBsZSB1c2luZyBgY2x1c3RlclByb2ZpbGVyYCBmb3IgR1NFQSwgc2VlIFtfSW50cm8gdG8gREdFOiBGdW5jdGlvbmFsIEFuYWx5c2lzLl8gZnJvbSBIYXJ2YXJkIENoYW4gQmlvaW5mb3JtYXRpY3MgQ29yZSBUcmFpbmluZy5dKGh0dHBzOi8vaGJjdHJhaW5pbmcuZ2l0aHViLmlvL0RHRV93b3Jrc2hvcC9sZXNzb25zLzA5X2Z1bmN0aW9uYWxfYW5hbHlzaXMuaHRtbCkKKiBUaGUgd2F5IHdlJ2xsIHVzZSBgY2x1c3RlclByb2ZpbGVyYCBoZXJlIHVzZXMgYGZnc2VhYCAoRmFzdCBHZW5lIFNldCBFbnJpY2htZW50IEFuYWx5c2lzKSB1bmRlciB0aGUgaG9vZC4KWW91IGNhbiByZWFkIG1vcmUgYWJvdXQgZmdzZWEgaW4gW0tvcm90a2V2aWNoIF9ldCBhbC5fICgyMDIxKV0oaHR0cHM6Ly9kb2kub3JnLzEwLjExMDEvMDYwMDEyKS4KKiBbcmVmaW5lLmJpbyBleGFtcGxlcyBHZW5lIHNldCBlbnJpY2htZW50IGFuYWx5c2lzIC0gUk5BLXNlcV0oaHR0cHM6Ly9hbGV4c2xlbW9uYWRlLmdpdGh1Yi5pby9yZWZpbmViaW8tZXhhbXBsZXMvMDMtcm5hc2VxL3BhdGh3YXktYW5hbHlzaXNfcm5hc2VxXzAyX2dzZWEuaHRtbCkgZnJvbSB3aGljaCB0aGlzIG1hdGVyaWFsIGhhcyBiZWVuIGFkYXB0ZWQuCgojIyBTZXQgdXAKCiMjIyBMaWJyYXJpZXMKCmBgYHtyIGxpYnJhcmllc30KIyBQYWNrYWdlIHRvIHJ1biBHU0VBCmxpYnJhcnkoY2x1c3RlclByb2ZpbGVyKQojIFBhY2thZ2UgdGhhdCBjb250YWlucyB0aGUgTVNpZ0RCIGdlbmUgc2V0cyBpbiB0aWR5IGZvcm1hdApsaWJyYXJ5KG1zaWdkYnIpCmBgYAoKIyMjIERpcmVjdG9yaWVzIGFuZCBGaWxlcwoKIyMjIyBEaXJlY3RvcmllcwoKYGBge3IgY3JlYXRlX2RpciwgbGl2ZSA9IFRSVUV9CiMgV2UnbGwgdXNlIHRoZSBtYXJrZXIgZ2VuZXMgYXMgR1NFQSBpbnB1dApybXNfYW5hbHlzaXNfZGlyIDwtIGZpbGUucGF0aCgiYW5hbHlzaXMiLCAicm1zIikKCiMgV2UnbGwgY3JlYXRlIGEgZGlyZWN0b3J5IHRvIHNwZWNpZmljYWxseSBob2xkIHRoZSBwYXRod2F5IHJlc3VsdHMgaWYgaXQgZG9lc24ndAojIGV4aXN0IHlldApyZXN1bHRzX2RpciA8LSBmaWxlLnBhdGgocm1zX2FuYWx5c2lzX2RpciwgInBhdGh3YXktYW5hbHlzaXMiKQpmczo6ZGlyX2NyZWF0ZShyZXN1bHRzX2RpcikKYGBgCgojIyMjIElucHV0IGZpbGVzCgoKYGBge3IgaW5wdXRfZmlsZXN9CmlucHV0X2ZpbGUgPC0gZmlsZS5wYXRoKHJtc19hbmFseXNpc19kaXIsCiAgICAgICAgICAgICAgICAgICAgICAgICJkZXNlcSIsCiAgICAgICAgICAgICAgICAgICAgICAgICJybXNfbXlvYmxhc3RfZGVzZXFfcmVzdWx0cy50c3YiKQpgYGAKCiMjIyMgT3V0cHV0IGZpbGVzCgpXZSdsbCBzYXZlIG91ciB0YWJsZSBvZiBHU0VBIHJlc3VsdHMgYXMgYSBUU1YuCgpgYGB7ciBvdXRwdXRfZmlsZXN9Cm91dHB1dF9maWxlIDwtIGZpbGUucGF0aChyZXN1bHRzX2RpciwKICAgICAgICAgICAgICAgICAgICAgICAgICJybXNfbXlvYmxhc3RfZ3NlYV9yZXN1bHRzLnRzdiIpCmBgYAoKIyMgR2VuZSBzZXRzCgpXZSB3aWxsIHVzZSBnZW5lIHNldHMgZnJvbSB0aGUgW01vbGVjdWxhciBTaWduYXR1cmVzIERhdGFiYXNlIChNU2lnREIpXShodHRwczovL3d3dy5nc2VhLW1zaWdkYi5vcmcvZ3NlYS9tc2lnZGIvaW5kZXguanNwKSBmcm9tIHRoZSBCcm9hZCBJbnN0aXR1dGUgKFtTdWJyYW1hbmlhbiwgVGFtYXlvICpldCBhbC4qIDIwMDVdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMDczL3BuYXMuMDUwNjU4MDEwMikpLgpUaGUgW2Btc2lnZGJyYF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL21zaWdkYnIvaW5kZXguaHRtbCkgcGFja2FnZSBjb250YWlucyBNU2lnREIgZGF0YXNldHMgYWxyZWFkeSBpbiB0aGUgdGlkeSBmb3JtYXQgcmVxdWlyZWQgYnkgYGNsdXN0ZXJQcm9maWxlcmAgYW5kIHN1cHBvcnRzIG11bHRpcGxlIG9yZ2FuaXNtcy4KCkxldCdzIHRha2UgYSBsb29rIGF0IHdoYXQgb3JnYW5pc21zIHRoZSBwYWNrYWdlIHN1cHBvcnRzLgoKYGBge3Igc2hvd19zcGVjaWVzfQptc2lnZGJyX3NwZWNpZXMoKQpgYGAKCk1TaWdEQiBjb250YWlucyA4IGRpZmZlcmVudCBnZW5lIHNldCBjb2xsZWN0aW9ucy4KCiAgICBIOiBoYWxsbWFyayBnZW5lIHNldHMKICAgIEMxOiBwb3NpdGlvbmFsIGdlbmUgc2V0cwogICAgQzI6IGN1cmF0ZWQgZ2VuZSBzZXRzCiAgICBDMzogbW90aWYgZ2VuZSBzZXRzCiAgICBDNDogY29tcHV0YXRpb25hbCBnZW5lIHNldHMKICAgIEM1OiBHTyBnZW5lIHNldHMKICAgIEM2OiBvbmNvZ2VuaWMgc2lnbmF0dXJlcwogICAgQzc6IGltbXVub2xvZ2ljIHNpZ25hdHVyZXMKCldlJ2xsIHVzZSB0aGUgSGFsbG1hcmsgY29sbGVjdGlvbiBmb3IgR1NFQS4KSGVyZSdzIGFuIGV4Y2VycHQgb2YgdGhlIFtjb2xsZWN0aW9uIGRlc2NyaXB0aW9uXShodHRwczovL3d3dy5nc2VhLW1zaWdkYi5vcmcvZ3NlYS9tc2lnZGIvY29sbGVjdGlvbl9kZXRhaWxzLmpzcCNIKToKCj4gSGFsbG1hcmsgZ2VuZSBzZXRzIHN1bW1hcml6ZSBhbmQgcmVwcmVzZW50IHNwZWNpZmljIHdlbGwtZGVmaW5lZCBiaW9sb2dpY2FsIHN0YXRlcyBvciBwcm9jZXNzZXMgYW5kIGRpc3BsYXkgY29oZXJlbnQgZXhwcmVzc2lvbi4gVGhlc2UgZ2VuZSBzZXRzIHdlcmUgZ2VuZXJhdGVkIGJ5IGEgY29tcHV0YXRpb25hbCBtZXRob2RvbG9neSBiYXNlZCBvbiBpZGVudGlmeWluZyBnZW5lIHNldCBvdmVybGFwcyBhbmQgcmV0YWluaW5nIGdlbmVzIHRoYXQgZGlzcGxheSBjb29yZGluYXRlIGV4cHJlc3Npb24uIFRoZSBoYWxsbWFya3MgcmVkdWNlIG5vaXNlIGFuZCByZWR1bmRhbmN5IGFuZCBwcm92aWRlIGEgYmV0dGVyIGRlbGluZWF0ZWQgYmlvbG9naWNhbCBzcGFjZSBmb3IgR1NFQS4KCk5vdGFibHksIHRoZXJlIGFyZSBvbmx5IDUwIGdlbmUgc2V0cyBpbmNsdWRlZCBpbiB0aGlzIGNvbGxlY3Rpb24uClRoZSBmZXdlciBnZW5lIHNldHMgd2UgdGVzdCwgdGhlIGxvd2VyIG91ciBtdWx0aXBsZSBoeXBvdGhlc2lzIHRlc3RpbmcgYnVyZGVuLgoKV2UgY2FuIHJldHJpZXZlIG9ubHkgdGhlIEhhbGxtYXJrIGdlbmUgc2V0cyBieSBzcGVjaWZ5aW5nIGBjYXRlZ29yeSA9ICJIImAgdG8gdGhlIGBtc2lnZGJyKClgIGZ1bmN0aW9uLgoKYGBge3IgaW1tdW5vbG9naWNfc2V0cywgbGl2ZSA9IFRSVUV9CmhzX2hhbGxtYXJrc19kZiA8LSBtc2lnZGJyKHNwZWNpZXMgPSAiSG9tbyBzYXBpZW5zIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgY2F0ZWdvcnkgPSAiSCIpCmBgYAoKIyMgR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcwoKX0FkYXB0ZWQgZnJvbSBbcmVmaW5lLmJpbyBleGFtcGxlc10oaHR0cHM6Ly9naXRodWIuY29tL0FsZXhzTGVtb25hZGUvcmVmaW5lYmlvLWV4YW1wbGVzL2Jsb2IvMzNjZGVmZjY2ZDU3ZjlmZThlZTRmY2I1MTU2YWVhNGFjMmRjZTA3Zi8wMy1ybmFzZXEvcGF0aHdheS1hbmFseXNpc19ybmFzZXFfMDJfZ3NlYS5SbWQpXwoKIVtdKGRpYWdyYW1zL3N1YnJhbWFuaWFuX2ZpZzEuanBnKQoKKipGaWd1cmUgMS4gW1N1YnJhbWFuaWFuIF9ldCBhbC5fICgyMDA1KV0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwNzMvcG5hcy4wNTA2NTgwMTAyKS4qKgoKR1NFQSBjYWxjdWxhdGVzIGEgcGF0aHdheS1sZXZlbCBtZXRyaWMsIGNhbGxlZCBhbiBlbnJpY2htZW50IHNjb3JlIChzb21ldGltZXMgYWJicmV2aWF0ZWQgYXMgRVMpLCBieSByYW5raW5nIGdlbmVzIGJ5IGEgZ2VuZS1sZXZlbCBzdGF0aXN0aWMuClRoaXMgc2NvcmUgcmVmbGVjdHMgd2hldGhlciBvciBub3QgYSBnZW5lIHNldCBvciBwYXRod2F5IGlzIG92ZXItcmVwcmVzZW50ZWQgYXQgdGhlIHRvcCBvciBib3R0b20gb2YgdGhlIGdlbmUgcmFua2luZ3MgKFtTdWJyYW1hbmlhbiBfZXQgYWwuXyAyMDA1XShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpOyBbWXVdKGh0dHA6Ly95dWxhYi1zbXUudG9wL2NsdXN0ZXJQcm9maWxlci1ib29rL2NoYXB0ZXIyLmh0bWwjZ2VuZS1zZXQtZW5yaWNobWVudC1hbmFseXNpcykpCgpTcGVjaWZpY2FsbHksIGFsbCBnZW5lcyBhcmUgcmFua2VkIGZyb20gbW9zdCBwb3NpdGl2ZSB0byBtb3N0IG5lZ2F0aXZlIGJhc2VkIG9uIHRoZWlyIHN0YXRpc3RpYyBhbmQgYSBydW5uaW5nIHN1bSBpcyBjYWxjdWxhdGVkOgpTdGFydGluZyB3aXRoIHRoZSBtb3N0IGhpZ2hseSByYW5rZWQgZ2VuZXMsIHRoZSBydW5uaW5nIHN1bSBpbmNyZWFzZXMgZm9yIGVhY2ggZ2VuZSBpbiB0aGUgcGF0aHdheSBhbmQgZGVjcmVhc2VzIGZvciBlYWNoIGdlbmUgbm90IGluIHRoZSBwYXRod2F5LgpUaGUgZW5yaWNobWVudCBzY29yZSBmb3IgYSBwYXRod2F5IGlzIHRoZSBydW5uaW5nIHN1bSdzIG1heGltdW0gZGV2aWF0aW9uIGZyb20gemVyby4KR1NFQSBhbHNvIGFzc2Vzc2VzIHN0YXRpc3RpY2FsIHNpZ25pZmljYW5jZSBvZiB0aGUgc2NvcmVzIGZvciBlYWNoIHBhdGh3YXkgdGhyb3VnaCBwZXJtdXRhdGlvbiB0ZXN0aW5nLgpBcyBhIHJlc3VsdCwgZWFjaCBpbnB1dCBwYXRod2F5IHdpbGwgaGF2ZSBhIHAtdmFsdWUgYXNzb2NpYXRlZCB3aXRoIGl0IHRoYXQgaXMgdGhlbiBjb3JyZWN0ZWQgZm9yIG11bHRpcGxlIGh5cG90aGVzaXMgdGVzdGluZyAoW1N1YnJhbWFuaWFuIF9ldCBhbC5fIDIwMDVdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMDczL3BuYXMuMDUwNjU4MDEwMik7IFtZdV0oaHR0cDovL3l1bGFiLXNtdS50b3AvY2x1c3RlclByb2ZpbGVyLWJvb2svY2hhcHRlcjIuaHRtbCNnZW5lLXNldC1lbnJpY2htZW50LWFuYWx5c2lzKSkuCgpUaGUgaW1wbGVtZW50YXRpb24gb2YgR1NFQSB3ZSB1c2UgaW4gaGVyZSBleGFtcGxlcyByZXF1aXJlcyBhIGdlbmUgbGlzdCBvcmRlcmVkIGJ5IHNvbWUgc3RhdGlzdGljIGFuZCBpbnB1dCBnZW5lIHNldHMuCldoZW4geW91IHVzZSBwcmV2aW91c2x5IGNvbXB1dGVkIGdlbmUtbGV2ZWwgc3RhdGlzdGljcyB3aXRoIEdTRUEsIGl0IGlzIGNhbGxlZCBHU0VBIHByZS1yYW5rZWQuCgojIyBERVNlcTIgcmVzdWx0cwoKYGBge3IgcmVhZF9pbl9tYXJrZXJzLCBsaXZlID0gVFJVRX0KZGVzZXFfZGYgPC0gcmVhZHI6OnJlYWRfdHN2KGlucHV0X2ZpbGUpCmBgYAoKYGBge3IgZGVzZXFfaGVhZH0KaGVhZChkZXNlcV9kZikKYGBgCgpUaGlzIGRhdGEgZnJhbWUgdXNlcyBFbnNlbWJsIGdlbmUgaWRlbnRpZmllcnMuCldlJ2xsIG5lZWQgdG8gbWFrZSBzdXJlIG91ciBnZW5lIHNldHMgdXNlIHRoZSBzYW1lIGlkZW50aWZpZXJzLgpMZXQncyB0YWtlIGEgbG9vayBhdCB0aGUgZmlyc3QgZmV3IHJvd3Mgb2YgdGhlIGRhdGEgZnJhbWUgdGhhdCBjb250YWlucyB0aGUgaGFsbG1hcmsgZ2VuZXNldHMuCgpgYGB7ciBoYWxsbWFya19oZWFkLCBsaXZlID0gVFJVRX0KaGVhZChoc19oYWxsbWFya3NfZGYpCmBgYAoKV2UgY2FuIHNlZSB0aGF0IHRoZSBnZW5lIHNldHMgZnJvbSBgbXNpZ2RicmAgaGF2ZSBFbnNlbWJsIGdlbmUgaWRlbnRpZmllcnMgYXNzb2NpYXRlZCB3aXRoIHRoZW0sIHNvIHdlIGRvbid0IG5lZWQgdG8gZG8gYW55IGNvbnZlcnNpb24uCkhvd2V2ZXIsIHdlJ2xsIG5lZWQgdG8gcGFzcyB0aGUgY29ycmVjdCBjb2x1bW4gdG8gdGhlIGZ1bmN0aW9uIHRoYXQgcnVucyBHU0VBLgoKT25lIGdvb2QgdGhpbmcgYWJvdXQgRW5zZW1ibCBnZW5lIGlkZW50aWZpZXJzIGlzIHRoYXQgdGhleSBhcmUgbGVzcyBsaWtlbHkgdG8gYmUgZHVwbGljYXRlZCB0aGFuLCBmb3IgZXhhbXBsZSwgZ2VuZSBzeW1ib2xzLgooTXVsdGlwbGUgRW5zZW1ibCBnZW5lIGlkZW50aWZpZXJzIGNhbiBtYXAgdG8gdGhlIHNhbWUgc3ltYm9sLikKClRoZSBHU0VBIGFwcHJvYWNoIHJlcXVpcmVzIG9uIGRpc2NyaW1pbmF0aW5nIGJldHdlZW4gZ2VuZXMgdGhhdCBhcmUgaW4gYSBnZW5lIHNldCBhbmQgdGhvc2UgdGhhdCBhcmUgbm90LgpQcmFjdGljYWxseSBzcGVha2luZywgZ2VuZSBzZXRzIGFyZSBqdXN0IGNvbGxlY3Rpb25zIG9mIGdlbmUgaWRlbnRpZmllcnMhCldoZW4gdGhlIGZ1bmN0aW9uIHdlIHVzZSBmb3IgR1NFQSBwcmUtcmFua2VkIGdldHMgYSBsaXN0IHdpdGggZHVwbGljYXRlZCBnZW5lIGlkZW50aWZpZXJzLCBpdCBjYW4gcHJvZHVjZSB1bmV4cGVjdGVkIHJlc3VsdHMuClNvLCBsZXQncyBjaGVjayBmb3IgZHVwbGljYXRlcyBpbiB0aGUgZGF0YSBmcmFtZSBvZiBERVNlcTIgcmVzdWx0cy4KCmBgYHtyIGNoZWNrX2R1cGxpY2F0ZXMsIGxpdmUgPSBUUlVFfQphbnkoZHVwbGljYXRlZChkZXNlcV9kZiRlbnNlbWJsX2lkKSkKYGBgCgpUaGVyZSBhcmUgbm8gZHVwbGljYXRlcyBmb3IgdXMgdG8gd29ycnkgYWJvdXQhCgojIyMgUHJlLXJhbmtlZCBsaXN0CgpUaGUgYEdTRUEoKWAgZnVuY3Rpb24gdGFrZXMgYSBwcmUtcmFua2VkIChzb3J0ZWQpIG5hbWVkIHZlY3RvciBvZiBzdGF0aXN0aWNzLCB3aGVyZSB0aGUgbmFtZXMgaW4gdGhlIHZlY3RvciBhcmUgZ2VuZSBpZGVudGlmaWVycy4KVGhpcyBpcyBzdGVwIDEgLS0gZ2VuZS1sZXZlbCBzdGF0aXN0aWNzLgoKYGBge3IgbGZjX3ZlY3Rvcn0KbGZjX3ZlY3RvciA8LSBkZXNlcV9kZiB8PgogICMgRXh0cmFjdCBhIHZlY3RvciBvZiBgbG9nMkZvbGRDaGFuZ2VgIG5hbWVkIGJ5IGBlbnNlbWJsX2lkYAogIGRwbHlyOjpwdWxsKGxvZzJGb2xkQ2hhbmdlLCBuYW1lID0gZW5zZW1ibF9pZCkKbGZjX3ZlY3RvciA8LSBzb3J0KGxmY192ZWN0b3IsIGRlY3JlYXNpbmcgPSBUUlVFKQpgYGAKCkxldCdzIGxvb2sgYXQgdGhlIHRvcCByYW5rZWQgdmFsdWVzLgoKYGBge3IgaGVhZF9sZmMsIGxpdmUgPSBUUlVFfQojIExvb2sgYXQgZmlyc3QgZW50cmllcyBvZiB0aGUgbG9nIGZvbGQgY2hhbmdlIHZlY3RvcgpoZWFkKGxmY192ZWN0b3IpCmBgYAoKQW5kIHRoZSBib3R0b20gb2YgdGhlIGxpc3QuCgpgYGB7ciB0YWlsX2xmYywgbGl2ZSA9IFRSVUV9CiMgTG9vayBhdCB0aGUgbGFzdCBlbnRyaWVzIG9mIHRoZSBsb2cgZm9sZCBjaGFuZ2UgdmVjdG9yCnRhaWwobGZjX3ZlY3RvcikKYGBgCgojIyBSdW4gR1NFQQoKTm93IGZvciB0aGUgYW5hbHlzaXMhCgpXZSBjYW4gdXNlIHRoZSBgR1NFQSgpYCBmdW5jdGlvbiB0byBwZXJmb3JtIEdTRUEgd2l0aCBhbnkgZ2VuZXJpYyBzZXQgb2YgZ2VuZSBzZXRzLCBidXQgdGhlcmUgYXJlIHNldmVyYWwgZnVuY3Rpb25zIGZvciB1c2luZyBzcGVjaWZpYywgY29tbW9ubHkgdXNlZCBnZW5lIHNldHMgKGUuZy4sIGBnc2VLRUdHKClgKS4KCmBgYHtyIHJ1bl9nc2VhfQpnc2VhX3Jlc3VsdHMgPC0gR1NFQShnZW5lTGlzdCA9IGxmY192ZWN0b3IsICAjIG9yZGVyZWQgcmFua2VkIGdlbmUgbGlzdAogICAgICAgICAgICAgICAgICAgICBtaW5HU1NpemUgPSAyNSwgICMgbWluaW11bSBnZW5lIHNldCBzaXplCiAgICAgICAgICAgICAgICAgICAgIG1heEdTU2l6ZSA9IDUwMCwgICMgbWF4aW11bSBnZW5lIHNldCBzZXQKICAgICAgICAgICAgICAgICAgICAgcHZhbHVlQ3V0b2ZmID0gMC4wNSwKICAgICAgICAgICAgICAgICAgICAgcEFkanVzdE1ldGhvZCA9ICJCSCIsICAjIGNvcnJlY3Rpb24gZm9yIG11bHRpcGxlIGh5cG90aGVzaXMgdGVzdGluZwogICAgICAgICAgICAgICAgICAgICBURVJNMkdFTkUgPSBkcGx5cjo6c2VsZWN0KGhzX2hhbGxtYXJrc19kZiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBnc19uYW1lLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGVuc2VtYmxfZ2VuZSkpICAjIHBhc3MgdGhlIGNvcnJlY3QgaWRlbnRpZmllciBjb2x1bW4KYGBgCkxldCdzIHRha2UgYSBsb29rIGF0IHRoZSBHU0VBIHJlc3VsdHMuCgpgYGB7ciB2aWV3X2dzZWEsIGxpdmUgPSBUUlVFLCBldmFsID0gRkFMU0V9ClZpZXcoZ3NlYV9yZXN1bHRzQHJlc3VsdCB8PgogICAgICAgZHBseXI6OmFycmFuZ2UoZHBseXI6OmRlc2MoTkVTKSkKKQpgYGAKCk5vcm1hbGl6ZWQgZW5yaWNobWVudCBzY29yZXMgKE5FUykgYXJlIGVucmljaG1lbnQgc2NvcmVzIHRoYXQgYXJlIHNjYWxlZCB0byBtYWtlIGdlbmUgc2V0cyB0aGF0IGNvbnRhaW4gZGlmZmVyZW50IG51bWJlciBvZiBnZW5lcyBjb21wYXJhYmxlLgoKUGF0aHdheXMgd2l0aCBzaWduaWZpY2FudCwgaGlnaGx5IHBvc2l0aXZlIE5FUyBhcmUgZW5yaWNoZWQgaW4gRVJNUyBteW9ibGFzdHMsIHdoZXJlYXMgcGF0aHdheXMgd2l0aCBzaWduaWZpY2FudCwgaGlnaGx5IG5lZ2F0aXZlIE5FUyBhcmUgZW5yaWNoZWQgaW4gQVJNUyBteW9ibGFzdHMuCgpMZXQncyB3cml0ZSB0aGVzZSByZXN1bHRzIHRvIGZpbGUuCgpgYGB7ciB3cml0ZV9nc2VhfQpnc2VhX3Jlc3VsdHNAcmVzdWx0IHw+IHJlYWRyOjp3cml0ZV90c3Yob3V0cHV0X2ZpbGUpCmBgYAoKIyMjIFZpc3VhbGl6aW5nIEdTRUEgcmVzdWx0cwoKV2UgY2FuIHZpc3VhbGl6ZSBHU0VBIHJlc3VsdHMgZm9yIGluZGl2aWR1YWwgcGF0aHdheXMgb3IgZ2VuZSBzZXRzIHVzaW5nIGBlbnJpY2hwbG90Ojpnc2VhcGxvdCgpYC4KTGV0J3MgdGFrZSBhIGxvb2sgYXQgMyBkaWZmZXJlbnQgcGF0aHdheXMgLS0gb25lIHdpdGggYSBoaWdobHkgcG9zaXRpdmUgTkVTLCBvbmUgd2l0aCBhIGhpZ2hseSBuZWdhdGl2ZSBORVMsIGFuZCBvbmUgdGhhdCB3YXMgbm90IGEgc2lnbmlmaWNhbnQgcmVzdWx0IC0tIHRvIGdldCBtb3JlIGluc2lnaHQgaW50byBob3cgRVMgYXJlIGNhbGN1bGF0ZWQuCgojIyMjIEhpZ2hseSBQb3NpdGl2ZSBORVMKCkxldCdzIHRha2UgbG9vayBhdCBhIHBhdGh3YXkgd2l0aCBhIGhpZ2hseSBwb3NpdGl2ZSBORVMgKGBIQUxMTUFSS19NWUNfVEFSR0VUU19WMmApIHVzaW5nIGEgR1NFQSBwbG90LgoKYGBge3IgaGlnaGx5X3Bvc30KZW5yaWNocGxvdDo6Z3NlYXBsb3QoZ3NlYV9yZXN1bHRzLAogICAgICAgICAgICAgICAgICAgICBnZW5lU2V0SUQgPSAiSEFMTE1BUktfTVlDX1RBUkdFVFNfVjIiLAogICAgICAgICAgICAgICAgICAgICB0aXRsZSA9ICJIQUxMTUFSS19NWUNfVEFSR0VUU19WMiIsCiAgICAgICAgICAgICAgICAgICAgIGNvbG9yLmxpbmUgPSAiIzAwNjZGRiIpCmBgYAoKTm90aWNlIGhvdyB0aGUgZ2VuZXMgdGhhdCBhcmUgaW4gdGhlIGdlbmUgc2V0LCBpbmRpY2F0ZWQgYnkgdGhlIGJsYWNrIGJhcnMsIHRlbmQgdG8gYmUgb24gdGhlIGxlZnQgc2lkZSBvZiB0aGUgZ3JhcGggaW5kaWNhdGluZyB0aGF0IHRoZXkgaGF2ZSBwb3NpdGl2ZSBnZW5lLWxldmVsIHNjb3Jlcy4KCiMjIyMgSGlnaGx5IE5lZ2F0aXZlIE5FUwoKVGhlIGdlbmUgc2V0IGBIQUxMTUFSS19NWU9HRU5FU0lTYCBoYWQgYSBoaWdobHkgbmVnYXRpdmUgTkVTLgoKYGBge3IgaGlnaGx5X25lZ30KZW5yaWNocGxvdDo6Z3NlYXBsb3QoZ3NlYV9yZXN1bHRzLAogICAgICAgICAgICAgICAgICAgICBnZW5lU2V0SUQgPSAiSEFMTE1BUktfTVlPR0VORVNJUyIsCiAgICAgICAgICAgICAgICAgICAgIHRpdGxlID0gIkhBTExNQVJLX01ZT0dFTkVTSVMiLAogICAgICAgICAgICAgICAgICAgICBjb2xvci5saW5lID0gIiMwMDY2RkYiKQpgYGAKClRoaXMgZ2VuZSBzZXQgc2hvd3MgdGhlIG9wcG9zaXRlIHBhdHRlcm4gLS0gZ2VuZXMgaW4gdGhlIHBhdGh3YXkgdGVuZCB0byBiZSBvbiB0aGUgcmlnaHQgc2lkZSBvZiB0aGUgZ3JhcGguCgojIyMjIEEgbm9uLXNpZ25pZmljYW50IHJlc3VsdAoKVGhlIGBAcmVzdWx0c2Agc2xvdCB3aWxsIG9ubHkgc2hvdyBnZW5lIHNldHMgdGhhdCBwYXNzIHRoZSBgcHZhbHVlQ3V0b2ZmYCB0aHJlc2hvbGQgd2Ugc3VwcGxpZWQgdG8gYEdTRUEoKWAsIGJ1dCB3ZSBjYW4gcGxvdCBhbnkgZ2VuZSBzZXQgc28gbG9uZyBhcyB3ZSBrbm93IGl0cyBuYW1lLgpMZXQncyBsb29rIGF0IGBIQUxMTUFSS19QNTNfUEFUSFdBWWAsIHdoaWNoIHdhcyBub3QgaW4gdGhlIHJlc3VsdHMgd2Ugdmlld2VkIGVhcmxpZXIuCgpgYGB7ciBwNTMsIGxpdmUgPSBUUlVFfQplbnJpY2hwbG90Ojpnc2VhcGxvdChnc2VhX3Jlc3VsdHMsCiAgICAgICAgICAgICAgICAgICAgIGdlbmVTZXRJRCA9ICJIQUxMTUFSS19QNTNfUEFUSFdBWSIsCiAgICAgICAgICAgICAgICAgICAgIHRpdGxlID0gIkhBTExNQVJLX1A1M19QQVRIV0FZIiwKICAgICAgICAgICAgICAgICAgICAgY29sb3IubGluZSA9ICIjMDA2NkZGIikKYGBgCgpHZW5lcyBpbiB0aGUgcGF0aHdheSBhcmUgZGlzdHJpYnV0ZWQgbW9yZSBldmVubHkgdGhyb3VnaG91dCB0aGUgcmFua2VkIGxpc3QsIHJlc3VsdGluZyBpbiBhIG1vcmUgIm1pZGRsaW5nIiBzY29yZS4KCipOb3RlOiBUaGUgcGxvdHMgcmV0dXJuZWQgYnkgYGVucmljaHBsb3Q6OmdzZWFwbG90YCBhcmUgZ2dwbG90cywgc28gd2UgY291bGQgdXNlIGBnZ3Bsb3QyOjpnZ3NhdmUoKWAgdG8gc2F2ZSB0aGVtIHRvIGZpbGUgaWYgd2Ugd2FudGVkIHRvLioKCiMjIFNlc3Npb24gSW5mbwoKYGBge3Igc2Vzc2lvbl9pbmZvfQpzZXNzaW9uSW5mbygpCmBgYAo=
+ + +
+
+ +
+ + + + + + + + + + + + + + + + + From c4ae9f947afb5de3b764fb5b03492ea491566d7b Mon Sep 17 00:00:00 2001 From: Jaclyn Taroni <19534205+jaclyn-taroni@users.noreply.github.com> Date: Mon, 25 Nov 2024 09:56:34 -0500 Subject: [PATCH 2/7] Remove 05 version of GSEA that uses marker genes --- .../05-gene_set_enrichment_analysis-live.Rmd | 344 -- .../05-gene_set_enrichment_analysis.Rmd | 356 -- .../05-gene_set_enrichment_analysis.nb.html | 3635 ----------------- 3 files changed, 4335 deletions(-) delete mode 100644 scRNA-seq-advanced/05-gene_set_enrichment_analysis-live.Rmd delete mode 100644 scRNA-seq-advanced/05-gene_set_enrichment_analysis.Rmd delete mode 100644 scRNA-seq-advanced/05-gene_set_enrichment_analysis.nb.html diff --git a/scRNA-seq-advanced/05-gene_set_enrichment_analysis-live.Rmd b/scRNA-seq-advanced/05-gene_set_enrichment_analysis-live.Rmd deleted file mode 100644 index 0435b77e..00000000 --- a/scRNA-seq-advanced/05-gene_set_enrichment_analysis-live.Rmd +++ /dev/null @@ -1,344 +0,0 @@ ---- -title: "Pathway analysis: Gene Set Enrichment Analysis (GSEA)" -output: - html_notebook: - toc: true - toc_float: true -author: CCDL for ALSF -date: 2021 ---- - -## Objectives - -This notebook will demonstrate how to: - -- Prepare tabular data of gene-level statistics for use with Gene Set Enrichment Analysis (GSEA) -- Access [Molecular Signatures Database gene set collections](https://www.gsea-msigdb.org/gsea/msigdb/collections.jsp) via the `msigdbr` package -- Perform GSEA with the `clusterProfiler` package -- Visualize GSEA results with the `enrichplot` package - ---- - -In this notebook, we'll analyze the marker genes from cluster 1, just as we did in the previous notebook. -Unlike ORA, GSEA allows us to use the full list of genes we could reliably measure, rather than picking some cutoff ourselves. - -GSEA is a functional class scoring (FCS) approach to pathway analysis that was first introduced in [Subramanian _et al._ (2005)](https://doi.org/10.1073/pnas.0506580102). -The rationale behind FCS approaches is that small changes in individual genes that participate in the same biological process or pathway can be significant and of biological interest. -FCS methods are better suited for identifying these pathways that show coordinated changes than ORA. -In ORA, we pick a cutoff that _typically_ only captures genes with large individual changes. - -There are 3 general steps in FCS methods ([Khatri _et al._ 2012]( https://doi.org/10.1371/journal.pcbi.1002375)): - -1. Calculate a gene-level statistic (here, we'll use the summary log fold changes in our marker genes table) -2. Gene-level statistics are aggregated into a pathway-level statistic -3. Assess the statistical significance of the pathway-level statistic - -We'll note here that GSEA was designed to detect small coordinated changes between _conditions_ in (necessarily bulk!) microarray data. -It may be more difficult to suss out small coordinated changes when we assay individual cells, due to technical or biological dropout. - -Individual cells may also differently sized ([Blasi *et al.* 2017](https://doi.org/10.1088/1478-3975/aa609a)), be at different stages of the cell cycle ([Satija lab](https://satijalab.org/seurat/archive/v3.1/cell_cycle_vignette.html)), or experiencing different levels of cellular stress ([Luecken and Theis. 2019](https://doi.org/10.15252/msb.20188746)), some of which may be interesting in the context of cancer and therefore should not be corrected for. -(In practice, it is common to correct for cell cycle stage but we have not done that upstream.) -These aspects also have the potential to cloud our ability to detect small coordinated changes in other pathways. - -There is some literature that suggests that dropout (excessive zeros) observed in single cell data is a function of cell type heterogeneity ([Kim *et al.* 2020](https://doi.org/10.1186/s13059-020-02096-y)). -We're using GSEA to compare clusters of cells, which we (ideally) expect to be different biological states and/or cell types. -If many genes "drop out" between cell types, we may expect our statistics used as input to GSEA to be noisy. - -With all this in mind, it may be unsurprising that ORA, which generally captures genes with large individual changes and lower within-group variation (the most "statistically significant" genes), is well-suited to analyzing scRNA-seq data and the advantages of GSEA in a bulk analysis setting may not be fully available here. -Nonetheless, GSEA is used for pathway analysis of scRNA-seq data. - -Methods specifically for pathway analysis of scRNA-seq (e.g., [Ma *et al.* 2020](https://doi.org/10.1038/s41467-020-15298-6)) are being developed, so stay tuned! - -#### Other resources - -* For another example using `clusterProfiler` for GSEA, see [_Intro to DGE: Functional Analysis._ from Harvard Chan Bioinformatics Core Training.](https://hbctraining.github.io/DGE_workshop/lessons/09_functional_analysis.html) -* The way we'll use `clusterProfiler` here uses `fgsea` (Fast Gene Set Enrichment Analysis) under the hood. -You can read more about fgsea in [Korotkevich _et al._ (2021)](https://doi.org/10.1101/060012). -* [refine.bio examples Gene set enrichment analysis - RNA-seq](https://alexslemonade.github.io/refinebio-examples/03-rnaseq/pathway-analysis_rnaseq_02_gsea.html) from which this material has been adapted. - -## Set up - -### Libraries - -```{r libraries} -# Package to run GSEA -library(clusterProfiler) -# Package that contains the MSigDB gene sets in tidy format -library(msigdbr) -``` - -### Directories and Files - -#### Directories - -```{r create_dir, live = TRUE} -# We'll use the marker genes as GSEA input - -# We'll create a directory to specifically hold the pathway results if it doesn't -# exist yet - -``` - -#### Input files - -We're going to use the version of the cluster 1 marker genes table that has gene symbols, so we don't have to do gene identifier conversion again! - -```{r input_files} -input_file <- file.path(hodgkins_analysis_dir, - "markers", - "cluster01_markers_with_gene_symbols.tsv") -``` - -#### Output files - -We'll save our table of GSEA results as a TSV. - -```{r output_files} -output_file <- file.path(results_dir, - "cluster01_immunologic_gsea_results.tsv") -``` - -## Gene sets - -We will use gene sets from the [Molecular Signatures Database (MSigDB)](https://www.gsea-msigdb.org/gsea/msigdb/index.jsp) from the Broad Institute ([Subramanian, Tamayo *et al.* 2005](https://doi.org/10.1073/pnas.0506580102)). -The [`msigdbr`](https://cran.r-project.org/web/packages/msigdbr/index.html) package contains MSigDB datasets already in the tidy format required by `clusterProfiler` and supports multiple organisms. - -Let's take a look at what organisms the package supports. - -```{r show_species} -msigdbr_species() -``` - -MSigDB contains 8 different gene set collections. - - H: hallmark gene sets - C1: positional gene sets - C2: curated gene sets - C3: motif gene sets - C4: computational gene sets - C5: GO gene sets - C6: oncogenic signatures - C7: immunologic signatures - - -Because we're working with a sample that's partly comprised of immune cells, let's use the immunologic signatures to analyze our marker genes. -These signatures are generally derived from gene expression experiments of a broad set of cell types, perturbations, and in some cases responses to specific vaccines ([Godec *et al.* 2016](https://doi.org/10.1016/j.immuni.2015.12.006)); they usually have an "up" and "down" component. -You can [read more about the C7 collection at MSigDB](https://www.gsea-msigdb.org/gsea/msigdb/collection_details.jsp#C7). - -We might expect that analyzing our marker genes with this collection could give us some information not only about cell identity or type, but also phenotype or activation state. - -We can retrieve only the Immunologic gene sets by specifying `category = "C7"` to the `msigdbr()` function. -Again, we only want human gene sets here so we specify with that with the `species` argument. - -```{r immunologic_sets, live = TRUE} - -``` - -## Gene Set Enrichment Analysis - -_Adapted from [refine.bio examples](https://github.com/AlexsLemonade/refinebio-examples/blob/33cdeff66d57f9fe8ee4fcb5156aea4ac2dce07f/03-rnaseq/pathway-analysis_rnaseq_02_gsea.Rmd)_ - -![](diagrams/subramanian_fig1.jpg) - -**Figure 1. [Subramanian _et al._ (2005)](https://doi.org/10.1073/pnas.0506580102).** - -GSEA calculates a pathway-level metric, called an enrichment score (sometimes abbreviated as ES), by ranking genes by a gene-level statistic. -This score reflects whether or not a gene set or pathway is over-represented at the top or bottom of the gene rankings ([Subramanian _et al._ 2005](https://doi.org/10.1073/pnas.0506580102); [Yu](http://yulab-smu.top/clusterProfiler-book/chapter2.html#gene-set-enrichment-analysis)) - -Specifically, all genes are ranked from most positive to most negative based on their statistic and a running sum is calculated: -Starting with the most highly ranked genes, the running sum increases for each gene in the pathway and decreases for each gene not in the pathway. -The enrichment score for a pathway is the running sum's maximum deviation from zero. -GSEA also assesses statistical significance of the scores for each pathway through permutation testing. -As a result, each input pathway will have a p-value associated with it that is then corrected for multiple hypothesis testing ([Subramanian _et al._ 2005](https://doi.org/10.1073/pnas.0506580102); [Yu](http://yulab-smu.top/clusterProfiler-book/chapter2.html#gene-set-enrichment-analysis)). - -The implementation of GSEA we use in here examples requires a gene list ordered by some statistic and input gene sets. -When you use previously computed gene-level statistics with GSEA, it is called GSEA pre-ranked. - -## Marker gene results - -```{r read_in_markers, live = TRUE} - -``` - -```{r} -head(markers_df) -``` - -Since this data frame of marker genes includes gene symbols, we do not need to perform any kind of gene identifier conversion. -We do, however, need to check for duplicate gene symbols. -We can accomplish this with `duplicated()`, which returns a logical vector (e.g., `TRUE` or `FALSE`). -The function `sum()` will count `TRUE` values as 1s and `FALSE` as 0s, so using it with `duplicated()` will count the number of duplicate values. - -```{r any_duplicated, live = TRUE} - -``` - -Luckily this number is very low, such that we do not expect it to impact our results much, but this will still cause a problem when we go to run GSEA. - -### Removing duplicates - -The GSEA approach requires on discriminating between genes that are in a gene set and those that are not. -Practically speaking, gene sets are just collections of gene identifiers! -When the function we use for GSEA pre-ranked gets a list with duplicated gene identifiers, it can produce unexpected results. - -Compared to the total number of genes that are in our results, there are not a lot of duplicates but we'll still need to make a decision about how to handle them. - -Let's get a vector of the duplicated gene symbols so we can use it to explore our filtering steps. - -```{r gene_dups, live = TRUE} - -``` - -Now we'll look at the values for the duplicated gene symbols. - -```{r show_gene_dups} -markers_df |> - dplyr::filter(gene_symbol %in% duplicated_gene_symbols) |> - dplyr::arrange(gene_symbol) -``` - -We can see that the associated values vary for each row. - -Let's keep the gene symbols associated with the higher absolute value of the gene-level statistic we'll use for GSEA (`summary.logFC`). - -Retaining the instance of the gene symbols with the higher absolute value of a gene-level statistic means that we will retain the value that is likely to be more highly- or lowly-ranked or, put another way, the values less likely to be towards the middle of the ranked gene list. -We should keep this decision in mind when interpreting our results. -For example, if all the duplicate identifiers happened to be in a particular gene set, we may get an overly optimistic view of how perturbed that gene set is because we preferentially selected instances of the identifier that have a higher absolute value of the statistic used for ranking. - -In the next chunk, we are going to filter out the duplicated rows using the `dplyr::distinct()` function after sorting by absolute value of the summary log fold change. -This will keep the first row with the duplicated value thus keeping the row with the largest absolute value. - -```{r filter_dup_markers} -filtered_markers_df <- markers_df |> - # Sort so that the highest absolute values of the summary log2 fold change are - # at the top - dplyr::arrange(dplyr::desc(abs(summary.logFC))) |> - # Filter out the duplicated rows using `dplyr::distinct()` - dplyr::distinct(gene_symbol, .keep_all = TRUE) -``` - -Let's see what happened to our duplicate identifiers. - -```{r show_filtered_markers, live = TRUE} -# Subset to & arrange by gene symbols that were duplicated in the original -# data frame of results - -``` - -Now we're ready to prep our pre-ranked list for GSEA. - -### Pre-ranked list - -The `GSEA()` function takes a pre-ranked (sorted) named vector of statistics, where the names in the vector are gene identifiers. -This is step 1 -- gene-level statistics. - -Here, we're using the summary log fold change, which summarizes of all of the log fold changes from the pairwise comparisons between cluster 1 and other clusters into a single value ([ref](https://rdrr.io/bioc/scran/man/combineMarkers.html)). -As such, this statistic gives us information about the relative magnitude and directionality of each gene's expression in cluster 1 relative to all other clusters. - -```{r lfc_vector} -lfc_vector <- filtered_markers_df |> - # Extract a vector of `summary.logFC` named by `gene_symbol` - dplyr::pull(summary.logFC, name = gene_symbol) -lfc_vector <- sort(lfc_vector, decreasing = TRUE) -``` - -Let's look at the top ranked values. - -```{r head_lfc, live = TRUE} -# Look at first entries of the log fold change vector - -``` - -And the bottom of the list. - -```{r tail_lfc, live = TRUE} -# Look at the last entries of the log fold change vector - -``` - -## Run GSEA - -Now for the analysis! - -We can use the `GSEA()` function to perform GSEA with any generic set of gene sets, but there are several functions for using specific, commonly used gene sets (e.g., `gseKEGG()`). - -```{r run_gsea} -gsea_results <- GSEA(geneList = lfc_vector, # ordered ranked gene list - minGSSize = 25, # minimum gene set size - maxGSSize = 500, # maximum gene set set - pvalueCutoff = 0.05, - pAdjustMethod = "BH", # correction for multiple hypothesis testing - TERM2GENE = dplyr::select(hs_immunologic_df, - gs_name, - gene_symbol)) -``` -Let's take a look at the GSEA results. - -```{r view_gsea, live = TRUE, eval = FALSE} - -``` - -Normalized enrichment scores (NES) are enrichment scores that are scaled to make gene sets that contain different number of genes comparable. - -Let's write these results to file. - -```{r write_gsea} -gsea_results@result |> readr::write_tsv(output_file) -``` - -### Visualizing GSEA results - -We can visualize GSEA results for individual pathways or gene sets using `enrichplot::gseaplot()`. -Let's take a look at 3 different pathways -- one with a highly positive NES, one with a highly negative NES, and one that was not a significant result -- to get more insight into how ES are calculated. - -#### Highly Positive NES - -Let's take look at a pathway with a highly positive NES (`GSE29618_BCELL_VS_PDC_UP`) using a GSEA plot. -This gene set is comprised of ([ref](https://www.gsea-msigdb.org/gsea/msigdb/cards/GSE29618_BCELL_VS_PDC_UP)): - -> Genes up-regulated in comparison of B cells versus plasmacytoid dendritic cells (pDC) . - -```{r highly_pos} -enrichplot::gseaplot(gsea_results, - geneSetID = "GSE29618_BCELL_VS_PDC_UP", - title = "GSE29618_BCELL_VS_PDC_UP", - color.line = "#0066FF") -``` - -Notice how the genes that are in the gene set, indicated by the black bars, tend to be on the left side of the graph indicating that they have positive gene-level scores. - -#### Highly Negative NES - -The gene set `GSE29618_BCELL_VS_PDC_DAY7_FLU_VACCINE_DN` had a highly negative NES, and is comprised of ([ref](https://www.gsea-msigdb.org/gsea/msigdb/geneset_page.jsp?geneSetName=GSE29618_BCELL_VS_PDC_DAY7_FLU_VACCINE_DN)) - -> Genes down-regulated in comparison of B cells from influenza vaccinee at day 7 post-vaccination versus plasmacytoid dendritic cells (pDC) at day 7 post-vaccination. - -This is the "down" signature that corresponds to one of the "up" pathways with a highly positive NES, so in a way this is not adding new information! - -```{r highly_neg} -enrichplot::gseaplot(gsea_results, - geneSetID = "GSE29618_BCELL_VS_PDC_DAY7_FLU_VACCINE_DN", - title = "GSE29618_BCELL_VS_PDC_DAY7_FLU_VACCINE_DN", - color.line = "#0066FF") -``` - -This gene set shows the opposite pattern -- genes in the pathway tend to be on the right side of the graph. - -#### A non-significant result - -The `@results` slot will only show gene sets that pass the `pvalueCutoff` threshold we supplied to `GSEA()`, but we can plot any gene set so long as we know its name. -Let's look at [`GOLDRATH_EFF_VS_MEMORY_CD8_TCELL_DN`](http://www.gsea-msigdb.org/gsea/msigdb/geneset_page.jsp?geneSetName=GOLDRATH_EFF_VS_MEMORY_CD8_TCELL_UP&keywords=GOLDRATH), which examines the difference between effector CD8 T cells and memory CD8 T cells and was not in the results we viewed earlier. - -```{r nonsig, live = TRUE} - -``` - -Genes in the pathway are distributed more evenly throughout the ranked list, resulting in a more "middling" score. - -*Note: The plots returned by `enrichplot::gseaplot` are ggplots, so we could use `ggplot2::ggsave()` to save them to file if we wanted to.* - -## Session Info - -```{r session_info} -sessionInfo() -``` diff --git a/scRNA-seq-advanced/05-gene_set_enrichment_analysis.Rmd b/scRNA-seq-advanced/05-gene_set_enrichment_analysis.Rmd deleted file mode 100644 index ce9a36b2..00000000 --- a/scRNA-seq-advanced/05-gene_set_enrichment_analysis.Rmd +++ /dev/null @@ -1,356 +0,0 @@ ---- -title: "Pathway analysis: Gene Set Enrichment Analysis (GSEA)" -output: - html_notebook: - toc: true - toc_float: true -author: CCDL for ALSF -date: 2021 ---- - -## Objectives - -This notebook will demonstrate how to: - -- Prepare tabular data of gene-level statistics for use with Gene Set Enrichment Analysis (GSEA) -- Access [Molecular Signatures Database gene set collections](https://www.gsea-msigdb.org/gsea/msigdb/collections.jsp) via the `msigdbr` package -- Perform GSEA with the `clusterProfiler` package -- Visualize GSEA results with the `enrichplot` package - ---- - -In this notebook, we'll analyze the marker genes from cluster 1, just as we did in the previous notebook. -Unlike ORA, GSEA allows us to use the full list of genes we could reliably measure, rather than picking some cutoff ourselves. - -GSEA is a functional class scoring (FCS) approach to pathway analysis that was first introduced in [Subramanian _et al._ (2005)](https://doi.org/10.1073/pnas.0506580102). -The rationale behind FCS approaches is that small changes in individual genes that participate in the same biological process or pathway can be significant and of biological interest. -FCS methods are better suited for identifying these pathways that show coordinated changes than ORA. -In ORA, we pick a cutoff that _typically_ only captures genes with large individual changes. - -There are 3 general steps in FCS methods ([Khatri _et al._ 2012]( https://doi.org/10.1371/journal.pcbi.1002375)): - -1. Calculate a gene-level statistic (here, we'll use the summary log fold changes in our marker genes table) -2. Gene-level statistics are aggregated into a pathway-level statistic -3. Assess the statistical significance of the pathway-level statistic - -We'll note here that GSEA was designed to detect small coordinated changes between _conditions_ in (necessarily bulk!) microarray data. -It may be more difficult to suss out small coordinated changes when we assay individual cells, due to technical or biological dropout. - -Individual cells may also differently sized ([Blasi *et al.* 2017](https://doi.org/10.1088/1478-3975/aa609a)), be at different stages of the cell cycle ([Satija lab](https://satijalab.org/seurat/archive/v3.1/cell_cycle_vignette.html)), or experiencing different levels of cellular stress ([Luecken and Theis. 2019](https://doi.org/10.15252/msb.20188746)), some of which may be interesting in the context of cancer and therefore should not be corrected for. -(In practice, it is common to correct for cell cycle stage but we have not done that upstream.) -These aspects also have the potential to cloud our ability to detect small coordinated changes in other pathways. - -There is some literature that suggests that dropout (excessive zeros) observed in single cell data is a function of cell type heterogeneity ([Kim *et al.* 2020](https://doi.org/10.1186/s13059-020-02096-y)). -We're using GSEA to compare clusters of cells, which we (ideally) expect to be different biological states and/or cell types. -If many genes "drop out" between cell types, we may expect our statistics used as input to GSEA to be noisy. - -With all this in mind, it may be unsurprising that ORA, which generally captures genes with large individual changes and lower within-group variation (the most "statistically significant" genes), is well-suited to analyzing scRNA-seq data and the advantages of GSEA in a bulk analysis setting may not be fully available here. -Nonetheless, GSEA is used for pathway analysis of scRNA-seq data. - -Methods specifically for pathway analysis of scRNA-seq (e.g., [Ma *et al.* 2020](https://doi.org/10.1038/s41467-020-15298-6)) are being developed, so stay tuned! - -#### Other resources - -* For another example using `clusterProfiler` for GSEA, see [_Intro to DGE: Functional Analysis._ from Harvard Chan Bioinformatics Core Training.](https://hbctraining.github.io/DGE_workshop/lessons/09_functional_analysis.html) -* The way we'll use `clusterProfiler` here uses `fgsea` (Fast Gene Set Enrichment Analysis) under the hood. -You can read more about fgsea in [Korotkevich _et al._ (2021)](https://doi.org/10.1101/060012). -* [refine.bio examples Gene set enrichment analysis - RNA-seq](https://alexslemonade.github.io/refinebio-examples/03-rnaseq/pathway-analysis_rnaseq_02_gsea.html) from which this material has been adapted. - -## Set up - -### Libraries - -```{r libraries} -# Package to run GSEA -library(clusterProfiler) -# Package that contains the MSigDB gene sets in tidy format -library(msigdbr) -``` - -### Directories and Files - -#### Directories - -```{r create_dir, live = TRUE} -# We'll use the marker genes as GSEA input -hodgkins_analysis_dir <- file.path("analysis", "hodgkins") - -# We'll create a directory to specifically hold the pathway results if it doesn't -# exist yet -results_dir <- file.path(hodgkins_analysis_dir, "pathway-analysis") -fs::dir_create(results_dir) -``` - -#### Input files - -We're going to use the version of the cluster 1 marker genes table that has gene symbols, so we don't have to do gene identifier conversion again! - -```{r input_files} -input_file <- file.path(hodgkins_analysis_dir, - "markers", - "cluster01_markers_with_gene_symbols.tsv") -``` - -#### Output files - -We'll save our table of GSEA results as a TSV. - -```{r output_files} -output_file <- file.path(results_dir, - "cluster01_immunologic_gsea_results.tsv") -``` - -## Gene sets - -We will use gene sets from the [Molecular Signatures Database (MSigDB)](https://www.gsea-msigdb.org/gsea/msigdb/index.jsp) from the Broad Institute ([Subramanian, Tamayo *et al.* 2005](https://doi.org/10.1073/pnas.0506580102)). -The [`msigdbr`](https://cran.r-project.org/web/packages/msigdbr/index.html) package contains MSigDB datasets already in the tidy format required by `clusterProfiler` and supports multiple organisms. - -Let's take a look at what organisms the package supports. - -```{r show_species} -msigdbr_species() -``` - -MSigDB contains 8 different gene set collections. - - H: hallmark gene sets - C1: positional gene sets - C2: curated gene sets - C3: motif gene sets - C4: computational gene sets - C5: GO gene sets - C6: oncogenic signatures - C7: immunologic signatures - - -Because we're working with a sample that's partly comprised of immune cells, let's use the immunologic signatures to analyze our marker genes. -These signatures are generally derived from gene expression experiments of a broad set of cell types, perturbations, and in some cases responses to specific vaccines ([Godec *et al.* 2016](https://doi.org/10.1016/j.immuni.2015.12.006)); they usually have an "up" and "down" component. -You can [read more about the C7 collection at MSigDB](https://www.gsea-msigdb.org/gsea/msigdb/collection_details.jsp#C7). - -We might expect that analyzing our marker genes with this collection could give us some information not only about cell identity or type, but also phenotype or activation state. - -We can retrieve only the Immunologic gene sets by specifying `category = "C7"` to the `msigdbr()` function. -Again, we only want human gene sets here so we specify with that with the `species` argument. - -```{r immunologic_sets, live = TRUE} -hs_immunologic_df <- msigdbr(species = "Homo sapiens", - category = "C7") -``` - -## Gene Set Enrichment Analysis - -_Adapted from [refine.bio examples](https://github.com/AlexsLemonade/refinebio-examples/blob/33cdeff66d57f9fe8ee4fcb5156aea4ac2dce07f/03-rnaseq/pathway-analysis_rnaseq_02_gsea.Rmd)_ - -![](diagrams/subramanian_fig1.jpg) - -**Figure 1. [Subramanian _et al._ (2005)](https://doi.org/10.1073/pnas.0506580102).** - -GSEA calculates a pathway-level metric, called an enrichment score (sometimes abbreviated as ES), by ranking genes by a gene-level statistic. -This score reflects whether or not a gene set or pathway is over-represented at the top or bottom of the gene rankings ([Subramanian _et al._ 2005](https://doi.org/10.1073/pnas.0506580102); [Yu](http://yulab-smu.top/clusterProfiler-book/chapter2.html#gene-set-enrichment-analysis)) - -Specifically, all genes are ranked from most positive to most negative based on their statistic and a running sum is calculated: -Starting with the most highly ranked genes, the running sum increases for each gene in the pathway and decreases for each gene not in the pathway. -The enrichment score for a pathway is the running sum's maximum deviation from zero. -GSEA also assesses statistical significance of the scores for each pathway through permutation testing. -As a result, each input pathway will have a p-value associated with it that is then corrected for multiple hypothesis testing ([Subramanian _et al._ 2005](https://doi.org/10.1073/pnas.0506580102); [Yu](http://yulab-smu.top/clusterProfiler-book/chapter2.html#gene-set-enrichment-analysis)). - -The implementation of GSEA we use in here examples requires a gene list ordered by some statistic and input gene sets. -When you use previously computed gene-level statistics with GSEA, it is called GSEA pre-ranked. - -## Marker gene results - -```{r read_in_markers, live = TRUE} -markers_df <- readr::read_tsv(input_file) -``` - -```{r} -head(markers_df) -``` - -Since this data frame of marker genes includes gene symbols, we do not need to perform any kind of gene identifier conversion. -We do, however, need to check for duplicate gene symbols. -We can accomplish this with `duplicated()`, which returns a logical vector (e.g., `TRUE` or `FALSE`). -The function `sum()` will count `TRUE` values as 1s and `FALSE` as 0s, so using it with `duplicated()` will count the number of duplicate values. - -```{r any_duplicated, live = TRUE} -sum(duplicated(markers_df$gene_symbol)) -``` - -Luckily this number is very low, such that we do not expect it to impact our results much, but this will still cause a problem when we go to run GSEA. - -### Removing duplicates - -The GSEA approach requires on discriminating between genes that are in a gene set and those that are not. -Practically speaking, gene sets are just collections of gene identifiers! -When the function we use for GSEA pre-ranked gets a list with duplicated gene identifiers, it can produce unexpected results. - -Compared to the total number of genes that are in our results, there are not a lot of duplicates but we'll still need to make a decision about how to handle them. - -Let's get a vector of the duplicated gene symbols so we can use it to explore our filtering steps. - -```{r gene_dups, live = TRUE} -duplicated_gene_symbols <- markers_df |> - dplyr::filter(duplicated(gene_symbol)) |> - dplyr::pull(gene_symbol) -``` - -Now we'll look at the values for the duplicated gene symbols. - -```{r show_gene_dups} -markers_df |> - dplyr::filter(gene_symbol %in% duplicated_gene_symbols) |> - dplyr::arrange(gene_symbol) -``` - -We can see that the associated values vary for each row. - -Let's keep the gene symbols associated with the higher absolute value of the gene-level statistic we'll use for GSEA (`summary.logFC`). - -Retaining the instance of the gene symbols with the higher absolute value of a gene-level statistic means that we will retain the value that is likely to be more highly- or lowly-ranked or, put another way, the values less likely to be towards the middle of the ranked gene list. -We should keep this decision in mind when interpreting our results. -For example, if all the duplicate identifiers happened to be in a particular gene set, we may get an overly optimistic view of how perturbed that gene set is because we preferentially selected instances of the identifier that have a higher absolute value of the statistic used for ranking. - -In the next chunk, we are going to filter out the duplicated rows using the `dplyr::distinct()` function after sorting by absolute value of the summary log fold change. -This will keep the first row with the duplicated value thus keeping the row with the largest absolute value. - -```{r filter_dup_markers} -filtered_markers_df <- markers_df |> - # Sort so that the highest absolute values of the summary log2 fold change are - # at the top - dplyr::arrange(dplyr::desc(abs(summary.logFC))) |> - # Filter out the duplicated rows using `dplyr::distinct()` - dplyr::distinct(gene_symbol, .keep_all = TRUE) -``` - -Let's see what happened to our duplicate identifiers. - -```{r show_filtered_markers, live = TRUE} -# Subset to & arrange by gene symbols that were duplicated in the original -# data frame of results -filtered_markers_df |> - dplyr::filter(gene_symbol %in% duplicated_gene_symbols) |> - dplyr::arrange(gene_symbol) -``` - -Now we're ready to prep our pre-ranked list for GSEA. - -### Pre-ranked list - -The `GSEA()` function takes a pre-ranked (sorted) named vector of statistics, where the names in the vector are gene identifiers. -This is step 1 -- gene-level statistics. - -Here, we're using the summary log fold change, which summarizes of all of the log fold changes from the pairwise comparisons between cluster 1 and other clusters into a single value ([ref](https://rdrr.io/bioc/scran/man/combineMarkers.html)). -As such, this statistic gives us information about the relative magnitude and directionality of each gene's expression in cluster 1 relative to all other clusters. - -```{r lfc_vector} -lfc_vector <- filtered_markers_df |> - # Extract a vector of `summary.logFC` named by `gene_symbol` - dplyr::pull(summary.logFC, name = gene_symbol) -lfc_vector <- sort(lfc_vector, decreasing = TRUE) -``` - -Let's look at the top ranked values. - -```{r head_lfc, live = TRUE} -# Look at first entries of the log fold change vector -head(lfc_vector) -``` - -And the bottom of the list. - -```{r tail_lfc, live = TRUE} -# Look at the last entries of the log fold change vector -tail(lfc_vector) -``` - -## Run GSEA - -Now for the analysis! - -We can use the `GSEA()` function to perform GSEA with any generic set of gene sets, but there are several functions for using specific, commonly used gene sets (e.g., `gseKEGG()`). - -```{r run_gsea} -gsea_results <- GSEA(geneList = lfc_vector, # ordered ranked gene list - minGSSize = 25, # minimum gene set size - maxGSSize = 500, # maximum gene set set - pvalueCutoff = 0.05, - pAdjustMethod = "BH", # correction for multiple hypothesis testing - TERM2GENE = dplyr::select(hs_immunologic_df, - gs_name, - gene_symbol)) -``` -Let's take a look at the GSEA results. - -```{r view_gsea, live = TRUE, eval = FALSE} -View(gsea_results@result |> - dplyr::arrange(dplyr::desc(NES)) -) -``` - -Normalized enrichment scores (NES) are enrichment scores that are scaled to make gene sets that contain different number of genes comparable. - -Let's write these results to file. - -```{r write_gsea} -gsea_results@result |> readr::write_tsv(output_file) -``` - -### Visualizing GSEA results - -We can visualize GSEA results for individual pathways or gene sets using `enrichplot::gseaplot()`. -Let's take a look at 3 different pathways -- one with a highly positive NES, one with a highly negative NES, and one that was not a significant result -- to get more insight into how ES are calculated. - -#### Highly Positive NES - -Let's take look at a pathway with a highly positive NES (`GSE29618_BCELL_VS_PDC_UP`) using a GSEA plot. -This gene set is comprised of ([ref](https://www.gsea-msigdb.org/gsea/msigdb/cards/GSE29618_BCELL_VS_PDC_UP)): - -> Genes up-regulated in comparison of B cells versus plasmacytoid dendritic cells (pDC) . - -```{r highly_pos} -enrichplot::gseaplot(gsea_results, - geneSetID = "GSE29618_BCELL_VS_PDC_UP", - title = "GSE29618_BCELL_VS_PDC_UP", - color.line = "#0066FF") -``` - -Notice how the genes that are in the gene set, indicated by the black bars, tend to be on the left side of the graph indicating that they have positive gene-level scores. - -#### Highly Negative NES - -The gene set `GSE29618_BCELL_VS_PDC_DAY7_FLU_VACCINE_DN` had a highly negative NES, and is comprised of ([ref](https://www.gsea-msigdb.org/gsea/msigdb/geneset_page.jsp?geneSetName=GSE29618_BCELL_VS_PDC_DAY7_FLU_VACCINE_DN)) - -> Genes down-regulated in comparison of B cells from influenza vaccinee at day 7 post-vaccination versus plasmacytoid dendritic cells (pDC) at day 7 post-vaccination. - -This is the "down" signature that corresponds to one of the "up" pathways with a highly positive NES, so in a way this is not adding new information! - -```{r highly_neg} -enrichplot::gseaplot(gsea_results, - geneSetID = "GSE29618_BCELL_VS_PDC_DAY7_FLU_VACCINE_DN", - title = "GSE29618_BCELL_VS_PDC_DAY7_FLU_VACCINE_DN", - color.line = "#0066FF") -``` - -This gene set shows the opposite pattern -- genes in the pathway tend to be on the right side of the graph. - -#### A non-significant result - -The `@results` slot will only show gene sets that pass the `pvalueCutoff` threshold we supplied to `GSEA()`, but we can plot any gene set so long as we know its name. -Let's look at [`GOLDRATH_EFF_VS_MEMORY_CD8_TCELL_DN`](http://www.gsea-msigdb.org/gsea/msigdb/geneset_page.jsp?geneSetName=GOLDRATH_EFF_VS_MEMORY_CD8_TCELL_UP&keywords=GOLDRATH), which examines the difference between effector CD8 T cells and memory CD8 T cells and was not in the results we viewed earlier. - -```{r nonsig, live = TRUE} -enrichplot::gseaplot(gsea_results, - geneSetID = "GOLDRATH_EFF_VS_MEMORY_CD8_TCELL_DN", - title = "GOLDRATH_EFF_VS_MEMORY_CD8_TCELL_DN", - color.line = "#0066FF") -``` - -Genes in the pathway are distributed more evenly throughout the ranked list, resulting in a more "middling" score. - -*Note: The plots returned by `enrichplot::gseaplot` are ggplots, so we could use `ggplot2::ggsave()` to save them to file if we wanted to.* - -## Session Info - -```{r session_info} -sessionInfo() -``` diff --git a/scRNA-seq-advanced/05-gene_set_enrichment_analysis.nb.html b/scRNA-seq-advanced/05-gene_set_enrichment_analysis.nb.html deleted file mode 100644 index 57cce690..00000000 --- a/scRNA-seq-advanced/05-gene_set_enrichment_analysis.nb.html +++ /dev/null @@ -1,3635 +0,0 @@ - - - - - - - - - - - - - - - -Pathway analysis: Gene Set Enrichment Analysis (GSEA) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -
-
-
-
-
- -
- - - - - - - - -
-

Objectives

-

This notebook will demonstrate how to:

-
    -
  • Prepare tabular data of gene-level statistics for use with Gene Set -Enrichment Analysis (GSEA)
  • -
  • Access Molecular -Signatures Database gene set collections via the -msigdbr package
  • -
  • Perform GSEA with the clusterProfiler package
  • -
  • Visualize GSEA results with the enrichplot package
  • -
-
-

In this notebook, we’ll analyze the marker genes from cluster 1, just -as we did in the previous notebook. Unlike ORA, GSEA allows us to use -the full list of genes we could reliably measure, rather than picking -some cutoff ourselves.

-

GSEA is a functional class scoring (FCS) approach to pathway analysis -that was first introduced in Subramanian et -al. (2005). The rationale behind FCS approaches is that small -changes in individual genes that participate in the same biological -process or pathway can be significant and of biological interest. FCS -methods are better suited for identifying these pathways that show -coordinated changes than ORA. In ORA, we pick a cutoff that -typically only captures genes with large individual -changes.

-

There are 3 general steps in FCS methods (Khatri et -al. 2012):

-
    -
  1. Calculate a gene-level statistic (here, we’ll use the summary log -fold changes in our marker genes table)
  2. -
  3. Gene-level statistics are aggregated into a pathway-level -statistic
  4. -
  5. Assess the statistical significance of the pathway-level -statistic
  6. -
-

We’ll note here that GSEA was designed to detect small coordinated -changes between conditions in (necessarily bulk!) microarray -data. It may be more difficult to suss out small coordinated changes -when we assay individual cells, due to technical or biological -dropout.

-

Individual cells may also differently sized (Blasi et al. -2017), be at different stages of the cell cycle (Satija -lab), or experiencing different levels of cellular stress (Luecken and Theis. -2019), some of which may be interesting in the context of cancer and -therefore should not be corrected for. (In practice, it is common to -correct for cell cycle stage but we have not done that upstream.) These -aspects also have the potential to cloud our ability to detect small -coordinated changes in other pathways.

-

There is some literature that suggests that dropout (excessive zeros) -observed in single cell data is a function of cell type heterogeneity -(Kim et -al. 2020). We’re using GSEA to compare clusters of cells, which -we (ideally) expect to be different biological states and/or cell types. -If many genes “drop out” between cell types, we may expect our -statistics used as input to GSEA to be noisy.

-

With all this in mind, it may be unsurprising that ORA, which -generally captures genes with large individual changes and lower -within-group variation (the most “statistically significant” genes), is -well-suited to analyzing scRNA-seq data and the advantages of GSEA in a -bulk analysis setting may not be fully available here. Nonetheless, GSEA -is used for pathway analysis of scRNA-seq data.

-

Methods specifically for pathway analysis of scRNA-seq (e.g., Ma et al. -2020) are being developed, so stay tuned!

-
-

Other resources

- -
-
-
-

Set up

-
-

Libraries

- - - -
# Package to run GSEA
-library(clusterProfiler)
- - -
- - -
clusterProfiler v4.12.0  For help: https://yulab-smu.top/biomedical-knowledge-mining-book/
-
-If you use clusterProfiler in published research, please cite:
-T Wu, E Hu, S Xu, M Chen, P Guo, Z Dai, T Feng, L Zhou, W Tang, L Zhan, X Fu, S Liu, X Bo, and G Yu. clusterProfiler 4.0: A universal enrichment tool for interpreting omics data. The Innovation. 2021, 2(3):100141
- - -

-Attaching package: 'clusterProfiler'
- - -
The following object is masked from 'package:stats':
-
-    filter
- - -
# Package that contains the MSigDB gene sets in tidy format
-library(msigdbr)
- - - -
-
-

Directories and Files

-
-

Directories

- - - -
# We'll use the marker genes as GSEA input
-hodgkins_analysis_dir <- file.path("analysis", "hodgkins")
-
-# We'll create a directory to specifically hold the pathway results if it doesn't
-# exist yet
-results_dir <- file.path(hodgkins_analysis_dir, "pathway-analysis")
-if (!dir.exists(results_dir)) {
-  dir.create(results_dir, recursive = TRUE)
-}
- - - -
-
-

Input files

-

We’re going to use the version of the cluster 1 marker genes table -that has gene symbols, so we don’t have to do gene identifier conversion -again!

- - - -
input_file <- file.path(hodgkins_analysis_dir,
-                        "markers",
-                        "cluster01_markers_with_gene_symbols.tsv")
- - - -
-
-

Output files

-

We’ll save our table of GSEA results as a TSV.

- - - -
output_file <- file.path(results_dir,
-                         "cluster01_immunologic_gsea_results.tsv")
- - - -
-
-
-
-

Gene sets

-

We will use gene sets from the Molecular -Signatures Database (MSigDB) from the Broad Institute (Subramanian, Tamayo -et al. 2005). The msigdbr -package contains MSigDB datasets already in the tidy format required by -clusterProfiler and supports multiple organisms.

-

Let’s take a look at what organisms the package supports.

- - - -
msigdbr_species()
- -
- -
- - -

MSigDB contains 8 different gene set collections.

-
H: hallmark gene sets
-C1: positional gene sets
-C2: curated gene sets
-C3: motif gene sets
-C4: computational gene sets
-C5: GO gene sets
-C6: oncogenic signatures
-C7: immunologic signatures
-

Because we’re working with a sample that’s partly comprised of immune -cells, let’s use the immunologic signatures to analyze our marker genes. -These signatures are generally derived from gene expression experiments -of a broad set of cell types, perturbations, and in some cases responses -to specific vaccines (Godec et -al. 2016); they usually have an “up” and “down” component. You -can read -more about the C7 collection at MSigDB.

-

We might expect that analyzing our marker genes with this collection -could give us some information not only about cell identity or type, but -also phenotype or activation state.

-

We can retrieve only the Immunologic gene sets by specifying -category = "C7" to the msigdbr() function. -Again, we only want human gene sets here so we specify with that with -the species argument.

- - - -
hs_immunologic_df <- msigdbr(species = "Homo sapiens",
-                             category = "C7")
- - - -
-
-

Gene Set Enrichment Analysis

-

Adapted from refine.bio -examples

-

-

Figure 1. Subramanian et -al. (2005).

-

GSEA calculates a pathway-level metric, called an enrichment score -(sometimes abbreviated as ES), by ranking genes by a gene-level -statistic. This score reflects whether or not a gene set or pathway is -over-represented at the top or bottom of the gene rankings (Subramanian et -al. 2005; Yu)

-

Specifically, all genes are ranked from most positive to most -negative based on their statistic and a running sum is calculated: -Starting with the most highly ranked genes, the running sum increases -for each gene in the pathway and decreases for each gene not in the -pathway. The enrichment score for a pathway is the running sum’s maximum -deviation from zero. GSEA also assesses statistical significance of the -scores for each pathway through permutation testing. As a result, each -input pathway will have a p-value associated with it that is then -corrected for multiple hypothesis testing (Subramanian et -al. 2005; Yu).

-

The implementation of GSEA we use in here examples requires a gene -list ordered by some statistic and input gene sets. When you use -previously computed gene-level statistics with GSEA, it is called GSEA -pre-ranked.

-
-
-

Marker gene results

- - - -
markers_df <- readr::read_tsv(input_file)
- - -
Rows: 7158 Columns: 17
-── Column specification ────────────────────────────────────────────────────────
-Delimiter: "\t"
-chr  (2): ensembl_id, gene_symbol
-dbl (15): p.value, FDR, summary.logFC, logFC.2, logFC.3, logFC.4, logFC.5, l...
-
-ℹ Use `spec()` to retrieve the full column specification for this data.
-ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
- - - - - - -
head(markers_df)
- -
- -
- - -

Since this data frame of marker genes includes gene symbols, we do -not need to perform any kind of gene identifier conversion. We do, -however, need to check for duplicate gene symbols. We can accomplish -this with duplicated(), which returns a logical vector -(e.g., TRUE or FALSE). The function -sum() will count TRUE values as 1s and -FALSE as 0s, so using it with duplicated() -will count the number of duplicate values.

- - - -
sum(duplicated(markers_df$gene_symbol))
- - -
[1] 1
- - - -

Luckily this number is very low, such that we do not expect it to -impact our results much, but this will still cause a problem when we go -to run GSEA.

-
-

Removing duplicates

-

The GSEA approach requires on discriminating between genes that are -in a gene set and those that are not. Practically speaking, gene sets -are just collections of gene identifiers! When the function we use for -GSEA pre-ranked gets a list with duplicated gene identifiers, it can -produce unexpected results.

-

Compared to the total number of genes that are in our results, there -are not a lot of duplicates but we’ll still need to make a decision -about how to handle them.

-

Let’s get a vector of the duplicated gene symbols so we can use it to -explore our filtering steps.

- - - -
duplicated_gene_symbols <- markers_df |>
-  dplyr::filter(duplicated(gene_symbol)) |>
-  dplyr::pull(gene_symbol)
- - - -

Now we’ll look at the values for the duplicated gene symbols.

- - - -
markers_df |>
-  dplyr::filter(gene_symbol %in% duplicated_gene_symbols) |>
-  dplyr::arrange(gene_symbol)
- -
- -
- - -

We can see that the associated values vary for each row.

-

Let’s keep the gene symbols associated with the higher absolute value -of the gene-level statistic we’ll use for GSEA -(summary.logFC).

-

Retaining the instance of the gene symbols with the higher absolute -value of a gene-level statistic means that we will retain the value that -is likely to be more highly- or lowly-ranked or, put another way, the -values less likely to be towards the middle of the ranked gene list. We -should keep this decision in mind when interpreting our results. For -example, if all the duplicate identifiers happened to be in a particular -gene set, we may get an overly optimistic view of how perturbed that -gene set is because we preferentially selected instances of the -identifier that have a higher absolute value of the statistic used for -ranking.

-

In the next chunk, we are going to filter out the duplicated rows -using the dplyr::distinct() function after sorting by -absolute value of the summary log fold change. This will keep the first -row with the duplicated value thus keeping the row with the largest -absolute value.

- - - -
filtered_markers_df <- markers_df |>
-  # Sort so that the highest absolute values of the summary log2 fold change are
-  # at the top
-  dplyr::arrange(dplyr::desc(abs(summary.logFC))) |>
-  # Filter out the duplicated rows using `dplyr::distinct()`
-  dplyr::distinct(gene_symbol, .keep_all = TRUE)
- - - -

Let’s see what happened to our duplicate identifiers.

- - - -
# Subset to & arrange by gene symbols that were duplicated in the original
-# data frame of results
-filtered_markers_df |>
-  dplyr::filter(gene_symbol %in% duplicated_gene_symbols) |>
-  dplyr::arrange(gene_symbol)
- -
- -
- - -

Now we’re ready to prep our pre-ranked list for GSEA.

-
-
-

Pre-ranked list

-

The GSEA() function takes a pre-ranked (sorted) named -vector of statistics, where the names in the vector are gene -identifiers. This is step 1 – gene-level statistics.

-

Here, we’re using the summary log fold change, which summarizes of -all of the log fold changes from the pairwise comparisons between -cluster 1 and other clusters into a single value (ref). As -such, this statistic gives us information about the relative magnitude -and directionality of each gene’s expression in cluster 1 relative to -all other clusters.

- - - -
lfc_vector <- filtered_markers_df |>
-  # Extract a vector of `summary.logFC` named by `gene_symbol`
-  dplyr::pull(summary.logFC, name = gene_symbol)
-lfc_vector <- sort(lfc_vector, decreasing = TRUE)
- - - -

Let’s look at the top ranked values.

- - - -
# Look at first entries of the log fold change vector
-head(lfc_vector)
- - -
    MS4A1      IGHM     CD79A      CD69      CD37 TNFRSF13C 
- 3.652490  3.244756  3.037044  2.776535  2.128713  2.029677 
- - - -

And the bottom of the list.

- - - -
# Look at the last entries of the log fold change vector
-tail(lfc_vector)
- - -
    RNF19A        ID3     PDLIM1   CYB561A3   HLA-DPB1       IRF8 
--0.7687903 -0.8241992 -0.8295044 -0.9030800 -1.0339844 -1.2398847 
- - - -
-
-
-

Run GSEA

-

Now for the analysis!

-

We can use the GSEA() function to perform GSEA with any -generic set of gene sets, but there are several functions for using -specific, commonly used gene sets (e.g., gseKEGG()).

- - - -
gsea_results <- GSEA(geneList = lfc_vector,  # ordered ranked gene list
-                     minGSSize = 25,  # minimum gene set size
-                     maxGSSize = 500,  # maximum gene set set
-                     pvalueCutoff = 0.05,
-                     pAdjustMethod = "BH",  # correction for multiple hypothesis testing
-                     TERM2GENE = dplyr::select(hs_immunologic_df,
-                                               gs_name,
-                                               gene_symbol))
- - -
using 'fgsea' for GSEA analysis, please cite Korotkevich et al (2019).
- - -
preparing geneSet collections...
- - -
GSEA analysis...
- - -
leading edge analysis...
- - -
done...
- - - -

Let’s take a look at the GSEA results.

- - - -
View(gsea_results@result |>
-       dplyr::arrange(dplyr::desc(NES))
-)
- - - -

Normalized enrichment scores (NES) are enrichment scores that are -scaled to make gene sets that contain different number of genes -comparable.

-

Let’s write these results to file.

- - - -
gsea_results@result |> readr::write_tsv(output_file)
- - - -
-

Visualizing GSEA results

-

We can visualize GSEA results for individual pathways or gene sets -using enrichplot::gseaplot(). Let’s take a look at 3 -different pathways – one with a highly positive NES, one with a highly -negative NES, and one that was not a significant result – to get more -insight into how ES are calculated.

-
-

Highly Positive NES

-

Let’s take look at a pathway with a highly positive NES -(GSE29618_BCELL_VS_PDC_UP) using a GSEA plot. This gene set -is comprised of (ref):

-
-

Genes up-regulated in comparison of B cells versus plasmacytoid -dendritic cells (pDC) .

-
- - - -
enrichplot::gseaplot(gsea_results,
-                     geneSetID = "GSE29618_BCELL_VS_PDC_UP",
-                     title = "GSE29618_BCELL_VS_PDC_UP",
-                     color.line = "#0066FF")
- - -

- - - -

Notice how the genes that are in the gene set, indicated by the black -bars, tend to be on the left side of the graph indicating that they have -positive gene-level scores.

-
-
-

Highly Negative NES

-

The gene set GSE29618_BCELL_VS_PDC_DAY7_FLU_VACCINE_DN -had a highly negative NES, and is comprised of (ref)

-
-

Genes down-regulated in comparison of B cells from influenza vaccinee -at day 7 post-vaccination versus plasmacytoid dendritic cells (pDC) at -day 7 post-vaccination.

-
-

This is the “down” signature that corresponds to one of the “up” -pathways with a highly positive NES, so in a way this is not adding new -information!

- - - -
enrichplot::gseaplot(gsea_results,
-                     geneSetID = "GSE29618_BCELL_VS_PDC_DAY7_FLU_VACCINE_DN",
-                     title = "GSE29618_BCELL_VS_PDC_DAY7_FLU_VACCINE_DN",
-                     color.line = "#0066FF")
- - -

- - - -

This gene set shows the opposite pattern – genes in the pathway tend -to be on the right side of the graph.

-
-
-

A non-significant result

-

The @results slot will only show gene sets that pass the -pvalueCutoff threshold we supplied to GSEA(), -but we can plot any gene set so long as we know its name. Let’s look at -GOLDRATH_EFF_VS_MEMORY_CD8_TCELL_DN, -which examines the difference between effector CD8 T cells and memory -CD8 T cells and was not in the results we viewed earlier.

- - - -
enrichplot::gseaplot(gsea_results,
-                     geneSetID = "GOLDRATH_EFF_VS_MEMORY_CD8_TCELL_DN",
-                     title = "GOLDRATH_EFF_VS_MEMORY_CD8_TCELL_DN",
-                     color.line = "#0066FF")
- - -

- - - -

Genes in the pathway are distributed more evenly throughout the -ranked list, resulting in a more “middling” score.

-

Note: The plots returned by enrichplot::gseaplot are -ggplots, so we could use ggplot2::ggsave() to save them to -file if we wanted to.

-
-
-
-
-

Session Info

- - - -
sessionInfo()
- - -
R version 4.4.1 (2024-06-14)
-Platform: x86_64-pc-linux-gnu
-Running under: Ubuntu 22.04.4 LTS
-
-Matrix products: default
-BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
-LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0
-
-locale:
- [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
- [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
- [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
- [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
- [9] LC_ADDRESS=C               LC_TELEPHONE=C            
-[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
-
-time zone: Etc/UTC
-tzcode source: system (glibc)
-
-attached base packages:
-[1] stats     graphics  grDevices utils     datasets  methods   base     
-
-other attached packages:
-[1] msigdbr_7.5.1          clusterProfiler_4.12.0 optparse_1.7.5        
-
-loaded via a namespace (and not attached):
-  [1] DBI_1.2.2               gson_0.1.0              shadowtext_0.1.3       
-  [4] gridExtra_2.3           rlang_1.1.3             magrittr_2.0.3         
-  [7] DOSE_3.30.0             compiler_4.4.1          RSQLite_2.3.6          
- [10] png_0.1-8               vctrs_0.6.5             reshape2_1.4.4         
- [13] stringr_1.5.1           pkgconfig_2.0.3         crayon_1.5.2           
- [16] fastmap_1.1.1           XVector_0.44.0          labeling_0.4.3         
- [19] ggraph_2.2.1            utf8_1.2.4              HDO.db_0.99.1          
- [22] rmarkdown_2.26          tzdb_0.4.0              enrichplot_1.24.0      
- [25] UCSC.utils_1.0.0        purrr_1.0.2             bit_4.0.5              
- [28] xfun_0.43               zlibbioc_1.50.0         cachem_1.0.8           
- [31] aplot_0.2.2             GenomeInfoDb_1.40.0     jsonlite_1.8.8         
- [34] blob_1.2.4              highr_0.10              BiocParallel_1.38.0    
- [37] tweenr_2.0.3            parallel_4.4.1          R6_2.5.1               
- [40] bslib_0.7.0             stringi_1.8.3           RColorBrewer_1.1-3     
- [43] jquerylib_0.1.4         GOSemSim_2.30.0         Rcpp_1.0.12            
- [46] knitr_1.46              readr_2.1.5             IRanges_2.38.0         
- [49] Matrix_1.7-0            splines_4.4.1           igraph_2.0.3           
- [52] tidyselect_1.2.1        qvalue_2.36.0           yaml_2.3.8             
- [55] viridis_0.6.5           codetools_0.2-20        lattice_0.22-6         
- [58] tibble_3.2.1            plyr_1.8.9              treeio_1.28.0          
- [61] Biobase_2.64.0          withr_3.0.0             KEGGREST_1.44.0        
- [64] evaluate_0.23           gridGraphics_0.5-1      scatterpie_0.2.2       
- [67] getopt_1.20.4           polyclip_1.10-6         Biostrings_2.72.0      
- [70] ggtree_3.12.0           pillar_1.9.0            stats4_4.4.1           
- [73] ggfun_0.1.4             generics_0.1.3          vroom_1.6.5            
- [76] hms_1.1.3               S4Vectors_0.42.0        ggplot2_3.5.1          
- [79] tidytree_0.4.6          munsell_0.5.1           scales_1.3.0           
- [82] glue_1.7.0              lazyeval_0.2.2          tools_4.4.1            
- [85] data.table_1.15.4       fgsea_1.30.0            babelgene_22.9         
- [88] fs_1.6.4                graphlayouts_1.1.1      fastmatch_1.1-4        
- [91] tidygraph_1.3.1         cowplot_1.1.3           grid_4.4.1             
- [94] ape_5.8                 tidyr_1.3.1             AnnotationDbi_1.66.0   
- [97] colorspace_2.1-0        nlme_3.1-164            patchwork_1.2.0        
-[100] GenomeInfoDbData_1.2.12
- [ reached getOption("max.print") -- omitted 20 entries ]
- - -
- -
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcyAoR1NFQSkiCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjEKLS0tCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOgoKLSBQcmVwYXJlIHRhYnVsYXIgZGF0YSBvZiBnZW5lLWxldmVsIHN0YXRpc3RpY3MgZm9yIHVzZSB3aXRoIEdlbmUgU2V0IEVucmljaG1lbnQgQW5hbHlzaXMgKEdTRUEpCi0gQWNjZXNzIFtNb2xlY3VsYXIgU2lnbmF0dXJlcyBEYXRhYmFzZSBnZW5lIHNldCBjb2xsZWN0aW9uc10oaHR0cHM6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2NvbGxlY3Rpb25zLmpzcCkgdmlhIHRoZSBgbXNpZ2RicmAgcGFja2FnZQotIFBlcmZvcm0gR1NFQSB3aXRoIHRoZSBgY2x1c3RlclByb2ZpbGVyYCBwYWNrYWdlCi0gVmlzdWFsaXplIEdTRUEgcmVzdWx0cyB3aXRoIHRoZSBgZW5yaWNocGxvdGAgcGFja2FnZQoKLS0tCgpJbiB0aGlzIG5vdGVib29rLCB3ZSdsbCBhbmFseXplIHRoZSBtYXJrZXIgZ2VuZXMgZnJvbSBjbHVzdGVyIDEsIGp1c3QgYXMgd2UgZGlkIGluIHRoZSBwcmV2aW91cyBub3RlYm9vay4KVW5saWtlIE9SQSwgR1NFQSBhbGxvd3MgdXMgdG8gdXNlIHRoZSBmdWxsIGxpc3Qgb2YgZ2VuZXMgd2UgY291bGQgcmVsaWFibHkgbWVhc3VyZSwgcmF0aGVyIHRoYW4gcGlja2luZyBzb21lIGN1dG9mZiBvdXJzZWx2ZXMuCgpHU0VBIGlzIGEgZnVuY3Rpb25hbCBjbGFzcyBzY29yaW5nIChGQ1MpIGFwcHJvYWNoIHRvIHBhdGh3YXkgYW5hbHlzaXMgdGhhdCB3YXMgZmlyc3QgaW50cm9kdWNlZCBpbiBbU3VicmFtYW5pYW4gX2V0IGFsLl8gKDIwMDUpXShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpLgpUaGUgcmF0aW9uYWxlIGJlaGluZCBGQ1MgYXBwcm9hY2hlcyBpcyB0aGF0IHNtYWxsIGNoYW5nZXMgaW4gaW5kaXZpZHVhbCBnZW5lcyB0aGF0IHBhcnRpY2lwYXRlIGluIHRoZSBzYW1lIGJpb2xvZ2ljYWwgcHJvY2VzcyBvciBwYXRod2F5IGNhbiBiZSBzaWduaWZpY2FudCBhbmQgb2YgYmlvbG9naWNhbCBpbnRlcmVzdC4KRkNTIG1ldGhvZHMgYXJlIGJldHRlciBzdWl0ZWQgZm9yIGlkZW50aWZ5aW5nIHRoZXNlIHBhdGh3YXlzIHRoYXQgc2hvdyBjb29yZGluYXRlZCBjaGFuZ2VzIHRoYW4gT1JBLgpJbiBPUkEsIHdlIHBpY2sgYSBjdXRvZmYgdGhhdCBfdHlwaWNhbGx5XyBvbmx5IGNhcHR1cmVzIGdlbmVzIHdpdGggbGFyZ2UgaW5kaXZpZHVhbCBjaGFuZ2VzLgoKVGhlcmUgYXJlIDMgZ2VuZXJhbCBzdGVwcyBpbiBGQ1MgbWV0aG9kcyAoW0toYXRyaSBfZXQgYWwuXyAyMDEyXSggaHR0cHM6Ly9kb2kub3JnLzEwLjEzNzEvam91cm5hbC5wY2JpLjEwMDIzNzUpKToKCjEuIENhbGN1bGF0ZSBhIGdlbmUtbGV2ZWwgc3RhdGlzdGljIChoZXJlLCB3ZSdsbCB1c2UgdGhlIHN1bW1hcnkgbG9nIGZvbGQgY2hhbmdlcyBpbiBvdXIgbWFya2VyIGdlbmVzIHRhYmxlKQoyLiBHZW5lLWxldmVsIHN0YXRpc3RpY3MgYXJlIGFnZ3JlZ2F0ZWQgaW50byBhIHBhdGh3YXktbGV2ZWwgc3RhdGlzdGljCjMuIEFzc2VzcyB0aGUgc3RhdGlzdGljYWwgc2lnbmlmaWNhbmNlIG9mIHRoZSBwYXRod2F5LWxldmVsIHN0YXRpc3RpYwoKV2UnbGwgbm90ZSBoZXJlIHRoYXQgR1NFQSB3YXMgZGVzaWduZWQgdG8gZGV0ZWN0IHNtYWxsIGNvb3JkaW5hdGVkIGNoYW5nZXMgYmV0d2VlbiBfY29uZGl0aW9uc18gaW4gKG5lY2Vzc2FyaWx5IGJ1bGshKSBtaWNyb2FycmF5IGRhdGEuCkl0IG1heSBiZSBtb3JlIGRpZmZpY3VsdCB0byBzdXNzIG91dCBzbWFsbCBjb29yZGluYXRlZCBjaGFuZ2VzIHdoZW4gd2UgYXNzYXkgaW5kaXZpZHVhbCBjZWxscywgZHVlIHRvIHRlY2huaWNhbCBvciBiaW9sb2dpY2FsIGRyb3BvdXQuCgpJbmRpdmlkdWFsIGNlbGxzIG1heSBhbHNvIGRpZmZlcmVudGx5IHNpemVkIChbQmxhc2kgKmV0IGFsLiogMjAxN10oaHR0cHM6Ly9kb2kub3JnLzEwLjEwODgvMTQ3OC0zOTc1L2FhNjA5YSkpLCBiZSBhdCBkaWZmZXJlbnQgc3RhZ2VzIG9mIHRoZSBjZWxsIGN5Y2xlIChbU2F0aWphIGxhYl0oaHR0cHM6Ly9zYXRpamFsYWIub3JnL3NldXJhdC9hcmNoaXZlL3YzLjEvY2VsbF9jeWNsZV92aWduZXR0ZS5odG1sKSksIG9yIGV4cGVyaWVuY2luZyBkaWZmZXJlbnQgbGV2ZWxzIG9mIGNlbGx1bGFyIHN0cmVzcyAoW0x1ZWNrZW4gYW5kIFRoZWlzLiAyMDE5XShodHRwczovL2RvaS5vcmcvMTAuMTUyNTIvbXNiLjIwMTg4NzQ2KSksIHNvbWUgb2Ygd2hpY2ggbWF5IGJlIGludGVyZXN0aW5nIGluIHRoZSBjb250ZXh0IG9mIGNhbmNlciBhbmQgdGhlcmVmb3JlIHNob3VsZCBub3QgYmUgY29ycmVjdGVkIGZvci4KKEluIHByYWN0aWNlLCBpdCBpcyBjb21tb24gdG8gY29ycmVjdCBmb3IgY2VsbCBjeWNsZSBzdGFnZSBidXQgd2UgaGF2ZSBub3QgZG9uZSB0aGF0IHVwc3RyZWFtLikKVGhlc2UgYXNwZWN0cyBhbHNvIGhhdmUgdGhlIHBvdGVudGlhbCB0byBjbG91ZCBvdXIgYWJpbGl0eSB0byBkZXRlY3Qgc21hbGwgY29vcmRpbmF0ZWQgY2hhbmdlcyBpbiBvdGhlciBwYXRod2F5cy4KClRoZXJlIGlzIHNvbWUgbGl0ZXJhdHVyZSB0aGF0IHN1Z2dlc3RzIHRoYXQgZHJvcG91dCAoZXhjZXNzaXZlIHplcm9zKSBvYnNlcnZlZCBpbiBzaW5nbGUgY2VsbCBkYXRhIGlzIGEgZnVuY3Rpb24gb2YgY2VsbCB0eXBlIGhldGVyb2dlbmVpdHkgKFtLaW0gKmV0IGFsLiogMjAyMF0oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvczEzMDU5LTAyMC0wMjA5Ni15KSkuCldlJ3JlIHVzaW5nIEdTRUEgdG8gY29tcGFyZSBjbHVzdGVycyBvZiBjZWxscywgd2hpY2ggd2UgKGlkZWFsbHkpIGV4cGVjdCB0byBiZSBkaWZmZXJlbnQgYmlvbG9naWNhbCBzdGF0ZXMgYW5kL29yIGNlbGwgdHlwZXMuCklmIG1hbnkgZ2VuZXMgImRyb3Agb3V0IiBiZXR3ZWVuIGNlbGwgdHlwZXMsIHdlIG1heSBleHBlY3Qgb3VyIHN0YXRpc3RpY3MgdXNlZCBhcyBpbnB1dCB0byBHU0VBIHRvIGJlIG5vaXN5LgoKV2l0aCBhbGwgdGhpcyBpbiBtaW5kLCBpdCBtYXkgYmUgdW5zdXJwcmlzaW5nIHRoYXQgT1JBLCB3aGljaCBnZW5lcmFsbHkgY2FwdHVyZXMgZ2VuZXMgd2l0aCBsYXJnZSBpbmRpdmlkdWFsIGNoYW5nZXMgYW5kIGxvd2VyIHdpdGhpbi1ncm91cCB2YXJpYXRpb24gKHRoZSBtb3N0ICJzdGF0aXN0aWNhbGx5IHNpZ25pZmljYW50IiBnZW5lcyksIGlzIHdlbGwtc3VpdGVkIHRvIGFuYWx5emluZyBzY1JOQS1zZXEgZGF0YSBhbmQgdGhlIGFkdmFudGFnZXMgb2YgR1NFQSBpbiBhIGJ1bGsgYW5hbHlzaXMgc2V0dGluZyBtYXkgbm90IGJlIGZ1bGx5IGF2YWlsYWJsZSBoZXJlLgpOb25ldGhlbGVzcywgR1NFQSBpcyB1c2VkIGZvciBwYXRod2F5IGFuYWx5c2lzIG9mIHNjUk5BLXNlcSBkYXRhLgoKTWV0aG9kcyBzcGVjaWZpY2FsbHkgZm9yIHBhdGh3YXkgYW5hbHlzaXMgb2Ygc2NSTkEtc2VxIChlLmcuLCBbTWEgKmV0IGFsLiogMjAyMF0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwMzgvczQxNDY3LTAyMC0xNTI5OC02KSkgYXJlIGJlaW5nIGRldmVsb3BlZCwgc28gc3RheSB0dW5lZCEKCiMjIyMgT3RoZXIgcmVzb3VyY2VzCgoqIEZvciBhbm90aGVyIGV4YW1wbGUgdXNpbmcgYGNsdXN0ZXJQcm9maWxlcmAgZm9yIEdTRUEsIHNlZSBbX0ludHJvIHRvIERHRTogRnVuY3Rpb25hbCBBbmFseXNpcy5fIGZyb20gSGFydmFyZCBDaGFuIEJpb2luZm9ybWF0aWNzIENvcmUgVHJhaW5pbmcuXShodHRwczovL2hiY3RyYWluaW5nLmdpdGh1Yi5pby9ER0Vfd29ya3Nob3AvbGVzc29ucy8wOV9mdW5jdGlvbmFsX2FuYWx5c2lzLmh0bWwpCiogVGhlIHdheSB3ZSdsbCB1c2UgYGNsdXN0ZXJQcm9maWxlcmAgaGVyZSB1c2VzIGBmZ3NlYWAgKEZhc3QgR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcykgdW5kZXIgdGhlIGhvb2QuCllvdSBjYW4gcmVhZCBtb3JlIGFib3V0IGZnc2VhIGluIFtLb3JvdGtldmljaCBfZXQgYWwuXyAoMjAyMSldKGh0dHBzOi8vZG9pLm9yZy8xMC4xMTAxLzA2MDAxMikuCiogW3JlZmluZS5iaW8gZXhhbXBsZXMgR2VuZSBzZXQgZW5yaWNobWVudCBhbmFseXNpcyAtIFJOQS1zZXFdKGh0dHBzOi8vYWxleHNsZW1vbmFkZS5naXRodWIuaW8vcmVmaW5lYmlvLWV4YW1wbGVzLzAzLXJuYXNlcS9wYXRod2F5LWFuYWx5c2lzX3JuYXNlcV8wMl9nc2VhLmh0bWwpIGZyb20gd2hpY2ggdGhpcyBtYXRlcmlhbCBoYXMgYmVlbiBhZGFwdGVkLgoKIyMgU2V0IHVwCgojIyMgTGlicmFyaWVzCgpgYGB7ciBsaWJyYXJpZXN9CiMgUGFja2FnZSB0byBydW4gR1NFQQpsaWJyYXJ5KGNsdXN0ZXJQcm9maWxlcikKIyBQYWNrYWdlIHRoYXQgY29udGFpbnMgdGhlIE1TaWdEQiBnZW5lIHNldHMgaW4gdGlkeSBmb3JtYXQKbGlicmFyeShtc2lnZGJyKQpgYGAKCiMjIyBEaXJlY3RvcmllcyBhbmQgRmlsZXMKCiMjIyMgRGlyZWN0b3JpZXMKCmBgYHtyIGNyZWF0ZV9kaXIsIGxpdmUgPSBUUlVFfQojIFdlJ2xsIHVzZSB0aGUgbWFya2VyIGdlbmVzIGFzIEdTRUEgaW5wdXQKaG9kZ2tpbnNfYW5hbHlzaXNfZGlyIDwtIGZpbGUucGF0aCgiYW5hbHlzaXMiLCAiaG9kZ2tpbnMiKQoKIyBXZSdsbCBjcmVhdGUgYSBkaXJlY3RvcnkgdG8gc3BlY2lmaWNhbGx5IGhvbGQgdGhlIHBhdGh3YXkgcmVzdWx0cyBpZiBpdCBkb2Vzbid0CiMgZXhpc3QgeWV0CnJlc3VsdHNfZGlyIDwtIGZpbGUucGF0aChob2Rna2luc19hbmFseXNpc19kaXIsICJwYXRod2F5LWFuYWx5c2lzIikKaWYgKCFkaXIuZXhpc3RzKHJlc3VsdHNfZGlyKSkgewogIGRpci5jcmVhdGUocmVzdWx0c19kaXIsIHJlY3Vyc2l2ZSA9IFRSVUUpCn0KYGBgCgojIyMjIElucHV0IGZpbGVzCgpXZSdyZSBnb2luZyB0byB1c2UgdGhlIHZlcnNpb24gb2YgdGhlIGNsdXN0ZXIgMSBtYXJrZXIgZ2VuZXMgdGFibGUgdGhhdCBoYXMgZ2VuZSBzeW1ib2xzLCBzbyB3ZSBkb24ndCBoYXZlIHRvIGRvIGdlbmUgaWRlbnRpZmllciBjb252ZXJzaW9uIGFnYWluIQoKYGBge3IgaW5wdXRfZmlsZXN9CmlucHV0X2ZpbGUgPC0gZmlsZS5wYXRoKGhvZGdraW5zX2FuYWx5c2lzX2RpciwKICAgICAgICAgICAgICAgICAgICAgICAgIm1hcmtlcnMiLAogICAgICAgICAgICAgICAgICAgICAgICAiY2x1c3RlcjAxX21hcmtlcnNfd2l0aF9nZW5lX3N5bWJvbHMudHN2IikKYGBgCgojIyMjIE91dHB1dCBmaWxlcwoKV2UnbGwgc2F2ZSBvdXIgdGFibGUgb2YgR1NFQSByZXN1bHRzIGFzIGEgVFNWLgoKYGBge3Igb3V0cHV0X2ZpbGVzfQpvdXRwdXRfZmlsZSA8LSBmaWxlLnBhdGgocmVzdWx0c19kaXIsCiAgICAgICAgICAgICAgICAgICAgICAgICAiY2x1c3RlcjAxX2ltbXVub2xvZ2ljX2dzZWFfcmVzdWx0cy50c3YiKQpgYGAKCiMjIEdlbmUgc2V0cwoKV2Ugd2lsbCB1c2UgZ2VuZSBzZXRzIGZyb20gdGhlIFtNb2xlY3VsYXIgU2lnbmF0dXJlcyBEYXRhYmFzZSAoTVNpZ0RCKV0oaHR0cHM6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2luZGV4LmpzcCkgZnJvbSB0aGUgQnJvYWQgSW5zdGl0dXRlIChbU3VicmFtYW5pYW4sIFRhbWF5byAqZXQgYWwuKiAyMDA1XShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpKS4KVGhlIFtgbXNpZ2RicmBdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy9tc2lnZGJyL2luZGV4Lmh0bWwpIHBhY2thZ2UgY29udGFpbnMgTVNpZ0RCIGRhdGFzZXRzIGFscmVhZHkgaW4gdGhlIHRpZHkgZm9ybWF0IHJlcXVpcmVkIGJ5IGBjbHVzdGVyUHJvZmlsZXJgIGFuZCBzdXBwb3J0cyBtdWx0aXBsZSBvcmdhbmlzbXMuCgpMZXQncyB0YWtlIGEgbG9vayBhdCB3aGF0IG9yZ2FuaXNtcyB0aGUgcGFja2FnZSBzdXBwb3J0cy4KCmBgYHtyIHNob3dfc3BlY2llc30KbXNpZ2Ricl9zcGVjaWVzKCkKYGBgCgpNU2lnREIgY29udGFpbnMgOCBkaWZmZXJlbnQgZ2VuZSBzZXQgY29sbGVjdGlvbnMuCgogICAgSDogaGFsbG1hcmsgZ2VuZSBzZXRzCiAgICBDMTogcG9zaXRpb25hbCBnZW5lIHNldHMKICAgIEMyOiBjdXJhdGVkIGdlbmUgc2V0cwogICAgQzM6IG1vdGlmIGdlbmUgc2V0cwogICAgQzQ6IGNvbXB1dGF0aW9uYWwgZ2VuZSBzZXRzCiAgICBDNTogR08gZ2VuZSBzZXRzCiAgICBDNjogb25jb2dlbmljIHNpZ25hdHVyZXMKICAgIEM3OiBpbW11bm9sb2dpYyBzaWduYXR1cmVzCgoKQmVjYXVzZSB3ZSdyZSB3b3JraW5nIHdpdGggYSBzYW1wbGUgdGhhdCdzIHBhcnRseSBjb21wcmlzZWQgb2YgaW1tdW5lIGNlbGxzLCBsZXQncyB1c2UgdGhlIGltbXVub2xvZ2ljIHNpZ25hdHVyZXMgdG8gYW5hbHl6ZSBvdXIgbWFya2VyIGdlbmVzLgpUaGVzZSBzaWduYXR1cmVzIGFyZSBnZW5lcmFsbHkgZGVyaXZlZCBmcm9tIGdlbmUgZXhwcmVzc2lvbiBleHBlcmltZW50cyBvZiBhIGJyb2FkIHNldCBvZiBjZWxsIHR5cGVzLCBwZXJ0dXJiYXRpb25zLCBhbmQgaW4gc29tZSBjYXNlcyByZXNwb25zZXMgdG8gc3BlY2lmaWMgdmFjY2luZXMgKFtHb2RlYyAqZXQgYWwuKiAyMDE2XShodHRwczovL2RvaS5vcmcvMTAuMTAxNi9qLmltbXVuaS4yMDE1LjEyLjAwNikpOyB0aGV5IHVzdWFsbHkgaGF2ZSBhbiAidXAiIGFuZCAiZG93biIgY29tcG9uZW50LgpZb3UgY2FuIFtyZWFkIG1vcmUgYWJvdXQgdGhlIEM3IGNvbGxlY3Rpb24gYXQgTVNpZ0RCXShodHRwczovL3d3dy5nc2VhLW1zaWdkYi5vcmcvZ3NlYS9tc2lnZGIvY29sbGVjdGlvbl9kZXRhaWxzLmpzcCNDNykuCgpXZSBtaWdodCBleHBlY3QgdGhhdCBhbmFseXppbmcgb3VyIG1hcmtlciBnZW5lcyB3aXRoIHRoaXMgY29sbGVjdGlvbiBjb3VsZCBnaXZlIHVzIHNvbWUgaW5mb3JtYXRpb24gbm90IG9ubHkgYWJvdXQgY2VsbCBpZGVudGl0eSBvciB0eXBlLCBidXQgYWxzbyBwaGVub3R5cGUgb3IgYWN0aXZhdGlvbiBzdGF0ZS4KCldlIGNhbiByZXRyaWV2ZSBvbmx5IHRoZSBJbW11bm9sb2dpYyBnZW5lIHNldHMgYnkgc3BlY2lmeWluZyBgY2F0ZWdvcnkgPSAiQzciYCB0byB0aGUgYG1zaWdkYnIoKWAgZnVuY3Rpb24uCkFnYWluLCB3ZSBvbmx5IHdhbnQgaHVtYW4gZ2VuZSBzZXRzIGhlcmUgc28gd2Ugc3BlY2lmeSB3aXRoIHRoYXQgd2l0aCB0aGUgYHNwZWNpZXNgIGFyZ3VtZW50LgoKYGBge3IgaW1tdW5vbG9naWNfc2V0cywgbGl2ZSA9IFRSVUV9CmhzX2ltbXVub2xvZ2ljX2RmIDwtIG1zaWdkYnIoc3BlY2llcyA9ICJIb21vIHNhcGllbnMiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNhdGVnb3J5ID0gIkM3IikKYGBgCgojIyBHZW5lIFNldCBFbnJpY2htZW50IEFuYWx5c2lzCgpfQWRhcHRlZCBmcm9tIFtyZWZpbmUuYmlvIGV4YW1wbGVzXShodHRwczovL2dpdGh1Yi5jb20vQWxleHNMZW1vbmFkZS9yZWZpbmViaW8tZXhhbXBsZXMvYmxvYi8zM2NkZWZmNjZkNTdmOWZlOGVlNGZjYjUxNTZhZWE0YWMyZGNlMDdmLzAzLXJuYXNlcS9wYXRod2F5LWFuYWx5c2lzX3JuYXNlcV8wMl9nc2VhLlJtZClfCgohW10oZGlhZ3JhbXMvc3VicmFtYW5pYW5fZmlnMS5qcGcpCgoqKkZpZ3VyZSAxLiBbU3VicmFtYW5pYW4gX2V0IGFsLl8gKDIwMDUpXShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpLioqCgpHU0VBIGNhbGN1bGF0ZXMgYSBwYXRod2F5LWxldmVsIG1ldHJpYywgY2FsbGVkIGFuIGVucmljaG1lbnQgc2NvcmUgKHNvbWV0aW1lcyBhYmJyZXZpYXRlZCBhcyBFUyksIGJ5IHJhbmtpbmcgZ2VuZXMgYnkgYSBnZW5lLWxldmVsIHN0YXRpc3RpYy4KVGhpcyBzY29yZSByZWZsZWN0cyB3aGV0aGVyIG9yIG5vdCBhIGdlbmUgc2V0IG9yIHBhdGh3YXkgaXMgb3Zlci1yZXByZXNlbnRlZCBhdCB0aGUgdG9wIG9yIGJvdHRvbSBvZiB0aGUgZ2VuZSByYW5raW5ncyAoW1N1YnJhbWFuaWFuIF9ldCBhbC5fIDIwMDVdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMDczL3BuYXMuMDUwNjU4MDEwMik7IFtZdV0oaHR0cDovL3l1bGFiLXNtdS50b3AvY2x1c3RlclByb2ZpbGVyLWJvb2svY2hhcHRlcjIuaHRtbCNnZW5lLXNldC1lbnJpY2htZW50LWFuYWx5c2lzKSkKClNwZWNpZmljYWxseSwgYWxsIGdlbmVzIGFyZSByYW5rZWQgZnJvbSBtb3N0IHBvc2l0aXZlIHRvIG1vc3QgbmVnYXRpdmUgYmFzZWQgb24gdGhlaXIgc3RhdGlzdGljIGFuZCBhIHJ1bm5pbmcgc3VtIGlzIGNhbGN1bGF0ZWQ6ClN0YXJ0aW5nIHdpdGggdGhlIG1vc3QgaGlnaGx5IHJhbmtlZCBnZW5lcywgdGhlIHJ1bm5pbmcgc3VtIGluY3JlYXNlcyBmb3IgZWFjaCBnZW5lIGluIHRoZSBwYXRod2F5IGFuZCBkZWNyZWFzZXMgZm9yIGVhY2ggZ2VuZSBub3QgaW4gdGhlIHBhdGh3YXkuClRoZSBlbnJpY2htZW50IHNjb3JlIGZvciBhIHBhdGh3YXkgaXMgdGhlIHJ1bm5pbmcgc3VtJ3MgbWF4aW11bSBkZXZpYXRpb24gZnJvbSB6ZXJvLgpHU0VBIGFsc28gYXNzZXNzZXMgc3RhdGlzdGljYWwgc2lnbmlmaWNhbmNlIG9mIHRoZSBzY29yZXMgZm9yIGVhY2ggcGF0aHdheSB0aHJvdWdoIHBlcm11dGF0aW9uIHRlc3RpbmcuCkFzIGEgcmVzdWx0LCBlYWNoIGlucHV0IHBhdGh3YXkgd2lsbCBoYXZlIGEgcC12YWx1ZSBhc3NvY2lhdGVkIHdpdGggaXQgdGhhdCBpcyB0aGVuIGNvcnJlY3RlZCBmb3IgbXVsdGlwbGUgaHlwb3RoZXNpcyB0ZXN0aW5nIChbU3VicmFtYW5pYW4gX2V0IGFsLl8gMjAwNV0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwNzMvcG5hcy4wNTA2NTgwMTAyKTsgW1l1XShodHRwOi8veXVsYWItc211LnRvcC9jbHVzdGVyUHJvZmlsZXItYm9vay9jaGFwdGVyMi5odG1sI2dlbmUtc2V0LWVucmljaG1lbnQtYW5hbHlzaXMpKS4KClRoZSBpbXBsZW1lbnRhdGlvbiBvZiBHU0VBIHdlIHVzZSBpbiBoZXJlIGV4YW1wbGVzIHJlcXVpcmVzIGEgZ2VuZSBsaXN0IG9yZGVyZWQgYnkgc29tZSBzdGF0aXN0aWMgYW5kIGlucHV0IGdlbmUgc2V0cy4KV2hlbiB5b3UgdXNlIHByZXZpb3VzbHkgY29tcHV0ZWQgZ2VuZS1sZXZlbCBzdGF0aXN0aWNzIHdpdGggR1NFQSwgaXQgaXMgY2FsbGVkIEdTRUEgcHJlLXJhbmtlZC4KCiMjIE1hcmtlciBnZW5lIHJlc3VsdHMKCmBgYHtyIHJlYWRfaW5fbWFya2VycywgbGl2ZSA9IFRSVUV9Cm1hcmtlcnNfZGYgPC0gcmVhZHI6OnJlYWRfdHN2KGlucHV0X2ZpbGUpCmBgYAoKYGBge3J9CmhlYWQobWFya2Vyc19kZikKYGBgCgpTaW5jZSB0aGlzIGRhdGEgZnJhbWUgb2YgbWFya2VyIGdlbmVzIGluY2x1ZGVzIGdlbmUgc3ltYm9scywgd2UgZG8gbm90IG5lZWQgdG8gcGVyZm9ybSBhbnkga2luZCBvZiBnZW5lIGlkZW50aWZpZXIgY29udmVyc2lvbi4KV2UgZG8sIGhvd2V2ZXIsIG5lZWQgdG8gY2hlY2sgZm9yIGR1cGxpY2F0ZSBnZW5lIHN5bWJvbHMuCldlIGNhbiBhY2NvbXBsaXNoIHRoaXMgd2l0aCBgZHVwbGljYXRlZCgpYCwgd2hpY2ggcmV0dXJucyBhIGxvZ2ljYWwgdmVjdG9yIChlLmcuLCBgVFJVRWAgb3IgYEZBTFNFYCkuClRoZSBmdW5jdGlvbiBgc3VtKClgIHdpbGwgY291bnQgYFRSVUVgIHZhbHVlcyBhcyAxcyBhbmQgYEZBTFNFYCBhcyAwcywgc28gdXNpbmcgaXQgd2l0aCBgZHVwbGljYXRlZCgpYCB3aWxsIGNvdW50IHRoZSBudW1iZXIgb2YgZHVwbGljYXRlIHZhbHVlcy4KCmBgYHtyIGFueV9kdXBsaWNhdGVkLCBsaXZlID0gVFJVRX0Kc3VtKGR1cGxpY2F0ZWQobWFya2Vyc19kZiRnZW5lX3N5bWJvbCkpCmBgYAoKTHVja2lseSB0aGlzIG51bWJlciBpcyB2ZXJ5IGxvdywgc3VjaCB0aGF0IHdlIGRvIG5vdCBleHBlY3QgaXQgdG8gaW1wYWN0IG91ciByZXN1bHRzIG11Y2gsIGJ1dCB0aGlzIHdpbGwgc3RpbGwgY2F1c2UgYSBwcm9ibGVtIHdoZW4gd2UgZ28gdG8gcnVuIEdTRUEuCgojIyMgUmVtb3ZpbmcgZHVwbGljYXRlcwoKVGhlIEdTRUEgYXBwcm9hY2ggcmVxdWlyZXMgb24gZGlzY3JpbWluYXRpbmcgYmV0d2VlbiBnZW5lcyB0aGF0IGFyZSBpbiBhIGdlbmUgc2V0IGFuZCB0aG9zZSB0aGF0IGFyZSBub3QuClByYWN0aWNhbGx5IHNwZWFraW5nLCBnZW5lIHNldHMgYXJlIGp1c3QgY29sbGVjdGlvbnMgb2YgZ2VuZSBpZGVudGlmaWVycyEKV2hlbiB0aGUgZnVuY3Rpb24gd2UgdXNlIGZvciBHU0VBIHByZS1yYW5rZWQgZ2V0cyBhIGxpc3Qgd2l0aCBkdXBsaWNhdGVkIGdlbmUgaWRlbnRpZmllcnMsIGl0IGNhbiBwcm9kdWNlIHVuZXhwZWN0ZWQgcmVzdWx0cy4KCkNvbXBhcmVkIHRvIHRoZSB0b3RhbCBudW1iZXIgb2YgZ2VuZXMgdGhhdCBhcmUgaW4gb3VyIHJlc3VsdHMsIHRoZXJlIGFyZSBub3QgYSBsb3Qgb2YgZHVwbGljYXRlcyBidXQgd2UnbGwgc3RpbGwgbmVlZCB0byBtYWtlIGEgZGVjaXNpb24gYWJvdXQgaG93IHRvIGhhbmRsZSB0aGVtLgoKTGV0J3MgZ2V0IGEgdmVjdG9yIG9mIHRoZSBkdXBsaWNhdGVkIGdlbmUgc3ltYm9scyBzbyB3ZSBjYW4gdXNlIGl0IHRvIGV4cGxvcmUgb3VyIGZpbHRlcmluZyBzdGVwcy4KCmBgYHtyIGdlbmVfZHVwcywgbGl2ZSA9IFRSVUV9CmR1cGxpY2F0ZWRfZ2VuZV9zeW1ib2xzIDwtIG1hcmtlcnNfZGYgfD4KICBkcGx5cjo6ZmlsdGVyKGR1cGxpY2F0ZWQoZ2VuZV9zeW1ib2wpKSB8PgogIGRwbHlyOjpwdWxsKGdlbmVfc3ltYm9sKQpgYGAKCk5vdyB3ZSdsbCBsb29rIGF0IHRoZSB2YWx1ZXMgZm9yIHRoZSBkdXBsaWNhdGVkIGdlbmUgc3ltYm9scy4KCmBgYHtyIHNob3dfZ2VuZV9kdXBzfQptYXJrZXJzX2RmIHw+CiAgZHBseXI6OmZpbHRlcihnZW5lX3N5bWJvbCAlaW4lIGR1cGxpY2F0ZWRfZ2VuZV9zeW1ib2xzKSB8PgogIGRwbHlyOjphcnJhbmdlKGdlbmVfc3ltYm9sKQpgYGAKCldlIGNhbiBzZWUgdGhhdCB0aGUgYXNzb2NpYXRlZCB2YWx1ZXMgdmFyeSBmb3IgZWFjaCByb3cuCgpMZXQncyBrZWVwIHRoZSBnZW5lIHN5bWJvbHMgYXNzb2NpYXRlZCB3aXRoIHRoZSBoaWdoZXIgYWJzb2x1dGUgdmFsdWUgb2YgdGhlIGdlbmUtbGV2ZWwgc3RhdGlzdGljIHdlJ2xsIHVzZSBmb3IgR1NFQSAoYHN1bW1hcnkubG9nRkNgKS4KClJldGFpbmluZyB0aGUgaW5zdGFuY2Ugb2YgdGhlIGdlbmUgc3ltYm9scyB3aXRoIHRoZSBoaWdoZXIgYWJzb2x1dGUgdmFsdWUgb2YgYSBnZW5lLWxldmVsIHN0YXRpc3RpYyBtZWFucyB0aGF0IHdlIHdpbGwgcmV0YWluIHRoZSB2YWx1ZSB0aGF0IGlzIGxpa2VseSB0byBiZSBtb3JlIGhpZ2hseS0gb3IgbG93bHktcmFua2VkIG9yLCBwdXQgYW5vdGhlciB3YXksIHRoZSB2YWx1ZXMgbGVzcyBsaWtlbHkgdG8gYmUgdG93YXJkcyB0aGUgbWlkZGxlIG9mIHRoZSByYW5rZWQgZ2VuZSBsaXN0LgpXZSBzaG91bGQga2VlcCB0aGlzIGRlY2lzaW9uIGluIG1pbmQgd2hlbiBpbnRlcnByZXRpbmcgb3VyIHJlc3VsdHMuCkZvciBleGFtcGxlLCBpZiBhbGwgdGhlIGR1cGxpY2F0ZSBpZGVudGlmaWVycyBoYXBwZW5lZCB0byBiZSBpbiBhIHBhcnRpY3VsYXIgZ2VuZSBzZXQsIHdlIG1heSBnZXQgYW4gb3Zlcmx5IG9wdGltaXN0aWMgdmlldyBvZiBob3cgcGVydHVyYmVkIHRoYXQgZ2VuZSBzZXQgaXMgYmVjYXVzZSB3ZSBwcmVmZXJlbnRpYWxseSBzZWxlY3RlZCBpbnN0YW5jZXMgb2YgdGhlIGlkZW50aWZpZXIgdGhhdCBoYXZlIGEgaGlnaGVyIGFic29sdXRlIHZhbHVlIG9mIHRoZSBzdGF0aXN0aWMgdXNlZCBmb3IgcmFua2luZy4KCkluIHRoZSBuZXh0IGNodW5rLCB3ZSBhcmUgZ29pbmcgdG8gZmlsdGVyIG91dCB0aGUgZHVwbGljYXRlZCByb3dzIHVzaW5nIHRoZSBgZHBseXI6OmRpc3RpbmN0KClgIGZ1bmN0aW9uIGFmdGVyIHNvcnRpbmcgYnkgYWJzb2x1dGUgdmFsdWUgb2YgdGhlIHN1bW1hcnkgbG9nIGZvbGQgY2hhbmdlLgpUaGlzIHdpbGwga2VlcCB0aGUgZmlyc3Qgcm93IHdpdGggdGhlIGR1cGxpY2F0ZWQgdmFsdWUgdGh1cyBrZWVwaW5nIHRoZSByb3cgd2l0aCB0aGUgbGFyZ2VzdCBhYnNvbHV0ZSB2YWx1ZS4KCmBgYHtyIGZpbHRlcl9kdXBfbWFya2Vyc30KZmlsdGVyZWRfbWFya2Vyc19kZiA8LSBtYXJrZXJzX2RmIHw+CiAgIyBTb3J0IHNvIHRoYXQgdGhlIGhpZ2hlc3QgYWJzb2x1dGUgdmFsdWVzIG9mIHRoZSBzdW1tYXJ5IGxvZzIgZm9sZCBjaGFuZ2UgYXJlCiAgIyBhdCB0aGUgdG9wCiAgZHBseXI6OmFycmFuZ2UoZHBseXI6OmRlc2MoYWJzKHN1bW1hcnkubG9nRkMpKSkgfD4KICAjIEZpbHRlciBvdXQgdGhlIGR1cGxpY2F0ZWQgcm93cyB1c2luZyBgZHBseXI6OmRpc3RpbmN0KClgCiAgZHBseXI6OmRpc3RpbmN0KGdlbmVfc3ltYm9sLCAua2VlcF9hbGwgPSBUUlVFKQpgYGAKCkxldCdzIHNlZSB3aGF0IGhhcHBlbmVkIHRvIG91ciBkdXBsaWNhdGUgaWRlbnRpZmllcnMuCgpgYGB7ciBzaG93X2ZpbHRlcmVkX21hcmtlcnMsIGxpdmUgPSBUUlVFfQojIFN1YnNldCB0byAmIGFycmFuZ2UgYnkgZ2VuZSBzeW1ib2xzIHRoYXQgd2VyZSBkdXBsaWNhdGVkIGluIHRoZSBvcmlnaW5hbAojIGRhdGEgZnJhbWUgb2YgcmVzdWx0cwpmaWx0ZXJlZF9tYXJrZXJzX2RmIHw+CiAgZHBseXI6OmZpbHRlcihnZW5lX3N5bWJvbCAlaW4lIGR1cGxpY2F0ZWRfZ2VuZV9zeW1ib2xzKSB8PgogIGRwbHlyOjphcnJhbmdlKGdlbmVfc3ltYm9sKQpgYGAKCk5vdyB3ZSdyZSByZWFkeSB0byBwcmVwIG91ciBwcmUtcmFua2VkIGxpc3QgZm9yIEdTRUEuCgojIyMgUHJlLXJhbmtlZCBsaXN0CgpUaGUgYEdTRUEoKWAgZnVuY3Rpb24gdGFrZXMgYSBwcmUtcmFua2VkIChzb3J0ZWQpIG5hbWVkIHZlY3RvciBvZiBzdGF0aXN0aWNzLCB3aGVyZSB0aGUgbmFtZXMgaW4gdGhlIHZlY3RvciBhcmUgZ2VuZSBpZGVudGlmaWVycy4KVGhpcyBpcyBzdGVwIDEgLS0gZ2VuZS1sZXZlbCBzdGF0aXN0aWNzLgoKSGVyZSwgd2UncmUgdXNpbmcgdGhlIHN1bW1hcnkgbG9nIGZvbGQgY2hhbmdlLCB3aGljaCBzdW1tYXJpemVzIG9mIGFsbCBvZiB0aGUgbG9nIGZvbGQgY2hhbmdlcyBmcm9tIHRoZSBwYWlyd2lzZSBjb21wYXJpc29ucyBiZXR3ZWVuIGNsdXN0ZXIgMSBhbmQgb3RoZXIgY2x1c3RlcnMgaW50byBhIHNpbmdsZSB2YWx1ZSAoW3JlZl0oaHR0cHM6Ly9yZHJyLmlvL2Jpb2Mvc2NyYW4vbWFuL2NvbWJpbmVNYXJrZXJzLmh0bWwpKS4KQXMgc3VjaCwgdGhpcyBzdGF0aXN0aWMgZ2l2ZXMgdXMgaW5mb3JtYXRpb24gYWJvdXQgdGhlIHJlbGF0aXZlIG1hZ25pdHVkZSBhbmQgZGlyZWN0aW9uYWxpdHkgb2YgZWFjaCBnZW5lJ3MgZXhwcmVzc2lvbiBpbiBjbHVzdGVyIDEgcmVsYXRpdmUgdG8gYWxsIG90aGVyIGNsdXN0ZXJzLgoKYGBge3IgbGZjX3ZlY3Rvcn0KbGZjX3ZlY3RvciA8LSBmaWx0ZXJlZF9tYXJrZXJzX2RmIHw+CiAgIyBFeHRyYWN0IGEgdmVjdG9yIG9mIGBzdW1tYXJ5LmxvZ0ZDYCBuYW1lZCBieSBgZ2VuZV9zeW1ib2xgCiAgZHBseXI6OnB1bGwoc3VtbWFyeS5sb2dGQywgbmFtZSA9IGdlbmVfc3ltYm9sKQpsZmNfdmVjdG9yIDwtIHNvcnQobGZjX3ZlY3RvciwgZGVjcmVhc2luZyA9IFRSVUUpCmBgYAoKTGV0J3MgbG9vayBhdCB0aGUgdG9wIHJhbmtlZCB2YWx1ZXMuCgpgYGB7ciBoZWFkX2xmYywgbGl2ZSA9IFRSVUV9CiMgTG9vayBhdCBmaXJzdCBlbnRyaWVzIG9mIHRoZSBsb2cgZm9sZCBjaGFuZ2UgdmVjdG9yCmhlYWQobGZjX3ZlY3RvcikKYGBgCgpBbmQgdGhlIGJvdHRvbSBvZiB0aGUgbGlzdC4KCmBgYHtyIHRhaWxfbGZjLCBsaXZlID0gVFJVRX0KIyBMb29rIGF0IHRoZSBsYXN0IGVudHJpZXMgb2YgdGhlIGxvZyBmb2xkIGNoYW5nZSB2ZWN0b3IKdGFpbChsZmNfdmVjdG9yKQpgYGAKCiMjIFJ1biBHU0VBCgpOb3cgZm9yIHRoZSBhbmFseXNpcyEKCldlIGNhbiB1c2UgdGhlIGBHU0VBKClgIGZ1bmN0aW9uIHRvIHBlcmZvcm0gR1NFQSB3aXRoIGFueSBnZW5lcmljIHNldCBvZiBnZW5lIHNldHMsIGJ1dCB0aGVyZSBhcmUgc2V2ZXJhbCBmdW5jdGlvbnMgZm9yIHVzaW5nIHNwZWNpZmljLCBjb21tb25seSB1c2VkIGdlbmUgc2V0cyAoZS5nLiwgYGdzZUtFR0coKWApLgoKYGBge3IgcnVuX2dzZWF9CmdzZWFfcmVzdWx0cyA8LSBHU0VBKGdlbmVMaXN0ID0gbGZjX3ZlY3RvciwgICMgb3JkZXJlZCByYW5rZWQgZ2VuZSBsaXN0CiAgICAgICAgICAgICAgICAgICAgIG1pbkdTU2l6ZSA9IDI1LCAgIyBtaW5pbXVtIGdlbmUgc2V0IHNpemUKICAgICAgICAgICAgICAgICAgICAgbWF4R1NTaXplID0gNTAwLCAgIyBtYXhpbXVtIGdlbmUgc2V0IHNldAogICAgICAgICAgICAgICAgICAgICBwdmFsdWVDdXRvZmYgPSAwLjA1LAogICAgICAgICAgICAgICAgICAgICBwQWRqdXN0TWV0aG9kID0gIkJIIiwgICMgY29ycmVjdGlvbiBmb3IgbXVsdGlwbGUgaHlwb3RoZXNpcyB0ZXN0aW5nCiAgICAgICAgICAgICAgICAgICAgIFRFUk0yR0VORSA9IGRwbHlyOjpzZWxlY3QoaHNfaW1tdW5vbG9naWNfZGYsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ3NfbmFtZSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBnZW5lX3N5bWJvbCkpCmBgYApMZXQncyB0YWtlIGEgbG9vayBhdCB0aGUgR1NFQSByZXN1bHRzLgoKYGBge3Igdmlld19nc2VhLCBsaXZlID0gVFJVRSwgZXZhbCA9IEZBTFNFfQpWaWV3KGdzZWFfcmVzdWx0c0ByZXN1bHQgfD4KICAgICAgIGRwbHlyOjphcnJhbmdlKGRwbHlyOjpkZXNjKE5FUykpCikKYGBgCgpOb3JtYWxpemVkIGVucmljaG1lbnQgc2NvcmVzIChORVMpIGFyZSBlbnJpY2htZW50IHNjb3JlcyB0aGF0IGFyZSBzY2FsZWQgdG8gbWFrZSBnZW5lIHNldHMgdGhhdCBjb250YWluIGRpZmZlcmVudCBudW1iZXIgb2YgZ2VuZXMgY29tcGFyYWJsZS4KCkxldCdzIHdyaXRlIHRoZXNlIHJlc3VsdHMgdG8gZmlsZS4KCmBgYHtyIHdyaXRlX2dzZWF9CmdzZWFfcmVzdWx0c0ByZXN1bHQgfD4gcmVhZHI6OndyaXRlX3RzdihvdXRwdXRfZmlsZSkKYGBgCgojIyMgVmlzdWFsaXppbmcgR1NFQSByZXN1bHRzCgpXZSBjYW4gdmlzdWFsaXplIEdTRUEgcmVzdWx0cyBmb3IgaW5kaXZpZHVhbCBwYXRod2F5cyBvciBnZW5lIHNldHMgdXNpbmcgYGVucmljaHBsb3Q6OmdzZWFwbG90KClgLgpMZXQncyB0YWtlIGEgbG9vayBhdCAzIGRpZmZlcmVudCBwYXRod2F5cyAtLSBvbmUgd2l0aCBhIGhpZ2hseSBwb3NpdGl2ZSBORVMsIG9uZSB3aXRoIGEgaGlnaGx5IG5lZ2F0aXZlIE5FUywgYW5kIG9uZSB0aGF0IHdhcyBub3QgYSBzaWduaWZpY2FudCByZXN1bHQgLS0gdG8gZ2V0IG1vcmUgaW5zaWdodCBpbnRvIGhvdyBFUyBhcmUgY2FsY3VsYXRlZC4KCiMjIyMgSGlnaGx5IFBvc2l0aXZlIE5FUwoKTGV0J3MgdGFrZSBsb29rIGF0IGEgcGF0aHdheSB3aXRoIGEgaGlnaGx5IHBvc2l0aXZlIE5FUyAoYEdTRTI5NjE4X0JDRUxMX1ZTX1BEQ19VUGApIHVzaW5nIGEgR1NFQSBwbG90LgpUaGlzIGdlbmUgc2V0IGlzIGNvbXByaXNlZCBvZiAoW3JlZl0oaHR0cHM6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2NhcmRzL0dTRTI5NjE4X0JDRUxMX1ZTX1BEQ19VUCkpOgoKPiBHZW5lcyB1cC1yZWd1bGF0ZWQgaW4gY29tcGFyaXNvbiBvZiBCIGNlbGxzIHZlcnN1cyBwbGFzbWFjeXRvaWQgZGVuZHJpdGljIGNlbGxzIChwREMpIC4KCmBgYHtyIGhpZ2hseV9wb3N9CmVucmljaHBsb3Q6OmdzZWFwbG90KGdzZWFfcmVzdWx0cywKICAgICAgICAgICAgICAgICAgICAgZ2VuZVNldElEID0gIkdTRTI5NjE4X0JDRUxMX1ZTX1BEQ19VUCIsCiAgICAgICAgICAgICAgICAgICAgIHRpdGxlID0gIkdTRTI5NjE4X0JDRUxMX1ZTX1BEQ19VUCIsCiAgICAgICAgICAgICAgICAgICAgIGNvbG9yLmxpbmUgPSAiIzAwNjZGRiIpCmBgYAoKTm90aWNlIGhvdyB0aGUgZ2VuZXMgdGhhdCBhcmUgaW4gdGhlIGdlbmUgc2V0LCBpbmRpY2F0ZWQgYnkgdGhlIGJsYWNrIGJhcnMsIHRlbmQgdG8gYmUgb24gdGhlIGxlZnQgc2lkZSBvZiB0aGUgZ3JhcGggaW5kaWNhdGluZyB0aGF0IHRoZXkgaGF2ZSBwb3NpdGl2ZSBnZW5lLWxldmVsIHNjb3Jlcy4KCiMjIyMgSGlnaGx5IE5lZ2F0aXZlIE5FUwoKVGhlIGdlbmUgc2V0IGBHU0UyOTYxOF9CQ0VMTF9WU19QRENfREFZN19GTFVfVkFDQ0lORV9ETmAgaGFkIGEgaGlnaGx5IG5lZ2F0aXZlIE5FUywgYW5kIGlzIGNvbXByaXNlZCBvZiAoW3JlZl0oaHR0cHM6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2dlbmVzZXRfcGFnZS5qc3A/Z2VuZVNldE5hbWU9R1NFMjk2MThfQkNFTExfVlNfUERDX0RBWTdfRkxVX1ZBQ0NJTkVfRE4pKQoKPiAgCUdlbmVzIGRvd24tcmVndWxhdGVkIGluIGNvbXBhcmlzb24gb2YgQiBjZWxscyBmcm9tIGluZmx1ZW56YSB2YWNjaW5lZSBhdCBkYXkgNyBwb3N0LXZhY2NpbmF0aW9uIHZlcnN1cyBwbGFzbWFjeXRvaWQgZGVuZHJpdGljIGNlbGxzIChwREMpIGF0IGRheSA3IHBvc3QtdmFjY2luYXRpb24uCgpUaGlzIGlzIHRoZSAiZG93biIgc2lnbmF0dXJlIHRoYXQgY29ycmVzcG9uZHMgdG8gb25lIG9mIHRoZSAidXAiIHBhdGh3YXlzIHdpdGggYSBoaWdobHkgcG9zaXRpdmUgTkVTLCBzbyBpbiBhIHdheSB0aGlzIGlzIG5vdCBhZGRpbmcgbmV3IGluZm9ybWF0aW9uIQoKYGBge3IgaGlnaGx5X25lZ30KZW5yaWNocGxvdDo6Z3NlYXBsb3QoZ3NlYV9yZXN1bHRzLAogICAgICAgICAgICAgICAgICAgICBnZW5lU2V0SUQgPSAiR1NFMjk2MThfQkNFTExfVlNfUERDX0RBWTdfRkxVX1ZBQ0NJTkVfRE4iLAogICAgICAgICAgICAgICAgICAgICB0aXRsZSA9ICJHU0UyOTYxOF9CQ0VMTF9WU19QRENfREFZN19GTFVfVkFDQ0lORV9ETiIsCiAgICAgICAgICAgICAgICAgICAgIGNvbG9yLmxpbmUgPSAiIzAwNjZGRiIpCmBgYAoKVGhpcyBnZW5lIHNldCBzaG93cyB0aGUgb3Bwb3NpdGUgcGF0dGVybiAtLSBnZW5lcyBpbiB0aGUgcGF0aHdheSB0ZW5kIHRvIGJlIG9uIHRoZSByaWdodCBzaWRlIG9mIHRoZSBncmFwaC4KCiMjIyMgQSBub24tc2lnbmlmaWNhbnQgcmVzdWx0CgpUaGUgYEByZXN1bHRzYCBzbG90IHdpbGwgb25seSBzaG93IGdlbmUgc2V0cyB0aGF0IHBhc3MgdGhlIGBwdmFsdWVDdXRvZmZgIHRocmVzaG9sZCB3ZSBzdXBwbGllZCB0byBgR1NFQSgpYCwgYnV0IHdlIGNhbiBwbG90IGFueSBnZW5lIHNldCBzbyBsb25nIGFzIHdlIGtub3cgaXRzIG5hbWUuCkxldCdzIGxvb2sgYXQgW2BHT0xEUkFUSF9FRkZfVlNfTUVNT1JZX0NEOF9UQ0VMTF9ETmBdKGh0dHA6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2dlbmVzZXRfcGFnZS5qc3A/Z2VuZVNldE5hbWU9R09MRFJBVEhfRUZGX1ZTX01FTU9SWV9DRDhfVENFTExfVVAma2V5d29yZHM9R09MRFJBVEgpLCB3aGljaCBleGFtaW5lcyB0aGUgZGlmZmVyZW5jZSBiZXR3ZWVuIGVmZmVjdG9yIENEOCBUIGNlbGxzIGFuZCBtZW1vcnkgQ0Q4IFQgY2VsbHMgYW5kIHdhcyBub3QgaW4gdGhlIHJlc3VsdHMgd2Ugdmlld2VkIGVhcmxpZXIuCgpgYGB7ciBub25zaWcsIGxpdmUgPSBUUlVFfQplbnJpY2hwbG90Ojpnc2VhcGxvdChnc2VhX3Jlc3VsdHMsCiAgICAgICAgICAgICAgICAgICAgIGdlbmVTZXRJRCA9ICJHT0xEUkFUSF9FRkZfVlNfTUVNT1JZX0NEOF9UQ0VMTF9ETiIsCiAgICAgICAgICAgICAgICAgICAgIHRpdGxlID0gIkdPTERSQVRIX0VGRl9WU19NRU1PUllfQ0Q4X1RDRUxMX0ROIiwKICAgICAgICAgICAgICAgICAgICAgY29sb3IubGluZSA9ICIjMDA2NkZGIikKYGBgCgpHZW5lcyBpbiB0aGUgcGF0aHdheSBhcmUgZGlzdHJpYnV0ZWQgbW9yZSBldmVubHkgdGhyb3VnaG91dCB0aGUgcmFua2VkIGxpc3QsIHJlc3VsdGluZyBpbiBhIG1vcmUgIm1pZGRsaW5nIiBzY29yZS4KCipOb3RlOiBUaGUgcGxvdHMgcmV0dXJuZWQgYnkgYGVucmljaHBsb3Q6OmdzZWFwbG90YCBhcmUgZ2dwbG90cywgc28gd2UgY291bGQgdXNlIGBnZ3Bsb3QyOjpnZ3NhdmUoKWAgdG8gc2F2ZSB0aGVtIHRvIGZpbGUgaWYgd2Ugd2FudGVkIHRvLioKCiMjIFNlc3Npb24gSW5mbwoKYGBge3Igc2Vzc2lvbl9pbmZvfQpzZXNzaW9uSW5mbygpCmBgYAo=
- - -
-
- -
- - - - - - - - - - - - - - - - - From 6afb72011832657260f5ac77daec4a4795a033dd Mon Sep 17 00:00:00 2001 From: Jaclyn Taroni <19534205+jaclyn-taroni@users.noreply.github.com> Date: Mon, 25 Nov 2024 09:56:58 -0500 Subject: [PATCH 3/7] Remove ORA notebook --- .../04-overrepresentation_analysis-live.Rmd | 416 -- .../04-overrepresentation_analysis.Rmd | 429 -- .../04-overrepresentation_analysis.nb.html | 3868 ----------------- 3 files changed, 4713 deletions(-) delete mode 100644 scRNA-seq-advanced/04-overrepresentation_analysis-live.Rmd delete mode 100644 scRNA-seq-advanced/04-overrepresentation_analysis.Rmd delete mode 100644 scRNA-seq-advanced/04-overrepresentation_analysis.nb.html diff --git a/scRNA-seq-advanced/04-overrepresentation_analysis-live.Rmd b/scRNA-seq-advanced/04-overrepresentation_analysis-live.Rmd deleted file mode 100644 index 8c4edd8a..00000000 --- a/scRNA-seq-advanced/04-overrepresentation_analysis-live.Rmd +++ /dev/null @@ -1,416 +0,0 @@ ---- -title: "Pathway analysis: Over-representation analysis (ORA)" -output: - html_notebook: - toc: true - toc_float: true -author: CCDL for ALSF -date: 2020 ---- - -## Objectives - -This notebook will demonstrate how to: - -- Perform gene identifier conversion with [`AnnotationDBI` annotation packages](https://bioconductor.org/packages/release/bioc/vignettes/AnnotationDbi/inst/doc/IntroToAnnotationPackages.pdf) -- Prepare gene sets for over-representation analysis, including an appropriate background set -- Perform over-representation analysis with the `clusterProfiler` package -- Introduce resources for ORA of scRNA-seq data, such as Gene Ontology - ---- - -In this notebook, we'll cover a type of pathway or gene set analysis called over-representation analysis (ORA). -The idea behind ORA is relatively straightforward: given a set of genes, do these genes overlap with a pathway more than we expect by chance? -The simplicity of only requiring an input gene set (sort of, more on that below) can be attractive. - -ORA has some limitations, outlined nicely (and more extensively!) in [Khatri _et al._ (2012)]( https://doi.org/10.1371/journal.pcbi.1002375). -One of the main issues with ORA is that typically all genes are treated as equal -- the context of the magnitude of a change we may be measuring is removed and each gene is treated as independent, which can sometimes result in an incorrect estimate of significance. - -We will use the [`clusterProfiler` package](https://bioconductor.org/packages/release/bioc/html/clusterProfiler.html) ([Yu *et al.* 2012](https://doi.org/10.1089/omi.2011.0118.)) to perform ORA. -`clusterProfiler` has many built-in functions that will run a specific type of analysis using a specific source of pathways/gene sets automatically, but for our purposes we're going to keep things as general as possible. -See the [`clusterProfiler` book](https://yulab-smu.github.io/clusterProfiler-book/index.html) for more information about the package's full suite of functionality. - -Because different bioinformatics tools often require different types of gene identifiers, we'll also cover how to convert between gene identifiers using [`AnnotationDbi`](https://bioconductor.org/packages/release/bioc/html/AnnotationDbi.html) Bioconductor packages in this notebook. -Check out the [_AnnotationDbi: Introduction To Bioconductor Annotation Packages_ (Carlson 2020.) vignette](https://bioconductor.org/packages/release/bioc/vignettes/AnnotationDbi/inst/doc/IntroToAnnotationPackages.pdf) for more information. - -#### Other resources - -* For another example using `clusterProfiler`, see [_Intro to DGE: Functional Analysis._ from Harvard Chan Bioinformatics Core Training.](https://hbctraining.github.io/DGE_workshop/lessons/09_functional_analysis.html) -* [`WebGestaltR`](https://cran.r-project.org/web/packages/WebGestaltR/) is another R package that can be used for ORA. -* OSCA has [a section on using Gene Ontology sets to annotate clusters](https://bioconductor.org/books/3.14/OSCA.basic/cell-type-annotation.html#assigning-cluster-labels-from-markers) using a different Bioconductor package. - -## Set up - -### Libraries - -```{r libraries} -# Package we'll use to run ORA -library(clusterProfiler) -# Package that contains MSigDB gene sets in tidy format -library(msigdbr) -# Homo sapiens annotation package we'll use for gene identifier conversion -library(org.Hs.eg.db) -``` - -### Directories and files - -#### Directories - -```{r create_ora_directory, live = TRUE} - -# We'll use the output of the clustering notebook for ORA - -# We'll create a directory to specifically hold the pathway results - -# Create a directory for an updated marker list file - -``` - -#### Input files - -We're going to perform ORA on the marker genes from cluster 1 we looked at in the previous notebook. - -```{r cluster_input} -input_file <- file.path(hodgkins_data_dir, - "markers", - "cluster01_markers.tsv") -``` - -We're going to use a file that is a cleaned version of the [CellMarker](http://bio-bigdata.hrbmu.edu.cn/CellMarker/) ([Zhang *et al.* 2018](https://doi.org/10.1093/nar/gky900)) human markers resources. -You can see how we obtained and cleaned this file in [this notebook](https://github.com/AlexsLemonade/training-modules/blob/master/scRNA-seq/setup/cellmarker_genes.Rmd). - -```{r cm_input} -cellmarker_file <- file.path("gene-sets", - "CellMarker_cleaned_human_markers.tsv") -``` - -#### Output files - -In the next notebook, we'll use the cluster 1 results. -Saving the post-gene identifier conversion results in this notebook will prevent us from duplicating effort (and code)! - -```{r gene_id_file} -cluster1_output_file <- file.path(hodgkins_analysis_dir, - "markers", - "cluster01_markers_with_gene_symbols.tsv") -``` - -We'll save the table of ORA results (e.g., p-values). - -```{r output_files, live = TRUE} - -``` - -## Read in marker genes - -Let's read in our marker genes from cluster 1. - -```{r read_in_markers, live = TRUE} - -``` - -And let's take a peek! - -```{r marker_head, live = TRUE} - -``` - -### Gene identifier conversion - -Our data frame of marker genes contains Ensembl gene identifiers. -We will need to convert from these identifiers into gene symbols or Entrez IDs for our use with cell marker data. - -We're going to convert our identifiers to gene symbols because they are a bit more human readable, but you can, with the change of a single argument, use the same code to convert to many other types of identifiers! - -The annotation package `org.Hs.eg.db` contains information for different identifiers. -`org.Hs.eg.db` is specific to _Homo sapiens_ -- this is what the `Hs` in the package name is referencing. -To perform gene identifier conversion in mouse (_Mus musculus_) we could use `org.Mm.eg.db`; -we would use `org.Dr.eg.db` for zebrafish (_Danio rerio_). - -We can see what types of IDs are available to us in an annotation package with `keytypes()`. - -```{r keytypes, live = TRUE} - -``` - -Even though we'll use this package to convert from Ensembl gene IDs (`ENSEMBL`) to gene symbols (`SYMBOL`), we could just as easily use it to convert from an Ensembl transcript ID (`ENSEMBLTRANS`) to Entrez IDs (`ENTREZID`). - -The function we will use to map from Ensembl gene IDs to gene symbols is called `mapIds()`. - -```{r map_to_symbol} -# This returns a named vector which we can convert to a data frame, where -# the keys (Ensembl IDs) are the names -symbols_vector <- mapIds(org.Hs.eg.db, # Specify the annotation package - # The vector of gene identifiers we want to - # map - keys = markers_df$gene, - # What type of gene identifiers we're starting - # with - keytype = "ENSEMBL", - # The type of gene identifier we want returned - column = "SYMBOL", - # In the case of 1:many mappings, return the - # first one. This is default behavior! - multiVals = "first") - -# We would like a data frame we can join to the DGE results -symbols_df <- data.frame( - ensembl_id = names(symbols_vector), - gene_symbol = symbols_vector -) |> - # Drop genes that don't have a gene symbol! - tidyr::drop_na() -``` - -This message is letting us know that sometimes Ensembl gene identifiers will map to multiple gene symbols. -In this case, it's also possible that a gene symbol will map to multiple Ensembl IDs. - -Now we are ready to add the gene symbols to our data frame with the marker genes. -We can use a _join_ function from the `dplyr` package to do this, which will use the Ensembl gene IDs in both data frames to determine how to join the rows. - -```{r add_symbols, live = TRUE} - - # An *inner* join will only return rows that are in both data frames - - # The name of the column that contains the Ensembl gene IDs - # in the left data frame and right data frame, effectively - # dropping genes that don't have gene symbols - -``` - -We're going to use this table again, with the gene symbols, in the next notebook so let's write it to a file. - -```{r save_gene_symbols, live = TRUE} - -``` - -## Over-representation Analysis (ORA) - -To test for over-representation, we can calculate a p-value with a hypergeometric test ([ref](https://yulab-smu.github.io/clusterProfiler-book/chapter2.html#over-representation-analysis)). - -\(p = 1 - \displaystyle\sum_{i = 0}^{k-1}\frac{ {M \choose i}{ {N-M} \choose {n-i} } } { {N \choose n} }\) - -Where `N` is the number of genes in the background distribution, `M` is the number of genes in a pathway or gene set, `n` is the number of genes we are interested in (our marker genes), and `k` is the number of genes that overlap between the pathway or gene set and our marker genes. - -Borrowing an example from [_clusterProfiler: universal enrichment tool for functional and comparative study_ (Yu )](http://yulab-smu.top/clusterProfiler-book/chapter2.html#over-representation-analysis): - -> **Example**: Suppose we have 17,980 genes detected in a Microarray study and 57 genes were differentially expressed. Among the differential expressed genes, 28 are annotated to a gene set. - -We'll call genes that are differentially expressed `gene_in_interest` and genes that are in the gene set `in_gene_set`. - -```{r gene_table} -gene_table <- data.frame( - gene_not_interest = c(2613, 15310), - gene_in_interest = c(28, 29) -) -rownames(gene_table) <- c("in_gene_set", "not_in_gene_set") - -gene_table -``` - -We can assess if the 28 overlapping genes mean that the differentially expressed genes are over-represented in the gene set with the hypergeometric distribution. -This corresponds to a one-sided Fisher's exact test. - -```{r fisher_test} -fisher.test(gene_table, alternative = "greater") -``` - -When we test **multiple pathways or gene sets**, the p-values then need to be **adjusted** for multiple hypothesis testing. - -### Marker gene preparation - -#### Top 100 cluster 1 genes - -We're interested in what pathways are over-represented in genes that mark, or are more highly expressed in, cluster 1. -We'll select the top 100 genes for this comparison. - -```{r cluster01_genes} -cluster01_markers <- markers_df |> - # Positive fold changes - dplyr::filter(summary.logFC > 0) |> - # Take the "top 100" when ordering by FDR (smallest FDR values) - dplyr::slice_min(order_by = FDR, n = 100) |> - # Extract a vector of the gene symbols - dplyr::pull(gene_symbol) -``` - -You may think that selecting the top 100 genes is pretty arbitrary and you are not wrong! -This also may seem strict, but let's think about our goal for this particular analysis: we hope that using ORA with resources that encode some knowledge about cell types will help us annotate cluster 1. -We're using the very top ranked genes in service of this goal. - -An alternative approach would be to have filtered to genes that are below a specific FDR threshold, but that would also be an arbitrary threshold that we chose. -This is a weakness of over-representation analysis; we generally need to pick a threshold for the genes that we include in our tests. - -As we mentioned in the clustering notebook, our statistical calculations are circular here. -We identified clusters based on gene expression and then identified genes that are differentially expressed between clusters. -We can use the FDR values for ranking, like we are here, but we shouldn't use them to define "significance." -You can read a little more about this in the [_Invalidity of p-values_ section of the OSCA book](https://bioconductor.org/books/3.14/OSCA.advanced/marker-detection-redux.html#p-value-invalidity). - -#### Background set - -As we saw above, calculating the p-value for ORA relies on the number of genes in the background distribution. -Sometimes folks consider genes from the entire genome to comprise the background, but in the example borrowed from the `clusterProfiler` authors, they state: - -> 17,980 genes detected in a Microarray study - -Where the key phrase is **genes detected**. - -If we were unable to include a gene in our marker gene comparisons because, for example, we could not reliably measure it, we shouldn't include in our background set! - -Our markers data frame has all of the genes that we did not filter out and have gene symbols corresponding to the Ensembl gene IDs, so we can use the `gene_symbol` column as our background set. - -```{r get_background_set, live = TRUE} - -``` - -### Gene Ontology ORA - -#### A note on GO - -The Gene Ontology (GO) ([Ashburner *et al.* 2000](https://dx.doi.org/10.1038/75556); [The Gene Ontology Consortium. 2021](https://doi.org/10.1093/nar/gkaa1113))is [an ontology](https://en.wikipedia.org/wiki/Ontology_(information_science)) that documents or describes how genes function in the biological domain and is comprised of 3 parts ([GO ontology documentation](http://geneontology.org/docs/ontology-documentation/)): - -* **Molecular function (MF):** activities that occur on the molecular level, often can be performed by individual genes -* **Biological process (BP):** programs carried out by multiple molecular entities; similar to what is sometimes thought of as a pathway, but with no notion of _dynamics_ -* **Cellular component (CC):** cellular structures and compartments - -GO is "loosely hierarchical" ([GO ontology documentation](http://geneontology.org/docs/ontology-documentation/)). -There are parent and child terms, where the "higher up" a term is, the more general it is. -This is somewhat abstract, so let's take a look at a GO term that's likely to be relevant to our Hodgkin's lymphoma data to get an idea of what this is like in practice: [`leukocyte activation`](http://www.informatics.jax.org/vocab/gene_ontology/GO:0045321). - -#### Run `enrichGO()` - -Now that we have our background set and our genes of interest, we're ready to run ORA using the `enrichGO()` function. - -```{r go_ora} -go_ora_results <- enrichGO(gene = cluster01_markers, # Top 100 genes - universe = background_set, # Background genes - keyType = "SYMBOL", # Our genes are gene symbols - OrgDb = org.Hs.eg.db, # Supports conversion, if needed - ont = "BP", # Biological process - pAdjustMethod = "BH", # FDR - pvalueCutoff = 0.00001) # Very stringent for viz -``` - -What is returned by `enrichGO()`? - -```{r view_go_ora, eval = FALSE} -View(go_ora_results) -``` - -The information we're most likely interested in is in the `result` slot. -Let's convert this into a data frame that we can write to file. - -```{r go_df} -go_result_df <- data.frame(go_ora_results@result) -``` - -Let's take a look at the GO sets with an adjusted p-value less than 0.00001. - -```{r filter_padj, live = TRUE} - -``` - -Some of these columns have names that are somewhat opaque and are used for visualization, so let's briefly talk about them here! - -| Column name | Numerator | Denominator | -|-------------|-----------|-------------| -| `GeneRatio` | Number of genes in the top 100 marker genes and the specific gene set (also in `Count` column) | Number of genes in the top 100 marker genes and _any_ gene set | -| `BgRatio` | Number of genes in the background set and the specific gene set | Number of genes in the background set and _any_ gene set | - -#### Visualizing results - -We can use a dot plot to visualize our significant enrichment results. - -```{r dotplot, live = TRUE} - -``` - -**What cell type do you think the cells from cluster 1 are?** - -We can use an [UpSet plot](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4720993/) to visualize the **overlap** between the gene sets that were returned as significant. - -```{r upsetplot, live = TRUE} - -``` - -It's important to keep in mind that gene sets or pathways aren't independent, either! -This is particularly true of GO terms, which are hierarchical in nature. - -### CellMarker ORA - -Next, we'll use another resource, CellMarker, as additional support for our conclusions based on GO. - -#### About CellMarker - -[CellMarker](http://bio-bigdata.hrbmu.edu.cn/CellMarker/) is a manually curated database of cell marker data for human and mouse ([Zhang *et al.* 2018](https://doi.org/10.1093/nar/gky900)). -Studies/evidence of the following nature were included in the manual curation process ([Zhang *et al.* 2018](https://doi.org/10.1093/nar/gky900)): - -* scRNA-seq -* Flow cytometry -* Immunostaining - -And there are both gene and protein markers. - -Let's read in the CellMarker data we've already cleaned: - -```{r read_in_cm, live = TRUE} - -``` -#### Run `enricher()` - -`clusterProfiler::enricher()` is a more general way to perform ORA -- we specify the pathways or gene sets ourselves. -We can use the it to run ORA with the CellMarker data, which contains gene symbols. -That's why we needed to perform gene identifier conversion earlier! - -```{r cm_ora, live = TRUE} - - # The pathway information should be a data frame with a term name or - # identifier and the gene identifiers - -``` - -#### Visualizing results - -Let's look at a bar plot of the significant results. - -```{r cm_barplot} -barplot(cellmarker_ora_results) + - # This is a ggplot, so we can use the following to label the x-axis! - ggplot2::labs(x = "count") -``` - -One thing that the [web version of CellMarker](http://bio-bigdata.hrbmu.edu.cn/CellMarker/) allows us to do is to get a better idea of _marker prevalence_. -Some of the genes that show up in our top 100 may not be very specific. -It's also likely that kidney is not very relevant to our analysis of Hodgkin's lymphoma, but lymph node is highly relevant. -However, blood cells in kidney are not _unexpected_ and, if we view the [CellMarker tissue type browser](http://bio-bigdata.hrbmu.edu.cn/CellMarker/browse.jsp), we can see that there are _a lot_ of genes annotated to kidney. -Every resource will have some limitations "baked in." - -We could do additional filtering of the gene sets to remove irrelevant tissues and to remove marker genes that are not particularly specific. -(Fewer gene sets would mean a lower multiple hypothesis testing burden.) -If some of the CellMarker marker genes come from immunostaining of terminally differentiated cells, we might not expect their transcripts to show up in scRNA-seq data, for example. -All things to keep in mind if you use this resource for your own work! - -### Write results to file - -Write the results to file for both GO and CellMarker gene sets. - -```{r write_results, live = TRUE} - -``` - -## Parting thoughts - -The goal of analyzing marker genes with ORA, like we did in this notebook, is to help us interpret our clustering results or annotate clusters. -We shouldn't use marker genes or any downstream analysis we perform with them to _justify_ cluster assignments; we generally want our cluster assignments to be data-driven. - -Just like it's important to be skeptical about p-values returned from the marker gene analysis itself, we should not get too attached to the idea of statistical significance here. -If we were to tweak the cutoffs (e.g., take the top 2500 genes), we might expect to get completely different significant results. -This is a limitation of ORA: we have to make some choices that are somewhat arbitrary. -It's a good idea to think about the goal of your analysis _upfront_ and pick cutoffs based on your goals. -That can help you avoid the temptation of trying a bunch of cutoffs until you get results that "look good." - -## Session Info - -```{r session_info} -sessionInfo() -``` diff --git a/scRNA-seq-advanced/04-overrepresentation_analysis.Rmd b/scRNA-seq-advanced/04-overrepresentation_analysis.Rmd deleted file mode 100644 index 277c0c71..00000000 --- a/scRNA-seq-advanced/04-overrepresentation_analysis.Rmd +++ /dev/null @@ -1,429 +0,0 @@ ---- -title: "Pathway analysis: Over-representation analysis (ORA)" -output: - html_notebook: - toc: true - toc_float: true -author: CCDL for ALSF -date: 2020 ---- - -## Objectives - -This notebook will demonstrate how to: - -- Perform gene identifier conversion with [`AnnotationDBI` annotation packages](https://bioconductor.org/packages/release/bioc/vignettes/AnnotationDbi/inst/doc/IntroToAnnotationPackages.pdf) -- Prepare gene sets for over-representation analysis, including an appropriate background set -- Perform over-representation analysis with the `clusterProfiler` package -- Introduce resources for ORA of scRNA-seq data, such as Gene Ontology - ---- - -In this notebook, we'll cover a type of pathway or gene set analysis called over-representation analysis (ORA). -The idea behind ORA is relatively straightforward: given a set of genes, do these genes overlap with a pathway more than we expect by chance? -The simplicity of only requiring an input gene set (sort of, more on that below) can be attractive. - -ORA has some limitations, outlined nicely (and more extensively!) in [Khatri _et al._ (2012)]( https://doi.org/10.1371/journal.pcbi.1002375). -One of the main issues with ORA is that typically all genes are treated as equal -- the context of the magnitude of a change we may be measuring is removed and each gene is treated as independent, which can sometimes result in an incorrect estimate of significance. - -We will use the [`clusterProfiler` package](https://bioconductor.org/packages/release/bioc/html/clusterProfiler.html) ([Yu *et al.* 2012](https://doi.org/10.1089/omi.2011.0118.)) to perform ORA. -`clusterProfiler` has many built-in functions that will run a specific type of analysis using a specific source of pathways/gene sets automatically, but for our purposes we're going to keep things as general as possible. -See the [`clusterProfiler` book](https://yulab-smu.github.io/clusterProfiler-book/index.html) for more information about the package's full suite of functionality. - -Because different bioinformatics tools often require different types of gene identifiers, we'll also cover how to convert between gene identifiers using [`AnnotationDbi`](https://bioconductor.org/packages/release/bioc/html/AnnotationDbi.html) Bioconductor packages in this notebook. -Check out the [_AnnotationDbi: Introduction To Bioconductor Annotation Packages_ (Carlson 2020.) vignette](https://bioconductor.org/packages/release/bioc/vignettes/AnnotationDbi/inst/doc/IntroToAnnotationPackages.pdf) for more information. - -#### Other resources - -* For another example using `clusterProfiler`, see [_Intro to DGE: Functional Analysis._ from Harvard Chan Bioinformatics Core Training.](https://hbctraining.github.io/DGE_workshop/lessons/09_functional_analysis.html) -* [`WebGestaltR`](https://cran.r-project.org/web/packages/WebGestaltR/) is another R package that can be used for ORA. -* OSCA has [a section on using Gene Ontology sets to annotate clusters](https://bioconductor.org/books/3.14/OSCA.basic/cell-type-annotation.html#assigning-cluster-labels-from-markers) using a different Bioconductor package. - -## Set up - -### Libraries - -```{r libraries} -# Package we'll use to run ORA -library(clusterProfiler) -# Package that contains MSigDB gene sets in tidy format -library(msigdbr) -# Homo sapiens annotation package we'll use for gene identifier conversion -library(org.Hs.eg.db) -``` - -### Directories and files - -#### Directories - -```{r create_ora_directory, live = TRUE} -hodgkins_data_dir <- file.path("data", "hodgkins") -# We'll use the output of the clustering notebook for ORA -hodgkins_analysis_dir <- file.path("analysis", "hodgkins") - -# We'll create a directory to specifically hold the pathway results -results_dir <- file.path(hodgkins_analysis_dir, "pathway-analysis") -fs::dir_create(results_dir) - -# Create a directory for an updated marker list file -fs::dir_create(file.path(hodgkins_analysis_dir, "markers")) -``` - -#### Input files - -We're going to perform ORA on the marker genes from cluster 1 we looked at in the previous notebook. - -```{r cluster_input} -input_file <- file.path(hodgkins_data_dir, - "markers", - "cluster01_markers.tsv") -``` - -We're going to use a file that is a cleaned version of the [CellMarker](http://bio-bigdata.hrbmu.edu.cn/CellMarker/) ([Zhang *et al.* 2018](https://doi.org/10.1093/nar/gky900)) human markers resources. -You can see how we obtained and cleaned this file in [this notebook](https://github.com/AlexsLemonade/training-modules/blob/master/scRNA-seq/setup/cellmarker_genes.Rmd). - -```{r cm_input} -cellmarker_file <- file.path("gene-sets", - "CellMarker_cleaned_human_markers.tsv") -``` - -#### Output files - -In the next notebook, we'll use the cluster 1 results. -Saving the post-gene identifier conversion results in this notebook will prevent us from duplicating effort (and code)! - -```{r gene_id_file} -cluster1_output_file <- file.path(hodgkins_analysis_dir, - "markers", - "cluster01_markers_with_gene_symbols.tsv") -``` - -We'll save the table of ORA results (e.g., p-values). - -```{r output_files, live = TRUE} -go_results_file <- file.path(results_dir, "cluster01_go_ora_results.tsv") -cm_results_file <- file.path(results_dir, - "cluster01_CellMarker_ora_results.tsv") -``` - -## Read in marker genes - -Let's read in our marker genes from cluster 1. - -```{r read_in_markers, live = TRUE} -markers_df <- readr::read_tsv(input_file) -``` - -And let's take a peek! - -```{r marker_head, live = TRUE} -head(markers_df) -``` - -### Gene identifier conversion - -Our data frame of marker genes contains Ensembl gene identifiers. -We will need to convert from these identifiers into gene symbols or Entrez IDs for our use with cell marker data. - -We're going to convert our identifiers to gene symbols because they are a bit more human readable, but you can, with the change of a single argument, use the same code to convert to many other types of identifiers! - -The annotation package `org.Hs.eg.db` contains information for different identifiers. -`org.Hs.eg.db` is specific to _Homo sapiens_ -- this is what the `Hs` in the package name is referencing. -To perform gene identifier conversion in mouse (_Mus musculus_) we could use `org.Mm.eg.db`; -we would use `org.Dr.eg.db` for zebrafish (_Danio rerio_). - -We can see what types of IDs are available to us in an annotation package with `keytypes()`. - -```{r keytypes, live = TRUE} -keytypes(org.Hs.eg.db) -``` - -Even though we'll use this package to convert from Ensembl gene IDs (`ENSEMBL`) to gene symbols (`SYMBOL`), we could just as easily use it to convert from an Ensembl transcript ID (`ENSEMBLTRANS`) to Entrez IDs (`ENTREZID`). - -The function we will use to map from Ensembl gene IDs to gene symbols is called `mapIds()`. - -```{r map_to_symbol} -# This returns a named vector which we can convert to a data frame, where -# the keys (Ensembl IDs) are the names -symbols_vector <- mapIds(org.Hs.eg.db, # Specify the annotation package - # The vector of gene identifiers we want to - # map - keys = markers_df$gene, - # What type of gene identifiers we're starting - # with - keytype = "ENSEMBL", - # The type of gene identifier we want returned - column = "SYMBOL", - # In the case of 1:many mappings, return the - # first one. This is default behavior! - multiVals = "first") - -# We would like a data frame we can join to the DGE results -symbols_df <- data.frame( - ensembl_id = names(symbols_vector), - gene_symbol = symbols_vector -) |> - # Drop genes that don't have a gene symbol! - tidyr::drop_na() -``` - -This message is letting us know that sometimes Ensembl gene identifiers will map to multiple gene symbols. -In this case, it's also possible that a gene symbol will map to multiple Ensembl IDs. - -Now we are ready to add the gene symbols to our data frame with the marker genes. -We can use a _join_ function from the `dplyr` package to do this, which will use the Ensembl gene IDs in both data frames to determine how to join the rows. - -```{r add_symbols, live = TRUE} -markers_df <- symbols_df |> - # An *inner* join will only return rows that are in both data frames - dplyr::inner_join(markers_df, - # The name of the column that contains the Ensembl gene IDs - # in the left data frame and right data frame, effectively - # dropping genes that don't have gene symbols - by = c("ensembl_id" = "gene")) -``` - -We're going to use this table again, with the gene symbols, in the next notebook so let's write it to a file. - -```{r save_gene_symbols, live = TRUE} -readr::write_tsv(markers_df, file = cluster1_output_file) -``` - -## Over-representation Analysis (ORA) - -To test for over-representation, we can calculate a p-value with a hypergeometric test ([ref](https://yulab-smu.github.io/clusterProfiler-book/chapter2.html#over-representation-analysis)). - -\(p = 1 - \displaystyle\sum_{i = 0}^{k-1}\frac{ {M \choose i}{ {N-M} \choose {n-i} } } { {N \choose n} }\) - -Where `N` is the number of genes in the background distribution, `M` is the number of genes in a pathway or gene set, `n` is the number of genes we are interested in (our marker genes), and `k` is the number of genes that overlap between the pathway or gene set and our marker genes. - -Borrowing an example from [_clusterProfiler: universal enrichment tool for functional and comparative study_ (Yu )](http://yulab-smu.top/clusterProfiler-book/chapter2.html#over-representation-analysis): - -> **Example**: Suppose we have 17,980 genes detected in a Microarray study and 57 genes were differentially expressed. Among the differential expressed genes, 28 are annotated to a gene set. - -We'll call genes that are differentially expressed `gene_in_interest` and genes that are in the gene set `in_gene_set`. - -```{r gene_table} -gene_table <- data.frame( - gene_not_interest = c(2613, 15310), - gene_in_interest = c(28, 29) -) -rownames(gene_table) <- c("in_gene_set", "not_in_gene_set") - -gene_table -``` - -We can assess if the 28 overlapping genes mean that the differentially expressed genes are over-represented in the gene set with the hypergeometric distribution. -This corresponds to a one-sided Fisher's exact test. - -```{r fisher_test} -fisher.test(gene_table, alternative = "greater") -``` - -When we test **multiple pathways or gene sets**, the p-values then need to be **adjusted** for multiple hypothesis testing. - -### Marker gene preparation - -#### Top 100 cluster 1 genes - -We're interested in what pathways are over-represented in genes that mark, or are more highly expressed in, cluster 1. -We'll select the top 100 genes for this comparison. - -```{r cluster01_genes} -cluster01_markers <- markers_df |> - # Positive fold changes - dplyr::filter(summary.logFC > 0) |> - # Take the "top 100" when ordering by FDR (smallest FDR values) - dplyr::slice_min(order_by = FDR, n = 100) |> - # Extract a vector of the gene symbols - dplyr::pull(gene_symbol) -``` - -You may think that selecting the top 100 genes is pretty arbitrary and you are not wrong! -This also may seem strict, but let's think about our goal for this particular analysis: we hope that using ORA with resources that encode some knowledge about cell types will help us annotate cluster 1. -We're using the very top ranked genes in service of this goal. - -An alternative approach would be to have filtered to genes that are below a specific FDR threshold, but that would also be an arbitrary threshold that we chose. -This is a weakness of over-representation analysis; we generally need to pick a threshold for the genes that we include in our tests. - -As we mentioned in the clustering notebook, our statistical calculations are circular here. -We identified clusters based on gene expression and then identified genes that are differentially expressed between clusters. -We can use the FDR values for ranking, like we are here, but we shouldn't use them to define "significance." -You can read a little more about this in the [_Invalidity of p-values_ section of the OSCA book](https://bioconductor.org/books/3.14/OSCA.advanced/marker-detection-redux.html#p-value-invalidity). - -#### Background set - -As we saw above, calculating the p-value for ORA relies on the number of genes in the background distribution. -Sometimes folks consider genes from the entire genome to comprise the background, but in the example borrowed from the `clusterProfiler` authors, they state: - -> 17,980 genes detected in a Microarray study - -Where the key phrase is **genes detected**. - -If we were unable to include a gene in our marker gene comparisons because, for example, we could not reliably measure it, we shouldn't include in our background set! - -Our markers data frame has all of the genes that we did not filter out and have gene symbols corresponding to the Ensembl gene IDs, so we can use the `gene_symbol` column as our background set. - -```{r get_background_set, live = TRUE} -background_set <- markers_df$gene_symbol -``` - -### Gene Ontology ORA - -#### A note on GO - -The Gene Ontology (GO) ([Ashburner *et al.* 2000](https://dx.doi.org/10.1038/75556); [The Gene Ontology Consortium. 2021](https://doi.org/10.1093/nar/gkaa1113))is [an ontology](https://en.wikipedia.org/wiki/Ontology_(information_science)) that documents or describes how genes function in the biological domain and is comprised of 3 parts ([GO ontology documentation](http://geneontology.org/docs/ontology-documentation/)): - -* **Molecular function (MF):** activities that occur on the molecular level, often can be performed by individual genes -* **Biological process (BP):** programs carried out by multiple molecular entities; similar to what is sometimes thought of as a pathway, but with no notion of _dynamics_ -* **Cellular component (CC):** cellular structures and compartments - -GO is "loosely hierarchical" ([GO ontology documentation](http://geneontology.org/docs/ontology-documentation/)). -There are parent and child terms, where the "higher up" a term is, the more general it is. -This is somewhat abstract, so let's take a look at a GO term that's likely to be relevant to our Hodgkin's lymphoma data to get an idea of what this is like in practice: [`leukocyte activation`](http://www.informatics.jax.org/vocab/gene_ontology/GO:0045321). - -#### Run `enrichGO()` - -Now that we have our background set and our genes of interest, we're ready to run ORA using the `enrichGO()` function. - -```{r go_ora} -go_ora_results <- enrichGO(gene = cluster01_markers, # Top 100 genes - universe = background_set, # Background genes - keyType = "SYMBOL", # Our genes are gene symbols - OrgDb = org.Hs.eg.db, # Supports conversion, if needed - ont = "BP", # Biological process - pAdjustMethod = "BH", # FDR - pvalueCutoff = 0.00001) # Very stringent for viz -``` - -What is returned by `enrichGO()`? - -```{r view_go_ora, eval = FALSE} -View(go_ora_results) -``` - -The information we're most likely interested in is in the `result` slot. -Let's convert this into a data frame that we can write to file. - -```{r go_df} -go_result_df <- data.frame(go_ora_results@result) -``` - -Let's take a look at the GO sets with an adjusted p-value less than 0.00001. - -```{r filter_padj, live = TRUE} -go_result_df |> - dplyr::filter(p.adjust < 0.00001) -``` - -Some of these columns have names that are somewhat opaque and are used for visualization, so let's briefly talk about them here! - -| Column name | Numerator | Denominator | -|-------------|-----------|-------------| -| `GeneRatio` | Number of genes in the top 100 marker genes and the specific gene set (also in `Count` column) | Number of genes in the top 100 marker genes and _any_ gene set | -| `BgRatio` | Number of genes in the background set and the specific gene set | Number of genes in the background set and _any_ gene set | - -#### Visualizing results - -We can use a dot plot to visualize our significant enrichment results. - -```{r dotplot, live = TRUE} -enrichplot::dotplot(go_ora_results) -``` - -**What cell type do you think the cells from cluster 1 are?** - -We can use an [UpSet plot](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4720993/) to visualize the **overlap** between the gene sets that were returned as significant. - -```{r upsetplot, live = TRUE} -enrichplot::upsetplot(go_ora_results) -``` - -It's important to keep in mind that gene sets or pathways aren't independent, either! -This is particularly true of GO terms, which are hierarchical in nature. - -### CellMarker ORA - -Next, we'll use another resource, CellMarker, as additional support for our conclusions based on GO. - -#### About CellMarker - -[CellMarker](http://bio-bigdata.hrbmu.edu.cn/CellMarker/) is a manually curated database of cell marker data for human and mouse ([Zhang *et al.* 2018](https://doi.org/10.1093/nar/gky900)). -Studies/evidence of the following nature were included in the manual curation process ([Zhang *et al.* 2018](https://doi.org/10.1093/nar/gky900)): - -* scRNA-seq -* Flow cytometry -* Immunostaining - -And there are both gene and protein markers. - -Let's read in the CellMarker data we've already cleaned: - -```{r read_in_cm, live = TRUE} -cellmarker_df <- readr::read_tsv(cellmarker_file) -``` -#### Run `enricher()` - -`clusterProfiler::enricher()` is a more general way to perform ORA -- we specify the pathways or gene sets ourselves. -We can use the it to run ORA with the CellMarker data, which contains gene symbols. -That's why we needed to perform gene identifier conversion earlier! - -```{r cm_ora, live = TRUE} -cellmarker_ora_results <- enricher( - gene = cluster01_markers, # Genes of interest - pvalueCutoff = 0.05, # More permissive cut off - pAdjustMethod = "BH", # FDR - universe = background_set, # Background set - # The pathway information should be a data frame with a term name or - # identifier and the gene identifiers - TERM2GENE = cellmarker_df -) -``` - -#### Visualizing results - -Let's look at a bar plot of the significant results. - -```{r cm_barplot} -barplot(cellmarker_ora_results) + - # This is a ggplot, so we can use the following to label the x-axis! - ggplot2::labs(x = "count") -``` - -One thing that the [web version of CellMarker](http://bio-bigdata.hrbmu.edu.cn/CellMarker/) allows us to do is to get a better idea of _marker prevalence_. -Some of the genes that show up in our top 100 may not be very specific. -It's also likely that kidney is not very relevant to our analysis of Hodgkin's lymphoma, but lymph node is highly relevant. -However, blood cells in kidney are not _unexpected_ and, if we view the [CellMarker tissue type browser](http://bio-bigdata.hrbmu.edu.cn/CellMarker/browse.jsp), we can see that there are _a lot_ of genes annotated to kidney. -Every resource will have some limitations "baked in." - -We could do additional filtering of the gene sets to remove irrelevant tissues and to remove marker genes that are not particularly specific. -(Fewer gene sets would mean a lower multiple hypothesis testing burden.) -If some of the CellMarker marker genes come from immunostaining of terminally differentiated cells, we might not expect their transcripts to show up in scRNA-seq data, for example. -All things to keep in mind if you use this resource for your own work! - -### Write results to file - -Write the results to file for both GO and CellMarker gene sets. - -```{r write_results, live = TRUE} -readr::write_tsv(go_result_df, file = go_results_file) -data.frame(cellmarker_ora_results@result) |> - readr::write_tsv(file = cm_results_file) -``` - -## Parting thoughts - -The goal of analyzing marker genes with ORA, like we did in this notebook, is to help us interpret our clustering results or annotate clusters. -We shouldn't use marker genes or any downstream analysis we perform with them to _justify_ cluster assignments; we generally want our cluster assignments to be data-driven. - -Just like it's important to be skeptical about p-values returned from the marker gene analysis itself, we should not get too attached to the idea of statistical significance here. -If we were to tweak the cutoffs (e.g., take the top 2500 genes), we might expect to get completely different significant results. -This is a limitation of ORA: we have to make some choices that are somewhat arbitrary. -It's a good idea to think about the goal of your analysis _upfront_ and pick cutoffs based on your goals. -That can help you avoid the temptation of trying a bunch of cutoffs until you get results that "look good." - -## Session Info - -```{r session_info} -sessionInfo() -``` diff --git a/scRNA-seq-advanced/04-overrepresentation_analysis.nb.html b/scRNA-seq-advanced/04-overrepresentation_analysis.nb.html deleted file mode 100644 index d064c476..00000000 --- a/scRNA-seq-advanced/04-overrepresentation_analysis.nb.html +++ /dev/null @@ -1,3868 +0,0 @@ - - - - - - - - - - - - - - - -Pathway analysis: Over-representation analysis (ORA) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -
-
-
-
-
- -
- - - - - - - - -
-

Objectives

-

This notebook will demonstrate how to:

-
    -
  • Perform gene identifier conversion with AnnotationDBI -annotation packages
  • -
  • Prepare gene sets for over-representation analysis, including an -appropriate background set
  • -
  • Perform over-representation analysis with the -clusterProfiler package
  • -
  • Introduce resources for ORA of scRNA-seq data, such as Gene -Ontology
  • -
-
-

In this notebook, we’ll cover a type of pathway or gene set analysis -called over-representation analysis (ORA). The idea behind ORA is -relatively straightforward: given a set of genes, do these genes overlap -with a pathway more than we expect by chance? The simplicity of only -requiring an input gene set (sort of, more on that below) can be -attractive.

-

ORA has some limitations, outlined nicely (and more extensively!) in -Khatri et -al. (2012). One of the main issues with ORA is that typically -all genes are treated as equal – the context of the magnitude of a -change we may be measuring is removed and each gene is treated as -independent, which can sometimes result in an incorrect estimate of -significance.

-

We will use the clusterProfiler -package (Yu et -al. 2012) to perform ORA. clusterProfiler has many -built-in functions that will run a specific type of analysis using a -specific source of pathways/gene sets automatically, but for our -purposes we’re going to keep things as general as possible. See the clusterProfiler -book for more information about the package’s full suite of -functionality.

-

Because different bioinformatics tools often require different types -of gene identifiers, we’ll also cover how to convert between gene -identifiers using AnnotationDbi -Bioconductor packages in this notebook. Check out the AnnotationDbi: -Introduction To Bioconductor Annotation Packages (Carlson 2020.) -vignette for more information.

-
-

Other resources

- -
-
-
-

Set up

-
-

Libraries

- - - -
# Package we'll use to run ORA
-library(clusterProfiler)
- - -
- - -
clusterProfiler v4.12.0  For help: https://yulab-smu.top/biomedical-knowledge-mining-book/
-
-If you use clusterProfiler in published research, please cite:
-T Wu, E Hu, S Xu, M Chen, P Guo, Z Dai, T Feng, L Zhou, W Tang, L Zhan, X Fu, S Liu, X Bo, and G Yu. clusterProfiler 4.0: A universal enrichment tool for interpreting omics data. The Innovation. 2021, 2(3):100141
- - -

-Attaching package: 'clusterProfiler'
- - -
The following object is masked from 'package:stats':
-
-    filter
- - -
# Package that contains MSigDB gene sets in tidy format
-library(msigdbr)
-# Homo sapiens annotation package we'll use for gene identifier conversion
-library(org.Hs.eg.db)
- - -
Loading required package: AnnotationDbi
- - -
Loading required package: stats4
- - -
Loading required package: BiocGenerics
- - -

-Attaching package: 'BiocGenerics'
- - -
The following objects are masked from 'package:stats':
-
-    IQR, mad, sd, var, xtabs
- - -
The following objects are masked from 'package:base':
-
-    anyDuplicated, aperm, append, as.data.frame, basename, cbind,
-    colnames, dirname, do.call, duplicated, eval, evalq, Filter, Find,
-    get, grep, grepl, intersect, is.unsorted, lapply, Map, mapply,
-    match, mget, order, paste, pmax, pmax.int, pmin, pmin.int,
-    Position, rank, rbind, Reduce, rownames, sapply, setdiff, table,
-    tapply, union, unique, unsplit, which.max, which.min
- - -
Loading required package: Biobase
- - -
Welcome to Bioconductor
-
-    Vignettes contain introductory material; view with
-    'browseVignettes()'. To cite Bioconductor, see
-    'citation("Biobase")', and for packages 'citation("pkgname")'.
- - -
Loading required package: IRanges
- - -
Loading required package: S4Vectors
- - -

-Attaching package: 'S4Vectors'
- - -
The following object is masked from 'package:clusterProfiler':
-
-    rename
- - -
The following object is masked from 'package:utils':
-
-    findMatches
- - -
The following objects are masked from 'package:base':
-
-    expand.grid, I, unname
- - -

-Attaching package: 'IRanges'
- - -
The following object is masked from 'package:clusterProfiler':
-
-    slice
- - -

-Attaching package: 'AnnotationDbi'
- - -
The following object is masked from 'package:clusterProfiler':
-
-    select
- - -
- - - -
-
-

Directories and files

-
-

Directories

- - - -
hodgkins_data_dir <- file.path("data", "hodgkins")
-# We'll use the output of the clustering notebook for ORA
-hodgkins_analysis_dir <- file.path("analysis", "hodgkins")
-
-# We'll create a directory to specifically hold the pathway results
-results_dir <- file.path(hodgkins_analysis_dir, "pathway-analysis")
-fs::dir_create(results_dir)
-
-# Create a directory for an updated marker list file
-fs::dir_create(file.path(hodgkins_analysis_dir, "markers"))
- - - -
-
-

Input files

-

We’re going to perform ORA on the marker genes from cluster 1 we -looked at in the previous notebook.

- - - -
input_file <- file.path(hodgkins_data_dir,
-                        "markers",
-                        "cluster01_markers.tsv")
- - - -

We’re going to use a file that is a cleaned version of the CellMarker (Zhang et al. -2018) human markers resources. You can see how we obtained and -cleaned this file in this -notebook.

- - - -
cellmarker_file <- file.path("gene-sets",
-                             "CellMarker_cleaned_human_markers.tsv")
- - - -
-
-

Output files

-

In the next notebook, we’ll use the cluster 1 results. Saving the -post-gene identifier conversion results in this notebook will prevent us -from duplicating effort (and code)!

- - - -
cluster1_output_file <- file.path(hodgkins_analysis_dir,
-                                  "markers",
-                                  "cluster01_markers_with_gene_symbols.tsv")
- - - -

We’ll save the table of ORA results (e.g., p-values).

- - - -
go_results_file <- file.path(results_dir, "cluster01_go_ora_results.tsv")
-cm_results_file <- file.path(results_dir,
-                             "cluster01_CellMarker_ora_results.tsv")
- - - -
-
-
-
-

Read in marker genes

-

Let’s read in our marker genes from cluster 1.

- - - -
markers_df <- readr::read_tsv(input_file)
- - -
Rows: 7264 Columns: 16
-── Column specification ────────────────────────────────────────────────────────
-Delimiter: "\t"
-chr  (1): gene
-dbl (15): p.value, FDR, summary.logFC, logFC.2, logFC.3, logFC.4, logFC.5, l...
-
-ℹ Use `spec()` to retrieve the full column specification for this data.
-ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
- - - -

And let’s take a peek!

- - - -
head(markers_df)
- -
- -
- - -
-

Gene identifier conversion

-

Our data frame of marker genes contains Ensembl gene identifiers. We -will need to convert from these identifiers into gene symbols or Entrez -IDs for our use with cell marker data.

-

We’re going to convert our identifiers to gene symbols because they -are a bit more human readable, but you can, with the change of a single -argument, use the same code to convert to many other types of -identifiers!

-

The annotation package org.Hs.eg.db contains information -for different identifiers. org.Hs.eg.db is specific to -Homo sapiens – this is what the Hs in the package -name is referencing. To perform gene identifier conversion in mouse -(Mus musculus) we could use org.Mm.eg.db; we would -use org.Dr.eg.db for zebrafish (Danio rerio).

-

We can see what types of IDs are available to us in an annotation -package with keytypes().

- - - -
keytypes(org.Hs.eg.db)
- - -
 [1] "ACCNUM"       "ALIAS"        "ENSEMBL"      "ENSEMBLPROT"  "ENSEMBLTRANS"
- [6] "ENTREZID"     "ENZYME"       "EVIDENCE"     "EVIDENCEALL"  "GENENAME"    
-[11] "GENETYPE"     "GO"           "GOALL"        "IPI"          "MAP"         
-[16] "OMIM"         "ONTOLOGY"     "ONTOLOGYALL"  "PATH"         "PFAM"        
-[21] "PMID"         "PROSITE"      "REFSEQ"       "SYMBOL"       "UCSCKG"      
-[26] "UNIPROT"     
- - -
ACCNUM
-ALIAS
-ENSEMBL
-ENSEMBLPROT
-ENSEMBLTRANS
-ENTREZID
-ENZYME
-EVIDENCE
-EVIDENCEALL
-GENENAME
-GENETYPE
-GO
-GOALL
-IPI
-MAP
-OMIM
-ONTOLOGY
-ONTOLOGYALL
-PATH
-PFAM
-PMID
-PROSITE
-REFSEQ
-SYMBOL
-UCSCKG
-UNIPROT
- - - -

Even though we’ll use this package to convert from Ensembl gene IDs -(ENSEMBL) to gene symbols (SYMBOL), we could -just as easily use it to convert from an Ensembl transcript ID -(ENSEMBLTRANS) to Entrez IDs (ENTREZID).

-

The function we will use to map from Ensembl gene IDs to gene symbols -is called mapIds().

- - - -
# This returns a named vector which we can convert to a data frame, where
-# the keys (Ensembl IDs) are the names
-symbols_vector <- mapIds(org.Hs.eg.db,  # Specify the annotation package
-                         # The vector of gene identifiers we want to
-                         # map
-                         keys = markers_df$gene,
-                         # What type of gene identifiers we're starting
-                         # with
-                         keytype = "ENSEMBL",
-                         # The type of gene identifier we want returned
-                         column = "SYMBOL",
-                         # In the case of 1:many mappings, return the
-                         # first one. This is default behavior!
-                         multiVals = "first")
- - -
'select()' returned 1:many mapping between keys and columns
- - -
# We would like a data frame we can join to the DGE results
-symbols_df <- data.frame(
-  ensembl_id = names(symbols_vector),
-  gene_symbol = symbols_vector
-) |>
-  # Drop genes that don't have a gene symbol!
-  tidyr::drop_na()
- - - -

This message is letting us know that sometimes Ensembl gene -identifiers will map to multiple gene symbols. In this case, it’s also -possible that a gene symbol will map to multiple Ensembl IDs.

-

Now we are ready to add the gene symbols to our data frame with the -marker genes. We can use a join function from the -dplyr package to do this, which will use the Ensembl gene -IDs in both data frames to determine how to join the rows.

- - - -
markers_df <- symbols_df |>
-  # An *inner* join will only return rows that are in both data frames
-  dplyr::inner_join(markers_df,
-                    # The name of the column that contains the Ensembl gene IDs
-                    # in the left data frame and right data frame, effectively
-                    # dropping genes that don't have gene symbols
-                    by = c("ensembl_id" = "gene"))
- - - -

We’re going to use this table again, with the gene symbols, in the -next notebook so let’s write it to a file.

- - - -
readr::write_tsv(markers_df, file = cluster1_output_file)
- - - -
-
-
-

Over-representation Analysis (ORA)

-

To test for over-representation, we can calculate a p-value with a -hypergeometric test (ref).

-

\(p = 1 - \displaystyle\sum_{i = -0}^{k-1}\frac{ {M \choose i}{ {N-M} \choose {n-i} } } { {N \choose n} -}\)

-

Where N is the number of genes in the background -distribution, M is the number of genes in a pathway or gene -set, n is the number of genes we are interested in (our -marker genes), and k is the number of genes that overlap -between the pathway or gene set and our marker genes.

-

Borrowing an example from clusterProfiler: -universal enrichment tool for functional and comparative study (Yu -):

-
-

Example: Suppose we have 17,980 genes detected in a -Microarray study and 57 genes were differentially expressed. Among the -differential expressed genes, 28 are annotated to a gene set.

-
-

We’ll call genes that are differentially expressed -gene_in_interest and genes that are in the gene set -in_gene_set.

- - - -
gene_table <- data.frame(
-  gene_not_interest = c(2613, 15310),
-  gene_in_interest = c(28, 29)
-)
-rownames(gene_table) <- c("in_gene_set", "not_in_gene_set")
-
-gene_table
- -
- -
- - -

We can assess if the 28 overlapping genes mean that the -differentially expressed genes are over-represented in the gene set with -the hypergeometric distribution. This corresponds to a one-sided -Fisher’s exact test.

- - - -
fisher.test(gene_table, alternative = "greater")
- - -

-    Fisher's Exact Test for Count Data
-
-data:  gene_table
-p-value = 1
-alternative hypothesis: true odds ratio is greater than 1
-95 percent confidence interval:
- 0.110242      Inf
-sample estimates:
-odds ratio 
- 0.1767937 
- - - -

When we test multiple pathways or gene sets, the -p-values then need to be adjusted for multiple -hypothesis testing.

-
-

Marker gene preparation

-
-

Top 100 cluster 1 genes

-

We’re interested in what pathways are over-represented in genes that -mark, or are more highly expressed in, cluster 1. We’ll select the top -100 genes for this comparison.

- - - -
cluster01_markers <- markers_df |>
-  # Positive fold changes
-  dplyr::filter(summary.logFC > 0) |>
-  # Take the "top 100" when ordering by FDR (smallest FDR values)
-  dplyr::slice_min(order_by = FDR, n = 100) |>
-  # Extract a vector of the gene symbols
-  dplyr::pull(gene_symbol)
- - - -

You may think that selecting the top 100 genes is pretty arbitrary -and you are not wrong! This also may seem strict, but let’s think about -our goal for this particular analysis: we hope that using ORA with -resources that encode some knowledge about cell types will help us -annotate cluster 1. We’re using the very top ranked genes in service of -this goal.

-

An alternative approach would be to have filtered to genes that are -below a specific FDR threshold, but that would also be an arbitrary -threshold that we chose. This is a weakness of over-representation -analysis; we generally need to pick a threshold for the genes that we -include in our tests.

-

As we mentioned in the clustering notebook, our statistical -calculations are circular here. We identified clusters based on gene -expression and then identified genes that are differentially expressed -between clusters. We can use the FDR values for ranking, like we are -here, but we shouldn’t use them to define “significance.” You can read a -little more about this in the Invalidity -of p-values section of the OSCA book.

-
-
-

Background set

-

As we saw above, calculating the p-value for ORA relies on the number -of genes in the background distribution. Sometimes folks consider genes -from the entire genome to comprise the background, but in the example -borrowed from the clusterProfiler authors, they state:

-
-

17,980 genes detected in a Microarray study

-
-

Where the key phrase is genes detected.

-

If we were unable to include a gene in our marker gene comparisons -because, for example, we could not reliably measure it, we shouldn’t -include in our background set!

-

Our markers data frame has all of the genes that we did not filter -out and have gene symbols corresponding to the Ensembl gene IDs, so we -can use the gene_symbol column as our background set.

- - - -
background_set <- markers_df$gene_symbol
- - - -
-
-
-

Gene Ontology ORA

-
-

A note on GO

-

The Gene Ontology (GO) (Ashburner et al. -2000; The Gene -Ontology Consortium. 2021)is an -ontology that documents or describes how genes function in the -biological domain and is comprised of 3 parts (GO ontology -documentation):

-
    -
  • Molecular function (MF): activities that occur on -the molecular level, often can be performed by individual genes
  • -
  • Biological process (BP): programs carried out by -multiple molecular entities; similar to what is sometimes thought of as -a pathway, but with no notion of dynamics
  • -
  • Cellular component (CC): cellular structures and -compartments
  • -
-

GO is “loosely hierarchical” (GO ontology -documentation). There are parent and child terms, where the “higher -up” a term is, the more general it is. This is somewhat abstract, so -let’s take a look at a GO term that’s likely to be relevant to our -Hodgkin’s lymphoma data to get an idea of what this is like in practice: -leukocyte activation.

-
-
-

Run enrichGO()

-

Now that we have our background set and our genes of interest, we’re -ready to run ORA using the enrichGO() function.

- - - -
go_ora_results <- enrichGO(gene = cluster01_markers,  # Top 100 genes
-                           universe = background_set,  # Background genes
-                           keyType = "SYMBOL",  # Our genes are gene symbols
-                           OrgDb = org.Hs.eg.db,  # Supports conversion, if needed
-                           ont = "BP",  # Biological process
-                           pAdjustMethod = "BH",  # FDR
-                           pvalueCutoff = 0.00001) # Very stringent for viz
- - - -

What is returned by enrichGO()?

- - - -
View(go_ora_results)
- - - -

The information we’re most likely interested in is in the -result slot. Let’s convert this into a data frame that we -can write to file.

- - - -
go_result_df <- data.frame(go_ora_results@result)
- - - -

Let’s take a look at the GO sets with an adjusted p-value less than -0.00001.

- - - -
go_result_df |>
-  dplyr::filter(p.adjust < 0.00001)
- -
- -
- - -

Some of these columns have names that are somewhat opaque and are -used for visualization, so let’s briefly talk about them here!

- ----- - - - - - - - - - - - - - - - - - - - -
Column nameNumeratorDenominator
GeneRatioNumber of genes in the top 100 marker genes and the specific gene -set (also in Count column)Number of genes in the top 100 marker genes and any gene -set
BgRatioNumber of genes in the background set and the specific gene setNumber of genes in the background set and any gene set
-
-
-

Visualizing results

-

We can use a dot plot to visualize our significant enrichment -results.

- - - -
enrichplot::dotplot(go_ora_results)
- - -

- - - -

What cell type do you think the cells from cluster 1 -are?

-

We can use an UpSet -plot to visualize the overlap between the gene sets -that were returned as significant.

- - - -
enrichplot::upsetplot(go_ora_results)
- - -

- - - -

It’s important to keep in mind that gene sets or pathways aren’t -independent, either! This is particularly true of GO terms, which are -hierarchical in nature.

-
-
-
-

CellMarker ORA

-

Next, we’ll use another resource, CellMarker, as additional support -for our conclusions based on GO.

-
-

About CellMarker

-

CellMarker -is a manually curated database of cell marker data for human and mouse -(Zhang et al. -2018). Studies/evidence of the following nature were included in the -manual curation process (Zhang et al. -2018):

-
    -
  • scRNA-seq
  • -
  • Flow cytometry
  • -
  • Immunostaining
  • -
-

And there are both gene and protein markers.

-

Let’s read in the CellMarker data we’ve already cleaned:

- - - -
cellmarker_df <- readr::read_tsv(cellmarker_file)
- - -
Rows: 30208 Columns: 2
-── Column specification ────────────────────────────────────────────────────────
-Delimiter: "\t"
-chr (2): cell_marker, gene_symbol
-
-ℹ Use `spec()` to retrieve the full column specification for this data.
-ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
- - - -
-
-

Run enricher()

-

clusterProfiler::enricher() is a more general way to -perform ORA – we specify the pathways or gene sets ourselves. We can use -the it to run ORA with the CellMarker data, which contains gene symbols. -That’s why we needed to perform gene identifier conversion earlier!

- - - -
cellmarker_ora_results <- enricher(
-  gene = cluster01_markers,  # Genes of interest
-  pvalueCutoff = 0.05,  # More permissive cut off
-  pAdjustMethod = "BH",  # FDR
-  universe = background_set,  # Background set
-  # The pathway information should be a data frame with a term name or
-  # identifier and the gene identifiers
-  TERM2GENE = cellmarker_df
-)
- - - -
-
-

Visualizing results

-

Let’s look at a bar plot of the significant results.

- - - -
barplot(cellmarker_ora_results) +
-  # This is a ggplot, so we can use the following to label the x-axis!
-  ggplot2::labs(x = "count")
- - -

- - - -

One thing that the web version of -CellMarker allows us to do is to get a better idea of marker -prevalence. Some of the genes that show up in our top 100 may not -be very specific. It’s also likely that kidney is not very relevant to -our analysis of Hodgkin’s lymphoma, but lymph node is highly relevant. -However, blood cells in kidney are not unexpected and, if we -view the CellMarker -tissue type browser, we can see that there are a lot of -genes annotated to kidney. Every resource will have some limitations -“baked in.”

-

We could do additional filtering of the gene sets to remove -irrelevant tissues and to remove marker genes that are not particularly -specific. (Fewer gene sets would mean a lower multiple hypothesis -testing burden.) If some of the CellMarker marker genes come from -immunostaining of terminally differentiated cells, we might not expect -their transcripts to show up in scRNA-seq data, for example. All things -to keep in mind if you use this resource for your own work!

-
-
-
-

Write results to file

-

Write the results to file for both GO and CellMarker gene sets.

- - - -
readr::write_tsv(go_result_df, file = go_results_file)
-data.frame(cellmarker_ora_results@result) |>
-  readr::write_tsv(file = cm_results_file)
- - - -
-
-
-

Parting thoughts

-

The goal of analyzing marker genes with ORA, like we did in this -notebook, is to help us interpret our clustering results or annotate -clusters. We shouldn’t use marker genes or any downstream analysis we -perform with them to justify cluster assignments; we generally -want our cluster assignments to be data-driven.

-

Just like it’s important to be skeptical about p-values returned from -the marker gene analysis itself, we should not get too attached to the -idea of statistical significance here. If we were to tweak the cutoffs -(e.g., take the top 2500 genes), we might expect to get completely -different significant results. This is a limitation of ORA: we have to -make some choices that are somewhat arbitrary. It’s a good idea to think -about the goal of your analysis upfront and pick cutoffs based -on your goals. That can help you avoid the temptation of trying a bunch -of cutoffs until you get results that “look good.”

-
-
-

Session Info

- - - -
sessionInfo()
- - -
R version 4.4.1 (2024-06-14)
-Platform: x86_64-pc-linux-gnu
-Running under: Ubuntu 22.04.4 LTS
-
-Matrix products: default
-BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
-LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0
-
-locale:
- [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
- [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
- [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
- [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
- [9] LC_ADDRESS=C               LC_TELEPHONE=C            
-[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
-
-time zone: Etc/UTC
-tzcode source: system (glibc)
-
-attached base packages:
-[1] stats4    stats     graphics  grDevices utils     datasets  methods  
-[8] base     
-
-other attached packages:
-[1] org.Hs.eg.db_3.19.1    AnnotationDbi_1.66.0   IRanges_2.38.0        
-[4] S4Vectors_0.42.0       Biobase_2.64.0         BiocGenerics_0.50.0   
-[7] msigdbr_7.5.1          clusterProfiler_4.12.0 optparse_1.7.5        
-
-loaded via a namespace (and not attached):
-  [1] DBI_1.2.2               gson_0.1.0              shadowtext_0.1.3       
-  [4] gridExtra_2.3           rlang_1.1.3             magrittr_2.0.3         
-  [7] DOSE_3.30.0             compiler_4.4.1          RSQLite_2.3.6          
- [10] png_0.1-8               vctrs_0.6.5             reshape2_1.4.4         
- [13] stringr_1.5.1           pkgconfig_2.0.3         crayon_1.5.2           
- [16] fastmap_1.1.1           XVector_0.44.0          labeling_0.4.3         
- [19] ggraph_2.2.1            utf8_1.2.4              HDO.db_0.99.1          
- [22] rmarkdown_2.26          tzdb_0.4.0              enrichplot_1.24.0      
- [25] UCSC.utils_1.0.0        purrr_1.0.2             bit_4.0.5              
- [28] xfun_0.43               zlibbioc_1.50.0         cachem_1.0.8           
- [31] aplot_0.2.2             GenomeInfoDb_1.40.0     jsonlite_1.8.8         
- [34] blob_1.2.4              highr_0.10              BiocParallel_1.38.0    
- [37] tweenr_2.0.3            parallel_4.4.1          R6_2.5.1               
- [40] bslib_0.7.0             stringi_1.8.3           RColorBrewer_1.1-3     
- [43] jquerylib_0.1.4         GOSemSim_2.30.0         Rcpp_1.0.12            
- [46] knitr_1.46              readr_2.1.5             Matrix_1.7-0           
- [49] splines_4.4.1           igraph_2.0.3            tidyselect_1.2.1       
- [52] qvalue_2.36.0           yaml_2.3.8              viridis_0.6.5          
- [55] codetools_0.2-20        lattice_0.22-6          tibble_3.2.1           
- [58] plyr_1.8.9              treeio_1.28.0           withr_3.0.0            
- [61] KEGGREST_1.44.0         evaluate_0.23           gridGraphics_0.5-1     
- [64] scatterpie_0.2.2        getopt_1.20.4           polyclip_1.10-6        
- [67] ggupset_0.3.0           Biostrings_2.72.0       ggtree_3.12.0          
- [70] pillar_1.9.0            ggfun_0.1.4             generics_0.1.3         
- [73] vroom_1.6.5             hms_1.1.3               ggplot2_3.5.1          
- [76] tidytree_0.4.6          munsell_0.5.1           scales_1.3.0           
- [79] glue_1.7.0              lazyeval_0.2.2          tools_4.4.1            
- [82] data.table_1.15.4       fgsea_1.30.0            babelgene_22.9         
- [85] fs_1.6.4                graphlayouts_1.1.1      fastmatch_1.1-4        
- [88] tidygraph_1.3.1         cowplot_1.1.3           grid_4.4.1             
- [91] ape_5.8                 tidyr_1.3.1             colorspace_2.1-0       
- [94] nlme_3.1-164            patchwork_1.2.0         GenomeInfoDbData_1.2.12
- [97] ggforce_0.4.2           cli_3.6.2               fansi_1.0.6            
-[100] viridisLite_0.4.2      
- [ reached getOption("max.print") -- omitted 15 entries ]
- - -
- -
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogT3Zlci1yZXByZXNlbnRhdGlvbiBhbmFseXNpcyAoT1JBKSIKb3V0cHV0OgogIGh0bWxfbm90ZWJvb2s6CiAgICB0b2M6IHRydWUKICAgIHRvY19mbG9hdDogdHJ1ZQphdXRob3I6IENDREwgZm9yIEFMU0YKZGF0ZTogMjAyMAotLS0KCiMjIE9iamVjdGl2ZXMKClRoaXMgbm90ZWJvb2sgd2lsbCBkZW1vbnN0cmF0ZSBob3cgdG86CgotIFBlcmZvcm0gZ2VuZSBpZGVudGlmaWVyIGNvbnZlcnNpb24gd2l0aCBbYEFubm90YXRpb25EQklgIGFubm90YXRpb24gcGFja2FnZXNdKGh0dHBzOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvdmlnbmV0dGVzL0Fubm90YXRpb25EYmkvaW5zdC9kb2MvSW50cm9Ub0Fubm90YXRpb25QYWNrYWdlcy5wZGYpCi0gUHJlcGFyZSBnZW5lIHNldHMgZm9yIG92ZXItcmVwcmVzZW50YXRpb24gYW5hbHlzaXMsIGluY2x1ZGluZyBhbiBhcHByb3ByaWF0ZSBiYWNrZ3JvdW5kIHNldAotIFBlcmZvcm0gb3Zlci1yZXByZXNlbnRhdGlvbiBhbmFseXNpcyB3aXRoIHRoZSBgY2x1c3RlclByb2ZpbGVyYCBwYWNrYWdlCi0gSW50cm9kdWNlIHJlc291cmNlcyBmb3IgT1JBIG9mIHNjUk5BLXNlcSBkYXRhLCBzdWNoIGFzIEdlbmUgT250b2xvZ3kKCi0tLQoKSW4gdGhpcyBub3RlYm9vaywgd2UnbGwgY292ZXIgYSB0eXBlIG9mIHBhdGh3YXkgb3IgZ2VuZSBzZXQgYW5hbHlzaXMgY2FsbGVkIG92ZXItcmVwcmVzZW50YXRpb24gYW5hbHlzaXMgKE9SQSkuClRoZSBpZGVhIGJlaGluZCBPUkEgaXMgcmVsYXRpdmVseSBzdHJhaWdodGZvcndhcmQ6IGdpdmVuIGEgc2V0IG9mIGdlbmVzLCBkbyB0aGVzZSBnZW5lcyBvdmVybGFwIHdpdGggYSBwYXRod2F5IG1vcmUgdGhhbiB3ZSBleHBlY3QgYnkgY2hhbmNlPwpUaGUgc2ltcGxpY2l0eSBvZiBvbmx5IHJlcXVpcmluZyBhbiBpbnB1dCBnZW5lIHNldCAoc29ydCBvZiwgbW9yZSBvbiB0aGF0IGJlbG93KSBjYW4gYmUgYXR0cmFjdGl2ZS4KCk9SQSBoYXMgc29tZSBsaW1pdGF0aW9ucywgb3V0bGluZWQgbmljZWx5IChhbmQgbW9yZSBleHRlbnNpdmVseSEpIGluIFtLaGF0cmkgX2V0IGFsLl8gKDIwMTIpXSggaHR0cHM6Ly9kb2kub3JnLzEwLjEzNzEvam91cm5hbC5wY2JpLjEwMDIzNzUpLgpPbmUgb2YgdGhlIG1haW4gaXNzdWVzIHdpdGggT1JBIGlzIHRoYXQgdHlwaWNhbGx5IGFsbCBnZW5lcyBhcmUgdHJlYXRlZCBhcyBlcXVhbCAtLSB0aGUgY29udGV4dCBvZiB0aGUgbWFnbml0dWRlIG9mIGEgY2hhbmdlIHdlIG1heSBiZSBtZWFzdXJpbmcgaXMgcmVtb3ZlZCBhbmQgZWFjaCBnZW5lIGlzIHRyZWF0ZWQgYXMgaW5kZXBlbmRlbnQsIHdoaWNoIGNhbiBzb21ldGltZXMgcmVzdWx0IGluIGFuIGluY29ycmVjdCBlc3RpbWF0ZSBvZiBzaWduaWZpY2FuY2UuCgpXZSB3aWxsIHVzZSB0aGUgW2BjbHVzdGVyUHJvZmlsZXJgIHBhY2thZ2VdKGh0dHBzOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvaHRtbC9jbHVzdGVyUHJvZmlsZXIuaHRtbCkgKFtZdSAqZXQgYWwuKiAyMDEyXShodHRwczovL2RvaS5vcmcvMTAuMTA4OS9vbWkuMjAxMS4wMTE4LikpIHRvIHBlcmZvcm0gT1JBLgpgY2x1c3RlclByb2ZpbGVyYCBoYXMgbWFueSBidWlsdC1pbiBmdW5jdGlvbnMgdGhhdCB3aWxsIHJ1biBhIHNwZWNpZmljIHR5cGUgb2YgYW5hbHlzaXMgdXNpbmcgYSBzcGVjaWZpYyBzb3VyY2Ugb2YgcGF0aHdheXMvZ2VuZSBzZXRzIGF1dG9tYXRpY2FsbHksIGJ1dCBmb3Igb3VyIHB1cnBvc2VzIHdlJ3JlIGdvaW5nIHRvIGtlZXAgdGhpbmdzIGFzIGdlbmVyYWwgYXMgcG9zc2libGUuClNlZSB0aGUgW2BjbHVzdGVyUHJvZmlsZXJgIGJvb2tdKGh0dHBzOi8veXVsYWItc211LmdpdGh1Yi5pby9jbHVzdGVyUHJvZmlsZXItYm9vay9pbmRleC5odG1sKSBmb3IgbW9yZSBpbmZvcm1hdGlvbiBhYm91dCB0aGUgcGFja2FnZSdzIGZ1bGwgc3VpdGUgb2YgZnVuY3Rpb25hbGl0eS4KCkJlY2F1c2UgZGlmZmVyZW50IGJpb2luZm9ybWF0aWNzIHRvb2xzIG9mdGVuIHJlcXVpcmUgZGlmZmVyZW50IHR5cGVzIG9mIGdlbmUgaWRlbnRpZmllcnMsIHdlJ2xsIGFsc28gY292ZXIgaG93IHRvIGNvbnZlcnQgYmV0d2VlbiBnZW5lIGlkZW50aWZpZXJzIHVzaW5nIFtgQW5ub3RhdGlvbkRiaWBdKGh0dHBzOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvaHRtbC9Bbm5vdGF0aW9uRGJpLmh0bWwpIEJpb2NvbmR1Y3RvciBwYWNrYWdlcyBpbiB0aGlzIG5vdGVib29rLgpDaGVjayBvdXQgdGhlIFtfQW5ub3RhdGlvbkRiaTogSW50cm9kdWN0aW9uIFRvIEJpb2NvbmR1Y3RvciBBbm5vdGF0aW9uIFBhY2thZ2VzXyAoQ2FybHNvbiAyMDIwLikgdmlnbmV0dGVdKGh0dHBzOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvdmlnbmV0dGVzL0Fubm90YXRpb25EYmkvaW5zdC9kb2MvSW50cm9Ub0Fubm90YXRpb25QYWNrYWdlcy5wZGYpIGZvciBtb3JlIGluZm9ybWF0aW9uLgoKIyMjIyBPdGhlciByZXNvdXJjZXMKCiogRm9yIGFub3RoZXIgZXhhbXBsZSB1c2luZyBgY2x1c3RlclByb2ZpbGVyYCwgc2VlIFtfSW50cm8gdG8gREdFOiBGdW5jdGlvbmFsIEFuYWx5c2lzLl8gZnJvbSBIYXJ2YXJkIENoYW4gQmlvaW5mb3JtYXRpY3MgQ29yZSBUcmFpbmluZy5dKGh0dHBzOi8vaGJjdHJhaW5pbmcuZ2l0aHViLmlvL0RHRV93b3Jrc2hvcC9sZXNzb25zLzA5X2Z1bmN0aW9uYWxfYW5hbHlzaXMuaHRtbCkKKiBbYFdlYkdlc3RhbHRSYF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL1dlYkdlc3RhbHRSLykgaXMgYW5vdGhlciBSIHBhY2thZ2UgdGhhdCBjYW4gYmUgdXNlZCBmb3IgT1JBLgoqIE9TQ0EgaGFzIFthIHNlY3Rpb24gb24gdXNpbmcgR2VuZSBPbnRvbG9neSBzZXRzIHRvIGFubm90YXRlIGNsdXN0ZXJzXShodHRwczovL2Jpb2NvbmR1Y3Rvci5vcmcvYm9va3MvMy4xNC9PU0NBLmJhc2ljL2NlbGwtdHlwZS1hbm5vdGF0aW9uLmh0bWwjYXNzaWduaW5nLWNsdXN0ZXItbGFiZWxzLWZyb20tbWFya2VycykgdXNpbmcgYSBkaWZmZXJlbnQgQmlvY29uZHVjdG9yIHBhY2thZ2UuCgojIyBTZXQgdXAKCiMjIyBMaWJyYXJpZXMKCmBgYHtyIGxpYnJhcmllc30KIyBQYWNrYWdlIHdlJ2xsIHVzZSB0byBydW4gT1JBCmxpYnJhcnkoY2x1c3RlclByb2ZpbGVyKQojIFBhY2thZ2UgdGhhdCBjb250YWlucyBNU2lnREIgZ2VuZSBzZXRzIGluIHRpZHkgZm9ybWF0CmxpYnJhcnkobXNpZ2RicikKIyBIb21vIHNhcGllbnMgYW5ub3RhdGlvbiBwYWNrYWdlIHdlJ2xsIHVzZSBmb3IgZ2VuZSBpZGVudGlmaWVyIGNvbnZlcnNpb24KbGlicmFyeShvcmcuSHMuZWcuZGIpCmBgYAoKIyMjIERpcmVjdG9yaWVzIGFuZCBmaWxlcwoKIyMjIyBEaXJlY3RvcmllcwoKYGBge3IgY3JlYXRlX29yYV9kaXJlY3RvcnksIGxpdmUgPSBUUlVFfQpob2Rna2luc19kYXRhX2RpciA8LSBmaWxlLnBhdGgoImRhdGEiLCAiaG9kZ2tpbnMiKQojIFdlJ2xsIHVzZSB0aGUgb3V0cHV0IG9mIHRoZSBjbHVzdGVyaW5nIG5vdGVib29rIGZvciBPUkEKaG9kZ2tpbnNfYW5hbHlzaXNfZGlyIDwtIGZpbGUucGF0aCgiYW5hbHlzaXMiLCAiaG9kZ2tpbnMiKQoKIyBXZSdsbCBjcmVhdGUgYSBkaXJlY3RvcnkgdG8gc3BlY2lmaWNhbGx5IGhvbGQgdGhlIHBhdGh3YXkgcmVzdWx0cwpyZXN1bHRzX2RpciA8LSBmaWxlLnBhdGgoaG9kZ2tpbnNfYW5hbHlzaXNfZGlyLCAicGF0aHdheS1hbmFseXNpcyIpCmZzOjpkaXJfY3JlYXRlKHJlc3VsdHNfZGlyKQoKIyBDcmVhdGUgYSBkaXJlY3RvcnkgZm9yIGFuIHVwZGF0ZWQgbWFya2VyIGxpc3QgZmlsZQpmczo6ZGlyX2NyZWF0ZShmaWxlLnBhdGgoaG9kZ2tpbnNfYW5hbHlzaXNfZGlyLCAibWFya2VycyIpKQpgYGAKCiMjIyMgSW5wdXQgZmlsZXMKCldlJ3JlIGdvaW5nIHRvIHBlcmZvcm0gT1JBIG9uIHRoZSBtYXJrZXIgZ2VuZXMgZnJvbSBjbHVzdGVyIDEgd2UgbG9va2VkIGF0IGluIHRoZSBwcmV2aW91cyBub3RlYm9vay4KCmBgYHtyIGNsdXN0ZXJfaW5wdXR9CmlucHV0X2ZpbGUgPC0gZmlsZS5wYXRoKGhvZGdraW5zX2RhdGFfZGlyLAogICAgICAgICAgICAgICAgICAgICAgICAibWFya2VycyIsCiAgICAgICAgICAgICAgICAgICAgICAgICJjbHVzdGVyMDFfbWFya2Vycy50c3YiKQpgYGAKCldlJ3JlIGdvaW5nIHRvIHVzZSBhIGZpbGUgdGhhdCBpcyBhIGNsZWFuZWQgdmVyc2lvbiBvZiB0aGUgW0NlbGxNYXJrZXJdKGh0dHA6Ly9iaW8tYmlnZGF0YS5ocmJtdS5lZHUuY24vQ2VsbE1hcmtlci8pIChbWmhhbmcgKmV0IGFsLiogMjAxOF0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwOTMvbmFyL2dreTkwMCkpIGh1bWFuIG1hcmtlcnMgcmVzb3VyY2VzLgpZb3UgY2FuIHNlZSBob3cgd2Ugb2J0YWluZWQgYW5kIGNsZWFuZWQgdGhpcyBmaWxlIGluIFt0aGlzIG5vdGVib29rXShodHRwczovL2dpdGh1Yi5jb20vQWxleHNMZW1vbmFkZS90cmFpbmluZy1tb2R1bGVzL2Jsb2IvbWFzdGVyL3NjUk5BLXNlcS9zZXR1cC9jZWxsbWFya2VyX2dlbmVzLlJtZCkuCgpgYGB7ciBjbV9pbnB1dH0KY2VsbG1hcmtlcl9maWxlIDwtIGZpbGUucGF0aCgiZ2VuZS1zZXRzIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAiQ2VsbE1hcmtlcl9jbGVhbmVkX2h1bWFuX21hcmtlcnMudHN2IikKYGBgCgojIyMjIE91dHB1dCBmaWxlcwoKSW4gdGhlIG5leHQgbm90ZWJvb2ssIHdlJ2xsIHVzZSB0aGUgY2x1c3RlciAxIHJlc3VsdHMuClNhdmluZyB0aGUgcG9zdC1nZW5lIGlkZW50aWZpZXIgY29udmVyc2lvbiByZXN1bHRzIGluIHRoaXMgbm90ZWJvb2sgd2lsbCBwcmV2ZW50IHVzIGZyb20gZHVwbGljYXRpbmcgZWZmb3J0IChhbmQgY29kZSkhCgpgYGB7ciBnZW5lX2lkX2ZpbGV9CmNsdXN0ZXIxX291dHB1dF9maWxlIDwtIGZpbGUucGF0aChob2Rna2luc19hbmFseXNpc19kaXIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAibWFya2VycyIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAiY2x1c3RlcjAxX21hcmtlcnNfd2l0aF9nZW5lX3N5bWJvbHMudHN2IikKYGBgCgpXZSdsbCBzYXZlIHRoZSB0YWJsZSBvZiBPUkEgcmVzdWx0cyAoZS5nLiwgcC12YWx1ZXMpLgoKYGBge3Igb3V0cHV0X2ZpbGVzLCBsaXZlID0gVFJVRX0KZ29fcmVzdWx0c19maWxlIDwtIGZpbGUucGF0aChyZXN1bHRzX2RpciwgImNsdXN0ZXIwMV9nb19vcmFfcmVzdWx0cy50c3YiKQpjbV9yZXN1bHRzX2ZpbGUgPC0gZmlsZS5wYXRoKHJlc3VsdHNfZGlyLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICJjbHVzdGVyMDFfQ2VsbE1hcmtlcl9vcmFfcmVzdWx0cy50c3YiKQpgYGAKCiMjIFJlYWQgaW4gbWFya2VyIGdlbmVzCgpMZXQncyByZWFkIGluIG91ciBtYXJrZXIgZ2VuZXMgZnJvbSBjbHVzdGVyIDEuCgpgYGB7ciByZWFkX2luX21hcmtlcnMsIGxpdmUgPSBUUlVFfQptYXJrZXJzX2RmIDwtIHJlYWRyOjpyZWFkX3RzdihpbnB1dF9maWxlKQpgYGAKCkFuZCBsZXQncyB0YWtlIGEgcGVlayEKCmBgYHtyIG1hcmtlcl9oZWFkLCBsaXZlID0gVFJVRX0KaGVhZChtYXJrZXJzX2RmKQpgYGAKCiMjIyBHZW5lIGlkZW50aWZpZXIgY29udmVyc2lvbgoKT3VyIGRhdGEgZnJhbWUgb2YgbWFya2VyIGdlbmVzIGNvbnRhaW5zIEVuc2VtYmwgZ2VuZSBpZGVudGlmaWVycy4KV2Ugd2lsbCBuZWVkIHRvIGNvbnZlcnQgZnJvbSB0aGVzZSBpZGVudGlmaWVycyBpbnRvIGdlbmUgc3ltYm9scyBvciBFbnRyZXogSURzIGZvciBvdXIgdXNlIHdpdGggY2VsbCBtYXJrZXIgZGF0YS4KCldlJ3JlIGdvaW5nIHRvIGNvbnZlcnQgb3VyIGlkZW50aWZpZXJzIHRvIGdlbmUgc3ltYm9scyBiZWNhdXNlIHRoZXkgYXJlIGEgYml0IG1vcmUgaHVtYW4gcmVhZGFibGUsIGJ1dCB5b3UgY2FuLCB3aXRoIHRoZSBjaGFuZ2Ugb2YgYSBzaW5nbGUgYXJndW1lbnQsIHVzZSB0aGUgc2FtZSBjb2RlIHRvIGNvbnZlcnQgdG8gbWFueSBvdGhlciB0eXBlcyBvZiBpZGVudGlmaWVycyEKClRoZSBhbm5vdGF0aW9uIHBhY2thZ2UgYG9yZy5Icy5lZy5kYmAgY29udGFpbnMgaW5mb3JtYXRpb24gZm9yIGRpZmZlcmVudCBpZGVudGlmaWVycy4KYG9yZy5Icy5lZy5kYmAgaXMgc3BlY2lmaWMgdG8gX0hvbW8gc2FwaWVuc18gLS0gdGhpcyBpcyB3aGF0IHRoZSBgSHNgIGluIHRoZSBwYWNrYWdlIG5hbWUgaXMgcmVmZXJlbmNpbmcuClRvIHBlcmZvcm0gZ2VuZSBpZGVudGlmaWVyIGNvbnZlcnNpb24gaW4gbW91c2UgKF9NdXMgbXVzY3VsdXNfKSB3ZSBjb3VsZCB1c2UgYG9yZy5NbS5lZy5kYmA7CndlIHdvdWxkIHVzZSBgb3JnLkRyLmVnLmRiYCBmb3IgemVicmFmaXNoIChfRGFuaW8gcmVyaW9fKS4KCldlIGNhbiBzZWUgd2hhdCB0eXBlcyBvZiBJRHMgYXJlIGF2YWlsYWJsZSB0byB1cyBpbiBhbiBhbm5vdGF0aW9uIHBhY2thZ2Ugd2l0aCBga2V5dHlwZXMoKWAuCgpgYGB7ciBrZXl0eXBlcywgbGl2ZSA9IFRSVUV9CmtleXR5cGVzKG9yZy5Icy5lZy5kYikKYGBgCgpFdmVuIHRob3VnaCB3ZSdsbCB1c2UgdGhpcyBwYWNrYWdlIHRvIGNvbnZlcnQgZnJvbSBFbnNlbWJsIGdlbmUgSURzIChgRU5TRU1CTGApIHRvIGdlbmUgc3ltYm9scyAoYFNZTUJPTGApLCB3ZSBjb3VsZCBqdXN0IGFzIGVhc2lseSB1c2UgaXQgdG8gY29udmVydCBmcm9tIGFuIEVuc2VtYmwgdHJhbnNjcmlwdCBJRCAoYEVOU0VNQkxUUkFOU2ApIHRvIEVudHJleiBJRHMgKGBFTlRSRVpJRGApLgoKVGhlIGZ1bmN0aW9uIHdlIHdpbGwgdXNlIHRvIG1hcCBmcm9tIEVuc2VtYmwgZ2VuZSBJRHMgdG8gZ2VuZSBzeW1ib2xzIGlzIGNhbGxlZCBgbWFwSWRzKClgLgoKYGBge3IgbWFwX3RvX3N5bWJvbH0KIyBUaGlzIHJldHVybnMgYSBuYW1lZCB2ZWN0b3Igd2hpY2ggd2UgY2FuIGNvbnZlcnQgdG8gYSBkYXRhIGZyYW1lLCB3aGVyZQojIHRoZSBrZXlzIChFbnNlbWJsIElEcykgYXJlIHRoZSBuYW1lcwpzeW1ib2xzX3ZlY3RvciA8LSBtYXBJZHMob3JnLkhzLmVnLmRiLCAgIyBTcGVjaWZ5IHRoZSBhbm5vdGF0aW9uIHBhY2thZ2UKICAgICAgICAgICAgICAgICAgICAgICAgICMgVGhlIHZlY3RvciBvZiBnZW5lIGlkZW50aWZpZXJzIHdlIHdhbnQgdG8KICAgICAgICAgICAgICAgICAgICAgICAgICMgbWFwCiAgICAgICAgICAgICAgICAgICAgICAgICBrZXlzID0gbWFya2Vyc19kZiRnZW5lLAogICAgICAgICAgICAgICAgICAgICAgICAgIyBXaGF0IHR5cGUgb2YgZ2VuZSBpZGVudGlmaWVycyB3ZSdyZSBzdGFydGluZwogICAgICAgICAgICAgICAgICAgICAgICAgIyB3aXRoCiAgICAgICAgICAgICAgICAgICAgICAgICBrZXl0eXBlID0gIkVOU0VNQkwiLAogICAgICAgICAgICAgICAgICAgICAgICAgIyBUaGUgdHlwZSBvZiBnZW5lIGlkZW50aWZpZXIgd2Ugd2FudCByZXR1cm5lZAogICAgICAgICAgICAgICAgICAgICAgICAgY29sdW1uID0gIlNZTUJPTCIsCiAgICAgICAgICAgICAgICAgICAgICAgICAjIEluIHRoZSBjYXNlIG9mIDE6bWFueSBtYXBwaW5ncywgcmV0dXJuIHRoZQogICAgICAgICAgICAgICAgICAgICAgICAgIyBmaXJzdCBvbmUuIFRoaXMgaXMgZGVmYXVsdCBiZWhhdmlvciEKICAgICAgICAgICAgICAgICAgICAgICAgIG11bHRpVmFscyA9ICJmaXJzdCIpCgojIFdlIHdvdWxkIGxpa2UgYSBkYXRhIGZyYW1lIHdlIGNhbiBqb2luIHRvIHRoZSBER0UgcmVzdWx0cwpzeW1ib2xzX2RmIDwtIGRhdGEuZnJhbWUoCiAgZW5zZW1ibF9pZCA9IG5hbWVzKHN5bWJvbHNfdmVjdG9yKSwKICBnZW5lX3N5bWJvbCA9IHN5bWJvbHNfdmVjdG9yCikgfD4KICAjIERyb3AgZ2VuZXMgdGhhdCBkb24ndCBoYXZlIGEgZ2VuZSBzeW1ib2whCiAgdGlkeXI6OmRyb3BfbmEoKQpgYGAKClRoaXMgbWVzc2FnZSBpcyBsZXR0aW5nIHVzIGtub3cgdGhhdCBzb21ldGltZXMgRW5zZW1ibCBnZW5lIGlkZW50aWZpZXJzIHdpbGwgbWFwIHRvIG11bHRpcGxlIGdlbmUgc3ltYm9scy4KSW4gdGhpcyBjYXNlLCBpdCdzIGFsc28gcG9zc2libGUgdGhhdCBhIGdlbmUgc3ltYm9sIHdpbGwgbWFwIHRvIG11bHRpcGxlIEVuc2VtYmwgSURzLgoKTm93IHdlIGFyZSByZWFkeSB0byBhZGQgdGhlIGdlbmUgc3ltYm9scyB0byBvdXIgZGF0YSBmcmFtZSB3aXRoIHRoZSBtYXJrZXIgZ2VuZXMuCldlIGNhbiB1c2UgYSBfam9pbl8gZnVuY3Rpb24gZnJvbSB0aGUgYGRwbHlyYCBwYWNrYWdlIHRvIGRvIHRoaXMsIHdoaWNoIHdpbGwgdXNlIHRoZSBFbnNlbWJsIGdlbmUgSURzIGluIGJvdGggZGF0YSBmcmFtZXMgdG8gZGV0ZXJtaW5lIGhvdyB0byBqb2luIHRoZSByb3dzLgoKYGBge3IgYWRkX3N5bWJvbHMsIGxpdmUgPSBUUlVFfQptYXJrZXJzX2RmIDwtIHN5bWJvbHNfZGYgfD4KICAjIEFuICppbm5lciogam9pbiB3aWxsIG9ubHkgcmV0dXJuIHJvd3MgdGhhdCBhcmUgaW4gYm90aCBkYXRhIGZyYW1lcwogIGRwbHlyOjppbm5lcl9qb2luKG1hcmtlcnNfZGYsCiAgICAgICAgICAgICAgICAgICAgIyBUaGUgbmFtZSBvZiB0aGUgY29sdW1uIHRoYXQgY29udGFpbnMgdGhlIEVuc2VtYmwgZ2VuZSBJRHMKICAgICAgICAgICAgICAgICAgICAjIGluIHRoZSBsZWZ0IGRhdGEgZnJhbWUgYW5kIHJpZ2h0IGRhdGEgZnJhbWUsIGVmZmVjdGl2ZWx5CiAgICAgICAgICAgICAgICAgICAgIyBkcm9wcGluZyBnZW5lcyB0aGF0IGRvbid0IGhhdmUgZ2VuZSBzeW1ib2xzCiAgICAgICAgICAgICAgICAgICAgYnkgPSBjKCJlbnNlbWJsX2lkIiA9ICJnZW5lIikpCmBgYAoKV2UncmUgZ29pbmcgdG8gdXNlIHRoaXMgdGFibGUgYWdhaW4sIHdpdGggdGhlIGdlbmUgc3ltYm9scywgaW4gdGhlIG5leHQgbm90ZWJvb2sgc28gbGV0J3Mgd3JpdGUgaXQgdG8gYSBmaWxlLgoKYGBge3Igc2F2ZV9nZW5lX3N5bWJvbHMsIGxpdmUgPSBUUlVFfQpyZWFkcjo6d3JpdGVfdHN2KG1hcmtlcnNfZGYsIGZpbGUgPSBjbHVzdGVyMV9vdXRwdXRfZmlsZSkKYGBgCgojIyBPdmVyLXJlcHJlc2VudGF0aW9uIEFuYWx5c2lzIChPUkEpCgpUbyB0ZXN0IGZvciBvdmVyLXJlcHJlc2VudGF0aW9uLCB3ZSBjYW4gY2FsY3VsYXRlIGEgcC12YWx1ZSB3aXRoIGEgaHlwZXJnZW9tZXRyaWMgdGVzdCAoW3JlZl0oaHR0cHM6Ly95dWxhYi1zbXUuZ2l0aHViLmlvL2NsdXN0ZXJQcm9maWxlci1ib29rL2NoYXB0ZXIyLmh0bWwjb3Zlci1yZXByZXNlbnRhdGlvbi1hbmFseXNpcykpLgoKXChwID0gMSAtIFxkaXNwbGF5c3R5bGVcc3VtX3tpID0gMH1ee2stMX1cZnJhY3sge00gXGNob29zZSBpfXsge04tTX0gXGNob29zZSB7bi1pfSB9IH0geyB7TiBcY2hvb3NlIG59IH1cKQoKV2hlcmUgYE5gIGlzIHRoZSBudW1iZXIgb2YgZ2VuZXMgaW4gdGhlIGJhY2tncm91bmQgZGlzdHJpYnV0aW9uLCBgTWAgaXMgdGhlIG51bWJlciBvZiBnZW5lcyBpbiBhIHBhdGh3YXkgb3IgZ2VuZSBzZXQsIGBuYCBpcyB0aGUgbnVtYmVyIG9mIGdlbmVzIHdlIGFyZSBpbnRlcmVzdGVkIGluIChvdXIgbWFya2VyIGdlbmVzKSwgYW5kIGBrYCBpcyB0aGUgbnVtYmVyIG9mIGdlbmVzIHRoYXQgb3ZlcmxhcCBiZXR3ZWVuIHRoZSBwYXRod2F5IG9yIGdlbmUgc2V0IGFuZCBvdXIgbWFya2VyIGdlbmVzLgoKQm9ycm93aW5nIGFuIGV4YW1wbGUgZnJvbSBbX2NsdXN0ZXJQcm9maWxlcjogdW5pdmVyc2FsIGVucmljaG1lbnQgdG9vbCBmb3IgZnVuY3Rpb25hbCBhbmQgY29tcGFyYXRpdmUgc3R1ZHlfIChZdSApXShodHRwOi8veXVsYWItc211LnRvcC9jbHVzdGVyUHJvZmlsZXItYm9vay9jaGFwdGVyMi5odG1sI292ZXItcmVwcmVzZW50YXRpb24tYW5hbHlzaXMpOgoKPiAqKkV4YW1wbGUqKjogU3VwcG9zZSB3ZSBoYXZlIDE3LDk4MCBnZW5lcyBkZXRlY3RlZCBpbiBhIE1pY3JvYXJyYXkgc3R1ZHkgYW5kIDU3IGdlbmVzIHdlcmUgZGlmZmVyZW50aWFsbHkgZXhwcmVzc2VkLiBBbW9uZyB0aGUgZGlmZmVyZW50aWFsIGV4cHJlc3NlZCBnZW5lcywgMjggYXJlIGFubm90YXRlZCB0byBhIGdlbmUgc2V0LgoKV2UnbGwgY2FsbCBnZW5lcyB0aGF0IGFyZSBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgYGdlbmVfaW5faW50ZXJlc3RgIGFuZCBnZW5lcyB0aGF0IGFyZSBpbiB0aGUgZ2VuZSBzZXQgYGluX2dlbmVfc2V0YC4KCmBgYHtyIGdlbmVfdGFibGV9CmdlbmVfdGFibGUgPC0gZGF0YS5mcmFtZSgKICBnZW5lX25vdF9pbnRlcmVzdCA9IGMoMjYxMywgMTUzMTApLAogIGdlbmVfaW5faW50ZXJlc3QgPSBjKDI4LCAyOSkKKQpyb3duYW1lcyhnZW5lX3RhYmxlKSA8LSBjKCJpbl9nZW5lX3NldCIsICJub3RfaW5fZ2VuZV9zZXQiKQoKZ2VuZV90YWJsZQpgYGAKCldlIGNhbiBhc3Nlc3MgaWYgdGhlIDI4IG92ZXJsYXBwaW5nIGdlbmVzIG1lYW4gdGhhdCB0aGUgZGlmZmVyZW50aWFsbHkgZXhwcmVzc2VkIGdlbmVzIGFyZSBvdmVyLXJlcHJlc2VudGVkIGluIHRoZSBnZW5lIHNldCB3aXRoIHRoZSBoeXBlcmdlb21ldHJpYyBkaXN0cmlidXRpb24uClRoaXMgY29ycmVzcG9uZHMgdG8gYSBvbmUtc2lkZWQgRmlzaGVyJ3MgZXhhY3QgdGVzdC4KCmBgYHtyIGZpc2hlcl90ZXN0fQpmaXNoZXIudGVzdChnZW5lX3RhYmxlLCBhbHRlcm5hdGl2ZSA9ICJncmVhdGVyIikKYGBgCgpXaGVuIHdlIHRlc3QgKiptdWx0aXBsZSBwYXRod2F5cyBvciBnZW5lIHNldHMqKiwgdGhlIHAtdmFsdWVzIHRoZW4gbmVlZCB0byBiZSAqKmFkanVzdGVkKiogZm9yIG11bHRpcGxlIGh5cG90aGVzaXMgdGVzdGluZy4KCiMjIyBNYXJrZXIgZ2VuZSBwcmVwYXJhdGlvbgoKIyMjIyBUb3AgMTAwIGNsdXN0ZXIgMSBnZW5lcwoKV2UncmUgaW50ZXJlc3RlZCBpbiB3aGF0IHBhdGh3YXlzIGFyZSBvdmVyLXJlcHJlc2VudGVkIGluIGdlbmVzIHRoYXQgbWFyaywgb3IgYXJlIG1vcmUgaGlnaGx5IGV4cHJlc3NlZCBpbiwgY2x1c3RlciAxLgpXZSdsbCBzZWxlY3QgdGhlIHRvcCAxMDAgZ2VuZXMgZm9yIHRoaXMgY29tcGFyaXNvbi4KCmBgYHtyIGNsdXN0ZXIwMV9nZW5lc30KY2x1c3RlcjAxX21hcmtlcnMgPC0gbWFya2Vyc19kZiB8PgogICMgUG9zaXRpdmUgZm9sZCBjaGFuZ2VzCiAgZHBseXI6OmZpbHRlcihzdW1tYXJ5LmxvZ0ZDID4gMCkgfD4KICAjIFRha2UgdGhlICJ0b3AgMTAwIiB3aGVuIG9yZGVyaW5nIGJ5IEZEUiAoc21hbGxlc3QgRkRSIHZhbHVlcykKICBkcGx5cjo6c2xpY2VfbWluKG9yZGVyX2J5ID0gRkRSLCBuID0gMTAwKSB8PgogICMgRXh0cmFjdCBhIHZlY3RvciBvZiB0aGUgZ2VuZSBzeW1ib2xzCiAgZHBseXI6OnB1bGwoZ2VuZV9zeW1ib2wpCmBgYAoKWW91IG1heSB0aGluayB0aGF0IHNlbGVjdGluZyB0aGUgdG9wIDEwMCBnZW5lcyBpcyBwcmV0dHkgYXJiaXRyYXJ5IGFuZCB5b3UgYXJlIG5vdCB3cm9uZyEKVGhpcyBhbHNvIG1heSBzZWVtIHN0cmljdCwgYnV0IGxldCdzIHRoaW5rIGFib3V0IG91ciBnb2FsIGZvciB0aGlzIHBhcnRpY3VsYXIgYW5hbHlzaXM6IHdlIGhvcGUgdGhhdCB1c2luZyBPUkEgd2l0aCByZXNvdXJjZXMgdGhhdCBlbmNvZGUgc29tZSBrbm93bGVkZ2UgYWJvdXQgY2VsbCB0eXBlcyB3aWxsIGhlbHAgdXMgYW5ub3RhdGUgY2x1c3RlciAxLgpXZSdyZSB1c2luZyB0aGUgdmVyeSB0b3AgcmFua2VkIGdlbmVzIGluIHNlcnZpY2Ugb2YgdGhpcyBnb2FsLgoKQW4gYWx0ZXJuYXRpdmUgYXBwcm9hY2ggd291bGQgYmUgdG8gaGF2ZSBmaWx0ZXJlZCB0byBnZW5lcyB0aGF0IGFyZSBiZWxvdyBhIHNwZWNpZmljIEZEUiB0aHJlc2hvbGQsIGJ1dCB0aGF0IHdvdWxkIGFsc28gYmUgYW4gYXJiaXRyYXJ5IHRocmVzaG9sZCB0aGF0IHdlIGNob3NlLgpUaGlzIGlzIGEgd2Vha25lc3Mgb2Ygb3Zlci1yZXByZXNlbnRhdGlvbiBhbmFseXNpczsgd2UgZ2VuZXJhbGx5IG5lZWQgdG8gcGljayBhIHRocmVzaG9sZCBmb3IgdGhlIGdlbmVzIHRoYXQgd2UgaW5jbHVkZSBpbiBvdXIgdGVzdHMuCgpBcyB3ZSBtZW50aW9uZWQgaW4gdGhlIGNsdXN0ZXJpbmcgbm90ZWJvb2ssIG91ciBzdGF0aXN0aWNhbCBjYWxjdWxhdGlvbnMgYXJlIGNpcmN1bGFyIGhlcmUuCldlIGlkZW50aWZpZWQgY2x1c3RlcnMgYmFzZWQgb24gZ2VuZSBleHByZXNzaW9uIGFuZCB0aGVuIGlkZW50aWZpZWQgZ2VuZXMgdGhhdCBhcmUgZGlmZmVyZW50aWFsbHkgZXhwcmVzc2VkIGJldHdlZW4gY2x1c3RlcnMuCldlIGNhbiB1c2UgdGhlIEZEUiB2YWx1ZXMgZm9yIHJhbmtpbmcsIGxpa2Ugd2UgYXJlIGhlcmUsIGJ1dCB3ZSBzaG91bGRuJ3QgdXNlIHRoZW0gdG8gZGVmaW5lICJzaWduaWZpY2FuY2UuIgpZb3UgY2FuIHJlYWQgYSBsaXR0bGUgbW9yZSBhYm91dCB0aGlzIGluIHRoZSBbX0ludmFsaWRpdHkgb2YgcC12YWx1ZXNfIHNlY3Rpb24gb2YgdGhlIE9TQ0EgYm9va10oaHR0cHM6Ly9iaW9jb25kdWN0b3Iub3JnL2Jvb2tzLzMuMTQvT1NDQS5hZHZhbmNlZC9tYXJrZXItZGV0ZWN0aW9uLXJlZHV4Lmh0bWwjcC12YWx1ZS1pbnZhbGlkaXR5KS4KCiMjIyMgQmFja2dyb3VuZCBzZXQKCkFzIHdlIHNhdyBhYm92ZSwgY2FsY3VsYXRpbmcgdGhlIHAtdmFsdWUgZm9yIE9SQSByZWxpZXMgb24gdGhlIG51bWJlciBvZiBnZW5lcyBpbiB0aGUgYmFja2dyb3VuZCBkaXN0cmlidXRpb24uClNvbWV0aW1lcyBmb2xrcyBjb25zaWRlciBnZW5lcyBmcm9tIHRoZSBlbnRpcmUgZ2Vub21lIHRvIGNvbXByaXNlIHRoZSBiYWNrZ3JvdW5kLCBidXQgaW4gdGhlIGV4YW1wbGUgYm9ycm93ZWQgZnJvbSB0aGUgYGNsdXN0ZXJQcm9maWxlcmAgYXV0aG9ycywgdGhleSBzdGF0ZToKCj4gMTcsOTgwIGdlbmVzIGRldGVjdGVkIGluIGEgTWljcm9hcnJheSBzdHVkeQoKV2hlcmUgdGhlIGtleSBwaHJhc2UgaXMgKipnZW5lcyBkZXRlY3RlZCoqLgoKSWYgd2Ugd2VyZSB1bmFibGUgdG8gaW5jbHVkZSBhIGdlbmUgaW4gb3VyIG1hcmtlciBnZW5lIGNvbXBhcmlzb25zIGJlY2F1c2UsIGZvciBleGFtcGxlLCB3ZSBjb3VsZCBub3QgcmVsaWFibHkgbWVhc3VyZSBpdCwgd2Ugc2hvdWxkbid0IGluY2x1ZGUgaW4gb3VyIGJhY2tncm91bmQgc2V0IQoKT3VyIG1hcmtlcnMgZGF0YSBmcmFtZSBoYXMgYWxsIG9mIHRoZSBnZW5lcyB0aGF0IHdlIGRpZCBub3QgZmlsdGVyIG91dCBhbmQgaGF2ZSBnZW5lIHN5bWJvbHMgY29ycmVzcG9uZGluZyB0byB0aGUgRW5zZW1ibCBnZW5lIElEcywgc28gd2UgY2FuIHVzZSB0aGUgYGdlbmVfc3ltYm9sYCBjb2x1bW4gYXMgb3VyIGJhY2tncm91bmQgc2V0LgoKYGBge3IgZ2V0X2JhY2tncm91bmRfc2V0LCBsaXZlID0gVFJVRX0KYmFja2dyb3VuZF9zZXQgPC0gbWFya2Vyc19kZiRnZW5lX3N5bWJvbApgYGAKCiMjIyBHZW5lIE9udG9sb2d5IE9SQQoKIyMjIyBBIG5vdGUgb24gR08KClRoZSBHZW5lIE9udG9sb2d5IChHTykgKFtBc2hidXJuZXIgKmV0IGFsLiogMjAwMF0oaHR0cHM6Ly9keC5kb2kub3JnLzEwLjEwMzgvNzU1NTYpOyBbVGhlIEdlbmUgT250b2xvZ3kgQ29uc29ydGl1bS4gMjAyMV0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwOTMvbmFyL2drYWExMTEzKSlpcyBbYW4gb250b2xvZ3ldKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL09udG9sb2d5XyhpbmZvcm1hdGlvbl9zY2llbmNlKSkgdGhhdCBkb2N1bWVudHMgb3IgZGVzY3JpYmVzIGhvdyBnZW5lcyBmdW5jdGlvbiBpbiB0aGUgYmlvbG9naWNhbCBkb21haW4gYW5kIGlzIGNvbXByaXNlZCBvZiAzIHBhcnRzIChbR08gb250b2xvZ3kgZG9jdW1lbnRhdGlvbl0oaHR0cDovL2dlbmVvbnRvbG9neS5vcmcvZG9jcy9vbnRvbG9neS1kb2N1bWVudGF0aW9uLykpOgoKKiAqKk1vbGVjdWxhciBmdW5jdGlvbiAoTUYpOioqIGFjdGl2aXRpZXMgdGhhdCBvY2N1ciBvbiB0aGUgbW9sZWN1bGFyIGxldmVsLCBvZnRlbiBjYW4gYmUgcGVyZm9ybWVkIGJ5IGluZGl2aWR1YWwgZ2VuZXMKKiAqKkJpb2xvZ2ljYWwgcHJvY2VzcyAoQlApOioqIHByb2dyYW1zIGNhcnJpZWQgb3V0IGJ5IG11bHRpcGxlIG1vbGVjdWxhciBlbnRpdGllczsgc2ltaWxhciB0byB3aGF0IGlzIHNvbWV0aW1lcyB0aG91Z2h0IG9mIGFzIGEgcGF0aHdheSwgYnV0IHdpdGggbm8gbm90aW9uIG9mIF9keW5hbWljc18KKiAqKkNlbGx1bGFyIGNvbXBvbmVudCAoQ0MpOioqIGNlbGx1bGFyIHN0cnVjdHVyZXMgYW5kIGNvbXBhcnRtZW50cwoKR08gaXMgImxvb3NlbHkgaGllcmFyY2hpY2FsIiAoW0dPIG9udG9sb2d5IGRvY3VtZW50YXRpb25dKGh0dHA6Ly9nZW5lb250b2xvZ3kub3JnL2RvY3Mvb250b2xvZ3ktZG9jdW1lbnRhdGlvbi8pKS4KVGhlcmUgYXJlIHBhcmVudCBhbmQgY2hpbGQgdGVybXMsIHdoZXJlIHRoZSAiaGlnaGVyIHVwIiBhIHRlcm0gaXMsIHRoZSBtb3JlIGdlbmVyYWwgaXQgaXMuClRoaXMgaXMgc29tZXdoYXQgYWJzdHJhY3QsIHNvIGxldCdzIHRha2UgYSBsb29rIGF0IGEgR08gdGVybSB0aGF0J3MgbGlrZWx5IHRvIGJlIHJlbGV2YW50IHRvIG91ciBIb2Rna2luJ3MgbHltcGhvbWEgZGF0YSB0byBnZXQgYW4gaWRlYSBvZiB3aGF0IHRoaXMgaXMgbGlrZSBpbiBwcmFjdGljZTogW2BsZXVrb2N5dGUgYWN0aXZhdGlvbmBdKGh0dHA6Ly93d3cuaW5mb3JtYXRpY3MuamF4Lm9yZy92b2NhYi9nZW5lX29udG9sb2d5L0dPOjAwNDUzMjEpLgoKIyMjIyBSdW4gYGVucmljaEdPKClgCgpOb3cgdGhhdCB3ZSBoYXZlIG91ciBiYWNrZ3JvdW5kIHNldCBhbmQgb3VyIGdlbmVzIG9mIGludGVyZXN0LCB3ZSdyZSByZWFkeSB0byBydW4gT1JBIHVzaW5nIHRoZSBgZW5yaWNoR08oKWAgZnVuY3Rpb24uCgpgYGB7ciBnb19vcmF9CmdvX29yYV9yZXN1bHRzIDwtIGVucmljaEdPKGdlbmUgPSBjbHVzdGVyMDFfbWFya2VycywgICMgVG9wIDEwMCBnZW5lcwogICAgICAgICAgICAgICAgICAgICAgICAgICB1bml2ZXJzZSA9IGJhY2tncm91bmRfc2V0LCAgIyBCYWNrZ3JvdW5kIGdlbmVzCiAgICAgICAgICAgICAgICAgICAgICAgICAgIGtleVR5cGUgPSAiU1lNQk9MIiwgICMgT3VyIGdlbmVzIGFyZSBnZW5lIHN5bWJvbHMKICAgICAgICAgICAgICAgICAgICAgICAgICAgT3JnRGIgPSBvcmcuSHMuZWcuZGIsICAjIFN1cHBvcnRzIGNvbnZlcnNpb24sIGlmIG5lZWRlZAogICAgICAgICAgICAgICAgICAgICAgICAgICBvbnQgPSAiQlAiLCAgIyBCaW9sb2dpY2FsIHByb2Nlc3MKICAgICAgICAgICAgICAgICAgICAgICAgICAgcEFkanVzdE1ldGhvZCA9ICJCSCIsICAjIEZEUgogICAgICAgICAgICAgICAgICAgICAgICAgICBwdmFsdWVDdXRvZmYgPSAwLjAwMDAxKSAjIFZlcnkgc3RyaW5nZW50IGZvciB2aXoKYGBgCgpXaGF0IGlzIHJldHVybmVkIGJ5IGBlbnJpY2hHTygpYD8KCmBgYHtyIHZpZXdfZ29fb3JhLCBldmFsID0gRkFMU0V9ClZpZXcoZ29fb3JhX3Jlc3VsdHMpCmBgYAoKVGhlIGluZm9ybWF0aW9uIHdlJ3JlIG1vc3QgbGlrZWx5IGludGVyZXN0ZWQgaW4gaXMgaW4gdGhlIGByZXN1bHRgIHNsb3QuCkxldCdzIGNvbnZlcnQgdGhpcyBpbnRvIGEgZGF0YSBmcmFtZSB0aGF0IHdlIGNhbiB3cml0ZSB0byBmaWxlLgoKYGBge3IgZ29fZGZ9CmdvX3Jlc3VsdF9kZiA8LSBkYXRhLmZyYW1lKGdvX29yYV9yZXN1bHRzQHJlc3VsdCkKYGBgCgpMZXQncyB0YWtlIGEgbG9vayBhdCB0aGUgR08gc2V0cyB3aXRoIGFuIGFkanVzdGVkIHAtdmFsdWUgbGVzcyB0aGFuIDAuMDAwMDEuCgpgYGB7ciBmaWx0ZXJfcGFkaiwgbGl2ZSA9IFRSVUV9CmdvX3Jlc3VsdF9kZiB8PgogIGRwbHlyOjpmaWx0ZXIocC5hZGp1c3QgPCAwLjAwMDAxKQpgYGAKClNvbWUgb2YgdGhlc2UgY29sdW1ucyBoYXZlIG5hbWVzIHRoYXQgYXJlIHNvbWV3aGF0IG9wYXF1ZSBhbmQgYXJlIHVzZWQgZm9yIHZpc3VhbGl6YXRpb24sIHNvIGxldCdzIGJyaWVmbHkgdGFsayBhYm91dCB0aGVtIGhlcmUhCgp8IENvbHVtbiBuYW1lIHwgTnVtZXJhdG9yIHwgRGVub21pbmF0b3IgfAp8LS0tLS0tLS0tLS0tLXwtLS0tLS0tLS0tLXwtLS0tLS0tLS0tLS0tfAp8IGBHZW5lUmF0aW9gIHwgTnVtYmVyIG9mIGdlbmVzIGluIHRoZSB0b3AgMTAwIG1hcmtlciBnZW5lcyBhbmQgdGhlIHNwZWNpZmljIGdlbmUgc2V0IChhbHNvIGluIGBDb3VudGAgY29sdW1uKSB8IE51bWJlciBvZiBnZW5lcyBpbiB0aGUgdG9wIDEwMCBtYXJrZXIgZ2VuZXMgYW5kIF9hbnlfIGdlbmUgc2V0IHwKfCBgQmdSYXRpb2AgICB8IE51bWJlciBvZiBnZW5lcyBpbiB0aGUgYmFja2dyb3VuZCBzZXQgYW5kIHRoZSBzcGVjaWZpYyBnZW5lIHNldCB8IE51bWJlciBvZiBnZW5lcyBpbiB0aGUgYmFja2dyb3VuZCBzZXQgYW5kIF9hbnlfIGdlbmUgc2V0IHwKCiMjIyMgVmlzdWFsaXppbmcgcmVzdWx0cwoKV2UgY2FuIHVzZSBhIGRvdCBwbG90IHRvIHZpc3VhbGl6ZSBvdXIgc2lnbmlmaWNhbnQgZW5yaWNobWVudCByZXN1bHRzLgoKYGBge3IgZG90cGxvdCwgbGl2ZSA9IFRSVUV9CmVucmljaHBsb3Q6OmRvdHBsb3QoZ29fb3JhX3Jlc3VsdHMpCmBgYAoKKipXaGF0IGNlbGwgdHlwZSBkbyB5b3UgdGhpbmsgdGhlIGNlbGxzIGZyb20gY2x1c3RlciAxIGFyZT8qKgoKV2UgY2FuIHVzZSBhbiBbVXBTZXQgcGxvdF0oaHR0cHM6Ly93d3cubmNiaS5ubG0ubmloLmdvdi9wbWMvYXJ0aWNsZXMvUE1DNDcyMDk5My8pIHRvIHZpc3VhbGl6ZSB0aGUgKipvdmVybGFwKiogYmV0d2VlbiB0aGUgZ2VuZSBzZXRzIHRoYXQgd2VyZSByZXR1cm5lZCBhcyBzaWduaWZpY2FudC4KCmBgYHtyIHVwc2V0cGxvdCwgbGl2ZSA9IFRSVUV9CmVucmljaHBsb3Q6OnVwc2V0cGxvdChnb19vcmFfcmVzdWx0cykKYGBgCgpJdCdzIGltcG9ydGFudCB0byBrZWVwIGluIG1pbmQgdGhhdCBnZW5lIHNldHMgb3IgcGF0aHdheXMgYXJlbid0IGluZGVwZW5kZW50LCBlaXRoZXIhClRoaXMgaXMgcGFydGljdWxhcmx5IHRydWUgb2YgR08gdGVybXMsIHdoaWNoIGFyZSBoaWVyYXJjaGljYWwgaW4gbmF0dXJlLgoKIyMjIENlbGxNYXJrZXIgT1JBCgpOZXh0LCB3ZSdsbCB1c2UgYW5vdGhlciByZXNvdXJjZSwgQ2VsbE1hcmtlciwgYXMgYWRkaXRpb25hbCBzdXBwb3J0IGZvciBvdXIgY29uY2x1c2lvbnMgYmFzZWQgb24gR08uCgojIyMjIEFib3V0IENlbGxNYXJrZXIKCltDZWxsTWFya2VyXShodHRwOi8vYmlvLWJpZ2RhdGEuaHJibXUuZWR1LmNuL0NlbGxNYXJrZXIvKSBpcyBhIG1hbnVhbGx5IGN1cmF0ZWQgZGF0YWJhc2Ugb2YgY2VsbCBtYXJrZXIgZGF0YSBmb3IgaHVtYW4gYW5kIG1vdXNlIChbWmhhbmcgKmV0IGFsLiogMjAxOF0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwOTMvbmFyL2dreTkwMCkpLgpTdHVkaWVzL2V2aWRlbmNlIG9mIHRoZSBmb2xsb3dpbmcgbmF0dXJlIHdlcmUgaW5jbHVkZWQgaW4gdGhlIG1hbnVhbCBjdXJhdGlvbiBwcm9jZXNzIChbWmhhbmcgKmV0IGFsLiogMjAxOF0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwOTMvbmFyL2dreTkwMCkpOgoKKiBzY1JOQS1zZXEKKiBGbG93IGN5dG9tZXRyeQoqIEltbXVub3N0YWluaW5nCgpBbmQgdGhlcmUgYXJlIGJvdGggZ2VuZSBhbmQgcHJvdGVpbiBtYXJrZXJzLgoKTGV0J3MgcmVhZCBpbiB0aGUgQ2VsbE1hcmtlciBkYXRhIHdlJ3ZlIGFscmVhZHkgY2xlYW5lZDoKCmBgYHtyIHJlYWRfaW5fY20sIGxpdmUgPSBUUlVFfQpjZWxsbWFya2VyX2RmIDwtIHJlYWRyOjpyZWFkX3RzdihjZWxsbWFya2VyX2ZpbGUpCmBgYAojIyMjIFJ1biBgZW5yaWNoZXIoKWAKCmBjbHVzdGVyUHJvZmlsZXI6OmVucmljaGVyKClgIGlzIGEgbW9yZSBnZW5lcmFsIHdheSB0byBwZXJmb3JtIE9SQSAtLSB3ZSBzcGVjaWZ5IHRoZSBwYXRod2F5cyBvciBnZW5lIHNldHMgb3Vyc2VsdmVzLgpXZSBjYW4gdXNlIHRoZSBpdCB0byBydW4gT1JBIHdpdGggdGhlIENlbGxNYXJrZXIgZGF0YSwgd2hpY2ggY29udGFpbnMgZ2VuZSBzeW1ib2xzLgpUaGF0J3Mgd2h5IHdlIG5lZWRlZCB0byBwZXJmb3JtIGdlbmUgaWRlbnRpZmllciBjb252ZXJzaW9uIGVhcmxpZXIhCgpgYGB7ciBjbV9vcmEsIGxpdmUgPSBUUlVFfQpjZWxsbWFya2VyX29yYV9yZXN1bHRzIDwtIGVucmljaGVyKAogIGdlbmUgPSBjbHVzdGVyMDFfbWFya2VycywgICMgR2VuZXMgb2YgaW50ZXJlc3QKICBwdmFsdWVDdXRvZmYgPSAwLjA1LCAgIyBNb3JlIHBlcm1pc3NpdmUgY3V0IG9mZgogIHBBZGp1c3RNZXRob2QgPSAiQkgiLCAgIyBGRFIKICB1bml2ZXJzZSA9IGJhY2tncm91bmRfc2V0LCAgIyBCYWNrZ3JvdW5kIHNldAogICMgVGhlIHBhdGh3YXkgaW5mb3JtYXRpb24gc2hvdWxkIGJlIGEgZGF0YSBmcmFtZSB3aXRoIGEgdGVybSBuYW1lIG9yCiAgIyBpZGVudGlmaWVyIGFuZCB0aGUgZ2VuZSBpZGVudGlmaWVycwogIFRFUk0yR0VORSA9IGNlbGxtYXJrZXJfZGYKKQpgYGAKCiMjIyMgVmlzdWFsaXppbmcgcmVzdWx0cwoKTGV0J3MgbG9vayBhdCBhIGJhciBwbG90IG9mIHRoZSBzaWduaWZpY2FudCByZXN1bHRzLgoKYGBge3IgY21fYmFycGxvdH0KYmFycGxvdChjZWxsbWFya2VyX29yYV9yZXN1bHRzKSArCiAgIyBUaGlzIGlzIGEgZ2dwbG90LCBzbyB3ZSBjYW4gdXNlIHRoZSBmb2xsb3dpbmcgdG8gbGFiZWwgdGhlIHgtYXhpcyEKICBnZ3Bsb3QyOjpsYWJzKHggPSAiY291bnQiKQpgYGAKCk9uZSB0aGluZyB0aGF0IHRoZSBbd2ViIHZlcnNpb24gb2YgQ2VsbE1hcmtlcl0oaHR0cDovL2Jpby1iaWdkYXRhLmhyYm11LmVkdS5jbi9DZWxsTWFya2VyLykgYWxsb3dzIHVzIHRvIGRvIGlzIHRvIGdldCBhIGJldHRlciBpZGVhIG9mIF9tYXJrZXIgcHJldmFsZW5jZV8uClNvbWUgb2YgdGhlIGdlbmVzIHRoYXQgc2hvdyB1cCBpbiBvdXIgdG9wIDEwMCBtYXkgbm90IGJlIHZlcnkgc3BlY2lmaWMuCkl0J3MgYWxzbyBsaWtlbHkgdGhhdCBraWRuZXkgaXMgbm90IHZlcnkgcmVsZXZhbnQgdG8gb3VyIGFuYWx5c2lzIG9mIEhvZGdraW4ncyBseW1waG9tYSwgYnV0IGx5bXBoIG5vZGUgaXMgaGlnaGx5IHJlbGV2YW50LgpIb3dldmVyLCBibG9vZCBjZWxscyBpbiBraWRuZXkgYXJlIG5vdCBfdW5leHBlY3RlZF8gYW5kLCBpZiB3ZSB2aWV3IHRoZSBbQ2VsbE1hcmtlciB0aXNzdWUgdHlwZSBicm93c2VyXShodHRwOi8vYmlvLWJpZ2RhdGEuaHJibXUuZWR1LmNuL0NlbGxNYXJrZXIvYnJvd3NlLmpzcCksIHdlIGNhbiBzZWUgdGhhdCB0aGVyZSBhcmUgX2EgbG90XyBvZiBnZW5lcyBhbm5vdGF0ZWQgdG8ga2lkbmV5LgpFdmVyeSByZXNvdXJjZSB3aWxsIGhhdmUgc29tZSBsaW1pdGF0aW9ucyAiYmFrZWQgaW4uIgoKV2UgY291bGQgZG8gYWRkaXRpb25hbCBmaWx0ZXJpbmcgb2YgdGhlIGdlbmUgc2V0cyB0byByZW1vdmUgaXJyZWxldmFudCB0aXNzdWVzIGFuZCB0byByZW1vdmUgbWFya2VyIGdlbmVzIHRoYXQgYXJlIG5vdCBwYXJ0aWN1bGFybHkgc3BlY2lmaWMuCihGZXdlciBnZW5lIHNldHMgd291bGQgbWVhbiBhIGxvd2VyIG11bHRpcGxlIGh5cG90aGVzaXMgdGVzdGluZyBidXJkZW4uKQpJZiBzb21lIG9mIHRoZSBDZWxsTWFya2VyIG1hcmtlciBnZW5lcyBjb21lIGZyb20gaW1tdW5vc3RhaW5pbmcgb2YgdGVybWluYWxseSBkaWZmZXJlbnRpYXRlZCBjZWxscywgd2UgbWlnaHQgbm90IGV4cGVjdCB0aGVpciB0cmFuc2NyaXB0cyB0byBzaG93IHVwIGluIHNjUk5BLXNlcSBkYXRhLCBmb3IgZXhhbXBsZS4KQWxsIHRoaW5ncyB0byBrZWVwIGluIG1pbmQgaWYgeW91IHVzZSB0aGlzIHJlc291cmNlIGZvciB5b3VyIG93biB3b3JrIQoKIyMjIFdyaXRlIHJlc3VsdHMgdG8gZmlsZQoKV3JpdGUgdGhlIHJlc3VsdHMgdG8gZmlsZSBmb3IgYm90aCBHTyBhbmQgQ2VsbE1hcmtlciBnZW5lIHNldHMuCgpgYGB7ciB3cml0ZV9yZXN1bHRzLCBsaXZlID0gVFJVRX0KcmVhZHI6OndyaXRlX3Rzdihnb19yZXN1bHRfZGYsIGZpbGUgPSBnb19yZXN1bHRzX2ZpbGUpCmRhdGEuZnJhbWUoY2VsbG1hcmtlcl9vcmFfcmVzdWx0c0ByZXN1bHQpIHw+CiAgcmVhZHI6OndyaXRlX3RzdihmaWxlID0gY21fcmVzdWx0c19maWxlKQpgYGAKCiMjIFBhcnRpbmcgdGhvdWdodHMKClRoZSBnb2FsIG9mIGFuYWx5emluZyBtYXJrZXIgZ2VuZXMgd2l0aCBPUkEsIGxpa2Ugd2UgZGlkIGluIHRoaXMgbm90ZWJvb2ssIGlzIHRvIGhlbHAgdXMgaW50ZXJwcmV0IG91ciBjbHVzdGVyaW5nIHJlc3VsdHMgb3IgYW5ub3RhdGUgY2x1c3RlcnMuCldlIHNob3VsZG4ndCB1c2UgbWFya2VyIGdlbmVzIG9yIGFueSBkb3duc3RyZWFtIGFuYWx5c2lzIHdlIHBlcmZvcm0gd2l0aCB0aGVtIHRvIF9qdXN0aWZ5XyBjbHVzdGVyIGFzc2lnbm1lbnRzOyB3ZSBnZW5lcmFsbHkgd2FudCBvdXIgY2x1c3RlciBhc3NpZ25tZW50cyB0byBiZSBkYXRhLWRyaXZlbi4KCkp1c3QgbGlrZSBpdCdzIGltcG9ydGFudCB0byBiZSBza2VwdGljYWwgYWJvdXQgcC12YWx1ZXMgcmV0dXJuZWQgZnJvbSB0aGUgbWFya2VyIGdlbmUgYW5hbHlzaXMgaXRzZWxmLCB3ZSBzaG91bGQgbm90IGdldCB0b28gYXR0YWNoZWQgdG8gdGhlIGlkZWEgb2Ygc3RhdGlzdGljYWwgc2lnbmlmaWNhbmNlIGhlcmUuCklmIHdlIHdlcmUgdG8gdHdlYWsgdGhlIGN1dG9mZnMgKGUuZy4sIHRha2UgdGhlIHRvcCAyNTAwIGdlbmVzKSwgd2UgbWlnaHQgZXhwZWN0IHRvIGdldCBjb21wbGV0ZWx5IGRpZmZlcmVudCBzaWduaWZpY2FudCByZXN1bHRzLgpUaGlzIGlzIGEgbGltaXRhdGlvbiBvZiBPUkE6IHdlIGhhdmUgdG8gbWFrZSBzb21lIGNob2ljZXMgdGhhdCBhcmUgc29tZXdoYXQgYXJiaXRyYXJ5LgpJdCdzIGEgZ29vZCBpZGVhIHRvIHRoaW5rIGFib3V0IHRoZSBnb2FsIG9mIHlvdXIgYW5hbHlzaXMgX3VwZnJvbnRfIGFuZCBwaWNrIGN1dG9mZnMgYmFzZWQgb24geW91ciBnb2Fscy4KVGhhdCBjYW4gaGVscCB5b3UgYXZvaWQgdGhlIHRlbXB0YXRpb24gb2YgdHJ5aW5nIGEgYnVuY2ggb2YgY3V0b2ZmcyB1bnRpbCB5b3UgZ2V0IHJlc3VsdHMgdGhhdCAibG9vayBnb29kLiIKCiMjIFNlc3Npb24gSW5mbwoKYGBge3Igc2Vzc2lvbl9pbmZvfQpzZXNzaW9uSW5mbygpCmBgYAo=
- - -
-
- -
- - - - - - - - - - - - - - - - - From 086d1aefdcc6787b81fbd9fcd1f996775682a99f Mon Sep 17 00:00:00 2001 From: Jaclyn Taroni <19534205+jaclyn-taroni@users.noreply.github.com> Date: Mon, 25 Nov 2024 15:17:17 +0000 Subject: [PATCH 4/7] geneset -> gene sets and rerun --- .../04-gene_set_enrichment_analysis.Rmd | 2 +- .../04-gene_set_enrichment_analysis.nb.html | 100 +++++++++++------- 2 files changed, 64 insertions(+), 38 deletions(-) diff --git a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd index 0f300979..0ae64adb 100644 --- a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd +++ b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd @@ -149,7 +149,7 @@ head(deseq_df) This data frame uses Ensembl gene identifiers. We'll need to make sure our gene sets use the same identifiers. -Let's take a look at the first few rows of the data frame that contains the hallmark genesets. +Let's take a look at the first few rows of the data frame that contains the hallmark gene sets. ```{r hallmark_head, live = TRUE} head(hs_hallmarks_df) diff --git a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html index 618e4c60..a825edd6 100644 --- a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html +++ b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html @@ -2949,7 +2949,7 @@

Libraries

library(clusterProfiler)
- +
clusterProfiler v4.12.0  For help: https://yulab-smu.top/biomedical-knowledge-mining-book/
 
 If you use clusterProfiler in published research, please cite:
@@ -2957,6 +2957,18 @@ 

Libraries

Attaching package: ‘clusterProfiler’ +The following object is masked from ‘package:AnnotationDbi’: + + select + +The following object is masked from ‘package:IRanges’: + + slice + +The following object is masked from ‘package:S4Vectors’: + + rename + The following object is masked from ‘package:stats’: filter
@@ -3102,12 +3114,12 @@

DESeq2 results

deseq_df <- readr::read_tsv(input_file)
- +
Rows: 60319 Columns: 27
-── Column specification ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
+── Column specification ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Delimiter: "\t"
 chr  (2): ensembl_id, gene_symbol
-dbl (25): baseMean, log2FoldChange, lfcSE, pvalue, padj, SCPCL000478.mean, SCPCL000478.detected, SCPCL000479.mean, SCPCL000479.detected, SCPCL000480.mean, SCPCL000480.detected, SCPCL000481.m...
+dbl (25): baseMean, log2FoldChange, lfcSE, pvalue, padj, SCPCL000478.mean, SCPCL000478.detected, SCPCL000479.mean, SCPCL000479.det...
 
 ℹ Use `spec()` to retrieve the full column specification for this data.
 ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
@@ -3130,8 +3142,8 @@

DESeq2 results

This data frame uses Ensembl gene identifiers. We’ll need to make sure our gene sets use the same identifiers. Let’s take a look at the -first few rows of the data frame that contains the hallmark -genesets.

+first few rows of the data frame that contains the hallmark gene +sets.

@@ -3351,7 +3363,7 @@

Session Info

sessionInfo()
- +
R version 4.4.0 (2024-04-24)
 Platform: x86_64-pc-linux-gnu
 Running under: Ubuntu 22.04.4 LTS
@@ -3361,49 +3373,63 @@ 

Session Info

LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0 locale: - [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8 LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8 LC_PAPER=C.UTF-8 LC_NAME=C - [9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C + [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8 LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 + [6] LC_MESSAGES=C.UTF-8 LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C +[11] LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C time zone: Etc/UTC tzcode source: system (glibc) attached base packages: -[1] stats graphics grDevices datasets utils methods base +[1] stats4 stats graphics grDevices utils datasets methods base other attached packages: -[1] msigdbr_7.5.1 clusterProfiler_4.12.0 + [1] clusterProfiler_4.12.0 msigdbr_7.5.1 GSEABase_1.66.0 graph_1.82.0 + [5] annotate_1.82.0 XML_3.99-0.16.1 AnnotationDbi_1.66.0 AUCell_1.26.0 + [9] SingleCellExperiment_1.26.0 SummarizedExperiment_1.34.0 Biobase_2.64.0 GenomicRanges_1.56.0 +[13] GenomeInfoDb_1.40.0 IRanges_2.38.0 S4Vectors_0.42.0 BiocGenerics_0.50.0 +[17] MatrixGenerics_1.16.0 matrixStats_1.3.0 loaded via a namespace (and not attached): - [1] RColorBrewer_1.1-3 rstudioapi_0.16.0 jsonlite_1.8.8 magrittr_2.0.3 estimability_1.5 ggbeeswarm_0.7.2 - [7] farver_2.1.1 rmarkdown_2.26 fs_1.6.4 zlibbioc_1.50.0 vctrs_0.6.5 memoise_2.0.1 - [13] DelayedMatrixStats_1.26.0 ggtree_3.12.0 htmltools_0.5.8.1 S4Arrays_1.4.0 BiocNeighbors_1.22.0 SparseArray_1.4.0 - [19] gridGraphics_0.5-1 plyr_1.8.9 emmeans_1.10.1 cachem_1.0.8 igraph_2.0.3 lifecycle_1.0.4 - [25] pkgconfig_2.0.3 gson_0.1.0 rsvd_1.0.5 Matrix_1.7-0 R6_2.5.1 fastmap_1.1.1 - [31] GenomeInfoDbData_1.2.12 MatrixGenerics_1.16.0 digest_0.6.35 aplot_0.2.2 enrichplot_1.24.0 colorspace_2.1-0 - [37] patchwork_1.2.0 AnnotationDbi_1.66.0 S4Vectors_0.42.0 scater_1.32.0 qusage_2.38.0 irlba_2.3.5.1 - [43] GenomicRanges_1.56.0 RSQLite_2.3.6 beachmat_2.20.0 labeling_0.4.3 fansi_1.0.6 httr_1.4.7 - [49] polyclip_1.10-6 abind_1.4-5 compiler_4.4.0 bit64_4.0.5 withr_3.0.0 BiocParallel_1.38.0 - [55] viridis_0.6.5 DBI_1.2.2 ggforce_0.4.2 MASS_7.3-60.2 DelayedArray_0.30.0 HDO.db_0.99.1 - [61] tools_4.4.0 vipor_0.4.7 beeswarm_0.4.0 ape_5.8 scatterpie_0.2.2 fftw_1.0-8 - [67] glue_1.7.0 nlme_3.1-164 GOSemSim_2.30.0 grid_4.4.0 shadowtext_0.1.3 reshape2_1.4.4 - [73] fgsea_1.30.0 generics_0.1.3 gtable_0.3.5 tzdb_0.4.0 tidyr_1.3.1 data.table_1.15.4 - [79] hms_1.1.3 ScaledMatrix_1.12.0 BiocSingular_1.20.0 tidygraph_1.3.1 utf8_1.2.4 XVector_0.44.0 - [85] BiocGenerics_0.50.0 ggrepel_0.9.5 pillar_1.9.0 stringr_1.5.1 vroom_1.6.5 babelgene_22.9 - [91] limma_3.60.0 yulab.utils_0.1.4 splines_4.4.0 dplyr_1.1.4 tweenr_2.0.3 treeio_1.28.0 - [97] lattice_0.22-6 renv_1.0.7 bit_4.0.5 tidyselect_1.2.1 GO.db_3.19.1 SingleCellExperiment_1.26.0 -[103] Biostrings_2.72.0 scuttle_1.14.0 knitr_1.46 gridExtra_2.3 IRanges_2.38.0 SummarizedExperiment_1.34.0 -[109] stats4_4.4.0 xfun_0.43 graphlayouts_1.1.1 Biobase_2.64.0 statmod_1.5.0 matrixStats_1.3.0 -[115] stringi_1.8.3 UCSC.utils_1.0.0 lazyeval_0.2.2 ggfun_0.1.4 yaml_2.3.8 evaluate_0.23 -[121] codetools_0.2-20 ggraph_2.2.1 tibble_3.2.1 qvalue_2.36.0 BiocManager_1.30.22 ggplotify_0.1.2 -[127] cli_3.6.2 xtable_1.8-4 munsell_0.5.1 Rcpp_1.0.12 GenomeInfoDb_1.40.0 coda_0.19-4.1 -[133] png_0.1-8 parallel_4.4.0 ggplot2_3.5.1 readr_2.1.5 blob_1.2.4 DOSE_3.30.0 -[139] sparseMatrixStats_1.16.0 mvtnorm_1.2-4 viridisLite_0.4.2 tidytree_0.4.6 scales_1.3.0 purrr_1.0.2 -[145] crayon_1.5.2 rlang_1.1.3 cowplot_1.1.3 fastmatch_1.1-4 KEGGREST_1.44.0
+ [1] RColorBrewer_1.1-3 rstudioapi_0.16.0 jsonlite_1.8.8 magrittr_2.0.3 + [5] ggbeeswarm_0.7.2 farver_2.1.1 rmarkdown_2.26 fs_1.6.4 + [9] zlibbioc_1.50.0 vctrs_0.6.5 memoise_2.0.1 DelayedMatrixStats_1.26.0 + [13] ggtree_3.12.0 htmltools_0.5.8.1 S4Arrays_1.4.0 BiocNeighbors_1.22.0 + [17] gridGraphics_0.5-1 SparseArray_1.4.0 sass_0.4.9 bslib_0.7.0 + [21] htmlwidgets_1.6.4 plyr_1.8.9 cachem_1.0.8 igraph_2.0.3 + [25] lifecycle_1.0.4 pkgconfig_2.0.3 gson_0.1.0 rsvd_1.0.5 + [29] Matrix_1.7-0 R6_2.5.1 fastmap_1.1.1 GenomeInfoDbData_1.2.12 + [33] aplot_0.2.2 digest_0.6.35 enrichplot_1.24.0 colorspace_2.1-0 + [37] patchwork_1.2.0 scater_1.32.0 irlba_2.3.5.1 RSQLite_2.3.6 + [41] beachmat_2.20.0 labeling_0.4.3 fansi_1.0.6 httr_1.4.7 + [45] polyclip_1.10-6 abind_1.4-5 compiler_4.4.0 bit64_4.0.5 + [49] withr_3.0.0 BiocParallel_1.38.0 viridis_0.6.5 DBI_1.2.2 + [53] ggforce_0.4.2 R.utils_2.12.3 MASS_7.3-60.2 DelayedArray_0.30.0 + [57] HDO.db_0.99.1 tools_4.4.0 vipor_0.4.7 scatterpie_0.2.2 + [61] ape_5.8 beeswarm_0.4.0 R.oo_1.26.0 glue_1.7.0 + [65] nlme_3.1-164 GOSemSim_2.30.0 shadowtext_0.1.3 grid_4.4.0 + [69] reshape2_1.4.4 fgsea_1.30.0 generics_0.1.3 gtable_0.3.5 + [73] tzdb_0.4.0 R.methodsS3_1.8.2 tidyr_1.3.1 data.table_1.15.4 + [77] hms_1.1.3 BiocSingular_1.20.0 tidygraph_1.3.1 ScaledMatrix_1.12.0 + [81] utf8_1.2.4 XVector_0.44.0 stringr_1.5.1 ggrepel_0.9.5 + [85] pillar_1.9.0 vroom_1.6.5 yulab.utils_0.1.4 babelgene_22.9 + [89] splines_4.4.0 dplyr_1.1.4 tweenr_2.0.3 treeio_1.28.0 + [93] lattice_0.22-6 renv_1.0.7 bit_4.0.5 tidyselect_1.2.1 + [97] GO.db_3.19.1 Biostrings_2.72.0 scuttle_1.14.0 knitr_1.46 +[101] gridExtra_2.3 xfun_0.43 graphlayouts_1.1.1 stringi_1.8.3 +[105] UCSC.utils_1.0.0 lazyeval_0.2.2 ggfun_0.1.4 yaml_2.3.8 +[109] evaluate_0.23 codetools_0.2-20 ggraph_2.2.1 qvalue_2.36.0 +[113] tibble_3.2.1 ggplotify_0.1.2 cli_3.6.2 xtable_1.8-4 +[117] jquerylib_0.1.4 munsell_0.5.1 Rcpp_1.0.12 png_0.1-8 +[121] parallel_4.4.0 ggplot2_3.5.1 readr_2.1.5 blob_1.2.4 +[125] DOSE_3.30.0 sparseMatrixStats_1.16.0 tidytree_0.4.6 viridisLite_0.4.2 +[129] scales_1.3.0 purrr_1.0.2 crayon_1.5.2 rlang_1.1.3 +[133] fastmatch_1.1-4 cowplot_1.1.3 KEGGREST_1.44.0 -
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcyAoR1NFQSkiCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjQKLS0tCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOgoKLSBQcmVwYXJlIHRhYnVsYXIgZGF0YSBvZiBnZW5lLWxldmVsIHN0YXRpc3RpY3MgZm9yIHVzZSB3aXRoIEdlbmUgU2V0IEVucmljaG1lbnQgQW5hbHlzaXMgKEdTRUEpCi0gQWNjZXNzIFtNb2xlY3VsYXIgU2lnbmF0dXJlcyBEYXRhYmFzZSBnZW5lIHNldCBjb2xsZWN0aW9uc10oaHR0cHM6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2NvbGxlY3Rpb25zLmpzcCkgdmlhIHRoZSBgbXNpZ2RicmAgcGFja2FnZQotIFBlcmZvcm0gR1NFQSB3aXRoIHRoZSBgY2x1c3RlclByb2ZpbGVyYCBwYWNrYWdlCi0gVmlzdWFsaXplIEdTRUEgcmVzdWx0cyB3aXRoIHRoZSBgZW5yaWNocGxvdGAgcGFja2FnZQoKLS0tCgpJbiB0aGlzIG5vdGVib29rLCB3ZSdsbCBhbmFseXplIHRoZSBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiByZXN1bHRzIGZyb20gdGhlIGxhc3Qgbm90ZWJvb2suCgpHU0VBIGlzIGEgZnVuY3Rpb25hbCBjbGFzcyBzY29yaW5nIChGQ1MpIGFwcHJvYWNoIHRvIHBhdGh3YXkgYW5hbHlzaXMgdGhhdCB3YXMgZmlyc3QgaW50cm9kdWNlZCBpbiBbU3VicmFtYW5pYW4gX2V0IGFsLl8gKDIwMDUpXShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpLgpUaGUgcmF0aW9uYWxlIGJlaGluZCBGQ1MgYXBwcm9hY2hlcyBpcyB0aGF0IHNtYWxsIGNoYW5nZXMgaW4gaW5kaXZpZHVhbCBnZW5lcyB0aGF0IHBhcnRpY2lwYXRlIGluIHRoZSBzYW1lIGJpb2xvZ2ljYWwgcHJvY2VzcyBvciBwYXRod2F5IGNhbiBiZSBzaWduaWZpY2FudCBhbmQgb2YgYmlvbG9naWNhbCBpbnRlcmVzdC4KClRoZXJlIGFyZSAzIGdlbmVyYWwgc3RlcHMgaW4gRkNTIG1ldGhvZHMgKFtLaGF0cmkgX2V0IGFsLl8gMjAxMl0oIGh0dHBzOi8vZG9pLm9yZy8xMC4xMzcxL2pvdXJuYWwucGNiaS4xMDAyMzc1KSk6CgoxLiBDYWxjdWxhdGUgYSBnZW5lLWxldmVsIHN0YXRpc3RpYyAoaGVyZSwgd2UnbGwgdXNlIHRoZSBzdW1tYXJ5IGxvZyBmb2xkIGNoYW5nZXMgaW4gb3VyIERFU2VxMiByZXN1bHRzKQoyLiBHZW5lLWxldmVsIHN0YXRpc3RpY3MgYXJlIGFnZ3JlZ2F0ZWQgaW50byBhIHBhdGh3YXktbGV2ZWwgc3RhdGlzdGljCjMuIEFzc2VzcyB0aGUgc3RhdGlzdGljYWwgc2lnbmlmaWNhbmNlIG9mIHRoZSBwYXRod2F5LWxldmVsIHN0YXRpc3RpYwoKIyMjIyBPdGhlciByZXNvdXJjZXMKCiogRm9yIGFub3RoZXIgZXhhbXBsZSB1c2luZyBgY2x1c3RlclByb2ZpbGVyYCBmb3IgR1NFQSwgc2VlIFtfSW50cm8gdG8gREdFOiBGdW5jdGlvbmFsIEFuYWx5c2lzLl8gZnJvbSBIYXJ2YXJkIENoYW4gQmlvaW5mb3JtYXRpY3MgQ29yZSBUcmFpbmluZy5dKGh0dHBzOi8vaGJjdHJhaW5pbmcuZ2l0aHViLmlvL0RHRV93b3Jrc2hvcC9sZXNzb25zLzA5X2Z1bmN0aW9uYWxfYW5hbHlzaXMuaHRtbCkKKiBUaGUgd2F5IHdlJ2xsIHVzZSBgY2x1c3RlclByb2ZpbGVyYCBoZXJlIHVzZXMgYGZnc2VhYCAoRmFzdCBHZW5lIFNldCBFbnJpY2htZW50IEFuYWx5c2lzKSB1bmRlciB0aGUgaG9vZC4KWW91IGNhbiByZWFkIG1vcmUgYWJvdXQgZmdzZWEgaW4gW0tvcm90a2V2aWNoIF9ldCBhbC5fICgyMDIxKV0oaHR0cHM6Ly9kb2kub3JnLzEwLjExMDEvMDYwMDEyKS4KKiBbcmVmaW5lLmJpbyBleGFtcGxlcyBHZW5lIHNldCBlbnJpY2htZW50IGFuYWx5c2lzIC0gUk5BLXNlcV0oaHR0cHM6Ly9hbGV4c2xlbW9uYWRlLmdpdGh1Yi5pby9yZWZpbmViaW8tZXhhbXBsZXMvMDMtcm5hc2VxL3BhdGh3YXktYW5hbHlzaXNfcm5hc2VxXzAyX2dzZWEuaHRtbCkgZnJvbSB3aGljaCB0aGlzIG1hdGVyaWFsIGhhcyBiZWVuIGFkYXB0ZWQuCgojIyBTZXQgdXAKCiMjIyBMaWJyYXJpZXMKCmBgYHtyIGxpYnJhcmllc30KIyBQYWNrYWdlIHRvIHJ1biBHU0VBCmxpYnJhcnkoY2x1c3RlclByb2ZpbGVyKQojIFBhY2thZ2UgdGhhdCBjb250YWlucyB0aGUgTVNpZ0RCIGdlbmUgc2V0cyBpbiB0aWR5IGZvcm1hdApsaWJyYXJ5KG1zaWdkYnIpCmBgYAoKIyMjIERpcmVjdG9yaWVzIGFuZCBGaWxlcwoKIyMjIyBEaXJlY3RvcmllcwoKYGBge3IgY3JlYXRlX2RpciwgbGl2ZSA9IFRSVUV9CiMgV2UnbGwgdXNlIHRoZSBtYXJrZXIgZ2VuZXMgYXMgR1NFQSBpbnB1dApybXNfYW5hbHlzaXNfZGlyIDwtIGZpbGUucGF0aCgiYW5hbHlzaXMiLCAicm1zIikKCiMgV2UnbGwgY3JlYXRlIGEgZGlyZWN0b3J5IHRvIHNwZWNpZmljYWxseSBob2xkIHRoZSBwYXRod2F5IHJlc3VsdHMgaWYgaXQgZG9lc24ndAojIGV4aXN0IHlldApyZXN1bHRzX2RpciA8LSBmaWxlLnBhdGgocm1zX2FuYWx5c2lzX2RpciwgInBhdGh3YXktYW5hbHlzaXMiKQpmczo6ZGlyX2NyZWF0ZShyZXN1bHRzX2RpcikKYGBgCgojIyMjIElucHV0IGZpbGVzCgoKYGBge3IgaW5wdXRfZmlsZXN9CmlucHV0X2ZpbGUgPC0gZmlsZS5wYXRoKHJtc19hbmFseXNpc19kaXIsCiAgICAgICAgICAgICAgICAgICAgICAgICJkZXNlcSIsCiAgICAgICAgICAgICAgICAgICAgICAgICJybXNfbXlvYmxhc3RfZGVzZXFfcmVzdWx0cy50c3YiKQpgYGAKCiMjIyMgT3V0cHV0IGZpbGVzCgpXZSdsbCBzYXZlIG91ciB0YWJsZSBvZiBHU0VBIHJlc3VsdHMgYXMgYSBUU1YuCgpgYGB7ciBvdXRwdXRfZmlsZXN9Cm91dHB1dF9maWxlIDwtIGZpbGUucGF0aChyZXN1bHRzX2RpciwKICAgICAgICAgICAgICAgICAgICAgICAgICJybXNfbXlvYmxhc3RfZ3NlYV9yZXN1bHRzLnRzdiIpCmBgYAoKIyMgR2VuZSBzZXRzCgpXZSB3aWxsIHVzZSBnZW5lIHNldHMgZnJvbSB0aGUgW01vbGVjdWxhciBTaWduYXR1cmVzIERhdGFiYXNlIChNU2lnREIpXShodHRwczovL3d3dy5nc2VhLW1zaWdkYi5vcmcvZ3NlYS9tc2lnZGIvaW5kZXguanNwKSBmcm9tIHRoZSBCcm9hZCBJbnN0aXR1dGUgKFtTdWJyYW1hbmlhbiwgVGFtYXlvICpldCBhbC4qIDIwMDVdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMDczL3BuYXMuMDUwNjU4MDEwMikpLgpUaGUgW2Btc2lnZGJyYF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL21zaWdkYnIvaW5kZXguaHRtbCkgcGFja2FnZSBjb250YWlucyBNU2lnREIgZGF0YXNldHMgYWxyZWFkeSBpbiB0aGUgdGlkeSBmb3JtYXQgcmVxdWlyZWQgYnkgYGNsdXN0ZXJQcm9maWxlcmAgYW5kIHN1cHBvcnRzIG11bHRpcGxlIG9yZ2FuaXNtcy4KCkxldCdzIHRha2UgYSBsb29rIGF0IHdoYXQgb3JnYW5pc21zIHRoZSBwYWNrYWdlIHN1cHBvcnRzLgoKYGBge3Igc2hvd19zcGVjaWVzfQptc2lnZGJyX3NwZWNpZXMoKQpgYGAKCk1TaWdEQiBjb250YWlucyA4IGRpZmZlcmVudCBnZW5lIHNldCBjb2xsZWN0aW9ucy4KCiAgICBIOiBoYWxsbWFyayBnZW5lIHNldHMKICAgIEMxOiBwb3NpdGlvbmFsIGdlbmUgc2V0cwogICAgQzI6IGN1cmF0ZWQgZ2VuZSBzZXRzCiAgICBDMzogbW90aWYgZ2VuZSBzZXRzCiAgICBDNDogY29tcHV0YXRpb25hbCBnZW5lIHNldHMKICAgIEM1OiBHTyBnZW5lIHNldHMKICAgIEM2OiBvbmNvZ2VuaWMgc2lnbmF0dXJlcwogICAgQzc6IGltbXVub2xvZ2ljIHNpZ25hdHVyZXMKCldlJ2xsIHVzZSB0aGUgSGFsbG1hcmsgY29sbGVjdGlvbiBmb3IgR1NFQS4KSGVyZSdzIGFuIGV4Y2VycHQgb2YgdGhlIFtjb2xsZWN0aW9uIGRlc2NyaXB0aW9uXShodHRwczovL3d3dy5nc2VhLW1zaWdkYi5vcmcvZ3NlYS9tc2lnZGIvY29sbGVjdGlvbl9kZXRhaWxzLmpzcCNIKToKCj4gSGFsbG1hcmsgZ2VuZSBzZXRzIHN1bW1hcml6ZSBhbmQgcmVwcmVzZW50IHNwZWNpZmljIHdlbGwtZGVmaW5lZCBiaW9sb2dpY2FsIHN0YXRlcyBvciBwcm9jZXNzZXMgYW5kIGRpc3BsYXkgY29oZXJlbnQgZXhwcmVzc2lvbi4gVGhlc2UgZ2VuZSBzZXRzIHdlcmUgZ2VuZXJhdGVkIGJ5IGEgY29tcHV0YXRpb25hbCBtZXRob2RvbG9neSBiYXNlZCBvbiBpZGVudGlmeWluZyBnZW5lIHNldCBvdmVybGFwcyBhbmQgcmV0YWluaW5nIGdlbmVzIHRoYXQgZGlzcGxheSBjb29yZGluYXRlIGV4cHJlc3Npb24uIFRoZSBoYWxsbWFya3MgcmVkdWNlIG5vaXNlIGFuZCByZWR1bmRhbmN5IGFuZCBwcm92aWRlIGEgYmV0dGVyIGRlbGluZWF0ZWQgYmlvbG9naWNhbCBzcGFjZSBmb3IgR1NFQS4KCk5vdGFibHksIHRoZXJlIGFyZSBvbmx5IDUwIGdlbmUgc2V0cyBpbmNsdWRlZCBpbiB0aGlzIGNvbGxlY3Rpb24uClRoZSBmZXdlciBnZW5lIHNldHMgd2UgdGVzdCwgdGhlIGxvd2VyIG91ciBtdWx0aXBsZSBoeXBvdGhlc2lzIHRlc3RpbmcgYnVyZGVuLgoKV2UgY2FuIHJldHJpZXZlIG9ubHkgdGhlIEhhbGxtYXJrIGdlbmUgc2V0cyBieSBzcGVjaWZ5aW5nIGBjYXRlZ29yeSA9ICJIImAgdG8gdGhlIGBtc2lnZGJyKClgIGZ1bmN0aW9uLgoKYGBge3IgaW1tdW5vbG9naWNfc2V0cywgbGl2ZSA9IFRSVUV9CmhzX2hhbGxtYXJrc19kZiA8LSBtc2lnZGJyKHNwZWNpZXMgPSAiSG9tbyBzYXBpZW5zIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgY2F0ZWdvcnkgPSAiSCIpCmBgYAoKIyMgR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcwoKX0FkYXB0ZWQgZnJvbSBbcmVmaW5lLmJpbyBleGFtcGxlc10oaHR0cHM6Ly9naXRodWIuY29tL0FsZXhzTGVtb25hZGUvcmVmaW5lYmlvLWV4YW1wbGVzL2Jsb2IvMzNjZGVmZjY2ZDU3ZjlmZThlZTRmY2I1MTU2YWVhNGFjMmRjZTA3Zi8wMy1ybmFzZXEvcGF0aHdheS1hbmFseXNpc19ybmFzZXFfMDJfZ3NlYS5SbWQpXwoKIVtdKGRpYWdyYW1zL3N1YnJhbWFuaWFuX2ZpZzEuanBnKQoKKipGaWd1cmUgMS4gW1N1YnJhbWFuaWFuIF9ldCBhbC5fICgyMDA1KV0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwNzMvcG5hcy4wNTA2NTgwMTAyKS4qKgoKR1NFQSBjYWxjdWxhdGVzIGEgcGF0aHdheS1sZXZlbCBtZXRyaWMsIGNhbGxlZCBhbiBlbnJpY2htZW50IHNjb3JlIChzb21ldGltZXMgYWJicmV2aWF0ZWQgYXMgRVMpLCBieSByYW5raW5nIGdlbmVzIGJ5IGEgZ2VuZS1sZXZlbCBzdGF0aXN0aWMuClRoaXMgc2NvcmUgcmVmbGVjdHMgd2hldGhlciBvciBub3QgYSBnZW5lIHNldCBvciBwYXRod2F5IGlzIG92ZXItcmVwcmVzZW50ZWQgYXQgdGhlIHRvcCBvciBib3R0b20gb2YgdGhlIGdlbmUgcmFua2luZ3MgKFtTdWJyYW1hbmlhbiBfZXQgYWwuXyAyMDA1XShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpOyBbWXVdKGh0dHA6Ly95dWxhYi1zbXUudG9wL2NsdXN0ZXJQcm9maWxlci1ib29rL2NoYXB0ZXIyLmh0bWwjZ2VuZS1zZXQtZW5yaWNobWVudC1hbmFseXNpcykpCgpTcGVjaWZpY2FsbHksIGFsbCBnZW5lcyBhcmUgcmFua2VkIGZyb20gbW9zdCBwb3NpdGl2ZSB0byBtb3N0IG5lZ2F0aXZlIGJhc2VkIG9uIHRoZWlyIHN0YXRpc3RpYyBhbmQgYSBydW5uaW5nIHN1bSBpcyBjYWxjdWxhdGVkOgpTdGFydGluZyB3aXRoIHRoZSBtb3N0IGhpZ2hseSByYW5rZWQgZ2VuZXMsIHRoZSBydW5uaW5nIHN1bSBpbmNyZWFzZXMgZm9yIGVhY2ggZ2VuZSBpbiB0aGUgcGF0aHdheSBhbmQgZGVjcmVhc2VzIGZvciBlYWNoIGdlbmUgbm90IGluIHRoZSBwYXRod2F5LgpUaGUgZW5yaWNobWVudCBzY29yZSBmb3IgYSBwYXRod2F5IGlzIHRoZSBydW5uaW5nIHN1bSdzIG1heGltdW0gZGV2aWF0aW9uIGZyb20gemVyby4KR1NFQSBhbHNvIGFzc2Vzc2VzIHN0YXRpc3RpY2FsIHNpZ25pZmljYW5jZSBvZiB0aGUgc2NvcmVzIGZvciBlYWNoIHBhdGh3YXkgdGhyb3VnaCBwZXJtdXRhdGlvbiB0ZXN0aW5nLgpBcyBhIHJlc3VsdCwgZWFjaCBpbnB1dCBwYXRod2F5IHdpbGwgaGF2ZSBhIHAtdmFsdWUgYXNzb2NpYXRlZCB3aXRoIGl0IHRoYXQgaXMgdGhlbiBjb3JyZWN0ZWQgZm9yIG11bHRpcGxlIGh5cG90aGVzaXMgdGVzdGluZyAoW1N1YnJhbWFuaWFuIF9ldCBhbC5fIDIwMDVdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMDczL3BuYXMuMDUwNjU4MDEwMik7IFtZdV0oaHR0cDovL3l1bGFiLXNtdS50b3AvY2x1c3RlclByb2ZpbGVyLWJvb2svY2hhcHRlcjIuaHRtbCNnZW5lLXNldC1lbnJpY2htZW50LWFuYWx5c2lzKSkuCgpUaGUgaW1wbGVtZW50YXRpb24gb2YgR1NFQSB3ZSB1c2UgaW4gaGVyZSBleGFtcGxlcyByZXF1aXJlcyBhIGdlbmUgbGlzdCBvcmRlcmVkIGJ5IHNvbWUgc3RhdGlzdGljIGFuZCBpbnB1dCBnZW5lIHNldHMuCldoZW4geW91IHVzZSBwcmV2aW91c2x5IGNvbXB1dGVkIGdlbmUtbGV2ZWwgc3RhdGlzdGljcyB3aXRoIEdTRUEsIGl0IGlzIGNhbGxlZCBHU0VBIHByZS1yYW5rZWQuCgojIyBERVNlcTIgcmVzdWx0cwoKYGBge3IgcmVhZF9pbl9tYXJrZXJzLCBsaXZlID0gVFJVRX0KZGVzZXFfZGYgPC0gcmVhZHI6OnJlYWRfdHN2KGlucHV0X2ZpbGUpCmBgYAoKYGBge3IgZGVzZXFfaGVhZH0KaGVhZChkZXNlcV9kZikKYGBgCgpUaGlzIGRhdGEgZnJhbWUgdXNlcyBFbnNlbWJsIGdlbmUgaWRlbnRpZmllcnMuCldlJ2xsIG5lZWQgdG8gbWFrZSBzdXJlIG91ciBnZW5lIHNldHMgdXNlIHRoZSBzYW1lIGlkZW50aWZpZXJzLgpMZXQncyB0YWtlIGEgbG9vayBhdCB0aGUgZmlyc3QgZmV3IHJvd3Mgb2YgdGhlIGRhdGEgZnJhbWUgdGhhdCBjb250YWlucyB0aGUgaGFsbG1hcmsgZ2VuZXNldHMuCgpgYGB7ciBoYWxsbWFya19oZWFkLCBsaXZlID0gVFJVRX0KaGVhZChoc19oYWxsbWFya3NfZGYpCmBgYAoKV2UgY2FuIHNlZSB0aGF0IHRoZSBnZW5lIHNldHMgZnJvbSBgbXNpZ2RicmAgaGF2ZSBFbnNlbWJsIGdlbmUgaWRlbnRpZmllcnMgYXNzb2NpYXRlZCB3aXRoIHRoZW0sIHNvIHdlIGRvbid0IG5lZWQgdG8gZG8gYW55IGNvbnZlcnNpb24uCkhvd2V2ZXIsIHdlJ2xsIG5lZWQgdG8gcGFzcyB0aGUgY29ycmVjdCBjb2x1bW4gdG8gdGhlIGZ1bmN0aW9uIHRoYXQgcnVucyBHU0VBLgoKT25lIGdvb2QgdGhpbmcgYWJvdXQgRW5zZW1ibCBnZW5lIGlkZW50aWZpZXJzIGlzIHRoYXQgdGhleSBhcmUgbGVzcyBsaWtlbHkgdG8gYmUgZHVwbGljYXRlZCB0aGFuLCBmb3IgZXhhbXBsZSwgZ2VuZSBzeW1ib2xzLgooTXVsdGlwbGUgRW5zZW1ibCBnZW5lIGlkZW50aWZpZXJzIGNhbiBtYXAgdG8gdGhlIHNhbWUgc3ltYm9sLikKClRoZSBHU0VBIGFwcHJvYWNoIHJlcXVpcmVzIG9uIGRpc2NyaW1pbmF0aW5nIGJldHdlZW4gZ2VuZXMgdGhhdCBhcmUgaW4gYSBnZW5lIHNldCBhbmQgdGhvc2UgdGhhdCBhcmUgbm90LgpQcmFjdGljYWxseSBzcGVha2luZywgZ2VuZSBzZXRzIGFyZSBqdXN0IGNvbGxlY3Rpb25zIG9mIGdlbmUgaWRlbnRpZmllcnMhCldoZW4gdGhlIGZ1bmN0aW9uIHdlIHVzZSBmb3IgR1NFQSBwcmUtcmFua2VkIGdldHMgYSBsaXN0IHdpdGggZHVwbGljYXRlZCBnZW5lIGlkZW50aWZpZXJzLCBpdCBjYW4gcHJvZHVjZSB1bmV4cGVjdGVkIHJlc3VsdHMuClNvLCBsZXQncyBjaGVjayBmb3IgZHVwbGljYXRlcyBpbiB0aGUgZGF0YSBmcmFtZSBvZiBERVNlcTIgcmVzdWx0cy4KCmBgYHtyIGNoZWNrX2R1cGxpY2F0ZXMsIGxpdmUgPSBUUlVFfQphbnkoZHVwbGljYXRlZChkZXNlcV9kZiRlbnNlbWJsX2lkKSkKYGBgCgpUaGVyZSBhcmUgbm8gZHVwbGljYXRlcyBmb3IgdXMgdG8gd29ycnkgYWJvdXQhCgojIyMgUHJlLXJhbmtlZCBsaXN0CgpUaGUgYEdTRUEoKWAgZnVuY3Rpb24gdGFrZXMgYSBwcmUtcmFua2VkIChzb3J0ZWQpIG5hbWVkIHZlY3RvciBvZiBzdGF0aXN0aWNzLCB3aGVyZSB0aGUgbmFtZXMgaW4gdGhlIHZlY3RvciBhcmUgZ2VuZSBpZGVudGlmaWVycy4KVGhpcyBpcyBzdGVwIDEgLS0gZ2VuZS1sZXZlbCBzdGF0aXN0aWNzLgoKYGBge3IgbGZjX3ZlY3Rvcn0KbGZjX3ZlY3RvciA8LSBkZXNlcV9kZiB8PgogICMgRXh0cmFjdCBhIHZlY3RvciBvZiBgbG9nMkZvbGRDaGFuZ2VgIG5hbWVkIGJ5IGBlbnNlbWJsX2lkYAogIGRwbHlyOjpwdWxsKGxvZzJGb2xkQ2hhbmdlLCBuYW1lID0gZW5zZW1ibF9pZCkKbGZjX3ZlY3RvciA8LSBzb3J0KGxmY192ZWN0b3IsIGRlY3JlYXNpbmcgPSBUUlVFKQpgYGAKCkxldCdzIGxvb2sgYXQgdGhlIHRvcCByYW5rZWQgdmFsdWVzLgoKYGBge3IgaGVhZF9sZmMsIGxpdmUgPSBUUlVFfQojIExvb2sgYXQgZmlyc3QgZW50cmllcyBvZiB0aGUgbG9nIGZvbGQgY2hhbmdlIHZlY3RvcgpoZWFkKGxmY192ZWN0b3IpCmBgYAoKQW5kIHRoZSBib3R0b20gb2YgdGhlIGxpc3QuCgpgYGB7ciB0YWlsX2xmYywgbGl2ZSA9IFRSVUV9CiMgTG9vayBhdCB0aGUgbGFzdCBlbnRyaWVzIG9mIHRoZSBsb2cgZm9sZCBjaGFuZ2UgdmVjdG9yCnRhaWwobGZjX3ZlY3RvcikKYGBgCgojIyBSdW4gR1NFQQoKTm93IGZvciB0aGUgYW5hbHlzaXMhCgpXZSBjYW4gdXNlIHRoZSBgR1NFQSgpYCBmdW5jdGlvbiB0byBwZXJmb3JtIEdTRUEgd2l0aCBhbnkgZ2VuZXJpYyBzZXQgb2YgZ2VuZSBzZXRzLCBidXQgdGhlcmUgYXJlIHNldmVyYWwgZnVuY3Rpb25zIGZvciB1c2luZyBzcGVjaWZpYywgY29tbW9ubHkgdXNlZCBnZW5lIHNldHMgKGUuZy4sIGBnc2VLRUdHKClgKS4KCmBgYHtyIHJ1bl9nc2VhfQpnc2VhX3Jlc3VsdHMgPC0gR1NFQShnZW5lTGlzdCA9IGxmY192ZWN0b3IsICAjIG9yZGVyZWQgcmFua2VkIGdlbmUgbGlzdAogICAgICAgICAgICAgICAgICAgICBtaW5HU1NpemUgPSAyNSwgICMgbWluaW11bSBnZW5lIHNldCBzaXplCiAgICAgICAgICAgICAgICAgICAgIG1heEdTU2l6ZSA9IDUwMCwgICMgbWF4aW11bSBnZW5lIHNldCBzZXQKICAgICAgICAgICAgICAgICAgICAgcHZhbHVlQ3V0b2ZmID0gMC4wNSwKICAgICAgICAgICAgICAgICAgICAgcEFkanVzdE1ldGhvZCA9ICJCSCIsICAjIGNvcnJlY3Rpb24gZm9yIG11bHRpcGxlIGh5cG90aGVzaXMgdGVzdGluZwogICAgICAgICAgICAgICAgICAgICBURVJNMkdFTkUgPSBkcGx5cjo6c2VsZWN0KGhzX2hhbGxtYXJrc19kZiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBnc19uYW1lLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGVuc2VtYmxfZ2VuZSkpICAjIHBhc3MgdGhlIGNvcnJlY3QgaWRlbnRpZmllciBjb2x1bW4KYGBgCkxldCdzIHRha2UgYSBsb29rIGF0IHRoZSBHU0VBIHJlc3VsdHMuCgpgYGB7ciB2aWV3X2dzZWEsIGxpdmUgPSBUUlVFLCBldmFsID0gRkFMU0V9ClZpZXcoZ3NlYV9yZXN1bHRzQHJlc3VsdCB8PgogICAgICAgZHBseXI6OmFycmFuZ2UoZHBseXI6OmRlc2MoTkVTKSkKKQpgYGAKCk5vcm1hbGl6ZWQgZW5yaWNobWVudCBzY29yZXMgKE5FUykgYXJlIGVucmljaG1lbnQgc2NvcmVzIHRoYXQgYXJlIHNjYWxlZCB0byBtYWtlIGdlbmUgc2V0cyB0aGF0IGNvbnRhaW4gZGlmZmVyZW50IG51bWJlciBvZiBnZW5lcyBjb21wYXJhYmxlLgoKUGF0aHdheXMgd2l0aCBzaWduaWZpY2FudCwgaGlnaGx5IHBvc2l0aXZlIE5FUyBhcmUgZW5yaWNoZWQgaW4gRVJNUyBteW9ibGFzdHMsIHdoZXJlYXMgcGF0aHdheXMgd2l0aCBzaWduaWZpY2FudCwgaGlnaGx5IG5lZ2F0aXZlIE5FUyBhcmUgZW5yaWNoZWQgaW4gQVJNUyBteW9ibGFzdHMuCgpMZXQncyB3cml0ZSB0aGVzZSByZXN1bHRzIHRvIGZpbGUuCgpgYGB7ciB3cml0ZV9nc2VhfQpnc2VhX3Jlc3VsdHNAcmVzdWx0IHw+IHJlYWRyOjp3cml0ZV90c3Yob3V0cHV0X2ZpbGUpCmBgYAoKIyMjIFZpc3VhbGl6aW5nIEdTRUEgcmVzdWx0cwoKV2UgY2FuIHZpc3VhbGl6ZSBHU0VBIHJlc3VsdHMgZm9yIGluZGl2aWR1YWwgcGF0aHdheXMgb3IgZ2VuZSBzZXRzIHVzaW5nIGBlbnJpY2hwbG90Ojpnc2VhcGxvdCgpYC4KTGV0J3MgdGFrZSBhIGxvb2sgYXQgMyBkaWZmZXJlbnQgcGF0aHdheXMgLS0gb25lIHdpdGggYSBoaWdobHkgcG9zaXRpdmUgTkVTLCBvbmUgd2l0aCBhIGhpZ2hseSBuZWdhdGl2ZSBORVMsIGFuZCBvbmUgdGhhdCB3YXMgbm90IGEgc2lnbmlmaWNhbnQgcmVzdWx0IC0tIHRvIGdldCBtb3JlIGluc2lnaHQgaW50byBob3cgRVMgYXJlIGNhbGN1bGF0ZWQuCgojIyMjIEhpZ2hseSBQb3NpdGl2ZSBORVMKCkxldCdzIHRha2UgbG9vayBhdCBhIHBhdGh3YXkgd2l0aCBhIGhpZ2hseSBwb3NpdGl2ZSBORVMgKGBIQUxMTUFSS19NWUNfVEFSR0VUU19WMmApIHVzaW5nIGEgR1NFQSBwbG90LgoKYGBge3IgaGlnaGx5X3Bvc30KZW5yaWNocGxvdDo6Z3NlYXBsb3QoZ3NlYV9yZXN1bHRzLAogICAgICAgICAgICAgICAgICAgICBnZW5lU2V0SUQgPSAiSEFMTE1BUktfTVlDX1RBUkdFVFNfVjIiLAogICAgICAgICAgICAgICAgICAgICB0aXRsZSA9ICJIQUxMTUFSS19NWUNfVEFSR0VUU19WMiIsCiAgICAgICAgICAgICAgICAgICAgIGNvbG9yLmxpbmUgPSAiIzAwNjZGRiIpCmBgYAoKTm90aWNlIGhvdyB0aGUgZ2VuZXMgdGhhdCBhcmUgaW4gdGhlIGdlbmUgc2V0LCBpbmRpY2F0ZWQgYnkgdGhlIGJsYWNrIGJhcnMsIHRlbmQgdG8gYmUgb24gdGhlIGxlZnQgc2lkZSBvZiB0aGUgZ3JhcGggaW5kaWNhdGluZyB0aGF0IHRoZXkgaGF2ZSBwb3NpdGl2ZSBnZW5lLWxldmVsIHNjb3Jlcy4KCiMjIyMgSGlnaGx5IE5lZ2F0aXZlIE5FUwoKVGhlIGdlbmUgc2V0IGBIQUxMTUFSS19NWU9HRU5FU0lTYCBoYWQgYSBoaWdobHkgbmVnYXRpdmUgTkVTLgoKYGBge3IgaGlnaGx5X25lZ30KZW5yaWNocGxvdDo6Z3NlYXBsb3QoZ3NlYV9yZXN1bHRzLAogICAgICAgICAgICAgICAgICAgICBnZW5lU2V0SUQgPSAiSEFMTE1BUktfTVlPR0VORVNJUyIsCiAgICAgICAgICAgICAgICAgICAgIHRpdGxlID0gIkhBTExNQVJLX01ZT0dFTkVTSVMiLAogICAgICAgICAgICAgICAgICAgICBjb2xvci5saW5lID0gIiMwMDY2RkYiKQpgYGAKClRoaXMgZ2VuZSBzZXQgc2hvd3MgdGhlIG9wcG9zaXRlIHBhdHRlcm4gLS0gZ2VuZXMgaW4gdGhlIHBhdGh3YXkgdGVuZCB0byBiZSBvbiB0aGUgcmlnaHQgc2lkZSBvZiB0aGUgZ3JhcGguCgojIyMjIEEgbm9uLXNpZ25pZmljYW50IHJlc3VsdAoKVGhlIGBAcmVzdWx0c2Agc2xvdCB3aWxsIG9ubHkgc2hvdyBnZW5lIHNldHMgdGhhdCBwYXNzIHRoZSBgcHZhbHVlQ3V0b2ZmYCB0aHJlc2hvbGQgd2Ugc3VwcGxpZWQgdG8gYEdTRUEoKWAsIGJ1dCB3ZSBjYW4gcGxvdCBhbnkgZ2VuZSBzZXQgc28gbG9uZyBhcyB3ZSBrbm93IGl0cyBuYW1lLgpMZXQncyBsb29rIGF0IGBIQUxMTUFSS19QNTNfUEFUSFdBWWAsIHdoaWNoIHdhcyBub3QgaW4gdGhlIHJlc3VsdHMgd2Ugdmlld2VkIGVhcmxpZXIuCgpgYGB7ciBwNTMsIGxpdmUgPSBUUlVFfQplbnJpY2hwbG90Ojpnc2VhcGxvdChnc2VhX3Jlc3VsdHMsCiAgICAgICAgICAgICAgICAgICAgIGdlbmVTZXRJRCA9ICJIQUxMTUFSS19QNTNfUEFUSFdBWSIsCiAgICAgICAgICAgICAgICAgICAgIHRpdGxlID0gIkhBTExNQVJLX1A1M19QQVRIV0FZIiwKICAgICAgICAgICAgICAgICAgICAgY29sb3IubGluZSA9ICIjMDA2NkZGIikKYGBgCgpHZW5lcyBpbiB0aGUgcGF0aHdheSBhcmUgZGlzdHJpYnV0ZWQgbW9yZSBldmVubHkgdGhyb3VnaG91dCB0aGUgcmFua2VkIGxpc3QsIHJlc3VsdGluZyBpbiBhIG1vcmUgIm1pZGRsaW5nIiBzY29yZS4KCipOb3RlOiBUaGUgcGxvdHMgcmV0dXJuZWQgYnkgYGVucmljaHBsb3Q6OmdzZWFwbG90YCBhcmUgZ2dwbG90cywgc28gd2UgY291bGQgdXNlIGBnZ3Bsb3QyOjpnZ3NhdmUoKWAgdG8gc2F2ZSB0aGVtIHRvIGZpbGUgaWYgd2Ugd2FudGVkIHRvLioKCiMjIFNlc3Npb24gSW5mbwoKYGBge3Igc2Vzc2lvbl9pbmZvfQpzZXNzaW9uSW5mbygpCmBgYAo=
+
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcyAoR1NFQSkiCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjQKLS0tCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOgoKLSBQcmVwYXJlIHRhYnVsYXIgZGF0YSBvZiBnZW5lLWxldmVsIHN0YXRpc3RpY3MgZm9yIHVzZSB3aXRoIEdlbmUgU2V0IEVucmljaG1lbnQgQW5hbHlzaXMgKEdTRUEpCi0gQWNjZXNzIFtNb2xlY3VsYXIgU2lnbmF0dXJlcyBEYXRhYmFzZSBnZW5lIHNldCBjb2xsZWN0aW9uc10oaHR0cHM6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2NvbGxlY3Rpb25zLmpzcCkgdmlhIHRoZSBgbXNpZ2RicmAgcGFja2FnZQotIFBlcmZvcm0gR1NFQSB3aXRoIHRoZSBgY2x1c3RlclByb2ZpbGVyYCBwYWNrYWdlCi0gVmlzdWFsaXplIEdTRUEgcmVzdWx0cyB3aXRoIHRoZSBgZW5yaWNocGxvdGAgcGFja2FnZQoKLS0tCgpJbiB0aGlzIG5vdGVib29rLCB3ZSdsbCBhbmFseXplIHRoZSBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiByZXN1bHRzIGZyb20gdGhlIGxhc3Qgbm90ZWJvb2suCgpHU0VBIGlzIGEgZnVuY3Rpb25hbCBjbGFzcyBzY29yaW5nIChGQ1MpIGFwcHJvYWNoIHRvIHBhdGh3YXkgYW5hbHlzaXMgdGhhdCB3YXMgZmlyc3QgaW50cm9kdWNlZCBpbiBbU3VicmFtYW5pYW4gX2V0IGFsLl8gKDIwMDUpXShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpLgpUaGUgcmF0aW9uYWxlIGJlaGluZCBGQ1MgYXBwcm9hY2hlcyBpcyB0aGF0IHNtYWxsIGNoYW5nZXMgaW4gaW5kaXZpZHVhbCBnZW5lcyB0aGF0IHBhcnRpY2lwYXRlIGluIHRoZSBzYW1lIGJpb2xvZ2ljYWwgcHJvY2VzcyBvciBwYXRod2F5IGNhbiBiZSBzaWduaWZpY2FudCBhbmQgb2YgYmlvbG9naWNhbCBpbnRlcmVzdC4KClRoZXJlIGFyZSAzIGdlbmVyYWwgc3RlcHMgaW4gRkNTIG1ldGhvZHMgKFtLaGF0cmkgX2V0IGFsLl8gMjAxMl0oIGh0dHBzOi8vZG9pLm9yZy8xMC4xMzcxL2pvdXJuYWwucGNiaS4xMDAyMzc1KSk6CgoxLiBDYWxjdWxhdGUgYSBnZW5lLWxldmVsIHN0YXRpc3RpYyAoaGVyZSwgd2UnbGwgdXNlIHRoZSBzdW1tYXJ5IGxvZyBmb2xkIGNoYW5nZXMgaW4gb3VyIERFU2VxMiByZXN1bHRzKQoyLiBHZW5lLWxldmVsIHN0YXRpc3RpY3MgYXJlIGFnZ3JlZ2F0ZWQgaW50byBhIHBhdGh3YXktbGV2ZWwgc3RhdGlzdGljCjMuIEFzc2VzcyB0aGUgc3RhdGlzdGljYWwgc2lnbmlmaWNhbmNlIG9mIHRoZSBwYXRod2F5LWxldmVsIHN0YXRpc3RpYwoKIyMjIyBPdGhlciByZXNvdXJjZXMKCiogRm9yIGFub3RoZXIgZXhhbXBsZSB1c2luZyBgY2x1c3RlclByb2ZpbGVyYCBmb3IgR1NFQSwgc2VlIFtfSW50cm8gdG8gREdFOiBGdW5jdGlvbmFsIEFuYWx5c2lzLl8gZnJvbSBIYXJ2YXJkIENoYW4gQmlvaW5mb3JtYXRpY3MgQ29yZSBUcmFpbmluZy5dKGh0dHBzOi8vaGJjdHJhaW5pbmcuZ2l0aHViLmlvL0RHRV93b3Jrc2hvcC9sZXNzb25zLzA5X2Z1bmN0aW9uYWxfYW5hbHlzaXMuaHRtbCkKKiBUaGUgd2F5IHdlJ2xsIHVzZSBgY2x1c3RlclByb2ZpbGVyYCBoZXJlIHVzZXMgYGZnc2VhYCAoRmFzdCBHZW5lIFNldCBFbnJpY2htZW50IEFuYWx5c2lzKSB1bmRlciB0aGUgaG9vZC4KWW91IGNhbiByZWFkIG1vcmUgYWJvdXQgZmdzZWEgaW4gW0tvcm90a2V2aWNoIF9ldCBhbC5fICgyMDIxKV0oaHR0cHM6Ly9kb2kub3JnLzEwLjExMDEvMDYwMDEyKS4KKiBbcmVmaW5lLmJpbyBleGFtcGxlcyBHZW5lIHNldCBlbnJpY2htZW50IGFuYWx5c2lzIC0gUk5BLXNlcV0oaHR0cHM6Ly9hbGV4c2xlbW9uYWRlLmdpdGh1Yi5pby9yZWZpbmViaW8tZXhhbXBsZXMvMDMtcm5hc2VxL3BhdGh3YXktYW5hbHlzaXNfcm5hc2VxXzAyX2dzZWEuaHRtbCkgZnJvbSB3aGljaCB0aGlzIG1hdGVyaWFsIGhhcyBiZWVuIGFkYXB0ZWQuCgojIyBTZXQgdXAKCiMjIyBMaWJyYXJpZXMKCmBgYHtyIGxpYnJhcmllc30KIyBQYWNrYWdlIHRvIHJ1biBHU0VBCmxpYnJhcnkoY2x1c3RlclByb2ZpbGVyKQojIFBhY2thZ2UgdGhhdCBjb250YWlucyB0aGUgTVNpZ0RCIGdlbmUgc2V0cyBpbiB0aWR5IGZvcm1hdApsaWJyYXJ5KG1zaWdkYnIpCmBgYAoKIyMjIERpcmVjdG9yaWVzIGFuZCBGaWxlcwoKIyMjIyBEaXJlY3RvcmllcwoKYGBge3IgY3JlYXRlX2RpciwgbGl2ZSA9IFRSVUV9CiMgV2UnbGwgdXNlIHRoZSBtYXJrZXIgZ2VuZXMgYXMgR1NFQSBpbnB1dApybXNfYW5hbHlzaXNfZGlyIDwtIGZpbGUucGF0aCgiYW5hbHlzaXMiLCAicm1zIikKCiMgV2UnbGwgY3JlYXRlIGEgZGlyZWN0b3J5IHRvIHNwZWNpZmljYWxseSBob2xkIHRoZSBwYXRod2F5IHJlc3VsdHMgaWYgaXQgZG9lc24ndAojIGV4aXN0IHlldApyZXN1bHRzX2RpciA8LSBmaWxlLnBhdGgocm1zX2FuYWx5c2lzX2RpciwgInBhdGh3YXktYW5hbHlzaXMiKQpmczo6ZGlyX2NyZWF0ZShyZXN1bHRzX2RpcikKYGBgCgojIyMjIElucHV0IGZpbGVzCgoKYGBge3IgaW5wdXRfZmlsZXN9CmlucHV0X2ZpbGUgPC0gZmlsZS5wYXRoKHJtc19hbmFseXNpc19kaXIsCiAgICAgICAgICAgICAgICAgICAgICAgICJkZXNlcSIsCiAgICAgICAgICAgICAgICAgICAgICAgICJybXNfbXlvYmxhc3RfZGVzZXFfcmVzdWx0cy50c3YiKQpgYGAKCiMjIyMgT3V0cHV0IGZpbGVzCgpXZSdsbCBzYXZlIG91ciB0YWJsZSBvZiBHU0VBIHJlc3VsdHMgYXMgYSBUU1YuCgpgYGB7ciBvdXRwdXRfZmlsZXN9Cm91dHB1dF9maWxlIDwtIGZpbGUucGF0aChyZXN1bHRzX2RpciwKICAgICAgICAgICAgICAgICAgICAgICAgICJybXNfbXlvYmxhc3RfZ3NlYV9yZXN1bHRzLnRzdiIpCmBgYAoKIyMgR2VuZSBzZXRzCgpXZSB3aWxsIHVzZSBnZW5lIHNldHMgZnJvbSB0aGUgW01vbGVjdWxhciBTaWduYXR1cmVzIERhdGFiYXNlIChNU2lnREIpXShodHRwczovL3d3dy5nc2VhLW1zaWdkYi5vcmcvZ3NlYS9tc2lnZGIvaW5kZXguanNwKSBmcm9tIHRoZSBCcm9hZCBJbnN0aXR1dGUgKFtTdWJyYW1hbmlhbiwgVGFtYXlvICpldCBhbC4qIDIwMDVdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMDczL3BuYXMuMDUwNjU4MDEwMikpLgpUaGUgW2Btc2lnZGJyYF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL21zaWdkYnIvaW5kZXguaHRtbCkgcGFja2FnZSBjb250YWlucyBNU2lnREIgZGF0YXNldHMgYWxyZWFkeSBpbiB0aGUgdGlkeSBmb3JtYXQgcmVxdWlyZWQgYnkgYGNsdXN0ZXJQcm9maWxlcmAgYW5kIHN1cHBvcnRzIG11bHRpcGxlIG9yZ2FuaXNtcy4KCkxldCdzIHRha2UgYSBsb29rIGF0IHdoYXQgb3JnYW5pc21zIHRoZSBwYWNrYWdlIHN1cHBvcnRzLgoKYGBge3Igc2hvd19zcGVjaWVzfQptc2lnZGJyX3NwZWNpZXMoKQpgYGAKCk1TaWdEQiBjb250YWlucyA4IGRpZmZlcmVudCBnZW5lIHNldCBjb2xsZWN0aW9ucy4KCiAgICBIOiBoYWxsbWFyayBnZW5lIHNldHMKICAgIEMxOiBwb3NpdGlvbmFsIGdlbmUgc2V0cwogICAgQzI6IGN1cmF0ZWQgZ2VuZSBzZXRzCiAgICBDMzogbW90aWYgZ2VuZSBzZXRzCiAgICBDNDogY29tcHV0YXRpb25hbCBnZW5lIHNldHMKICAgIEM1OiBHTyBnZW5lIHNldHMKICAgIEM2OiBvbmNvZ2VuaWMgc2lnbmF0dXJlcwogICAgQzc6IGltbXVub2xvZ2ljIHNpZ25hdHVyZXMKCldlJ2xsIHVzZSB0aGUgSGFsbG1hcmsgY29sbGVjdGlvbiBmb3IgR1NFQS4KSGVyZSdzIGFuIGV4Y2VycHQgb2YgdGhlIFtjb2xsZWN0aW9uIGRlc2NyaXB0aW9uXShodHRwczovL3d3dy5nc2VhLW1zaWdkYi5vcmcvZ3NlYS9tc2lnZGIvY29sbGVjdGlvbl9kZXRhaWxzLmpzcCNIKToKCj4gSGFsbG1hcmsgZ2VuZSBzZXRzIHN1bW1hcml6ZSBhbmQgcmVwcmVzZW50IHNwZWNpZmljIHdlbGwtZGVmaW5lZCBiaW9sb2dpY2FsIHN0YXRlcyBvciBwcm9jZXNzZXMgYW5kIGRpc3BsYXkgY29oZXJlbnQgZXhwcmVzc2lvbi4gVGhlc2UgZ2VuZSBzZXRzIHdlcmUgZ2VuZXJhdGVkIGJ5IGEgY29tcHV0YXRpb25hbCBtZXRob2RvbG9neSBiYXNlZCBvbiBpZGVudGlmeWluZyBnZW5lIHNldCBvdmVybGFwcyBhbmQgcmV0YWluaW5nIGdlbmVzIHRoYXQgZGlzcGxheSBjb29yZGluYXRlIGV4cHJlc3Npb24uIFRoZSBoYWxsbWFya3MgcmVkdWNlIG5vaXNlIGFuZCByZWR1bmRhbmN5IGFuZCBwcm92aWRlIGEgYmV0dGVyIGRlbGluZWF0ZWQgYmlvbG9naWNhbCBzcGFjZSBmb3IgR1NFQS4KCk5vdGFibHksIHRoZXJlIGFyZSBvbmx5IDUwIGdlbmUgc2V0cyBpbmNsdWRlZCBpbiB0aGlzIGNvbGxlY3Rpb24uClRoZSBmZXdlciBnZW5lIHNldHMgd2UgdGVzdCwgdGhlIGxvd2VyIG91ciBtdWx0aXBsZSBoeXBvdGhlc2lzIHRlc3RpbmcgYnVyZGVuLgoKV2UgY2FuIHJldHJpZXZlIG9ubHkgdGhlIEhhbGxtYXJrIGdlbmUgc2V0cyBieSBzcGVjaWZ5aW5nIGBjYXRlZ29yeSA9ICJIImAgdG8gdGhlIGBtc2lnZGJyKClgIGZ1bmN0aW9uLgoKYGBge3IgaW1tdW5vbG9naWNfc2V0cywgbGl2ZSA9IFRSVUV9CmhzX2hhbGxtYXJrc19kZiA8LSBtc2lnZGJyKHNwZWNpZXMgPSAiSG9tbyBzYXBpZW5zIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgY2F0ZWdvcnkgPSAiSCIpCmBgYAoKIyMgR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcwoKX0FkYXB0ZWQgZnJvbSBbcmVmaW5lLmJpbyBleGFtcGxlc10oaHR0cHM6Ly9naXRodWIuY29tL0FsZXhzTGVtb25hZGUvcmVmaW5lYmlvLWV4YW1wbGVzL2Jsb2IvMzNjZGVmZjY2ZDU3ZjlmZThlZTRmY2I1MTU2YWVhNGFjMmRjZTA3Zi8wMy1ybmFzZXEvcGF0aHdheS1hbmFseXNpc19ybmFzZXFfMDJfZ3NlYS5SbWQpXwoKIVtdKGRpYWdyYW1zL3N1YnJhbWFuaWFuX2ZpZzEuanBnKQoKKipGaWd1cmUgMS4gW1N1YnJhbWFuaWFuIF9ldCBhbC5fICgyMDA1KV0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwNzMvcG5hcy4wNTA2NTgwMTAyKS4qKgoKR1NFQSBjYWxjdWxhdGVzIGEgcGF0aHdheS1sZXZlbCBtZXRyaWMsIGNhbGxlZCBhbiBlbnJpY2htZW50IHNjb3JlIChzb21ldGltZXMgYWJicmV2aWF0ZWQgYXMgRVMpLCBieSByYW5raW5nIGdlbmVzIGJ5IGEgZ2VuZS1sZXZlbCBzdGF0aXN0aWMuClRoaXMgc2NvcmUgcmVmbGVjdHMgd2hldGhlciBvciBub3QgYSBnZW5lIHNldCBvciBwYXRod2F5IGlzIG92ZXItcmVwcmVzZW50ZWQgYXQgdGhlIHRvcCBvciBib3R0b20gb2YgdGhlIGdlbmUgcmFua2luZ3MgKFtTdWJyYW1hbmlhbiBfZXQgYWwuXyAyMDA1XShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpOyBbWXVdKGh0dHA6Ly95dWxhYi1zbXUudG9wL2NsdXN0ZXJQcm9maWxlci1ib29rL2NoYXB0ZXIyLmh0bWwjZ2VuZS1zZXQtZW5yaWNobWVudC1hbmFseXNpcykpCgpTcGVjaWZpY2FsbHksIGFsbCBnZW5lcyBhcmUgcmFua2VkIGZyb20gbW9zdCBwb3NpdGl2ZSB0byBtb3N0IG5lZ2F0aXZlIGJhc2VkIG9uIHRoZWlyIHN0YXRpc3RpYyBhbmQgYSBydW5uaW5nIHN1bSBpcyBjYWxjdWxhdGVkOgpTdGFydGluZyB3aXRoIHRoZSBtb3N0IGhpZ2hseSByYW5rZWQgZ2VuZXMsIHRoZSBydW5uaW5nIHN1bSBpbmNyZWFzZXMgZm9yIGVhY2ggZ2VuZSBpbiB0aGUgcGF0aHdheSBhbmQgZGVjcmVhc2VzIGZvciBlYWNoIGdlbmUgbm90IGluIHRoZSBwYXRod2F5LgpUaGUgZW5yaWNobWVudCBzY29yZSBmb3IgYSBwYXRod2F5IGlzIHRoZSBydW5uaW5nIHN1bSdzIG1heGltdW0gZGV2aWF0aW9uIGZyb20gemVyby4KR1NFQSBhbHNvIGFzc2Vzc2VzIHN0YXRpc3RpY2FsIHNpZ25pZmljYW5jZSBvZiB0aGUgc2NvcmVzIGZvciBlYWNoIHBhdGh3YXkgdGhyb3VnaCBwZXJtdXRhdGlvbiB0ZXN0aW5nLgpBcyBhIHJlc3VsdCwgZWFjaCBpbnB1dCBwYXRod2F5IHdpbGwgaGF2ZSBhIHAtdmFsdWUgYXNzb2NpYXRlZCB3aXRoIGl0IHRoYXQgaXMgdGhlbiBjb3JyZWN0ZWQgZm9yIG11bHRpcGxlIGh5cG90aGVzaXMgdGVzdGluZyAoW1N1YnJhbWFuaWFuIF9ldCBhbC5fIDIwMDVdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMDczL3BuYXMuMDUwNjU4MDEwMik7IFtZdV0oaHR0cDovL3l1bGFiLXNtdS50b3AvY2x1c3RlclByb2ZpbGVyLWJvb2svY2hhcHRlcjIuaHRtbCNnZW5lLXNldC1lbnJpY2htZW50LWFuYWx5c2lzKSkuCgpUaGUgaW1wbGVtZW50YXRpb24gb2YgR1NFQSB3ZSB1c2UgaW4gaGVyZSBleGFtcGxlcyByZXF1aXJlcyBhIGdlbmUgbGlzdCBvcmRlcmVkIGJ5IHNvbWUgc3RhdGlzdGljIGFuZCBpbnB1dCBnZW5lIHNldHMuCldoZW4geW91IHVzZSBwcmV2aW91c2x5IGNvbXB1dGVkIGdlbmUtbGV2ZWwgc3RhdGlzdGljcyB3aXRoIEdTRUEsIGl0IGlzIGNhbGxlZCBHU0VBIHByZS1yYW5rZWQuCgojIyBERVNlcTIgcmVzdWx0cwoKYGBge3IgcmVhZF9pbl9tYXJrZXJzLCBsaXZlID0gVFJVRX0KZGVzZXFfZGYgPC0gcmVhZHI6OnJlYWRfdHN2KGlucHV0X2ZpbGUpCmBgYAoKYGBge3IgZGVzZXFfaGVhZH0KaGVhZChkZXNlcV9kZikKYGBgCgpUaGlzIGRhdGEgZnJhbWUgdXNlcyBFbnNlbWJsIGdlbmUgaWRlbnRpZmllcnMuCldlJ2xsIG5lZWQgdG8gbWFrZSBzdXJlIG91ciBnZW5lIHNldHMgdXNlIHRoZSBzYW1lIGlkZW50aWZpZXJzLgpMZXQncyB0YWtlIGEgbG9vayBhdCB0aGUgZmlyc3QgZmV3IHJvd3Mgb2YgdGhlIGRhdGEgZnJhbWUgdGhhdCBjb250YWlucyB0aGUgaGFsbG1hcmsgZ2VuZSBzZXRzLgoKYGBge3IgaGFsbG1hcmtfaGVhZCwgbGl2ZSA9IFRSVUV9CmhlYWQoaHNfaGFsbG1hcmtzX2RmKQpgYGAKCldlIGNhbiBzZWUgdGhhdCB0aGUgZ2VuZSBzZXRzIGZyb20gYG1zaWdkYnJgIGhhdmUgRW5zZW1ibCBnZW5lIGlkZW50aWZpZXJzIGFzc29jaWF0ZWQgd2l0aCB0aGVtLCBzbyB3ZSBkb24ndCBuZWVkIHRvIGRvIGFueSBjb252ZXJzaW9uLgpIb3dldmVyLCB3ZSdsbCBuZWVkIHRvIHBhc3MgdGhlIGNvcnJlY3QgY29sdW1uIHRvIHRoZSBmdW5jdGlvbiB0aGF0IHJ1bnMgR1NFQS4KCk9uZSBnb29kIHRoaW5nIGFib3V0IEVuc2VtYmwgZ2VuZSBpZGVudGlmaWVycyBpcyB0aGF0IHRoZXkgYXJlIGxlc3MgbGlrZWx5IHRvIGJlIGR1cGxpY2F0ZWQgdGhhbiwgZm9yIGV4YW1wbGUsIGdlbmUgc3ltYm9scy4KKE11bHRpcGxlIEVuc2VtYmwgZ2VuZSBpZGVudGlmaWVycyBjYW4gbWFwIHRvIHRoZSBzYW1lIHN5bWJvbC4pCgpUaGUgR1NFQSBhcHByb2FjaCByZXF1aXJlcyBvbiBkaXNjcmltaW5hdGluZyBiZXR3ZWVuIGdlbmVzIHRoYXQgYXJlIGluIGEgZ2VuZSBzZXQgYW5kIHRob3NlIHRoYXQgYXJlIG5vdC4KUHJhY3RpY2FsbHkgc3BlYWtpbmcsIGdlbmUgc2V0cyBhcmUganVzdCBjb2xsZWN0aW9ucyBvZiBnZW5lIGlkZW50aWZpZXJzIQpXaGVuIHRoZSBmdW5jdGlvbiB3ZSB1c2UgZm9yIEdTRUEgcHJlLXJhbmtlZCBnZXRzIGEgbGlzdCB3aXRoIGR1cGxpY2F0ZWQgZ2VuZSBpZGVudGlmaWVycywgaXQgY2FuIHByb2R1Y2UgdW5leHBlY3RlZCByZXN1bHRzLgpTbywgbGV0J3MgY2hlY2sgZm9yIGR1cGxpY2F0ZXMgaW4gdGhlIGRhdGEgZnJhbWUgb2YgREVTZXEyIHJlc3VsdHMuCgpgYGB7ciBjaGVja19kdXBsaWNhdGVzLCBsaXZlID0gVFJVRX0KYW55KGR1cGxpY2F0ZWQoZGVzZXFfZGYkZW5zZW1ibF9pZCkpCmBgYAoKVGhlcmUgYXJlIG5vIGR1cGxpY2F0ZXMgZm9yIHVzIHRvIHdvcnJ5IGFib3V0IQoKIyMjIFByZS1yYW5rZWQgbGlzdAoKVGhlIGBHU0VBKClgIGZ1bmN0aW9uIHRha2VzIGEgcHJlLXJhbmtlZCAoc29ydGVkKSBuYW1lZCB2ZWN0b3Igb2Ygc3RhdGlzdGljcywgd2hlcmUgdGhlIG5hbWVzIGluIHRoZSB2ZWN0b3IgYXJlIGdlbmUgaWRlbnRpZmllcnMuClRoaXMgaXMgc3RlcCAxIC0tIGdlbmUtbGV2ZWwgc3RhdGlzdGljcy4KCmBgYHtyIGxmY192ZWN0b3J9CmxmY192ZWN0b3IgPC0gZGVzZXFfZGYgfD4KICAjIEV4dHJhY3QgYSB2ZWN0b3Igb2YgYGxvZzJGb2xkQ2hhbmdlYCBuYW1lZCBieSBgZW5zZW1ibF9pZGAKICBkcGx5cjo6cHVsbChsb2cyRm9sZENoYW5nZSwgbmFtZSA9IGVuc2VtYmxfaWQpCmxmY192ZWN0b3IgPC0gc29ydChsZmNfdmVjdG9yLCBkZWNyZWFzaW5nID0gVFJVRSkKYGBgCgpMZXQncyBsb29rIGF0IHRoZSB0b3AgcmFua2VkIHZhbHVlcy4KCmBgYHtyIGhlYWRfbGZjLCBsaXZlID0gVFJVRX0KIyBMb29rIGF0IGZpcnN0IGVudHJpZXMgb2YgdGhlIGxvZyBmb2xkIGNoYW5nZSB2ZWN0b3IKaGVhZChsZmNfdmVjdG9yKQpgYGAKCkFuZCB0aGUgYm90dG9tIG9mIHRoZSBsaXN0LgoKYGBge3IgdGFpbF9sZmMsIGxpdmUgPSBUUlVFfQojIExvb2sgYXQgdGhlIGxhc3QgZW50cmllcyBvZiB0aGUgbG9nIGZvbGQgY2hhbmdlIHZlY3Rvcgp0YWlsKGxmY192ZWN0b3IpCmBgYAoKIyMgUnVuIEdTRUEKCk5vdyBmb3IgdGhlIGFuYWx5c2lzIQoKV2UgY2FuIHVzZSB0aGUgYEdTRUEoKWAgZnVuY3Rpb24gdG8gcGVyZm9ybSBHU0VBIHdpdGggYW55IGdlbmVyaWMgc2V0IG9mIGdlbmUgc2V0cywgYnV0IHRoZXJlIGFyZSBzZXZlcmFsIGZ1bmN0aW9ucyBmb3IgdXNpbmcgc3BlY2lmaWMsIGNvbW1vbmx5IHVzZWQgZ2VuZSBzZXRzIChlLmcuLCBgZ3NlS0VHRygpYCkuCgpgYGB7ciBydW5fZ3NlYX0KZ3NlYV9yZXN1bHRzIDwtIEdTRUEoZ2VuZUxpc3QgPSBsZmNfdmVjdG9yLCAgIyBvcmRlcmVkIHJhbmtlZCBnZW5lIGxpc3QKICAgICAgICAgICAgICAgICAgICAgbWluR1NTaXplID0gMjUsICAjIG1pbmltdW0gZ2VuZSBzZXQgc2l6ZQogICAgICAgICAgICAgICAgICAgICBtYXhHU1NpemUgPSA1MDAsICAjIG1heGltdW0gZ2VuZSBzZXQgc2V0CiAgICAgICAgICAgICAgICAgICAgIHB2YWx1ZUN1dG9mZiA9IDAuMDUsCiAgICAgICAgICAgICAgICAgICAgIHBBZGp1c3RNZXRob2QgPSAiQkgiLCAgIyBjb3JyZWN0aW9uIGZvciBtdWx0aXBsZSBoeXBvdGhlc2lzIHRlc3RpbmcKICAgICAgICAgICAgICAgICAgICAgVEVSTTJHRU5FID0gZHBseXI6OnNlbGVjdChoc19oYWxsbWFya3NfZGYsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ3NfbmFtZSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBlbnNlbWJsX2dlbmUpKSAgIyBwYXNzIHRoZSBjb3JyZWN0IGlkZW50aWZpZXIgY29sdW1uCmBgYApMZXQncyB0YWtlIGEgbG9vayBhdCB0aGUgR1NFQSByZXN1bHRzLgoKYGBge3Igdmlld19nc2VhLCBsaXZlID0gVFJVRSwgZXZhbCA9IEZBTFNFfQpWaWV3KGdzZWFfcmVzdWx0c0ByZXN1bHQgfD4KICAgICAgIGRwbHlyOjphcnJhbmdlKGRwbHlyOjpkZXNjKE5FUykpCikKYGBgCgpOb3JtYWxpemVkIGVucmljaG1lbnQgc2NvcmVzIChORVMpIGFyZSBlbnJpY2htZW50IHNjb3JlcyB0aGF0IGFyZSBzY2FsZWQgdG8gbWFrZSBnZW5lIHNldHMgdGhhdCBjb250YWluIGRpZmZlcmVudCBudW1iZXIgb2YgZ2VuZXMgY29tcGFyYWJsZS4KClBhdGh3YXlzIHdpdGggc2lnbmlmaWNhbnQsIGhpZ2hseSBwb3NpdGl2ZSBORVMgYXJlIGVucmljaGVkIGluIEVSTVMgbXlvYmxhc3RzLCB3aGVyZWFzIHBhdGh3YXlzIHdpdGggc2lnbmlmaWNhbnQsIGhpZ2hseSBuZWdhdGl2ZSBORVMgYXJlIGVucmljaGVkIGluIEFSTVMgbXlvYmxhc3RzLgoKTGV0J3Mgd3JpdGUgdGhlc2UgcmVzdWx0cyB0byBmaWxlLgoKYGBge3Igd3JpdGVfZ3NlYX0KZ3NlYV9yZXN1bHRzQHJlc3VsdCB8PiByZWFkcjo6d3JpdGVfdHN2KG91dHB1dF9maWxlKQpgYGAKCiMjIyBWaXN1YWxpemluZyBHU0VBIHJlc3VsdHMKCldlIGNhbiB2aXN1YWxpemUgR1NFQSByZXN1bHRzIGZvciBpbmRpdmlkdWFsIHBhdGh3YXlzIG9yIGdlbmUgc2V0cyB1c2luZyBgZW5yaWNocGxvdDo6Z3NlYXBsb3QoKWAuCkxldCdzIHRha2UgYSBsb29rIGF0IDMgZGlmZmVyZW50IHBhdGh3YXlzIC0tIG9uZSB3aXRoIGEgaGlnaGx5IHBvc2l0aXZlIE5FUywgb25lIHdpdGggYSBoaWdobHkgbmVnYXRpdmUgTkVTLCBhbmQgb25lIHRoYXQgd2FzIG5vdCBhIHNpZ25pZmljYW50IHJlc3VsdCAtLSB0byBnZXQgbW9yZSBpbnNpZ2h0IGludG8gaG93IEVTIGFyZSBjYWxjdWxhdGVkLgoKIyMjIyBIaWdobHkgUG9zaXRpdmUgTkVTCgpMZXQncyB0YWtlIGxvb2sgYXQgYSBwYXRod2F5IHdpdGggYSBoaWdobHkgcG9zaXRpdmUgTkVTIChgSEFMTE1BUktfTVlDX1RBUkdFVFNfVjJgKSB1c2luZyBhIEdTRUEgcGxvdC4KCmBgYHtyIGhpZ2hseV9wb3N9CmVucmljaHBsb3Q6OmdzZWFwbG90KGdzZWFfcmVzdWx0cywKICAgICAgICAgICAgICAgICAgICAgZ2VuZVNldElEID0gIkhBTExNQVJLX01ZQ19UQVJHRVRTX1YyIiwKICAgICAgICAgICAgICAgICAgICAgdGl0bGUgPSAiSEFMTE1BUktfTVlDX1RBUkdFVFNfVjIiLAogICAgICAgICAgICAgICAgICAgICBjb2xvci5saW5lID0gIiMwMDY2RkYiKQpgYGAKCk5vdGljZSBob3cgdGhlIGdlbmVzIHRoYXQgYXJlIGluIHRoZSBnZW5lIHNldCwgaW5kaWNhdGVkIGJ5IHRoZSBibGFjayBiYXJzLCB0ZW5kIHRvIGJlIG9uIHRoZSBsZWZ0IHNpZGUgb2YgdGhlIGdyYXBoIGluZGljYXRpbmcgdGhhdCB0aGV5IGhhdmUgcG9zaXRpdmUgZ2VuZS1sZXZlbCBzY29yZXMuCgojIyMjIEhpZ2hseSBOZWdhdGl2ZSBORVMKClRoZSBnZW5lIHNldCBgSEFMTE1BUktfTVlPR0VORVNJU2AgaGFkIGEgaGlnaGx5IG5lZ2F0aXZlIE5FUy4KCmBgYHtyIGhpZ2hseV9uZWd9CmVucmljaHBsb3Q6OmdzZWFwbG90KGdzZWFfcmVzdWx0cywKICAgICAgICAgICAgICAgICAgICAgZ2VuZVNldElEID0gIkhBTExNQVJLX01ZT0dFTkVTSVMiLAogICAgICAgICAgICAgICAgICAgICB0aXRsZSA9ICJIQUxMTUFSS19NWU9HRU5FU0lTIiwKICAgICAgICAgICAgICAgICAgICAgY29sb3IubGluZSA9ICIjMDA2NkZGIikKYGBgCgpUaGlzIGdlbmUgc2V0IHNob3dzIHRoZSBvcHBvc2l0ZSBwYXR0ZXJuIC0tIGdlbmVzIGluIHRoZSBwYXRod2F5IHRlbmQgdG8gYmUgb24gdGhlIHJpZ2h0IHNpZGUgb2YgdGhlIGdyYXBoLgoKIyMjIyBBIG5vbi1zaWduaWZpY2FudCByZXN1bHQKClRoZSBgQHJlc3VsdHNgIHNsb3Qgd2lsbCBvbmx5IHNob3cgZ2VuZSBzZXRzIHRoYXQgcGFzcyB0aGUgYHB2YWx1ZUN1dG9mZmAgdGhyZXNob2xkIHdlIHN1cHBsaWVkIHRvIGBHU0VBKClgLCBidXQgd2UgY2FuIHBsb3QgYW55IGdlbmUgc2V0IHNvIGxvbmcgYXMgd2Uga25vdyBpdHMgbmFtZS4KTGV0J3MgbG9vayBhdCBgSEFMTE1BUktfUDUzX1BBVEhXQVlgLCB3aGljaCB3YXMgbm90IGluIHRoZSByZXN1bHRzIHdlIHZpZXdlZCBlYXJsaWVyLgoKYGBge3IgcDUzLCBsaXZlID0gVFJVRX0KZW5yaWNocGxvdDo6Z3NlYXBsb3QoZ3NlYV9yZXN1bHRzLAogICAgICAgICAgICAgICAgICAgICBnZW5lU2V0SUQgPSAiSEFMTE1BUktfUDUzX1BBVEhXQVkiLAogICAgICAgICAgICAgICAgICAgICB0aXRsZSA9ICJIQUxMTUFSS19QNTNfUEFUSFdBWSIsCiAgICAgICAgICAgICAgICAgICAgIGNvbG9yLmxpbmUgPSAiIzAwNjZGRiIpCmBgYAoKR2VuZXMgaW4gdGhlIHBhdGh3YXkgYXJlIGRpc3RyaWJ1dGVkIG1vcmUgZXZlbmx5IHRocm91Z2hvdXQgdGhlIHJhbmtlZCBsaXN0LCByZXN1bHRpbmcgaW4gYSBtb3JlICJtaWRkbGluZyIgc2NvcmUuCgoqTm90ZTogVGhlIHBsb3RzIHJldHVybmVkIGJ5IGBlbnJpY2hwbG90Ojpnc2VhcGxvdGAgYXJlIGdncGxvdHMsIHNvIHdlIGNvdWxkIHVzZSBgZ2dwbG90Mjo6Z2dzYXZlKClgIHRvIHNhdmUgdGhlbSB0byBmaWxlIGlmIHdlIHdhbnRlZCB0by4qCgojIyBTZXNzaW9uIEluZm8KCmBgYHtyIHNlc3Npb25faW5mb30Kc2Vzc2lvbkluZm8oKQpgYGAK
From fb4a2a8b4bb70f1f3b5e44fd17c26c95cb7f9cbd Mon Sep 17 00:00:00 2001 From: Jaclyn Taroni <19534205+jaclyn-taroni@users.noreply.github.com> Date: Mon, 25 Nov 2024 10:24:55 -0500 Subject: [PATCH 5/7] Make corresponding changes to render live script --- scripts/render-live.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/scripts/render-live.sh b/scripts/render-live.sh index 2cb07fc4..c95e12f4 100644 --- a/scripts/render-live.sh +++ b/scripts/render-live.sh @@ -35,8 +35,7 @@ files=( scRNA-seq-advanced/01-read_filter_normalize_scRNA.Rmd scRNA-seq-advanced/02-dataset_integration.Rmd scRNA-seq-advanced/03-differential_expression.Rmd - scRNA-seq-advanced/04-overrepresentation_analysis.Rmd - scRNA-seq-advanced/05-gene_set_enrichment_analysis.Rmd + scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd # machine-learning/01-openpbta_heatmap.Rmd # machine-learning/02-openpbta_consensus_clustering.Rmd # machine-learning/03-openpbta_PLIER.Rmd From fff81bf861600a31f96a211d352253c9a8709442 Mon Sep 17 00:00:00 2001 From: Jaclyn Taroni <19534205+jaclyn-taroni@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:22:14 -0500 Subject: [PATCH 6/7] Apply suggestions from code review Co-authored-by: Stephanie Spielman --- scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd index 0ae64adb..5b519495 100644 --- a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd +++ b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd @@ -24,18 +24,18 @@ In this notebook, we'll analyze the differential expression results from the las GSEA is a functional class scoring (FCS) approach to pathway analysis that was first introduced in [Subramanian _et al._ (2005)](https://doi.org/10.1073/pnas.0506580102). The rationale behind FCS approaches is that small changes in individual genes that participate in the same biological process or pathway can be significant and of biological interest. -There are 3 general steps in FCS methods ([Khatri _et al._ 2012]( https://doi.org/10.1371/journal.pcbi.1002375)): +There are 3 general steps in FCS methods ([Khatri _et al._ 2012](https://doi.org/10.1371/journal.pcbi.1002375)): 1. Calculate a gene-level statistic (here, we'll use the summary log fold changes in our DESeq2 results) -2. Gene-level statistics are aggregated into a pathway-level statistic +2. Aggregate gene-level statistics into a pathway-level statistic 3. Assess the statistical significance of the pathway-level statistic #### Other resources * For another example using `clusterProfiler` for GSEA, see [_Intro to DGE: Functional Analysis._ from Harvard Chan Bioinformatics Core Training.](https://hbctraining.github.io/DGE_workshop/lessons/09_functional_analysis.html) * The way we'll use `clusterProfiler` here uses `fgsea` (Fast Gene Set Enrichment Analysis) under the hood. -You can read more about fgsea in [Korotkevich _et al._ (2021)](https://doi.org/10.1101/060012). -* [refine.bio examples Gene set enrichment analysis - RNA-seq](https://alexslemonade.github.io/refinebio-examples/03-rnaseq/pathway-analysis_rnaseq_02_gsea.html) from which this material has been adapted. +You can read more about `fgsea` in [Korotkevich _et al._ (2021)](https://doi.org/10.1101/060012). +* See the [refine.bio examples for "Gene set enrichment analysis - RNA-seq"](https://alexslemonade.github.io/refinebio-examples/03-rnaseq/pathway-analysis_rnaseq_02_gsea.html) from which this material has been adapted. ## Set up From 44f3b761366124417510894ef28aa3e02f595d88 Mon Sep 17 00:00:00 2001 From: Jaclyn Taroni <19534205+jaclyn-taroni@users.noreply.github.com> Date: Mon, 25 Nov 2024 17:54:11 +0000 Subject: [PATCH 7/7] Respond to review and rerun --- .../04-gene_set_enrichment_analysis.Rmd | 5 +- .../04-gene_set_enrichment_analysis.nb.html | 123 +++++++----------- 2 files changed, 52 insertions(+), 76 deletions(-) diff --git a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd index 5b519495..5029c88d 100644 --- a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd +++ b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.Rmd @@ -32,7 +32,7 @@ There are 3 general steps in FCS methods ([Khatri _et al._ 2012](https://doi.org #### Other resources -* For another example using `clusterProfiler` for GSEA, see [_Intro to DGE: Functional Analysis._ from Harvard Chan Bioinformatics Core Training.](https://hbctraining.github.io/DGE_workshop/lessons/09_functional_analysis.html) +* For another example using `clusterProfiler` for GSEA, see [_Intro to DGE: Functional Analysis._ from Harvard Chan Bioinformatics Core Training.](https://hbctraining.github.io/Training-modules/DGE-functional-analysis/lessons/02_functional_analysis.html) * The way we'll use `clusterProfiler` here uses `fgsea` (Fast Gene Set Enrichment Analysis) under the hood. You can read more about `fgsea` in [Korotkevich _et al._ (2021)](https://doi.org/10.1101/060012). * See the [refine.bio examples for "Gene set enrichment analysis - RNA-seq"](https://alexslemonade.github.io/refinebio-examples/03-rnaseq/pathway-analysis_rnaseq_02_gsea.html) from which this material has been adapted. @@ -158,6 +158,9 @@ head(hs_hallmarks_df) We can see that the gene sets from `msigdbr` have Ensembl gene identifiers associated with them, so we don't need to do any conversion. However, we'll need to pass the correct column to the function that runs GSEA. +If we needed to do gene identifier conversion, we would likely use the `AnnotationDbi` package. +You can see an example in this Harvard Chan Bioinformatics Core Training material: + One good thing about Ensembl gene identifiers is that they are less likely to be duplicated than, for example, gene symbols. (Multiple Ensembl gene identifiers can map to the same symbol.) diff --git a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html index a825edd6..5197164f 100644 --- a/scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html +++ b/scRNA-seq-advanced/04-gene_set_enrichment_analysis.nb.html @@ -2911,8 +2911,7 @@

Objectives

  1. Calculate a gene-level statistic (here, we’ll use the summary log fold changes in our DESeq2 results)
  2. -
  3. Gene-level statistics are aggregated into a pathway-level -statistic
  4. +
  5. Aggregate gene-level statistics into a pathway-level statistic
  6. Assess the statistical significance of the pathway-level statistic
@@ -2920,16 +2919,16 @@

Objectives

Other resources

@@ -2939,36 +2938,22 @@

Set up

Libraries

- -
# Package to run GSEA
- - -
Warning message:
-replacing previous import ‘S4Arrays::makeNindexFromArrayViewport’ by ‘DelayedArray::makeNindexFromArrayViewport’ when loading ‘SummarizedExperiment’ 
- - -
library(clusterProfiler)
+ +
# Package to run GSEA
+library(clusterProfiler)
- -
clusterProfiler v4.12.0  For help: https://yulab-smu.top/biomedical-knowledge-mining-book/
+
+

+Registered S3 method overwritten by 'data.table':
+  method           from
+  print.data.table     
+clusterProfiler v4.12.0  For help: https://yulab-smu.top/biomedical-knowledge-mining-book/
 
 If you use clusterProfiler in published research, please cite:
 T Wu, E Hu, S Xu, M Chen, P Guo, Z Dai, T Feng, L Zhou, W Tang, L Zhan, X Fu, S Liu, X Bo, and G Yu. clusterProfiler 4.0: A universal enrichment tool for interpreting omics data. The Innovation. 2021, 2(3):100141
 
 Attaching package: ‘clusterProfiler’
 
-The following object is masked from ‘package:AnnotationDbi’:
-
-    select
-
-The following object is masked from ‘package:IRanges’:
-
-    slice
-
-The following object is masked from ‘package:S4Vectors’:
-
-    rename
-
 The following object is masked from ‘package:stats’:
 
     filter
@@ -3114,12 +3099,12 @@

DESeq2 results

deseq_df <- readr::read_tsv(input_file)
- +
Rows: 60319 Columns: 27
-── Column specification ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
+── Column specification ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Delimiter: "\t"
 chr  (2): ensembl_id, gene_symbol
-dbl (25): baseMean, log2FoldChange, lfcSE, pvalue, padj, SCPCL000478.mean, SCPCL000478.detected, SCPCL000479.mean, SCPCL000479.det...
+dbl (25): baseMean, log2FoldChange, lfcSE, pvalue, padj, SCPCL000478.mean, SCPCL000478.detected, SCPCL000479.mean, SCPCL000479.de...
 
 ℹ Use `spec()` to retrieve the full column specification for this data.
 ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
@@ -3162,6 +3147,9 @@

DESeq2 results

gene identifiers associated with them, so we don’t need to do any conversion. However, we’ll need to pass the correct column to the function that runs GSEA.

+

If we needed to do gene identifier conversion, we would likely use +the AnnotationDbi package. You can see an example in this +Harvard Chan Bioinformatics Core Training material: https://hbctraining.github.io/DGE_workshop_salmon_online/lessons/AnnotationDbi_lesson.html

One good thing about Ensembl gene identifiers is that they are less likely to be duplicated than, for example, gene symbols. (Multiple Ensembl gene identifiers can map to the same symbol.)

@@ -3363,7 +3351,7 @@

Session Info

sessionInfo()
- +
R version 4.4.0 (2024-04-24)
 Platform: x86_64-pc-linux-gnu
 Running under: Ubuntu 22.04.4 LTS
@@ -3381,55 +3369,40 @@ 

Session Info

tzcode source: system (glibc) attached base packages: -[1] stats4 stats graphics grDevices utils datasets methods base +[1] stats graphics grDevices datasets utils methods base other attached packages: - [1] clusterProfiler_4.12.0 msigdbr_7.5.1 GSEABase_1.66.0 graph_1.82.0 - [5] annotate_1.82.0 XML_3.99-0.16.1 AnnotationDbi_1.66.0 AUCell_1.26.0 - [9] SingleCellExperiment_1.26.0 SummarizedExperiment_1.34.0 Biobase_2.64.0 GenomicRanges_1.56.0 -[13] GenomeInfoDb_1.40.0 IRanges_2.38.0 S4Vectors_0.42.0 BiocGenerics_0.50.0 -[17] MatrixGenerics_1.16.0 matrixStats_1.3.0 +[1] msigdbr_7.5.1 clusterProfiler_4.12.0 loaded via a namespace (and not attached): - [1] RColorBrewer_1.1-3 rstudioapi_0.16.0 jsonlite_1.8.8 magrittr_2.0.3 - [5] ggbeeswarm_0.7.2 farver_2.1.1 rmarkdown_2.26 fs_1.6.4 - [9] zlibbioc_1.50.0 vctrs_0.6.5 memoise_2.0.1 DelayedMatrixStats_1.26.0 - [13] ggtree_3.12.0 htmltools_0.5.8.1 S4Arrays_1.4.0 BiocNeighbors_1.22.0 - [17] gridGraphics_0.5-1 SparseArray_1.4.0 sass_0.4.9 bslib_0.7.0 - [21] htmlwidgets_1.6.4 plyr_1.8.9 cachem_1.0.8 igraph_2.0.3 - [25] lifecycle_1.0.4 pkgconfig_2.0.3 gson_0.1.0 rsvd_1.0.5 - [29] Matrix_1.7-0 R6_2.5.1 fastmap_1.1.1 GenomeInfoDbData_1.2.12 - [33] aplot_0.2.2 digest_0.6.35 enrichplot_1.24.0 colorspace_2.1-0 - [37] patchwork_1.2.0 scater_1.32.0 irlba_2.3.5.1 RSQLite_2.3.6 - [41] beachmat_2.20.0 labeling_0.4.3 fansi_1.0.6 httr_1.4.7 - [45] polyclip_1.10-6 abind_1.4-5 compiler_4.4.0 bit64_4.0.5 - [49] withr_3.0.0 BiocParallel_1.38.0 viridis_0.6.5 DBI_1.2.2 - [53] ggforce_0.4.2 R.utils_2.12.3 MASS_7.3-60.2 DelayedArray_0.30.0 - [57] HDO.db_0.99.1 tools_4.4.0 vipor_0.4.7 scatterpie_0.2.2 - [61] ape_5.8 beeswarm_0.4.0 R.oo_1.26.0 glue_1.7.0 - [65] nlme_3.1-164 GOSemSim_2.30.0 shadowtext_0.1.3 grid_4.4.0 - [69] reshape2_1.4.4 fgsea_1.30.0 generics_0.1.3 gtable_0.3.5 - [73] tzdb_0.4.0 R.methodsS3_1.8.2 tidyr_1.3.1 data.table_1.15.4 - [77] hms_1.1.3 BiocSingular_1.20.0 tidygraph_1.3.1 ScaledMatrix_1.12.0 - [81] utf8_1.2.4 XVector_0.44.0 stringr_1.5.1 ggrepel_0.9.5 - [85] pillar_1.9.0 vroom_1.6.5 yulab.utils_0.1.4 babelgene_22.9 - [89] splines_4.4.0 dplyr_1.1.4 tweenr_2.0.3 treeio_1.28.0 - [93] lattice_0.22-6 renv_1.0.7 bit_4.0.5 tidyselect_1.2.1 - [97] GO.db_3.19.1 Biostrings_2.72.0 scuttle_1.14.0 knitr_1.46 -[101] gridExtra_2.3 xfun_0.43 graphlayouts_1.1.1 stringi_1.8.3 -[105] UCSC.utils_1.0.0 lazyeval_0.2.2 ggfun_0.1.4 yaml_2.3.8 -[109] evaluate_0.23 codetools_0.2-20 ggraph_2.2.1 qvalue_2.36.0 -[113] tibble_3.2.1 ggplotify_0.1.2 cli_3.6.2 xtable_1.8-4 -[117] jquerylib_0.1.4 munsell_0.5.1 Rcpp_1.0.12 png_0.1-8 -[121] parallel_4.4.0 ggplot2_3.5.1 readr_2.1.5 blob_1.2.4 -[125] DOSE_3.30.0 sparseMatrixStats_1.16.0 tidytree_0.4.6 viridisLite_0.4.2 -[129] scales_1.3.0 purrr_1.0.2 crayon_1.5.2 rlang_1.1.3 -[133] fastmatch_1.1-4 cowplot_1.1.3 KEGGREST_1.44.0
+ [1] DBI_1.2.2 gson_0.1.0 shadowtext_0.1.3 gridExtra_2.3 rlang_1.1.3 + [6] magrittr_2.0.3 DOSE_3.30.0 compiler_4.4.0 RSQLite_2.3.6 png_0.1-8 + [11] vctrs_0.6.5 reshape2_1.4.4 stringr_1.5.1 pkgconfig_2.0.3 crayon_1.5.2 + [16] fastmap_1.1.1 XVector_0.44.0 labeling_0.4.3 ggraph_2.2.1 utf8_1.2.4 + [21] HDO.db_0.99.1 tzdb_0.4.0 enrichplot_1.24.0 UCSC.utils_1.0.0 purrr_1.0.2 + [26] bit_4.0.5 xfun_0.43 zlibbioc_1.50.0 cachem_1.0.8 aplot_0.2.2 + [31] GenomeInfoDb_1.40.0 jsonlite_1.8.8 blob_1.2.4 BiocParallel_1.38.0 tweenr_2.0.3 + [36] parallel_4.4.0 R6_2.5.1 stringi_1.8.3 RColorBrewer_1.1-3 GOSemSim_2.30.0 + [41] Rcpp_1.0.12 knitr_1.46 readr_2.1.5 IRanges_2.38.0 Matrix_1.7-0 + [46] splines_4.4.0 igraph_2.0.3 tidyselect_1.2.1 qvalue_2.36.0 rstudioapi_0.16.0 + [51] yaml_2.3.8 viridis_0.6.5 codetools_0.2-20 lattice_0.22-6 tibble_3.2.1 + [56] plyr_1.8.9 treeio_1.28.0 Biobase_2.64.0 withr_3.0.0 KEGGREST_1.44.0 + [61] gridGraphics_0.5-1 scatterpie_0.2.2 polyclip_1.10-6 Biostrings_2.72.0 pillar_1.9.0 + [66] BiocManager_1.30.22 ggtree_3.12.0 renv_1.0.7 stats4_4.4.0 ggfun_0.1.4 + [71] generics_0.1.3 vroom_1.6.5 hms_1.1.3 S4Vectors_0.42.0 ggplot2_3.5.1 + [76] tidytree_0.4.6 munsell_0.5.1 scales_1.3.0 glue_1.7.0 lazyeval_0.2.2 + [81] tools_4.4.0 data.table_1.15.4 fgsea_1.30.0 babelgene_22.9 fs_1.6.4 + [86] graphlayouts_1.1.1 fastmatch_1.1-4 tidygraph_1.3.1 cowplot_1.1.3 grid_4.4.0 + [91] ape_5.8 tidyr_1.3.1 AnnotationDbi_1.66.0 colorspace_2.1-0 nlme_3.1-164 + [96] GenomeInfoDbData_1.2.12 patchwork_1.2.0 ggforce_0.4.2 cli_3.6.2 fansi_1.0.6 +[101] viridisLite_0.4.2 dplyr_1.1.4 gtable_0.3.5 yulab.utils_0.1.4 digest_0.6.35 +[106] BiocGenerics_0.50.0 ggrepel_0.9.5 ggplotify_0.1.2 farver_2.1.1 memoise_2.0.1 +[111] lifecycle_1.0.4 httr_1.4.7 GO.db_3.19.1 bit64_4.0.5 MASS_7.3-60.2
-
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcyAoR1NFQSkiCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjQKLS0tCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOgoKLSBQcmVwYXJlIHRhYnVsYXIgZGF0YSBvZiBnZW5lLWxldmVsIHN0YXRpc3RpY3MgZm9yIHVzZSB3aXRoIEdlbmUgU2V0IEVucmljaG1lbnQgQW5hbHlzaXMgKEdTRUEpCi0gQWNjZXNzIFtNb2xlY3VsYXIgU2lnbmF0dXJlcyBEYXRhYmFzZSBnZW5lIHNldCBjb2xsZWN0aW9uc10oaHR0cHM6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2NvbGxlY3Rpb25zLmpzcCkgdmlhIHRoZSBgbXNpZ2RicmAgcGFja2FnZQotIFBlcmZvcm0gR1NFQSB3aXRoIHRoZSBgY2x1c3RlclByb2ZpbGVyYCBwYWNrYWdlCi0gVmlzdWFsaXplIEdTRUEgcmVzdWx0cyB3aXRoIHRoZSBgZW5yaWNocGxvdGAgcGFja2FnZQoKLS0tCgpJbiB0aGlzIG5vdGVib29rLCB3ZSdsbCBhbmFseXplIHRoZSBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiByZXN1bHRzIGZyb20gdGhlIGxhc3Qgbm90ZWJvb2suCgpHU0VBIGlzIGEgZnVuY3Rpb25hbCBjbGFzcyBzY29yaW5nIChGQ1MpIGFwcHJvYWNoIHRvIHBhdGh3YXkgYW5hbHlzaXMgdGhhdCB3YXMgZmlyc3QgaW50cm9kdWNlZCBpbiBbU3VicmFtYW5pYW4gX2V0IGFsLl8gKDIwMDUpXShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpLgpUaGUgcmF0aW9uYWxlIGJlaGluZCBGQ1MgYXBwcm9hY2hlcyBpcyB0aGF0IHNtYWxsIGNoYW5nZXMgaW4gaW5kaXZpZHVhbCBnZW5lcyB0aGF0IHBhcnRpY2lwYXRlIGluIHRoZSBzYW1lIGJpb2xvZ2ljYWwgcHJvY2VzcyBvciBwYXRod2F5IGNhbiBiZSBzaWduaWZpY2FudCBhbmQgb2YgYmlvbG9naWNhbCBpbnRlcmVzdC4KClRoZXJlIGFyZSAzIGdlbmVyYWwgc3RlcHMgaW4gRkNTIG1ldGhvZHMgKFtLaGF0cmkgX2V0IGFsLl8gMjAxMl0oIGh0dHBzOi8vZG9pLm9yZy8xMC4xMzcxL2pvdXJuYWwucGNiaS4xMDAyMzc1KSk6CgoxLiBDYWxjdWxhdGUgYSBnZW5lLWxldmVsIHN0YXRpc3RpYyAoaGVyZSwgd2UnbGwgdXNlIHRoZSBzdW1tYXJ5IGxvZyBmb2xkIGNoYW5nZXMgaW4gb3VyIERFU2VxMiByZXN1bHRzKQoyLiBHZW5lLWxldmVsIHN0YXRpc3RpY3MgYXJlIGFnZ3JlZ2F0ZWQgaW50byBhIHBhdGh3YXktbGV2ZWwgc3RhdGlzdGljCjMuIEFzc2VzcyB0aGUgc3RhdGlzdGljYWwgc2lnbmlmaWNhbmNlIG9mIHRoZSBwYXRod2F5LWxldmVsIHN0YXRpc3RpYwoKIyMjIyBPdGhlciByZXNvdXJjZXMKCiogRm9yIGFub3RoZXIgZXhhbXBsZSB1c2luZyBgY2x1c3RlclByb2ZpbGVyYCBmb3IgR1NFQSwgc2VlIFtfSW50cm8gdG8gREdFOiBGdW5jdGlvbmFsIEFuYWx5c2lzLl8gZnJvbSBIYXJ2YXJkIENoYW4gQmlvaW5mb3JtYXRpY3MgQ29yZSBUcmFpbmluZy5dKGh0dHBzOi8vaGJjdHJhaW5pbmcuZ2l0aHViLmlvL0RHRV93b3Jrc2hvcC9sZXNzb25zLzA5X2Z1bmN0aW9uYWxfYW5hbHlzaXMuaHRtbCkKKiBUaGUgd2F5IHdlJ2xsIHVzZSBgY2x1c3RlclByb2ZpbGVyYCBoZXJlIHVzZXMgYGZnc2VhYCAoRmFzdCBHZW5lIFNldCBFbnJpY2htZW50IEFuYWx5c2lzKSB1bmRlciB0aGUgaG9vZC4KWW91IGNhbiByZWFkIG1vcmUgYWJvdXQgZmdzZWEgaW4gW0tvcm90a2V2aWNoIF9ldCBhbC5fICgyMDIxKV0oaHR0cHM6Ly9kb2kub3JnLzEwLjExMDEvMDYwMDEyKS4KKiBbcmVmaW5lLmJpbyBleGFtcGxlcyBHZW5lIHNldCBlbnJpY2htZW50IGFuYWx5c2lzIC0gUk5BLXNlcV0oaHR0cHM6Ly9hbGV4c2xlbW9uYWRlLmdpdGh1Yi5pby9yZWZpbmViaW8tZXhhbXBsZXMvMDMtcm5hc2VxL3BhdGh3YXktYW5hbHlzaXNfcm5hc2VxXzAyX2dzZWEuaHRtbCkgZnJvbSB3aGljaCB0aGlzIG1hdGVyaWFsIGhhcyBiZWVuIGFkYXB0ZWQuCgojIyBTZXQgdXAKCiMjIyBMaWJyYXJpZXMKCmBgYHtyIGxpYnJhcmllc30KIyBQYWNrYWdlIHRvIHJ1biBHU0VBCmxpYnJhcnkoY2x1c3RlclByb2ZpbGVyKQojIFBhY2thZ2UgdGhhdCBjb250YWlucyB0aGUgTVNpZ0RCIGdlbmUgc2V0cyBpbiB0aWR5IGZvcm1hdApsaWJyYXJ5KG1zaWdkYnIpCmBgYAoKIyMjIERpcmVjdG9yaWVzIGFuZCBGaWxlcwoKIyMjIyBEaXJlY3RvcmllcwoKYGBge3IgY3JlYXRlX2RpciwgbGl2ZSA9IFRSVUV9CiMgV2UnbGwgdXNlIHRoZSBtYXJrZXIgZ2VuZXMgYXMgR1NFQSBpbnB1dApybXNfYW5hbHlzaXNfZGlyIDwtIGZpbGUucGF0aCgiYW5hbHlzaXMiLCAicm1zIikKCiMgV2UnbGwgY3JlYXRlIGEgZGlyZWN0b3J5IHRvIHNwZWNpZmljYWxseSBob2xkIHRoZSBwYXRod2F5IHJlc3VsdHMgaWYgaXQgZG9lc24ndAojIGV4aXN0IHlldApyZXN1bHRzX2RpciA8LSBmaWxlLnBhdGgocm1zX2FuYWx5c2lzX2RpciwgInBhdGh3YXktYW5hbHlzaXMiKQpmczo6ZGlyX2NyZWF0ZShyZXN1bHRzX2RpcikKYGBgCgojIyMjIElucHV0IGZpbGVzCgoKYGBge3IgaW5wdXRfZmlsZXN9CmlucHV0X2ZpbGUgPC0gZmlsZS5wYXRoKHJtc19hbmFseXNpc19kaXIsCiAgICAgICAgICAgICAgICAgICAgICAgICJkZXNlcSIsCiAgICAgICAgICAgICAgICAgICAgICAgICJybXNfbXlvYmxhc3RfZGVzZXFfcmVzdWx0cy50c3YiKQpgYGAKCiMjIyMgT3V0cHV0IGZpbGVzCgpXZSdsbCBzYXZlIG91ciB0YWJsZSBvZiBHU0VBIHJlc3VsdHMgYXMgYSBUU1YuCgpgYGB7ciBvdXRwdXRfZmlsZXN9Cm91dHB1dF9maWxlIDwtIGZpbGUucGF0aChyZXN1bHRzX2RpciwKICAgICAgICAgICAgICAgICAgICAgICAgICJybXNfbXlvYmxhc3RfZ3NlYV9yZXN1bHRzLnRzdiIpCmBgYAoKIyMgR2VuZSBzZXRzCgpXZSB3aWxsIHVzZSBnZW5lIHNldHMgZnJvbSB0aGUgW01vbGVjdWxhciBTaWduYXR1cmVzIERhdGFiYXNlIChNU2lnREIpXShodHRwczovL3d3dy5nc2VhLW1zaWdkYi5vcmcvZ3NlYS9tc2lnZGIvaW5kZXguanNwKSBmcm9tIHRoZSBCcm9hZCBJbnN0aXR1dGUgKFtTdWJyYW1hbmlhbiwgVGFtYXlvICpldCBhbC4qIDIwMDVdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMDczL3BuYXMuMDUwNjU4MDEwMikpLgpUaGUgW2Btc2lnZGJyYF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL21zaWdkYnIvaW5kZXguaHRtbCkgcGFja2FnZSBjb250YWlucyBNU2lnREIgZGF0YXNldHMgYWxyZWFkeSBpbiB0aGUgdGlkeSBmb3JtYXQgcmVxdWlyZWQgYnkgYGNsdXN0ZXJQcm9maWxlcmAgYW5kIHN1cHBvcnRzIG11bHRpcGxlIG9yZ2FuaXNtcy4KCkxldCdzIHRha2UgYSBsb29rIGF0IHdoYXQgb3JnYW5pc21zIHRoZSBwYWNrYWdlIHN1cHBvcnRzLgoKYGBge3Igc2hvd19zcGVjaWVzfQptc2lnZGJyX3NwZWNpZXMoKQpgYGAKCk1TaWdEQiBjb250YWlucyA4IGRpZmZlcmVudCBnZW5lIHNldCBjb2xsZWN0aW9ucy4KCiAgICBIOiBoYWxsbWFyayBnZW5lIHNldHMKICAgIEMxOiBwb3NpdGlvbmFsIGdlbmUgc2V0cwogICAgQzI6IGN1cmF0ZWQgZ2VuZSBzZXRzCiAgICBDMzogbW90aWYgZ2VuZSBzZXRzCiAgICBDNDogY29tcHV0YXRpb25hbCBnZW5lIHNldHMKICAgIEM1OiBHTyBnZW5lIHNldHMKICAgIEM2OiBvbmNvZ2VuaWMgc2lnbmF0dXJlcwogICAgQzc6IGltbXVub2xvZ2ljIHNpZ25hdHVyZXMKCldlJ2xsIHVzZSB0aGUgSGFsbG1hcmsgY29sbGVjdGlvbiBmb3IgR1NFQS4KSGVyZSdzIGFuIGV4Y2VycHQgb2YgdGhlIFtjb2xsZWN0aW9uIGRlc2NyaXB0aW9uXShodHRwczovL3d3dy5nc2VhLW1zaWdkYi5vcmcvZ3NlYS9tc2lnZGIvY29sbGVjdGlvbl9kZXRhaWxzLmpzcCNIKToKCj4gSGFsbG1hcmsgZ2VuZSBzZXRzIHN1bW1hcml6ZSBhbmQgcmVwcmVzZW50IHNwZWNpZmljIHdlbGwtZGVmaW5lZCBiaW9sb2dpY2FsIHN0YXRlcyBvciBwcm9jZXNzZXMgYW5kIGRpc3BsYXkgY29oZXJlbnQgZXhwcmVzc2lvbi4gVGhlc2UgZ2VuZSBzZXRzIHdlcmUgZ2VuZXJhdGVkIGJ5IGEgY29tcHV0YXRpb25hbCBtZXRob2RvbG9neSBiYXNlZCBvbiBpZGVudGlmeWluZyBnZW5lIHNldCBvdmVybGFwcyBhbmQgcmV0YWluaW5nIGdlbmVzIHRoYXQgZGlzcGxheSBjb29yZGluYXRlIGV4cHJlc3Npb24uIFRoZSBoYWxsbWFya3MgcmVkdWNlIG5vaXNlIGFuZCByZWR1bmRhbmN5IGFuZCBwcm92aWRlIGEgYmV0dGVyIGRlbGluZWF0ZWQgYmlvbG9naWNhbCBzcGFjZSBmb3IgR1NFQS4KCk5vdGFibHksIHRoZXJlIGFyZSBvbmx5IDUwIGdlbmUgc2V0cyBpbmNsdWRlZCBpbiB0aGlzIGNvbGxlY3Rpb24uClRoZSBmZXdlciBnZW5lIHNldHMgd2UgdGVzdCwgdGhlIGxvd2VyIG91ciBtdWx0aXBsZSBoeXBvdGhlc2lzIHRlc3RpbmcgYnVyZGVuLgoKV2UgY2FuIHJldHJpZXZlIG9ubHkgdGhlIEhhbGxtYXJrIGdlbmUgc2V0cyBieSBzcGVjaWZ5aW5nIGBjYXRlZ29yeSA9ICJIImAgdG8gdGhlIGBtc2lnZGJyKClgIGZ1bmN0aW9uLgoKYGBge3IgaW1tdW5vbG9naWNfc2V0cywgbGl2ZSA9IFRSVUV9CmhzX2hhbGxtYXJrc19kZiA8LSBtc2lnZGJyKHNwZWNpZXMgPSAiSG9tbyBzYXBpZW5zIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgY2F0ZWdvcnkgPSAiSCIpCmBgYAoKIyMgR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcwoKX0FkYXB0ZWQgZnJvbSBbcmVmaW5lLmJpbyBleGFtcGxlc10oaHR0cHM6Ly9naXRodWIuY29tL0FsZXhzTGVtb25hZGUvcmVmaW5lYmlvLWV4YW1wbGVzL2Jsb2IvMzNjZGVmZjY2ZDU3ZjlmZThlZTRmY2I1MTU2YWVhNGFjMmRjZTA3Zi8wMy1ybmFzZXEvcGF0aHdheS1hbmFseXNpc19ybmFzZXFfMDJfZ3NlYS5SbWQpXwoKIVtdKGRpYWdyYW1zL3N1YnJhbWFuaWFuX2ZpZzEuanBnKQoKKipGaWd1cmUgMS4gW1N1YnJhbWFuaWFuIF9ldCBhbC5fICgyMDA1KV0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwNzMvcG5hcy4wNTA2NTgwMTAyKS4qKgoKR1NFQSBjYWxjdWxhdGVzIGEgcGF0aHdheS1sZXZlbCBtZXRyaWMsIGNhbGxlZCBhbiBlbnJpY2htZW50IHNjb3JlIChzb21ldGltZXMgYWJicmV2aWF0ZWQgYXMgRVMpLCBieSByYW5raW5nIGdlbmVzIGJ5IGEgZ2VuZS1sZXZlbCBzdGF0aXN0aWMuClRoaXMgc2NvcmUgcmVmbGVjdHMgd2hldGhlciBvciBub3QgYSBnZW5lIHNldCBvciBwYXRod2F5IGlzIG92ZXItcmVwcmVzZW50ZWQgYXQgdGhlIHRvcCBvciBib3R0b20gb2YgdGhlIGdlbmUgcmFua2luZ3MgKFtTdWJyYW1hbmlhbiBfZXQgYWwuXyAyMDA1XShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpOyBbWXVdKGh0dHA6Ly95dWxhYi1zbXUudG9wL2NsdXN0ZXJQcm9maWxlci1ib29rL2NoYXB0ZXIyLmh0bWwjZ2VuZS1zZXQtZW5yaWNobWVudC1hbmFseXNpcykpCgpTcGVjaWZpY2FsbHksIGFsbCBnZW5lcyBhcmUgcmFua2VkIGZyb20gbW9zdCBwb3NpdGl2ZSB0byBtb3N0IG5lZ2F0aXZlIGJhc2VkIG9uIHRoZWlyIHN0YXRpc3RpYyBhbmQgYSBydW5uaW5nIHN1bSBpcyBjYWxjdWxhdGVkOgpTdGFydGluZyB3aXRoIHRoZSBtb3N0IGhpZ2hseSByYW5rZWQgZ2VuZXMsIHRoZSBydW5uaW5nIHN1bSBpbmNyZWFzZXMgZm9yIGVhY2ggZ2VuZSBpbiB0aGUgcGF0aHdheSBhbmQgZGVjcmVhc2VzIGZvciBlYWNoIGdlbmUgbm90IGluIHRoZSBwYXRod2F5LgpUaGUgZW5yaWNobWVudCBzY29yZSBmb3IgYSBwYXRod2F5IGlzIHRoZSBydW5uaW5nIHN1bSdzIG1heGltdW0gZGV2aWF0aW9uIGZyb20gemVyby4KR1NFQSBhbHNvIGFzc2Vzc2VzIHN0YXRpc3RpY2FsIHNpZ25pZmljYW5jZSBvZiB0aGUgc2NvcmVzIGZvciBlYWNoIHBhdGh3YXkgdGhyb3VnaCBwZXJtdXRhdGlvbiB0ZXN0aW5nLgpBcyBhIHJlc3VsdCwgZWFjaCBpbnB1dCBwYXRod2F5IHdpbGwgaGF2ZSBhIHAtdmFsdWUgYXNzb2NpYXRlZCB3aXRoIGl0IHRoYXQgaXMgdGhlbiBjb3JyZWN0ZWQgZm9yIG11bHRpcGxlIGh5cG90aGVzaXMgdGVzdGluZyAoW1N1YnJhbWFuaWFuIF9ldCBhbC5fIDIwMDVdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMDczL3BuYXMuMDUwNjU4MDEwMik7IFtZdV0oaHR0cDovL3l1bGFiLXNtdS50b3AvY2x1c3RlclByb2ZpbGVyLWJvb2svY2hhcHRlcjIuaHRtbCNnZW5lLXNldC1lbnJpY2htZW50LWFuYWx5c2lzKSkuCgpUaGUgaW1wbGVtZW50YXRpb24gb2YgR1NFQSB3ZSB1c2UgaW4gaGVyZSBleGFtcGxlcyByZXF1aXJlcyBhIGdlbmUgbGlzdCBvcmRlcmVkIGJ5IHNvbWUgc3RhdGlzdGljIGFuZCBpbnB1dCBnZW5lIHNldHMuCldoZW4geW91IHVzZSBwcmV2aW91c2x5IGNvbXB1dGVkIGdlbmUtbGV2ZWwgc3RhdGlzdGljcyB3aXRoIEdTRUEsIGl0IGlzIGNhbGxlZCBHU0VBIHByZS1yYW5rZWQuCgojIyBERVNlcTIgcmVzdWx0cwoKYGBge3IgcmVhZF9pbl9tYXJrZXJzLCBsaXZlID0gVFJVRX0KZGVzZXFfZGYgPC0gcmVhZHI6OnJlYWRfdHN2KGlucHV0X2ZpbGUpCmBgYAoKYGBge3IgZGVzZXFfaGVhZH0KaGVhZChkZXNlcV9kZikKYGBgCgpUaGlzIGRhdGEgZnJhbWUgdXNlcyBFbnNlbWJsIGdlbmUgaWRlbnRpZmllcnMuCldlJ2xsIG5lZWQgdG8gbWFrZSBzdXJlIG91ciBnZW5lIHNldHMgdXNlIHRoZSBzYW1lIGlkZW50aWZpZXJzLgpMZXQncyB0YWtlIGEgbG9vayBhdCB0aGUgZmlyc3QgZmV3IHJvd3Mgb2YgdGhlIGRhdGEgZnJhbWUgdGhhdCBjb250YWlucyB0aGUgaGFsbG1hcmsgZ2VuZSBzZXRzLgoKYGBge3IgaGFsbG1hcmtfaGVhZCwgbGl2ZSA9IFRSVUV9CmhlYWQoaHNfaGFsbG1hcmtzX2RmKQpgYGAKCldlIGNhbiBzZWUgdGhhdCB0aGUgZ2VuZSBzZXRzIGZyb20gYG1zaWdkYnJgIGhhdmUgRW5zZW1ibCBnZW5lIGlkZW50aWZpZXJzIGFzc29jaWF0ZWQgd2l0aCB0aGVtLCBzbyB3ZSBkb24ndCBuZWVkIHRvIGRvIGFueSBjb252ZXJzaW9uLgpIb3dldmVyLCB3ZSdsbCBuZWVkIHRvIHBhc3MgdGhlIGNvcnJlY3QgY29sdW1uIHRvIHRoZSBmdW5jdGlvbiB0aGF0IHJ1bnMgR1NFQS4KCk9uZSBnb29kIHRoaW5nIGFib3V0IEVuc2VtYmwgZ2VuZSBpZGVudGlmaWVycyBpcyB0aGF0IHRoZXkgYXJlIGxlc3MgbGlrZWx5IHRvIGJlIGR1cGxpY2F0ZWQgdGhhbiwgZm9yIGV4YW1wbGUsIGdlbmUgc3ltYm9scy4KKE11bHRpcGxlIEVuc2VtYmwgZ2VuZSBpZGVudGlmaWVycyBjYW4gbWFwIHRvIHRoZSBzYW1lIHN5bWJvbC4pCgpUaGUgR1NFQSBhcHByb2FjaCByZXF1aXJlcyBvbiBkaXNjcmltaW5hdGluZyBiZXR3ZWVuIGdlbmVzIHRoYXQgYXJlIGluIGEgZ2VuZSBzZXQgYW5kIHRob3NlIHRoYXQgYXJlIG5vdC4KUHJhY3RpY2FsbHkgc3BlYWtpbmcsIGdlbmUgc2V0cyBhcmUganVzdCBjb2xsZWN0aW9ucyBvZiBnZW5lIGlkZW50aWZpZXJzIQpXaGVuIHRoZSBmdW5jdGlvbiB3ZSB1c2UgZm9yIEdTRUEgcHJlLXJhbmtlZCBnZXRzIGEgbGlzdCB3aXRoIGR1cGxpY2F0ZWQgZ2VuZSBpZGVudGlmaWVycywgaXQgY2FuIHByb2R1Y2UgdW5leHBlY3RlZCByZXN1bHRzLgpTbywgbGV0J3MgY2hlY2sgZm9yIGR1cGxpY2F0ZXMgaW4gdGhlIGRhdGEgZnJhbWUgb2YgREVTZXEyIHJlc3VsdHMuCgpgYGB7ciBjaGVja19kdXBsaWNhdGVzLCBsaXZlID0gVFJVRX0KYW55KGR1cGxpY2F0ZWQoZGVzZXFfZGYkZW5zZW1ibF9pZCkpCmBgYAoKVGhlcmUgYXJlIG5vIGR1cGxpY2F0ZXMgZm9yIHVzIHRvIHdvcnJ5IGFib3V0IQoKIyMjIFByZS1yYW5rZWQgbGlzdAoKVGhlIGBHU0VBKClgIGZ1bmN0aW9uIHRha2VzIGEgcHJlLXJhbmtlZCAoc29ydGVkKSBuYW1lZCB2ZWN0b3Igb2Ygc3RhdGlzdGljcywgd2hlcmUgdGhlIG5hbWVzIGluIHRoZSB2ZWN0b3IgYXJlIGdlbmUgaWRlbnRpZmllcnMuClRoaXMgaXMgc3RlcCAxIC0tIGdlbmUtbGV2ZWwgc3RhdGlzdGljcy4KCmBgYHtyIGxmY192ZWN0b3J9CmxmY192ZWN0b3IgPC0gZGVzZXFfZGYgfD4KICAjIEV4dHJhY3QgYSB2ZWN0b3Igb2YgYGxvZzJGb2xkQ2hhbmdlYCBuYW1lZCBieSBgZW5zZW1ibF9pZGAKICBkcGx5cjo6cHVsbChsb2cyRm9sZENoYW5nZSwgbmFtZSA9IGVuc2VtYmxfaWQpCmxmY192ZWN0b3IgPC0gc29ydChsZmNfdmVjdG9yLCBkZWNyZWFzaW5nID0gVFJVRSkKYGBgCgpMZXQncyBsb29rIGF0IHRoZSB0b3AgcmFua2VkIHZhbHVlcy4KCmBgYHtyIGhlYWRfbGZjLCBsaXZlID0gVFJVRX0KIyBMb29rIGF0IGZpcnN0IGVudHJpZXMgb2YgdGhlIGxvZyBmb2xkIGNoYW5nZSB2ZWN0b3IKaGVhZChsZmNfdmVjdG9yKQpgYGAKCkFuZCB0aGUgYm90dG9tIG9mIHRoZSBsaXN0LgoKYGBge3IgdGFpbF9sZmMsIGxpdmUgPSBUUlVFfQojIExvb2sgYXQgdGhlIGxhc3QgZW50cmllcyBvZiB0aGUgbG9nIGZvbGQgY2hhbmdlIHZlY3Rvcgp0YWlsKGxmY192ZWN0b3IpCmBgYAoKIyMgUnVuIEdTRUEKCk5vdyBmb3IgdGhlIGFuYWx5c2lzIQoKV2UgY2FuIHVzZSB0aGUgYEdTRUEoKWAgZnVuY3Rpb24gdG8gcGVyZm9ybSBHU0VBIHdpdGggYW55IGdlbmVyaWMgc2V0IG9mIGdlbmUgc2V0cywgYnV0IHRoZXJlIGFyZSBzZXZlcmFsIGZ1bmN0aW9ucyBmb3IgdXNpbmcgc3BlY2lmaWMsIGNvbW1vbmx5IHVzZWQgZ2VuZSBzZXRzIChlLmcuLCBgZ3NlS0VHRygpYCkuCgpgYGB7ciBydW5fZ3NlYX0KZ3NlYV9yZXN1bHRzIDwtIEdTRUEoZ2VuZUxpc3QgPSBsZmNfdmVjdG9yLCAgIyBvcmRlcmVkIHJhbmtlZCBnZW5lIGxpc3QKICAgICAgICAgICAgICAgICAgICAgbWluR1NTaXplID0gMjUsICAjIG1pbmltdW0gZ2VuZSBzZXQgc2l6ZQogICAgICAgICAgICAgICAgICAgICBtYXhHU1NpemUgPSA1MDAsICAjIG1heGltdW0gZ2VuZSBzZXQgc2V0CiAgICAgICAgICAgICAgICAgICAgIHB2YWx1ZUN1dG9mZiA9IDAuMDUsCiAgICAgICAgICAgICAgICAgICAgIHBBZGp1c3RNZXRob2QgPSAiQkgiLCAgIyBjb3JyZWN0aW9uIGZvciBtdWx0aXBsZSBoeXBvdGhlc2lzIHRlc3RpbmcKICAgICAgICAgICAgICAgICAgICAgVEVSTTJHRU5FID0gZHBseXI6OnNlbGVjdChoc19oYWxsbWFya3NfZGYsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ3NfbmFtZSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBlbnNlbWJsX2dlbmUpKSAgIyBwYXNzIHRoZSBjb3JyZWN0IGlkZW50aWZpZXIgY29sdW1uCmBgYApMZXQncyB0YWtlIGEgbG9vayBhdCB0aGUgR1NFQSByZXN1bHRzLgoKYGBge3Igdmlld19nc2VhLCBsaXZlID0gVFJVRSwgZXZhbCA9IEZBTFNFfQpWaWV3KGdzZWFfcmVzdWx0c0ByZXN1bHQgfD4KICAgICAgIGRwbHlyOjphcnJhbmdlKGRwbHlyOjpkZXNjKE5FUykpCikKYGBgCgpOb3JtYWxpemVkIGVucmljaG1lbnQgc2NvcmVzIChORVMpIGFyZSBlbnJpY2htZW50IHNjb3JlcyB0aGF0IGFyZSBzY2FsZWQgdG8gbWFrZSBnZW5lIHNldHMgdGhhdCBjb250YWluIGRpZmZlcmVudCBudW1iZXIgb2YgZ2VuZXMgY29tcGFyYWJsZS4KClBhdGh3YXlzIHdpdGggc2lnbmlmaWNhbnQsIGhpZ2hseSBwb3NpdGl2ZSBORVMgYXJlIGVucmljaGVkIGluIEVSTVMgbXlvYmxhc3RzLCB3aGVyZWFzIHBhdGh3YXlzIHdpdGggc2lnbmlmaWNhbnQsIGhpZ2hseSBuZWdhdGl2ZSBORVMgYXJlIGVucmljaGVkIGluIEFSTVMgbXlvYmxhc3RzLgoKTGV0J3Mgd3JpdGUgdGhlc2UgcmVzdWx0cyB0byBmaWxlLgoKYGBge3Igd3JpdGVfZ3NlYX0KZ3NlYV9yZXN1bHRzQHJlc3VsdCB8PiByZWFkcjo6d3JpdGVfdHN2KG91dHB1dF9maWxlKQpgYGAKCiMjIyBWaXN1YWxpemluZyBHU0VBIHJlc3VsdHMKCldlIGNhbiB2aXN1YWxpemUgR1NFQSByZXN1bHRzIGZvciBpbmRpdmlkdWFsIHBhdGh3YXlzIG9yIGdlbmUgc2V0cyB1c2luZyBgZW5yaWNocGxvdDo6Z3NlYXBsb3QoKWAuCkxldCdzIHRha2UgYSBsb29rIGF0IDMgZGlmZmVyZW50IHBhdGh3YXlzIC0tIG9uZSB3aXRoIGEgaGlnaGx5IHBvc2l0aXZlIE5FUywgb25lIHdpdGggYSBoaWdobHkgbmVnYXRpdmUgTkVTLCBhbmQgb25lIHRoYXQgd2FzIG5vdCBhIHNpZ25pZmljYW50IHJlc3VsdCAtLSB0byBnZXQgbW9yZSBpbnNpZ2h0IGludG8gaG93IEVTIGFyZSBjYWxjdWxhdGVkLgoKIyMjIyBIaWdobHkgUG9zaXRpdmUgTkVTCgpMZXQncyB0YWtlIGxvb2sgYXQgYSBwYXRod2F5IHdpdGggYSBoaWdobHkgcG9zaXRpdmUgTkVTIChgSEFMTE1BUktfTVlDX1RBUkdFVFNfVjJgKSB1c2luZyBhIEdTRUEgcGxvdC4KCmBgYHtyIGhpZ2hseV9wb3N9CmVucmljaHBsb3Q6OmdzZWFwbG90KGdzZWFfcmVzdWx0cywKICAgICAgICAgICAgICAgICAgICAgZ2VuZVNldElEID0gIkhBTExNQVJLX01ZQ19UQVJHRVRTX1YyIiwKICAgICAgICAgICAgICAgICAgICAgdGl0bGUgPSAiSEFMTE1BUktfTVlDX1RBUkdFVFNfVjIiLAogICAgICAgICAgICAgICAgICAgICBjb2xvci5saW5lID0gIiMwMDY2RkYiKQpgYGAKCk5vdGljZSBob3cgdGhlIGdlbmVzIHRoYXQgYXJlIGluIHRoZSBnZW5lIHNldCwgaW5kaWNhdGVkIGJ5IHRoZSBibGFjayBiYXJzLCB0ZW5kIHRvIGJlIG9uIHRoZSBsZWZ0IHNpZGUgb2YgdGhlIGdyYXBoIGluZGljYXRpbmcgdGhhdCB0aGV5IGhhdmUgcG9zaXRpdmUgZ2VuZS1sZXZlbCBzY29yZXMuCgojIyMjIEhpZ2hseSBOZWdhdGl2ZSBORVMKClRoZSBnZW5lIHNldCBgSEFMTE1BUktfTVlPR0VORVNJU2AgaGFkIGEgaGlnaGx5IG5lZ2F0aXZlIE5FUy4KCmBgYHtyIGhpZ2hseV9uZWd9CmVucmljaHBsb3Q6OmdzZWFwbG90KGdzZWFfcmVzdWx0cywKICAgICAgICAgICAgICAgICAgICAgZ2VuZVNldElEID0gIkhBTExNQVJLX01ZT0dFTkVTSVMiLAogICAgICAgICAgICAgICAgICAgICB0aXRsZSA9ICJIQUxMTUFSS19NWU9HRU5FU0lTIiwKICAgICAgICAgICAgICAgICAgICAgY29sb3IubGluZSA9ICIjMDA2NkZGIikKYGBgCgpUaGlzIGdlbmUgc2V0IHNob3dzIHRoZSBvcHBvc2l0ZSBwYXR0ZXJuIC0tIGdlbmVzIGluIHRoZSBwYXRod2F5IHRlbmQgdG8gYmUgb24gdGhlIHJpZ2h0IHNpZGUgb2YgdGhlIGdyYXBoLgoKIyMjIyBBIG5vbi1zaWduaWZpY2FudCByZXN1bHQKClRoZSBgQHJlc3VsdHNgIHNsb3Qgd2lsbCBvbmx5IHNob3cgZ2VuZSBzZXRzIHRoYXQgcGFzcyB0aGUgYHB2YWx1ZUN1dG9mZmAgdGhyZXNob2xkIHdlIHN1cHBsaWVkIHRvIGBHU0VBKClgLCBidXQgd2UgY2FuIHBsb3QgYW55IGdlbmUgc2V0IHNvIGxvbmcgYXMgd2Uga25vdyBpdHMgbmFtZS4KTGV0J3MgbG9vayBhdCBgSEFMTE1BUktfUDUzX1BBVEhXQVlgLCB3aGljaCB3YXMgbm90IGluIHRoZSByZXN1bHRzIHdlIHZpZXdlZCBlYXJsaWVyLgoKYGBge3IgcDUzLCBsaXZlID0gVFJVRX0KZW5yaWNocGxvdDo6Z3NlYXBsb3QoZ3NlYV9yZXN1bHRzLAogICAgICAgICAgICAgICAgICAgICBnZW5lU2V0SUQgPSAiSEFMTE1BUktfUDUzX1BBVEhXQVkiLAogICAgICAgICAgICAgICAgICAgICB0aXRsZSA9ICJIQUxMTUFSS19QNTNfUEFUSFdBWSIsCiAgICAgICAgICAgICAgICAgICAgIGNvbG9yLmxpbmUgPSAiIzAwNjZGRiIpCmBgYAoKR2VuZXMgaW4gdGhlIHBhdGh3YXkgYXJlIGRpc3RyaWJ1dGVkIG1vcmUgZXZlbmx5IHRocm91Z2hvdXQgdGhlIHJhbmtlZCBsaXN0LCByZXN1bHRpbmcgaW4gYSBtb3JlICJtaWRkbGluZyIgc2NvcmUuCgoqTm90ZTogVGhlIHBsb3RzIHJldHVybmVkIGJ5IGBlbnJpY2hwbG90Ojpnc2VhcGxvdGAgYXJlIGdncGxvdHMsIHNvIHdlIGNvdWxkIHVzZSBgZ2dwbG90Mjo6Z2dzYXZlKClgIHRvIHNhdmUgdGhlbSB0byBmaWxlIGlmIHdlIHdhbnRlZCB0by4qCgojIyBTZXNzaW9uIEluZm8KCmBgYHtyIHNlc3Npb25faW5mb30Kc2Vzc2lvbkluZm8oKQpgYGAK
+
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcyAoR1NFQSkiCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjQKLS0tCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOgoKLSBQcmVwYXJlIHRhYnVsYXIgZGF0YSBvZiBnZW5lLWxldmVsIHN0YXRpc3RpY3MgZm9yIHVzZSB3aXRoIEdlbmUgU2V0IEVucmljaG1lbnQgQW5hbHlzaXMgKEdTRUEpCi0gQWNjZXNzIFtNb2xlY3VsYXIgU2lnbmF0dXJlcyBEYXRhYmFzZSBnZW5lIHNldCBjb2xsZWN0aW9uc10oaHR0cHM6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2NvbGxlY3Rpb25zLmpzcCkgdmlhIHRoZSBgbXNpZ2RicmAgcGFja2FnZQotIFBlcmZvcm0gR1NFQSB3aXRoIHRoZSBgY2x1c3RlclByb2ZpbGVyYCBwYWNrYWdlCi0gVmlzdWFsaXplIEdTRUEgcmVzdWx0cyB3aXRoIHRoZSBgZW5yaWNocGxvdGAgcGFja2FnZQoKLS0tCgpJbiB0aGlzIG5vdGVib29rLCB3ZSdsbCBhbmFseXplIHRoZSBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiByZXN1bHRzIGZyb20gdGhlIGxhc3Qgbm90ZWJvb2suCgpHU0VBIGlzIGEgZnVuY3Rpb25hbCBjbGFzcyBzY29yaW5nIChGQ1MpIGFwcHJvYWNoIHRvIHBhdGh3YXkgYW5hbHlzaXMgdGhhdCB3YXMgZmlyc3QgaW50cm9kdWNlZCBpbiBbU3VicmFtYW5pYW4gX2V0IGFsLl8gKDIwMDUpXShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpLgpUaGUgcmF0aW9uYWxlIGJlaGluZCBGQ1MgYXBwcm9hY2hlcyBpcyB0aGF0IHNtYWxsIGNoYW5nZXMgaW4gaW5kaXZpZHVhbCBnZW5lcyB0aGF0IHBhcnRpY2lwYXRlIGluIHRoZSBzYW1lIGJpb2xvZ2ljYWwgcHJvY2VzcyBvciBwYXRod2F5IGNhbiBiZSBzaWduaWZpY2FudCBhbmQgb2YgYmlvbG9naWNhbCBpbnRlcmVzdC4KClRoZXJlIGFyZSAzIGdlbmVyYWwgc3RlcHMgaW4gRkNTIG1ldGhvZHMgKFtLaGF0cmkgX2V0IGFsLl8gMjAxMl0oaHR0cHM6Ly9kb2kub3JnLzEwLjEzNzEvam91cm5hbC5wY2JpLjEwMDIzNzUpKToKCjEuIENhbGN1bGF0ZSBhIGdlbmUtbGV2ZWwgc3RhdGlzdGljIChoZXJlLCB3ZSdsbCB1c2UgdGhlIHN1bW1hcnkgbG9nIGZvbGQgY2hhbmdlcyBpbiBvdXIgREVTZXEyIHJlc3VsdHMpCjIuIEFnZ3JlZ2F0ZSBnZW5lLWxldmVsIHN0YXRpc3RpY3MgaW50byBhIHBhdGh3YXktbGV2ZWwgc3RhdGlzdGljCjMuIEFzc2VzcyB0aGUgc3RhdGlzdGljYWwgc2lnbmlmaWNhbmNlIG9mIHRoZSBwYXRod2F5LWxldmVsIHN0YXRpc3RpYwoKIyMjIyBPdGhlciByZXNvdXJjZXMKCiogRm9yIGFub3RoZXIgZXhhbXBsZSB1c2luZyBgY2x1c3RlclByb2ZpbGVyYCBmb3IgR1NFQSwgc2VlIFtfSW50cm8gdG8gREdFOiBGdW5jdGlvbmFsIEFuYWx5c2lzLl8gZnJvbSBIYXJ2YXJkIENoYW4gQmlvaW5mb3JtYXRpY3MgQ29yZSBUcmFpbmluZy5dKGh0dHBzOi8vaGJjdHJhaW5pbmcuZ2l0aHViLmlvL1RyYWluaW5nLW1vZHVsZXMvREdFLWZ1bmN0aW9uYWwtYW5hbHlzaXMvbGVzc29ucy8wMl9mdW5jdGlvbmFsX2FuYWx5c2lzLmh0bWwpCiogVGhlIHdheSB3ZSdsbCB1c2UgYGNsdXN0ZXJQcm9maWxlcmAgaGVyZSB1c2VzIGBmZ3NlYWAgKEZhc3QgR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcykgdW5kZXIgdGhlIGhvb2QuCllvdSBjYW4gcmVhZCBtb3JlIGFib3V0IGBmZ3NlYWAgaW4gW0tvcm90a2V2aWNoIF9ldCBhbC5fICgyMDIxKV0oaHR0cHM6Ly9kb2kub3JnLzEwLjExMDEvMDYwMDEyKS4KKiBTZWUgdGhlIFtyZWZpbmUuYmlvIGV4YW1wbGVzIGZvciAiR2VuZSBzZXQgZW5yaWNobWVudCBhbmFseXNpcyAtIFJOQS1zZXEiXShodHRwczovL2FsZXhzbGVtb25hZGUuZ2l0aHViLmlvL3JlZmluZWJpby1leGFtcGxlcy8wMy1ybmFzZXEvcGF0aHdheS1hbmFseXNpc19ybmFzZXFfMDJfZ3NlYS5odG1sKSBmcm9tIHdoaWNoIHRoaXMgbWF0ZXJpYWwgaGFzIGJlZW4gYWRhcHRlZC4KCiMjIFNldCB1cAoKIyMjIExpYnJhcmllcwoKYGBge3IgbGlicmFyaWVzfQojIFBhY2thZ2UgdG8gcnVuIEdTRUEKbGlicmFyeShjbHVzdGVyUHJvZmlsZXIpCiMgUGFja2FnZSB0aGF0IGNvbnRhaW5zIHRoZSBNU2lnREIgZ2VuZSBzZXRzIGluIHRpZHkgZm9ybWF0CmxpYnJhcnkobXNpZ2RicikKYGBgCgojIyMgRGlyZWN0b3JpZXMgYW5kIEZpbGVzCgojIyMjIERpcmVjdG9yaWVzCgpgYGB7ciBjcmVhdGVfZGlyLCBsaXZlID0gVFJVRX0KIyBXZSdsbCB1c2UgdGhlIG1hcmtlciBnZW5lcyBhcyBHU0VBIGlucHV0CnJtc19hbmFseXNpc19kaXIgPC0gZmlsZS5wYXRoKCJhbmFseXNpcyIsICJybXMiKQoKIyBXZSdsbCBjcmVhdGUgYSBkaXJlY3RvcnkgdG8gc3BlY2lmaWNhbGx5IGhvbGQgdGhlIHBhdGh3YXkgcmVzdWx0cyBpZiBpdCBkb2Vzbid0CiMgZXhpc3QgeWV0CnJlc3VsdHNfZGlyIDwtIGZpbGUucGF0aChybXNfYW5hbHlzaXNfZGlyLCAicGF0aHdheS1hbmFseXNpcyIpCmZzOjpkaXJfY3JlYXRlKHJlc3VsdHNfZGlyKQpgYGAKCiMjIyMgSW5wdXQgZmlsZXMKCgpgYGB7ciBpbnB1dF9maWxlc30KaW5wdXRfZmlsZSA8LSBmaWxlLnBhdGgocm1zX2FuYWx5c2lzX2RpciwKICAgICAgICAgICAgICAgICAgICAgICAgImRlc2VxIiwKICAgICAgICAgICAgICAgICAgICAgICAgInJtc19teW9ibGFzdF9kZXNlcV9yZXN1bHRzLnRzdiIpCmBgYAoKIyMjIyBPdXRwdXQgZmlsZXMKCldlJ2xsIHNhdmUgb3VyIHRhYmxlIG9mIEdTRUEgcmVzdWx0cyBhcyBhIFRTVi4KCmBgYHtyIG91dHB1dF9maWxlc30Kb3V0cHV0X2ZpbGUgPC0gZmlsZS5wYXRoKHJlc3VsdHNfZGlyLAogICAgICAgICAgICAgICAgICAgICAgICAgInJtc19teW9ibGFzdF9nc2VhX3Jlc3VsdHMudHN2IikKYGBgCgojIyBHZW5lIHNldHMKCldlIHdpbGwgdXNlIGdlbmUgc2V0cyBmcm9tIHRoZSBbTW9sZWN1bGFyIFNpZ25hdHVyZXMgRGF0YWJhc2UgKE1TaWdEQildKGh0dHBzOi8vd3d3LmdzZWEtbXNpZ2RiLm9yZy9nc2VhL21zaWdkYi9pbmRleC5qc3ApIGZyb20gdGhlIEJyb2FkIEluc3RpdHV0ZSAoW1N1YnJhbWFuaWFuLCBUYW1heW8gKmV0IGFsLiogMjAwNV0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwNzMvcG5hcy4wNTA2NTgwMTAyKSkuClRoZSBbYG1zaWdkYnJgXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvbXNpZ2Rici9pbmRleC5odG1sKSBwYWNrYWdlIGNvbnRhaW5zIE1TaWdEQiBkYXRhc2V0cyBhbHJlYWR5IGluIHRoZSB0aWR5IGZvcm1hdCByZXF1aXJlZCBieSBgY2x1c3RlclByb2ZpbGVyYCBhbmQgc3VwcG9ydHMgbXVsdGlwbGUgb3JnYW5pc21zLgoKTGV0J3MgdGFrZSBhIGxvb2sgYXQgd2hhdCBvcmdhbmlzbXMgdGhlIHBhY2thZ2Ugc3VwcG9ydHMuCgpgYGB7ciBzaG93X3NwZWNpZXN9Cm1zaWdkYnJfc3BlY2llcygpCmBgYAoKTVNpZ0RCIGNvbnRhaW5zIDggZGlmZmVyZW50IGdlbmUgc2V0IGNvbGxlY3Rpb25zLgoKICAgIEg6IGhhbGxtYXJrIGdlbmUgc2V0cwogICAgQzE6IHBvc2l0aW9uYWwgZ2VuZSBzZXRzCiAgICBDMjogY3VyYXRlZCBnZW5lIHNldHMKICAgIEMzOiBtb3RpZiBnZW5lIHNldHMKICAgIEM0OiBjb21wdXRhdGlvbmFsIGdlbmUgc2V0cwogICAgQzU6IEdPIGdlbmUgc2V0cwogICAgQzY6IG9uY29nZW5pYyBzaWduYXR1cmVzCiAgICBDNzogaW1tdW5vbG9naWMgc2lnbmF0dXJlcwoKV2UnbGwgdXNlIHRoZSBIYWxsbWFyayBjb2xsZWN0aW9uIGZvciBHU0VBLgpIZXJlJ3MgYW4gZXhjZXJwdCBvZiB0aGUgW2NvbGxlY3Rpb24gZGVzY3JpcHRpb25dKGh0dHBzOi8vd3d3LmdzZWEtbXNpZ2RiLm9yZy9nc2VhL21zaWdkYi9jb2xsZWN0aW9uX2RldGFpbHMuanNwI0gpOgoKPiBIYWxsbWFyayBnZW5lIHNldHMgc3VtbWFyaXplIGFuZCByZXByZXNlbnQgc3BlY2lmaWMgd2VsbC1kZWZpbmVkIGJpb2xvZ2ljYWwgc3RhdGVzIG9yIHByb2Nlc3NlcyBhbmQgZGlzcGxheSBjb2hlcmVudCBleHByZXNzaW9uLiBUaGVzZSBnZW5lIHNldHMgd2VyZSBnZW5lcmF0ZWQgYnkgYSBjb21wdXRhdGlvbmFsIG1ldGhvZG9sb2d5IGJhc2VkIG9uIGlkZW50aWZ5aW5nIGdlbmUgc2V0IG92ZXJsYXBzIGFuZCByZXRhaW5pbmcgZ2VuZXMgdGhhdCBkaXNwbGF5IGNvb3JkaW5hdGUgZXhwcmVzc2lvbi4gVGhlIGhhbGxtYXJrcyByZWR1Y2Ugbm9pc2UgYW5kIHJlZHVuZGFuY3kgYW5kIHByb3ZpZGUgYSBiZXR0ZXIgZGVsaW5lYXRlZCBiaW9sb2dpY2FsIHNwYWNlIGZvciBHU0VBLgoKTm90YWJseSwgdGhlcmUgYXJlIG9ubHkgNTAgZ2VuZSBzZXRzIGluY2x1ZGVkIGluIHRoaXMgY29sbGVjdGlvbi4KVGhlIGZld2VyIGdlbmUgc2V0cyB3ZSB0ZXN0LCB0aGUgbG93ZXIgb3VyIG11bHRpcGxlIGh5cG90aGVzaXMgdGVzdGluZyBidXJkZW4uCgpXZSBjYW4gcmV0cmlldmUgb25seSB0aGUgSGFsbG1hcmsgZ2VuZSBzZXRzIGJ5IHNwZWNpZnlpbmcgYGNhdGVnb3J5ID0gIkgiYCB0byB0aGUgYG1zaWdkYnIoKWAgZnVuY3Rpb24uCgpgYGB7ciBpbW11bm9sb2dpY19zZXRzLCBsaXZlID0gVFJVRX0KaHNfaGFsbG1hcmtzX2RmIDwtIG1zaWdkYnIoc3BlY2llcyA9ICJIb21vIHNhcGllbnMiLAogICAgICAgICAgICAgICAgICAgICAgICAgICBjYXRlZ29yeSA9ICJIIikKYGBgCgojIyBHZW5lIFNldCBFbnJpY2htZW50IEFuYWx5c2lzCgpfQWRhcHRlZCBmcm9tIFtyZWZpbmUuYmlvIGV4YW1wbGVzXShodHRwczovL2dpdGh1Yi5jb20vQWxleHNMZW1vbmFkZS9yZWZpbmViaW8tZXhhbXBsZXMvYmxvYi8zM2NkZWZmNjZkNTdmOWZlOGVlNGZjYjUxNTZhZWE0YWMyZGNlMDdmLzAzLXJuYXNlcS9wYXRod2F5LWFuYWx5c2lzX3JuYXNlcV8wMl9nc2VhLlJtZClfCgohW10oZGlhZ3JhbXMvc3VicmFtYW5pYW5fZmlnMS5qcGcpCgoqKkZpZ3VyZSAxLiBbU3VicmFtYW5pYW4gX2V0IGFsLl8gKDIwMDUpXShodHRwczovL2RvaS5vcmcvMTAuMTA3My9wbmFzLjA1MDY1ODAxMDIpLioqCgpHU0VBIGNhbGN1bGF0ZXMgYSBwYXRod2F5LWxldmVsIG1ldHJpYywgY2FsbGVkIGFuIGVucmljaG1lbnQgc2NvcmUgKHNvbWV0aW1lcyBhYmJyZXZpYXRlZCBhcyBFUyksIGJ5IHJhbmtpbmcgZ2VuZXMgYnkgYSBnZW5lLWxldmVsIHN0YXRpc3RpYy4KVGhpcyBzY29yZSByZWZsZWN0cyB3aGV0aGVyIG9yIG5vdCBhIGdlbmUgc2V0IG9yIHBhdGh3YXkgaXMgb3Zlci1yZXByZXNlbnRlZCBhdCB0aGUgdG9wIG9yIGJvdHRvbSBvZiB0aGUgZ2VuZSByYW5raW5ncyAoW1N1YnJhbWFuaWFuIF9ldCBhbC5fIDIwMDVdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMDczL3BuYXMuMDUwNjU4MDEwMik7IFtZdV0oaHR0cDovL3l1bGFiLXNtdS50b3AvY2x1c3RlclByb2ZpbGVyLWJvb2svY2hhcHRlcjIuaHRtbCNnZW5lLXNldC1lbnJpY2htZW50LWFuYWx5c2lzKSkKClNwZWNpZmljYWxseSwgYWxsIGdlbmVzIGFyZSByYW5rZWQgZnJvbSBtb3N0IHBvc2l0aXZlIHRvIG1vc3QgbmVnYXRpdmUgYmFzZWQgb24gdGhlaXIgc3RhdGlzdGljIGFuZCBhIHJ1bm5pbmcgc3VtIGlzIGNhbGN1bGF0ZWQ6ClN0YXJ0aW5nIHdpdGggdGhlIG1vc3QgaGlnaGx5IHJhbmtlZCBnZW5lcywgdGhlIHJ1bm5pbmcgc3VtIGluY3JlYXNlcyBmb3IgZWFjaCBnZW5lIGluIHRoZSBwYXRod2F5IGFuZCBkZWNyZWFzZXMgZm9yIGVhY2ggZ2VuZSBub3QgaW4gdGhlIHBhdGh3YXkuClRoZSBlbnJpY2htZW50IHNjb3JlIGZvciBhIHBhdGh3YXkgaXMgdGhlIHJ1bm5pbmcgc3VtJ3MgbWF4aW11bSBkZXZpYXRpb24gZnJvbSB6ZXJvLgpHU0VBIGFsc28gYXNzZXNzZXMgc3RhdGlzdGljYWwgc2lnbmlmaWNhbmNlIG9mIHRoZSBzY29yZXMgZm9yIGVhY2ggcGF0aHdheSB0aHJvdWdoIHBlcm11dGF0aW9uIHRlc3RpbmcuCkFzIGEgcmVzdWx0LCBlYWNoIGlucHV0IHBhdGh3YXkgd2lsbCBoYXZlIGEgcC12YWx1ZSBhc3NvY2lhdGVkIHdpdGggaXQgdGhhdCBpcyB0aGVuIGNvcnJlY3RlZCBmb3IgbXVsdGlwbGUgaHlwb3RoZXNpcyB0ZXN0aW5nIChbU3VicmFtYW5pYW4gX2V0IGFsLl8gMjAwNV0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwNzMvcG5hcy4wNTA2NTgwMTAyKTsgW1l1XShodHRwOi8veXVsYWItc211LnRvcC9jbHVzdGVyUHJvZmlsZXItYm9vay9jaGFwdGVyMi5odG1sI2dlbmUtc2V0LWVucmljaG1lbnQtYW5hbHlzaXMpKS4KClRoZSBpbXBsZW1lbnRhdGlvbiBvZiBHU0VBIHdlIHVzZSBpbiBoZXJlIGV4YW1wbGVzIHJlcXVpcmVzIGEgZ2VuZSBsaXN0IG9yZGVyZWQgYnkgc29tZSBzdGF0aXN0aWMgYW5kIGlucHV0IGdlbmUgc2V0cy4KV2hlbiB5b3UgdXNlIHByZXZpb3VzbHkgY29tcHV0ZWQgZ2VuZS1sZXZlbCBzdGF0aXN0aWNzIHdpdGggR1NFQSwgaXQgaXMgY2FsbGVkIEdTRUEgcHJlLXJhbmtlZC4KCiMjIERFU2VxMiByZXN1bHRzCgpgYGB7ciByZWFkX2luX21hcmtlcnMsIGxpdmUgPSBUUlVFfQpkZXNlcV9kZiA8LSByZWFkcjo6cmVhZF90c3YoaW5wdXRfZmlsZSkKYGBgCgpgYGB7ciBkZXNlcV9oZWFkfQpoZWFkKGRlc2VxX2RmKQpgYGAKClRoaXMgZGF0YSBmcmFtZSB1c2VzIEVuc2VtYmwgZ2VuZSBpZGVudGlmaWVycy4KV2UnbGwgbmVlZCB0byBtYWtlIHN1cmUgb3VyIGdlbmUgc2V0cyB1c2UgdGhlIHNhbWUgaWRlbnRpZmllcnMuCkxldCdzIHRha2UgYSBsb29rIGF0IHRoZSBmaXJzdCBmZXcgcm93cyBvZiB0aGUgZGF0YSBmcmFtZSB0aGF0IGNvbnRhaW5zIHRoZSBoYWxsbWFyayBnZW5lIHNldHMuCgpgYGB7ciBoYWxsbWFya19oZWFkLCBsaXZlID0gVFJVRX0KaGVhZChoc19oYWxsbWFya3NfZGYpCmBgYAoKV2UgY2FuIHNlZSB0aGF0IHRoZSBnZW5lIHNldHMgZnJvbSBgbXNpZ2RicmAgaGF2ZSBFbnNlbWJsIGdlbmUgaWRlbnRpZmllcnMgYXNzb2NpYXRlZCB3aXRoIHRoZW0sIHNvIHdlIGRvbid0IG5lZWQgdG8gZG8gYW55IGNvbnZlcnNpb24uCkhvd2V2ZXIsIHdlJ2xsIG5lZWQgdG8gcGFzcyB0aGUgY29ycmVjdCBjb2x1bW4gdG8gdGhlIGZ1bmN0aW9uIHRoYXQgcnVucyBHU0VBLgoKSWYgd2UgbmVlZGVkIHRvIGRvIGdlbmUgaWRlbnRpZmllciBjb252ZXJzaW9uLCB3ZSB3b3VsZCBsaWtlbHkgdXNlIHRoZSBgQW5ub3RhdGlvbkRiaWAgcGFja2FnZS4KWW91IGNhbiBzZWUgYW4gZXhhbXBsZSBpbiB0aGlzIEhhcnZhcmQgQ2hhbiBCaW9pbmZvcm1hdGljcyBDb3JlIFRyYWluaW5nIG1hdGVyaWFsOiA8aHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wX3NhbG1vbl9vbmxpbmUvbGVzc29ucy9Bbm5vdGF0aW9uRGJpX2xlc3Nvbi5odG1sPgoKT25lIGdvb2QgdGhpbmcgYWJvdXQgRW5zZW1ibCBnZW5lIGlkZW50aWZpZXJzIGlzIHRoYXQgdGhleSBhcmUgbGVzcyBsaWtlbHkgdG8gYmUgZHVwbGljYXRlZCB0aGFuLCBmb3IgZXhhbXBsZSwgZ2VuZSBzeW1ib2xzLgooTXVsdGlwbGUgRW5zZW1ibCBnZW5lIGlkZW50aWZpZXJzIGNhbiBtYXAgdG8gdGhlIHNhbWUgc3ltYm9sLikKClRoZSBHU0VBIGFwcHJvYWNoIHJlcXVpcmVzIG9uIGRpc2NyaW1pbmF0aW5nIGJldHdlZW4gZ2VuZXMgdGhhdCBhcmUgaW4gYSBnZW5lIHNldCBhbmQgdGhvc2UgdGhhdCBhcmUgbm90LgpQcmFjdGljYWxseSBzcGVha2luZywgZ2VuZSBzZXRzIGFyZSBqdXN0IGNvbGxlY3Rpb25zIG9mIGdlbmUgaWRlbnRpZmllcnMhCldoZW4gdGhlIGZ1bmN0aW9uIHdlIHVzZSBmb3IgR1NFQSBwcmUtcmFua2VkIGdldHMgYSBsaXN0IHdpdGggZHVwbGljYXRlZCBnZW5lIGlkZW50aWZpZXJzLCBpdCBjYW4gcHJvZHVjZSB1bmV4cGVjdGVkIHJlc3VsdHMuClNvLCBsZXQncyBjaGVjayBmb3IgZHVwbGljYXRlcyBpbiB0aGUgZGF0YSBmcmFtZSBvZiBERVNlcTIgcmVzdWx0cy4KCmBgYHtyIGNoZWNrX2R1cGxpY2F0ZXMsIGxpdmUgPSBUUlVFfQphbnkoZHVwbGljYXRlZChkZXNlcV9kZiRlbnNlbWJsX2lkKSkKYGBgCgpUaGVyZSBhcmUgbm8gZHVwbGljYXRlcyBmb3IgdXMgdG8gd29ycnkgYWJvdXQhCgojIyMgUHJlLXJhbmtlZCBsaXN0CgpUaGUgYEdTRUEoKWAgZnVuY3Rpb24gdGFrZXMgYSBwcmUtcmFua2VkIChzb3J0ZWQpIG5hbWVkIHZlY3RvciBvZiBzdGF0aXN0aWNzLCB3aGVyZSB0aGUgbmFtZXMgaW4gdGhlIHZlY3RvciBhcmUgZ2VuZSBpZGVudGlmaWVycy4KVGhpcyBpcyBzdGVwIDEgLS0gZ2VuZS1sZXZlbCBzdGF0aXN0aWNzLgoKYGBge3IgbGZjX3ZlY3Rvcn0KbGZjX3ZlY3RvciA8LSBkZXNlcV9kZiB8PgogICMgRXh0cmFjdCBhIHZlY3RvciBvZiBgbG9nMkZvbGRDaGFuZ2VgIG5hbWVkIGJ5IGBlbnNlbWJsX2lkYAogIGRwbHlyOjpwdWxsKGxvZzJGb2xkQ2hhbmdlLCBuYW1lID0gZW5zZW1ibF9pZCkKbGZjX3ZlY3RvciA8LSBzb3J0KGxmY192ZWN0b3IsIGRlY3JlYXNpbmcgPSBUUlVFKQpgYGAKCkxldCdzIGxvb2sgYXQgdGhlIHRvcCByYW5rZWQgdmFsdWVzLgoKYGBge3IgaGVhZF9sZmMsIGxpdmUgPSBUUlVFfQojIExvb2sgYXQgZmlyc3QgZW50cmllcyBvZiB0aGUgbG9nIGZvbGQgY2hhbmdlIHZlY3RvcgpoZWFkKGxmY192ZWN0b3IpCmBgYAoKQW5kIHRoZSBib3R0b20gb2YgdGhlIGxpc3QuCgpgYGB7ciB0YWlsX2xmYywgbGl2ZSA9IFRSVUV9CiMgTG9vayBhdCB0aGUgbGFzdCBlbnRyaWVzIG9mIHRoZSBsb2cgZm9sZCBjaGFuZ2UgdmVjdG9yCnRhaWwobGZjX3ZlY3RvcikKYGBgCgojIyBSdW4gR1NFQQoKTm93IGZvciB0aGUgYW5hbHlzaXMhCgpXZSBjYW4gdXNlIHRoZSBgR1NFQSgpYCBmdW5jdGlvbiB0byBwZXJmb3JtIEdTRUEgd2l0aCBhbnkgZ2VuZXJpYyBzZXQgb2YgZ2VuZSBzZXRzLCBidXQgdGhlcmUgYXJlIHNldmVyYWwgZnVuY3Rpb25zIGZvciB1c2luZyBzcGVjaWZpYywgY29tbW9ubHkgdXNlZCBnZW5lIHNldHMgKGUuZy4sIGBnc2VLRUdHKClgKS4KCmBgYHtyIHJ1bl9nc2VhfQpnc2VhX3Jlc3VsdHMgPC0gR1NFQShnZW5lTGlzdCA9IGxmY192ZWN0b3IsICAjIG9yZGVyZWQgcmFua2VkIGdlbmUgbGlzdAogICAgICAgICAgICAgICAgICAgICBtaW5HU1NpemUgPSAyNSwgICMgbWluaW11bSBnZW5lIHNldCBzaXplCiAgICAgICAgICAgICAgICAgICAgIG1heEdTU2l6ZSA9IDUwMCwgICMgbWF4aW11bSBnZW5lIHNldCBzZXQKICAgICAgICAgICAgICAgICAgICAgcHZhbHVlQ3V0b2ZmID0gMC4wNSwKICAgICAgICAgICAgICAgICAgICAgcEFkanVzdE1ldGhvZCA9ICJCSCIsICAjIGNvcnJlY3Rpb24gZm9yIG11bHRpcGxlIGh5cG90aGVzaXMgdGVzdGluZwogICAgICAgICAgICAgICAgICAgICBURVJNMkdFTkUgPSBkcGx5cjo6c2VsZWN0KGhzX2hhbGxtYXJrc19kZiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBnc19uYW1lLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGVuc2VtYmxfZ2VuZSkpICAjIHBhc3MgdGhlIGNvcnJlY3QgaWRlbnRpZmllciBjb2x1bW4KYGBgCkxldCdzIHRha2UgYSBsb29rIGF0IHRoZSBHU0VBIHJlc3VsdHMuCgpgYGB7ciB2aWV3X2dzZWEsIGxpdmUgPSBUUlVFLCBldmFsID0gRkFMU0V9ClZpZXcoZ3NlYV9yZXN1bHRzQHJlc3VsdCB8PgogICAgICAgZHBseXI6OmFycmFuZ2UoZHBseXI6OmRlc2MoTkVTKSkKKQpgYGAKCk5vcm1hbGl6ZWQgZW5yaWNobWVudCBzY29yZXMgKE5FUykgYXJlIGVucmljaG1lbnQgc2NvcmVzIHRoYXQgYXJlIHNjYWxlZCB0byBtYWtlIGdlbmUgc2V0cyB0aGF0IGNvbnRhaW4gZGlmZmVyZW50IG51bWJlciBvZiBnZW5lcyBjb21wYXJhYmxlLgoKUGF0aHdheXMgd2l0aCBzaWduaWZpY2FudCwgaGlnaGx5IHBvc2l0aXZlIE5FUyBhcmUgZW5yaWNoZWQgaW4gRVJNUyBteW9ibGFzdHMsIHdoZXJlYXMgcGF0aHdheXMgd2l0aCBzaWduaWZpY2FudCwgaGlnaGx5IG5lZ2F0aXZlIE5FUyBhcmUgZW5yaWNoZWQgaW4gQVJNUyBteW9ibGFzdHMuCgpMZXQncyB3cml0ZSB0aGVzZSByZXN1bHRzIHRvIGZpbGUuCgpgYGB7ciB3cml0ZV9nc2VhfQpnc2VhX3Jlc3VsdHNAcmVzdWx0IHw+IHJlYWRyOjp3cml0ZV90c3Yob3V0cHV0X2ZpbGUpCmBgYAoKIyMjIFZpc3VhbGl6aW5nIEdTRUEgcmVzdWx0cwoKV2UgY2FuIHZpc3VhbGl6ZSBHU0VBIHJlc3VsdHMgZm9yIGluZGl2aWR1YWwgcGF0aHdheXMgb3IgZ2VuZSBzZXRzIHVzaW5nIGBlbnJpY2hwbG90Ojpnc2VhcGxvdCgpYC4KTGV0J3MgdGFrZSBhIGxvb2sgYXQgMyBkaWZmZXJlbnQgcGF0aHdheXMgLS0gb25lIHdpdGggYSBoaWdobHkgcG9zaXRpdmUgTkVTLCBvbmUgd2l0aCBhIGhpZ2hseSBuZWdhdGl2ZSBORVMsIGFuZCBvbmUgdGhhdCB3YXMgbm90IGEgc2lnbmlmaWNhbnQgcmVzdWx0IC0tIHRvIGdldCBtb3JlIGluc2lnaHQgaW50byBob3cgRVMgYXJlIGNhbGN1bGF0ZWQuCgojIyMjIEhpZ2hseSBQb3NpdGl2ZSBORVMKCkxldCdzIHRha2UgbG9vayBhdCBhIHBhdGh3YXkgd2l0aCBhIGhpZ2hseSBwb3NpdGl2ZSBORVMgKGBIQUxMTUFSS19NWUNfVEFSR0VUU19WMmApIHVzaW5nIGEgR1NFQSBwbG90LgoKYGBge3IgaGlnaGx5X3Bvc30KZW5yaWNocGxvdDo6Z3NlYXBsb3QoZ3NlYV9yZXN1bHRzLAogICAgICAgICAgICAgICAgICAgICBnZW5lU2V0SUQgPSAiSEFMTE1BUktfTVlDX1RBUkdFVFNfVjIiLAogICAgICAgICAgICAgICAgICAgICB0aXRsZSA9ICJIQUxMTUFSS19NWUNfVEFSR0VUU19WMiIsCiAgICAgICAgICAgICAgICAgICAgIGNvbG9yLmxpbmUgPSAiIzAwNjZGRiIpCmBgYAoKTm90aWNlIGhvdyB0aGUgZ2VuZXMgdGhhdCBhcmUgaW4gdGhlIGdlbmUgc2V0LCBpbmRpY2F0ZWQgYnkgdGhlIGJsYWNrIGJhcnMsIHRlbmQgdG8gYmUgb24gdGhlIGxlZnQgc2lkZSBvZiB0aGUgZ3JhcGggaW5kaWNhdGluZyB0aGF0IHRoZXkgaGF2ZSBwb3NpdGl2ZSBnZW5lLWxldmVsIHNjb3Jlcy4KCiMjIyMgSGlnaGx5IE5lZ2F0aXZlIE5FUwoKVGhlIGdlbmUgc2V0IGBIQUxMTUFSS19NWU9HRU5FU0lTYCBoYWQgYSBoaWdobHkgbmVnYXRpdmUgTkVTLgoKYGBge3IgaGlnaGx5X25lZ30KZW5yaWNocGxvdDo6Z3NlYXBsb3QoZ3NlYV9yZXN1bHRzLAogICAgICAgICAgICAgICAgICAgICBnZW5lU2V0SUQgPSAiSEFMTE1BUktfTVlPR0VORVNJUyIsCiAgICAgICAgICAgICAgICAgICAgIHRpdGxlID0gIkhBTExNQVJLX01ZT0dFTkVTSVMiLAogICAgICAgICAgICAgICAgICAgICBjb2xvci5saW5lID0gIiMwMDY2RkYiKQpgYGAKClRoaXMgZ2VuZSBzZXQgc2hvd3MgdGhlIG9wcG9zaXRlIHBhdHRlcm4gLS0gZ2VuZXMgaW4gdGhlIHBhdGh3YXkgdGVuZCB0byBiZSBvbiB0aGUgcmlnaHQgc2lkZSBvZiB0aGUgZ3JhcGguCgojIyMjIEEgbm9uLXNpZ25pZmljYW50IHJlc3VsdAoKVGhlIGBAcmVzdWx0c2Agc2xvdCB3aWxsIG9ubHkgc2hvdyBnZW5lIHNldHMgdGhhdCBwYXNzIHRoZSBgcHZhbHVlQ3V0b2ZmYCB0aHJlc2hvbGQgd2Ugc3VwcGxpZWQgdG8gYEdTRUEoKWAsIGJ1dCB3ZSBjYW4gcGxvdCBhbnkgZ2VuZSBzZXQgc28gbG9uZyBhcyB3ZSBrbm93IGl0cyBuYW1lLgpMZXQncyBsb29rIGF0IGBIQUxMTUFSS19QNTNfUEFUSFdBWWAsIHdoaWNoIHdhcyBub3QgaW4gdGhlIHJlc3VsdHMgd2Ugdmlld2VkIGVhcmxpZXIuCgpgYGB7ciBwNTMsIGxpdmUgPSBUUlVFfQplbnJpY2hwbG90Ojpnc2VhcGxvdChnc2VhX3Jlc3VsdHMsCiAgICAgICAgICAgICAgICAgICAgIGdlbmVTZXRJRCA9ICJIQUxMTUFSS19QNTNfUEFUSFdBWSIsCiAgICAgICAgICAgICAgICAgICAgIHRpdGxlID0gIkhBTExNQVJLX1A1M19QQVRIV0FZIiwKICAgICAgICAgICAgICAgICAgICAgY29sb3IubGluZSA9ICIjMDA2NkZGIikKYGBgCgpHZW5lcyBpbiB0aGUgcGF0aHdheSBhcmUgZGlzdHJpYnV0ZWQgbW9yZSBldmVubHkgdGhyb3VnaG91dCB0aGUgcmFua2VkIGxpc3QsIHJlc3VsdGluZyBpbiBhIG1vcmUgIm1pZGRsaW5nIiBzY29yZS4KCipOb3RlOiBUaGUgcGxvdHMgcmV0dXJuZWQgYnkgYGVucmljaHBsb3Q6OmdzZWFwbG90YCBhcmUgZ2dwbG90cywgc28gd2UgY291bGQgdXNlIGBnZ3Bsb3QyOjpnZ3NhdmUoKWAgdG8gc2F2ZSB0aGVtIHRvIGZpbGUgaWYgd2Ugd2FudGVkIHRvLioKCiMjIFNlc3Npb24gSW5mbwoKYGBge3Igc2Vzc2lvbl9pbmZvfQpzZXNzaW9uSW5mbygpCmBgYAo=