participatory-inequality-appendix.qmd

# Appendix {.unnumbered}

# Data Collection and Processing {#collection}

@tbl-master-table <!--and @fig-docket-table--> shows the number of rulemaking processes (dockets) and comments in our data by agency. Identifying comments from organizations and matching those organizations to metadata about their resources required significant effort. To match comments to organizations found in various databases, we first extract entity names from the text or from comment metadata where available. We then use a custom probabilistic matching algorithm that was iteratively built to correctly match organizations in these data using a combination of term-frequency times inverse document frequency (TF-IDF) and Jaccard distance. For each commenter, we start with the most uncommon token (word) in the entity name string and search for names in each dataset that have that token. For example, if Klamath First Federal Bank submitted a comment, the algorithm first looks for names with the token "Klamath." We then rank the resulting candidate matches using a modified Jaccard index that scores each token in the commenter's name that matches a token in the candidate name in inverse proportion to the token's frequency in the commenter dataset (normalizing by the sum of the inverse frequencies of all the tokens in the commenter's name, matching or otherwise) so that 'more informative' words contribute more to the match score. We then set a threshold match score that, upon inspection, yields correct matches. Finally, we then hand-validated over 30,000 comments, including all matches that occurred ten times or more and a sample of others and implement a custom set of corrections based on this validated set.

```{=html}
<!--As noted, the data we use in our analysis is the subset of all comments that match an organization with one or more types of wealth data.
Practically, acquiring wealth information for all commenters---including thousands of individuals who submit form comments as part of mass comment campaigns (with no little identifying information)---would have been impossible. Moreover, because nearly all individual commenters are mobilized by an organization, form comments are best conceptualized as supporting the more sophisticated comments we focus on in our analysis here [@JudgeLord2021].-->
```

```{r}
#| label: "tbl-master-table"
#| tbl-cap: "Comments, Comment Attachments, Comment Sophistication, Comment Lobbying Success, and Commenter Wealth Data on Rules Implementing the Dodd-Frank Act"
#| fig.pos: "b"
#| out-extra: "\\floatnote{Wealth data are only available for comments linked to organizations.}"


# source("code/setup.R")

read_csv(here::here("data", "master_tables_status.csv")) %>% 
  mutate(attachments_table = replace_na(attachments_table, 0)) %>% 
  rename(`Attachments` = attachments_table,
         `Sophistication Measures` =`term counts`,
         Comments = `comments table`,
         Agency = agency_acronym) %>%
  mutate(`Success Measures` = `Sophistication Measures`) %>% 
  left_join(
    commenter_assets %>% 
      drop_na(best_match_name) %>% 
      count(Agency, name = "Wealth Data") 
    ) %>% 
  mutate(`Wealth Data` = replace_na(`Wealth Data`, 0) ) %>%
  drop_na(Agency) %>% #kablebox()
  knitr::kable(#table.attr = 'data-quarto-disable-processing="true"',
    booktabs = T,
    align = "l",
    linesep = "\\addlinespace")
```

<!-- CUT FOR AJPS 20-PAGE LIMIT 
\clearpage

## Entity extraction and matching {#matching}

@fig-docket-table shows the number of rulemaking dockets and the number of comments matched to organizations with resource data by agency.

```{r}
#| label: "fig-docket-table"
#| fig-cap: "Dockets and Comments Matched to Asset Data by Agency"
#| out-width: "70%"
knitr::include_graphics(here::here("figs/docket-table-1.png"))
```

<!-- TODO: PAGE DESCRIBING MATCHING METHOD-->


# Additional Descriptives {#descriptives}

<!-- CUT FOR AJPS 
## Types of Non-Profits

@fig-commenters-noncommenters-section shows that the relationship between assets and commenting shown in @fig-commenters-noncommenters also appears when we look at different types of non-profits. Specifically, we focus on the difference between business leagues --- 501(c)(6)s --- and charities --- 501(c)(3)s. The difference between the groups means are statistically significant using a one-way ANOVA, although we note that in the six pairwise t-tests between all the groups the differences between the 501(c)(6)s that commented and the 501(c)(3) that did not and the 501(c)(6)s that commented and those that did not were not statistically significant.

```{r}
#| label: "fig-commenters-noncommenters-section"
#| fig-cap: "Non-profits that Did and Did Not Comment by Type"
#| out-width: "100%"
knitr::include_graphics(here::here("figs/nonprofit-density-by-section-1.png"))
```
--> 

## Non-profit revenue

@fig-commenters-noncommenters-revenue shows that the relationship between assets and commenting shown in @fig-commenters-noncommenters also appears when we look at revenue rather than assets. Indeed the relationship between revenue and commenting is much stronger than the relationship between assets and commenting. We focus on assets in the body of the text because it is more comparable to wealth measures from for-profit organizations.

```{r}
#| label: "fig-commenters-noncommenters-revenue"
#| fig-cap: "Revenue of Non-profits that Did and Did Not Comment"
#| layout-ncol: 2 
#| fig-subcap: 
#| - " "
#| - " "
knitr::include_graphics(here::here(c(
  "figs/nonprofit-density-2.png",
  "figs/ind-assoc-density-2.png")))
```

## Non-profit volunteers

The most likely context for a wealth-membership correlation would be among non-profit organizations. To test these relationships, we examine the active membership base of non-profits.
@fig-assets-vol shows that the a non-profit's assets and the number of volunteers it has are not especially correlated for the sample of organizations that commented on a Dodd-Frank rule. This offers further evidence that the relationship between wealth and lobbying success we observe should not be interpreted as larger membership organizations being more successful. Rather, it is wealthy organizations, regardless of membership that enjoy success rulemaking.

```{r}
#| label: "fig-assets-vol"
#| fig-cap: "Volunteers of Non-profits that Did and Did Not Comment"
#| out-width: "70%"
knitr::include_graphics(here::here("figs/assets-vol-1.png"))
```

@fig-efficacyXvol shows that the a non-profit's number of volunteer does not predict its level of lobbying success.

```{r}
#| label: "fig-efficacyXvol"
#| out-width: "60%"
#| fig-cap: "Efficacy by Number of Volunteers"
knitr::include_graphics(here::here("figs/efficacyXvol-2.png"))
```


<!---Cut for AJPS 20 page limit 
## Histograms

Figures [-@fig-commenters-noncommenters-counts], [-@fig-FDIC-density-by-class], and [-@fig-opensecrets-count] present histograms of wealth distributions by whether an organization commented on a Dodd-Frank rule.

```{r}
#| label: "fig-commenters-noncommenters-counts"
#| fig-cap: "Financial Resources of Organizations that Did and Did Not Comment"
#| layout-ncol: 2 
#| fig-subcap: 
#| - " "
#| - " "
#| - " "
#| - " "
knitr::include_graphics(here::here(c(
  "figs/nonprofit-count-1.png",
  "figs/creditunion-count-1.png",
  "figs/FDIC-count-select-1.png",
  "figs/compustat-count-1.png")))
```

### Variation within classes of banks

When we look within categories of banks, we see that the wealthier banks within each class are also more likely to submit comments on financial rules than similar banks with less wealth. @fig-FDIC-density-by-class shows that, within each class of bank (i.e., commercial banks, commercial banks, state banks, and savings associations), wealthier banks participate in financial rulemaking more than less wealthy banks.  While the differences within types of banks are fairly large, these differences in means only reach statistical significance at the 0.05 level for for-profit categories of banks.


@fig-FDIC-density-by-class shows wealth distributions for four prominent types of banks: commercial banks, commercial banks, state banks, and non-profit savings associations. The top-left panel of @fig-FDIC-density-by-class shows that commercial banks that comment are wealthier than those that did not comment. The modal commercial bank that commented has 40 percent more assets than the modal commercial bank that did not comment. The top-right panel of @fig-FDIC-density-by-class shows that commercial banks banks that comment are wealthier than those that did not comment. The modal commercial bank that commented has nearly twice the assets of the modal commercial bank that did not comment. Similarly, the bottom-left panel of @fig-FDIC-density-by-class shows that the average assets of state banks that commented were three times the average assets of the state banks that did not comment. While savings associations are less likely to comment than more profit-oriented banks, such as commercial banks (see @fig-mp-FDIC), the bottom-right panel of @fig-FDIC-density-by-class shows that when savings associations do comment, they tend to be the wealthier ones.

```{r}
#| label: "fig-FDIC-density-by-class"
#| fig-cap: "Financial Resources of Banks that Did and Did Not Comment"
#| layout-ncol: 2 
#| fig-subcap: 
#| - " "
#| - " "
#| - " "
#| - " "

knitr::include_graphics(here::here(c(
  "figs/FDIC-count-by-class-1.png",
  "figs/FDIC-count-by-class-2.png",
  "figs/FDIC-count-by-class-3.png",
  "figs/FDIC-count-by-class-4.png")))
```

### Variation among organizations that donate to campaigns and spend money on lobbying

```{r}
#| label: "fig-opensecrets-count"
#| fig-cap: "Political Spending of Organizations that Did and Did Not Comment"
#| layout-ncol: 2 
#| fig-subcap: 
#| - " "
#| - " "
knitr::include_graphics(here::here(c(
  "figs/opensecrets-count-1.png",
  "figs/opensecrets-lobbying-count-1.png")))
```

## Frequent commenters by number of Dodd-Frank rules

Note that commenting on *more rules* is not the same as submitting *more comments* overall. Many wealthy organizations only submit one sophisticated comment per rulemaking docket. Some organizations also submit many comments on the same rule as a form of public pressure. Pressure campaigns are mostly organized by public interest groups but are also occasionally organized by regulated companies [@JudgeLord2021]. For example, Axcess Financial (a payday lending company) and Advance Financial (a credit union) both mobilized over 1000 comments from their retail stores on the Consumer Financial Protection Bureau's Payday Loan Rule. Mobilizing public pressure is different from lobbying. Our analysis here focuses on the breadth, not the amplitude of lobbying.

```{r}
#| label: "fig-number-of-dockets"
#| fig-cap: "Number of Dockets on Which Each Type of Organization Commented"
#| layout-ncol: 2 
#| fig-subcap: 
#| - " "
#| - " "
#| - " "
#| - " "
#| - " "
#| - " "
knitr::include_graphics(here::here(c(
    "figs/nonprofit-rules-1.png",
    "figs/ind-assoc-rules-1.png",
    "figs/creditunion-rules-1.png",
    "figs/compustat-rules-1.png",
    "figs/fdic-rules-1.png",
    "figs/opensecrets-rules-1.png")))
```

```{r}
#| label: "fig-dockets-percentile-five"
#| fig-cap: "Frequent and Infrequent Commenters (By the Number of Dockets on Which Each Organization Commented) by Resources (Log Scale)"
#| layout-ncol: 2 
#| fig-subcap: 
#| - " "
#| - " "
#| - " "
#| - " "
#| - " "
#| - " "
knitr::include_graphics(here::here(c(
    "figs/nonprofit-rules-2.png",
    "figs/ind-assoc-rules-2.png",
    "figs/creditunion-rules-2.png",
    "figs/fdic-rules-2.png",
    "figs/compustat-rules-2.png",
    "figs/opensecrets-rules-2.png")))
```
--> 

\clearpage 

# Regression Tables {#tables}

## The Odds of Commenting by Wealth

@tbl-mp-all-table2 presents alternative specifications of the model shown in column 2 of @tbl-mp-all-table, using linear instead of logged terms and omitting interaction terms. In all specifications, the results are essentially the same. The only differences appear in the interaction terms between the linear and logged asset models. Banks remain more likely to comment than credit unions and other nonprofits across the distribution of support. In the linear models, this difference is larger at higher asset levels, whereas in the logged model this difference is smaller at higher (logged) asset levels.

@tbl-mp-assets-table <!--(@fig-mp-assets)--> shows the results of separate logit models predicting the log odds of commenting on a Dodd-Frank rule by assets for banks, credit unions, and non-profits. These models show that wealthier organizations of all three types are significantly more likely to comment. Of these three types of organizations, the marginal effect of assets on the log odds of commenting is the largest for banks.

\footnotesize
```{r}
#| label: tbl-mp-all-table2
#| tbl-cap: Log Odds of Commenting on Any Dodd-Frank Rule
# Models of commenting by assets
load(here::here("models", "mpAll.Rdata"))

modelsummary(models, notes = "Reference category = Banks")  #%>% kableExtra::kable_styling(latex_options="scale_down")
    #FIXME sometimes scale_down causes an error 
```
\small

<!--CUT FOR AJPS 
```{r}
#| label: "fig-mp-assets"
#| fig-cap: "Log Odds of Participating in Dodd-Frank Rulemaking by Assets"
#| out-width: "90%"
knitr::include_graphics(here::here("figs/mp-nonprofit-credit-unions-1.png"))
```
--> 

Column 1 of @tbl-mp-all-table presented a model that pooled data bank, credit union, and nonprofit assets. For robustness, @tbl-mp-assets-table presents regression tables estimating separate models for each type of organization.

\footnotesize
```{r}
#| label: "tbl-mp-assets-table"
#| tbl-cap: "Log Odds of Commenting on Any Dodd-Frank Rule"
load(here::here("models", "pr-of-comment.Rdata"))

modelsummary(models)#  %>% kableExtra::kable_styling(latex_options="scale_down") # %>% kableextra::kable_styling()
```
\small

@tbl-mp-FDIC and @tbl-mp-FDIC2 present alternative specifications to the model in column 3 of @tbl-mp-all-table. @tbl-mp-FDIC2 presents models without interactions and without logging assets. @tbl-mp-FDIC presents models that break out more categories of banks, rather than lumping them into commercial and non-commercial banks. @fig-mp-FDIC and @tbl-mp-FDIC show that commercial banks were disproportionately represented in Dodd-Frank rulemaking and non-commercial banks (e.g. savings associations) were less represented, even controlling for asset differences. This provides further support for the *Profit-motivated Participation* Hypothesis (H2).

Likewise, assets remain a significant predictor of whether an organization comments even controlling for differences in the type of bank institution. This provides additional evidence for the *Differential Participation* Hypothesis (H1).

```{r}
#| label: "fig-mp-FDIC"
#| fig-cap: "Predicted Probability of Participating in Dodd-Frank Rulemaking by Type of Bank"
#| out-width: "80%"
#TODO CREDIT UNIONS SHOULD BE IN THIS MODEL 
knitr::include_graphics(here::here("figs/mp-FDIC-predict-1.png"))
```

\footnotesize
```{r}
#| label: tbl-mp-FDIC2
#| tbl-cap: "Log Odds of Commenting on Any Dodd-Frank Rule by Bank Type"
#| out-width: "80%"
load(here::here("models", "mpFDIC2.Rdata"))

modelsummary(models, notes = "Non-commercial banks include savings associations and national associations.") #%>% kableExtra::kable_styling(latex_options="scale_down")
```

```{r}
#| label: "tbl-mp-FDIC"
#| tbl-cap:  "Log Odds of Commenting on Any Dodd-Frank Rule by Bank Type"
load(here::here("models", "mpFDIC.Rdata"))

modelsummary(models, notes = "Reference category = savings associations") # %>% kableextra::kable_styling()
```
\small

## The Odds of Commenting by Political Spending

@fig-mp-opensecrets (@tbl-mp-opensecrets-table) shows the results of logit models predicting the log odds of commenting on a Dodd-Frank rule by each organization's average or total political action committee contributions. These models show that organizations that spend more on political campaigns are significantly more likely to comment.

```{r}
#| label: "fig-mp-opensecrets"
#| fig-cap: "Predicted Probability of Participating in Dodd-Frank Rulemaking by Political Spending"
#| out-width: "30%"
knitr::include_graphics(here::here("figs/mp-opensecrets-predict-1.png"))
```

```{r}
#| label: "tbl-mp-opensecrets-table"
#| tbl-cap:  "Log Odds of Commenting on Any Dodd-Frank Rule by Bank Type"
load(here::here("models", "mp-opensecrets.Rdata"))

#FIXME 
modelsummary(models) # %>% kableextra::kable_styling()
```


<!--

````{=html}

@fig-mp-np-by-subsection shows that business leagues (Non-profit 06) were disproportionately represented in Dodd-Frank rulemaking compared to non business leagues (Non-profit 03).  We note, however, that their probabilities of commenting are much lower than any type of bank as shown in @fig-mp-FDIC.


```{r}
#| eval: false
#| label: "fig-mp-np-by-subsection"
#| fig-cap: "Predicted Probability of Participating in Dodd-Frank Rulemaking by Type of Non-Profit"
#| layout-ncol: 2 
#| fig-subcap: \phantomcaption
#TODO CREDIT UNIONS SHOULD BE IN THIS MODEL 
knitr::include_graphics(here::here("figs/only-np-sep-mp-all-predict-log-1.png"))
knitr::include_graphics(here::here("figs/np-sep-mp-all-predict-log-1.png.png"))
```

````

--->

# Frequent participants are wealthier than those who participate less frequently


We have shown that wealth is highly correlated with *whether* an organization comments. We also expect wealth inequalities in lobbying participation to persist even among those organizations that can pay the initial costs of rulemaking participation. Stated differently, when focused on those entities that have submitted at least one comment to a Dodd-Frank regulation, we argue that more wealthy organizations will, again, hold an advantage over less wealthy organizations by participating in *more* rulemaking processes. 

> *Differential Frequency of Participation Hypothesis:* Among organizations commenting on rulemaking, organizations with greater wealth comment on more rules.

We use Welch t-tests to assess differences between commenters and non-commenters frequent and infrequent commenters. We then use Poisson regression to model the count of rules on which an organization comments. 


The *Differential Frequency of Participation* Hypothesis posits that, among commenters, wealthy organizations participate more frequently. To test this hypothesis, we count the number of Dodd-Frank rules on which each participating organization commented.
@fig-dockets-percentile shows that organizations that comment on more rules tend to be wealthier. Given that most organizations comment on few rules, we sort commenters by the number of rules on which they comment and compare the wealth of the top 10% to the bottom 90%. In the appendix, we show similar results comparing organizations that commented on five or more rules to those that commented on fewer than five rules.

<!--TODO: Check these numbers once the plots in Fig 10 are formatted like Fig 5, with means and medians-->
Panel (a) of @fig-dockets-percentile shows that most of the non-profits in the top 10% of most frequent commenters had assets over \$1 million. In contrast, non-profits in the bottom 90% (i.e., low-frequency commenters) had assets under \$1 million.
Panel (b) of @fig-dockets-percentile shows that most of the credit unions in the top 10% of most frequent commenters had greater assets than the average credit union in the bottom 90%. 
Panel (c) of @fig-dockets-percentile shows that most of the industry associations in the top 10% of most frequent commenters had greater assets than the average industry association in the bottom 90%. Most of the industry associations in the top 10% of most frequent commenters had assets over \$10 million. In contrast, industry associations in the bottom 90% (i.e., low-frequency commenters) had assets under \$1 million.
Panel (d) of @fig-dockets-percentile shows that, even among banks, a large share of the most frequent commenters had assets over \$1 billion. Yet, nearly all banks that were less frequent commenters---most of which only commented on one rule---had far less than \$1 billion in assets. 
Panel (e) of @fig-dockets-percentile shows that, among publicly-traded companies, the majority of frequent commenters had market capitalization over \$10 billion. In contrast, most companies that were less frequent commenters had under \$10 billion in market capitalization.
Overall, while these differences are only statistically significant for industry associations and publicly-traded companies, the general pattern is in the direction predicted by Hypothesis 3: frequent commenters also tend to be more wealthy organizations.

```{r}
#| label: fig-dockets-percentile
#| fig-cap: "Frequent and Infrequent Commenters (By Percentile of the Number of Dockets on Which Each Organization Commented) by Resources (Log Scale)"
#| layout-ncol: 2
#| fig-subcap: 
#| - "Non-profits"
#| - "Credit Unions"
#| - "Industry Associations"
#| - "Banks"
#| - "Publicly-traded Companies"
knitr::include_graphics(here::here(c(
  "figs/nonprofit-rules-3.png",
  "figs/creditunion-rules-3.png",
  "figs/ind-assoc-rules-3.png",
    "figs/fdic-rules-3.png",
  "figs/compustat-rules-3.png")))
```


\clearpage

# Measuring Comment Sophistication with Legal Citations {#legal}

Our analyses investigating the *Differential Sophistication* (H5) and *Dividends of Sophistication* (H6) hypotheses rely on a measure of comment sophistication based on the number of technical terms used in a given comment. However, using technical terms is only one way to gauge sophistication. An alternate measure would be the number of legal citations in the comment. Wealthier organizations may be more influential by using sophistical legal arguments in commenting.

This section replicates the descriptive and regression analyses conducted in sections 4.2.3 and 4.2.4, using the number of legal citations as the measure of comment sophistication. We count the number of citations to the U.S. Code, Supreme Court cases, appellate and district court cases, the code of federal regulations, and the federal register. Like in the analyses relying on technical terms, we sum up citations across all the submitted documents of a commenter. @fig-efficacyXsophistication-legal shows a strong relationship between legal citations and comment lobbying success, again highlighting the comment from the Chamber of Commerce discussed in Section 3.

```{r}
#| label: "fig-efficacyXsophistication-legal"
#| fig-cap: "Lobbying Success by Comment Sophistication"
#| out-width: "70%"
knitr::include_graphics(here::here("figs/boxplot-efficacyXsophistication-legal-1.png"))
```

Our findings on wealth technical sophistication (H5) hold even with an alternative legal measure of sophistication. @fig-assets-terms-legal shows that the number of words from the comment added to the final rule is correlated with the number of legal citations. Like the analyses using technical terms, the figure also shows a positive correlation between the number of legal citations in a comment and the amount of text it shares with the final rule.

```{r}
#| label: "fig-assets-terms-legal"
#| layout-ncol: 2 
#| fig-cap: "Amount of Legal Language by Assets (Among Comments from Banks on Dodd-Frank Rules)"
#| fig-subcap: 
#| - " "
#| - " "
#| - " "
#| - " "
#| - " "

knitr::include_graphics(here::here(c(
  ## Nonprofits
  "figs/boxplot-assets-legal-2.png",
  ## Credit unions 
  "figs/boxplot-assets-legal-4.png",
  ## Industry Associations 
  "figs/boxplot-assets-legal-6.png",
  ## FDIC-insured banks 
  "figs/boxplot-assets-legal-10.png",
  ## Market cap (publicly-traded companies)
  "figs/boxplot-assets-legal-8.png")))
```

Findings regarding the relationship between technical sophistication and the number of words from a comment added to final rule (efficacy) are robust to alternative specifications. @tbl-efficacy-sophistication-pois re-estimates the models from @tbl-efficacy-sophistication using Poisson regression. While Poisson regression is more appropriate for counts of words added to the final rule, the coefficients are more difficult to interpret than the OLS results in @tbl-efficacy-sophistication-pois.

```{r}
#| label: "tbl-efficacy-sophistication-pois"
#| out-width: "80%"
#| tbl-cap: "Poisson Models of Lobbying Success by Comment Sophistication"

load(here::here("models", "mes-pois.Rdata"))

rows <- rows |> mutate(`1` = "Lobbying Success",
                       `2` = "Lobbying Success")

modelsummary(models, notes = " ")  #%>% kableExtra::kable_styling(latex_options="scale_down")
```


<!--
@fig-assets-terms-legal also corroborates with regression findings on technical sophistication. Here, comments from wealthier organizations tend to include more legal language, a pattern permeated across banks, publicly traded companies, and campaign donors. Similar to the relationship between technical terms and commenter wealth, most of the comments from publicly traded companies with ten or more legal citations were submitted by companies with over \$50 billion in market capitalization.

```{r}
#|eval: false
#| label: "fig-marketcap-terms-legal"
#| out-width: "90%"
#| fig-cap: "Amount of Legal Language by Market Capitalization (Among Comments on Dodd-Frank Rules)"
knitr::include_graphics(here::here("figs/mb-2.png"))

```

-->


Analyses on sophistication and influence (H6) also hold up when using a measure of legal sophistication. @fig-efficacyXsophistication-legal shows that comments using more legal language are more likely to contain text added in the final rule.

```{r}
#| label: "fig-efficacy-sophistication-second"
#| out-width: "80%"
#| fig-cap: "OLS Models of Lobbying Success by Legal Language"
knitr::include_graphics(here::here("figs/mes-2.png"))
```


<!---

## The relationship between assets and sophistication
```{r}
#| label: "fig-marketcap-terms"
#| out-width: "90%"
#| fig-cap: "Amount of Legal and Technical Language by Market Capitalization (Among Comments on Dodd-Frank Rules)"
knitr::include_graphics(here::here("figs/mt-2.png"))
```
--->


# Mediation

<!--the ability of the wealthy to mobilize lawyers and experts to make sophisticated and thus influential arguments on their behalf-->

Lobbying and PAC contributions are both forms of political spending that would influence the rulemaking process in a direct financial way. Technical expertise and legal expertise are both measured using dictionary techniques, looking at banking terms and legal citations, respectively. The expertise pathways each come from a sense that large companies employ experts who have meaningful advice to give to agencies, whether it be on finer points of the law or nuance in how regulations should be enforced to capture the correct behaviors.

Money itself can influence the efficacy, if there is a portion of efficacy that is purely a function of wealth. For example, if attention was driven by the wealth of a company, then this might be a direct relationship from the market cap of the company to the efficacy of their comments.

@fig-paths formalizes these potential pathways into a basic DAG.

```{mermaid}
%%| label: fig-paths
%%| fig-cap: "Stylized pathways of influence from wealth to efficacious comments."
%%| fig-width: 3
%%| out-width: 100%
flowchart LR
  A[Market Cap] --> B[Efficacy]
  A --> C(Lobbying) --> B
  A --> D(PAC contributions) --> B
  A --> E(Technical Expertise) --> B
  A --> F(Legal Expertise) --> B
```

To test the different pathways, we require some strong assumptions.

1.  There is a causal relationship from the market cap of a company to the efficacy of comments.
2.  The paths included in the analysis are the full enumeration of possible paths.
3.  The relationships can be described by linear models and binarized treatments.

Note that while these assumptions are possible, our estimation is not identified. Further, we are technologically limited by existing software for mediation analysis ([see the `mediation` R package](https://cran.r-project.org/package=mediation)).

@fig-mediation demonstrates that the Average Conditional Marginal Effect (ACME) for technical sophistication is nearly identical to the Total Effect of market capitalization on lobbying success. This means that technical sophistication explains nearly all of the greater success of wealthier companies. Legal sophistication also explains a large share of the total relationship when we use legal citations as an alternative mediator. This suggests that legal citations explain much of the greater success of wealthier companies.

```{r}
#| label: "fig-mediation"
#| layout-ncol: 2 
#| out-width: 85%
#| fig-subcap:
#| - " "
#| - " "
#| - " "
#| - " "
#| - " "
#| fig-cap: "Political Spending, Lobbying, Technical Sophistication, and Legal Sophistication as a Proposed Mediators Between Wealth and Lobbying Success"
knitr::include_graphics(here::here(c(
  "figs/mediation-marketcap-lobbying-1.png",
  "figs/mediation-marketcap-lobbying-2.png",
  "figs/mediation-marketcap-bluebook-1.png",
  "figs/mediation-marketcap-terms-1.png")))
```

\clearpage

\blandscape

```{r}
#| label: "tbl-mediation-table"
#| tbl-cap: "Models for Mediation Analysis"
# load(here::here("models", "models.m.Rdata")) # FIXME
load(here::here("models", "models.m4No0.Rdata"))
rows[,2] <- "PAC Spending"
rows[,3] <- "LDA Spending"
rows[,5] <- "Lobbying Success"
modelsummary(models.m4No0, notes = " ")
```

\elandscape

\clearpage

# Measuring Change Between Draft and Final Rules

In dealing with endogeneity, one methodological choice merits elaboration: we excluded text from the proposed rule when measuring lobbying success but not when measuring sophistication. This choice rests on the underlying concepts we are attempting to measure. In measuring text reuse, we aim to capture ideas that were not yet in the policy when the comment was submitted. Thus, text copied from the agency's proposal must be excluded. Indeed, text that appears in both the draft and final rule is what did *not* change. If a commenter attached a marked-up version of the proposed rule, we aim to exclude all but their suggested changes.

In contrast, in measuring sophistication, we aim to assess how much the commenter utilizes expertise to engage in technical policy debates. Here, attaching a marked-up version of the proposed rule captures the underlying concept of sophistication. Thus, our counts of technical banking terms do not exclude the text of the draft rule. Even if they are the agency's terms, engaging with its texts indicates sophistication. For example, the comment with the most legal terms from a bank contained a 4-page comment and 112 pages of attachments, 105 of which were the full proposed rule. These 105 pages were excluded from our measure of text reuse but included in the legal and banking terms count.

Our algorithm works in the following steps: Match each comment to the proposed rule that came after the proposed rule and before the final rule. For a simple sequence (1) proposed rule, (2) comment, (3) final rule, this step is perfunctory. Not all rulemaking sequences, however, follow this template. Some have multiple proposed rules. Suppose a rule has 2 proposed rules and a final rule and a comment came after the second proposed rule (i.e. (1) proposed rule, (2) proposed rule two, (3) comment, and (4) final rule). We would match the comment to the second proposed rule, since this comment is likely responding to that version of the proposed rule rather than the first one.

For the proposed-final pair, tokenize the proposed and final rule into sentences and then remove the text of the proposed rule from the final rule.
Re-tokenize both the proposed and final rules into 10-grams. Find the overlap between proposed and final rules, keeping track of the position index of the 10-grams. 10-grams that are consecutive in the proposed rule should be treated as consecutive n-grams for the purpose of counting consecutive words. In this manner, two consecutive 10-grams would have a sequence length of 11 and not 20.

This algorithm has a few notable strengths. First, it focuses on text that is both legally binding and the explanation for why it is legally binding. Second, the results are easy to interpret: efficacy is simply the number of words in a comment and its corresponding final rule. There are, however, several drawbacks. Notably, not all sequences in both comments and final rules are equally influential. Second, when there are multiple proposed and final rules, the algorithm relies on the date a comment is submitted to match the comment to the right proposed-final pair. This is problematic when a comment is written about one proposed rule but submitted after a subsequent proposed rule is released.



# Case Study of the Whistleblower Rule

As a part of the Dodd-Frank Act, Congress mandated that the SEC establish a whistleblower program to provide monetary incentives to individuals that report violations of securities laws. Eligible employees who provide “original, timely, and credible information that leads to a successful enforcement action” are eligible for awards ranging from 10-30% of the money collected when a fine exceeds \$1 million. Between 2012 and September 2018, the SEC received over 28,000 tips and disbursed \$326 million to 59 whistleblowers. Successful tips have halted frauds, Ponzi schemes, and accounting violations among other types of financial misconduct.

Among other provisions, the SEC proposed making certain individuals ineligible for the reward. One class of ineligible employees were people who learned about and reported violations based on information obtained from audits required under securities laws. In the proposed rule, the SEC operationalized this as:

> (4) The Commission will not consider information to be derived from your independent knowledge or independent analysis if you obtained the knowledge or the information upon which your analysis is based:
>
> ```{=html}
> <!-- -->
> ```
> (iii) Through the performance of an engagement required under the securities laws by an independent public accountant, if that information relates to a violation by the engagement client or the client’s directors, officers or other employees;

The American Bar Association had a problem with this provision because it did not explicitly exclude internal company personnel who support the auditors above. That is, the ABA feared that not explicitly excluding employees who help on audits would allow those employees to report fraud directly to the SEC and be treated as whistleblowers. The American Bar Association wrote:

The Committees believe that Proposed Rule 21F-4(b)(4)(iii) with respect to information obtained through the performance of an engagement required under the securities laws by an independent public accountant should also include information obtained by internal company personnel in connection with their role supporting an independent public accountant conducting an audit required under the securities laws (including both a financial statement audit and an audit of internal controls).

The SEC agreed and then cited the passage in the preamble of the final rule:

One commenter urged that the exclusion for independent public accountants should also extend to information obtained by internal company personnel in connection with their role supporting an independent public accountant conducting an audit required under the securities laws.

Crucially, they also changed the legally operative language of the final rule to explicitly exclude the aforementioned employees.

>(4) The Commission will not consider information to be derived from your independent knowledge or independent analysis in any of the following circumstances:
>
> ```{=html}
> <!-- -->
> ```
>(iii) In circumstances not covered by paragraphs (b)(4)(i) or (b)(4)(ii) of this section, if you obtained the information because you were: ...
>
> ```{=html}
> <!-- -->
> ```
(B) An employee whose principal duties involve compliance or internal audit responsibilities, or you were employed by or otherwise associated with a firm retained to perform compliance or internal audit functions for an entity;

## Summary Statistics on Citations in the Whistleblower Rule

A slightly different measure than efficacy is how many times the organization is cited by the Securities and Exchange Commission in the preambles of the final rules.  This table reveals three interesting patterns.  First, the majority of all trade associations, coalitions, corporations, and law firms who submitted comments were cited.  Second, comments from trade associations and coalitions were cited extensively.  Third, although a few citizens were cited, the modal citations on a comment from a citizen was 0. 

**Type of Organization**|**Cited**|**Out of**|**Avg Citations**|**Cited Once**
:-----:|:-----:|:-----:|:-----:|:-----:
Associations|34|39|12.2|4
Citizens|13|136|4.4|6
Coalitions|6|7|14|0
Corporations|11|16|5.1|0
Law Firms|22|29|4.4|8

: Citations to Comments on the Whistleblower Rule {#tbl-whistleblower}




<!-- OLD # Validation of text re-use as a measure of lobbying success


Below, we shed light on why several comments have efficacy values that are particularly high.

1)  <https://www.sec.gov/comments/s7-33-10/s73310-110.pdf>, Efficacy score = 10131

This letter is unusual in that its members (Americans for Limited Government, Ryder Systems, Inc., Financial Services Institute, Inc., U.S. Chamber of Commerce, Verizon, and White & Case, LLP) are all part of multiple coalitions and submitted multiple comments. The Chamber of Commerce, for example, submitted five comments and attended seven meetings and was a part of two separate coalitions.

2)  <https://www.sec.gov/comments/s7-33-10/s73310-35.pdf>, Efficacy score = 10131

This is the same letter as above, submitted twice to the SEC although recorded by the SEC as a separate comment. Reassuringly it has the same efficacy!

3)  <https://comments.cftc.gov/Handlers/PdfHandler.ashx?id=24275>, Efficacy score = 7593

4)  <https://comments.cftc.gov/Handlers/PdfHandler.ashx?id=24205>, Efficacy score = 7411

5)  <https://www.sec.gov/comments/s7-41-11/s74111-230.pdf>, Efficacy score = 7312

The following three comments (<https://comments.cftc.gov/Handlers/PdfHandler.ashx?id=24275>, <https://comments.cftc.gov/Handlers/PdfHandler.ashx?id=24205>, <https://www.sec.gov/comments/s7-41-11/s74111-230.pdf>) are all related. The CFTC, OCC, FRB, FDIC, and SEC all worked together to develop rules to prohibit banking companies from engaging in proprietary trading (trading securities with their own money instead of clients'). While all five agencies worked together to write a final rule, in the end the CFTC wrote one rule and the OCC, FRB, FDIC, and SEC wrote another. Despite the different final rules, the agencies "the CFTC and the other agencies have worked closely together to develop the same rule text and supplementary information, except for information specific to the CFTC or the other agencies, as applicable."

The comment <https://comments.cftc.gov/Handlers/PdfHandler.ashx?id=24275> is a 340-page letter from Occupy the SEC, a group affiliated with Occupy Wall Street, submitted on the CFTC's version of the final rule. It was cited 285 times in the preamble to the final rule.

The comment <https://comments.cftc.gov/Handlers/PdfHandler.ashx?id=24205> is a 325-page letter from Occupy the SEC and was submitted on the CFTC's version of the final rule. It was cited 284 times in the preamble to the final rule.

The comment <https://www.sec.gov/comments/s7-41-11/s74111-230.pdf> is a 325-page letter from Occupy the SEC and was submitted to the OCC, FRB, FDIC, and SEC's version of the rule. It was cited 284 times in the preamble to the final rule.

6)  <https://www.sec.gov/comments/s7-25-11/s72511-55.pdf>, Efficacy score = 5809

SIFMA is cited over 500 times by the CFTC in the preamble to the final rule.

7)  <https://www.sec.gov/comments/s7-18-11/s71811-32.pdf>, Efficacy score = 5563

Standard & Poors's letter to the SEC is cited over 200 times by the SEC in the preamble to the final rule.

8)  <https://comments.cftc.gov/Handlers/PdfHandler.ashx?id=25016>, Efficacy score = 4965

Not only did the FIA help develop the guidelines that were codified into the final rule, the FIA's letter was cited 98 separate times by the CFTC in the preamble to the final rule.
-->

{{< include assets/efficacy_appendix.txt >}}