Moving towards validating group names

[tobias]

This conversation started on the Clojurians Slack in response to concerns about Clojars being an avenue for shadowing internal library names.

I'm moving that discussion here to have a longer-term record of it and to invite others to contribute.

---

tcrawley: I've been thinking about this for a bit. I don't think it would be too difficult/onerous to move to a safer model. I hope to have time to write up a proposal for discussion this weekend.

dominicm: The key part is the validation right?
dominicm: I suppose some easy validations for Clojars without impeding users too much would be GH OAuth, etc.
dominicm: The sonatype process is a little onerous, requiring the creation of jira tickets (and we know how much clojurists hate jira) so I think clojars could automate that.

tcrawley: Well, I didn't have a chance to write up my thoughts this past weekend, but they are, roughly:

This is indeed a validation problem - we want to validate group names.

• org.clojars. groups - we don't have to do anything here, since we are already validating these
• com.github. groups - validating these would be straightforward since we already have "Log in with GitHub". We would just need to add some logic there, then deny deploys to those groups until someone that is group owner (or the group creator, if a push will create the group) logs in via GitHub with that username
• other reverse-hostname groups - We could validate these automatically with a TXT record. I think initially I would prefer doing so manually with folks filing GH issues, since that would let us see what validation methods work for folks, and wouldn't require as much upfront implementation (so could be released sooner - my availability to work on Clojars is essentially a small amount of time on the weekends ATM)
• That leaves non-reverse-hostname groups - these are (generally) less likely to shadow internal artifacts, especially when the group and artifact name are the same (cheshire/cheshire , hiccup/hiccup, etc). These require a bit of thought on how to validate. We could:
  • Allow deployment to continue to existing ones, but don't allow new ones to be created, pushing folks to use reverse-hostname groups
  • Allow them to be created unvalidated. This isn't a great option, since it would still allow potential shadowing, and would push more folks to using these types of groups since reverse-hostname groups now require validation, and it would be good for us to go the other way as a community.

An open question is do we disallow all deploys to an unvalidated group? Or just forbid creation of new artifacts until validation? Or just disallow new group creation until validation?

Another easier (implementation-wise) but more painful (for the community) option is to make clojars read-only, and have folks switch to deploying to OSSRH. That would require lots more work from library maintainers, and would change the names of almost everything, so probably isn't a great option. But it would move us to a more vetted/trusted model, and would take some of the burden of maintaining our own package system off of me/the community.

That's my $0.02 currently. I would love feedback on this from anyone interested. I can take the discussion here and move it to: https://github.com/clojars/clojars-web/discussions later

alexmiller: personally, I'd like to add friction/resistance to adding new foo/foo names and do whatever we can to encourage either hostname (or copyright/trademark "ownership") or conferred identity via hosts like github

tcrawley: I agree

alexmiller: I do not think we need or want to push people to OSSRH, it really is significantly more onerous due to several rules
alexmiller: not just signing, but silly things for clojurians like source/javadocs

tcrawley: I agree there as well, but wanted to present it because it is an option

[jcsims]

The domain maps to a known, managed, verifiable identifier for a company. Living in the JVM ecosystem, and the fact that Maven Central follows this same pattern, is attractive.

[jcsims]

A follow-on question to this how someone might claim a Group on Clojars, without requiring a deployed artifact (in this case, specifically to avoid shadowing of internal artifacts).

[tobias]

Good question! There are a couple of options:

• You can add the groupname to this list via a PR. But this would have to be undone if you ever did want to use the group.
• You can open an issue and an admin can create the group in the db with you as the owner. That will prevent anyone else from pushing to it unless you delegate rights to them.

[nnichols]

I like all of the reverse hostname formatted options, and I think pushing independent contributors/non-org groups towards com.github or com.gitlab probably incurs the least overhead for creating new groups. Additionally, the DNS record validation is a common enough pattern in artifact repositories IMO that it's pretty low-friction to adopt.

Re: Leaving non-reverse-hostname groups non-validated, I think the existing groups will have to stay in their existing state- lest there becomes some huge community effort to migrate all a ton of major libraries. I like that Clojars deployment is as low-friction as it currently is, and I think it's helped drive the OSS community around Clojars. It's probably less likely for these deps to be shadowed, but I would want a mechanism or something to promote using a "validated" group name. Maybe something like the verified commit badge?

[tobias]

I like the idea of a verified badge for verified groups.

[timothypratley]

I for one would welcome a "verified group ownership via clojars or github or dns" requirement; I think it is reasonable, clarifies expectations for library writers and consumers, and improves security.

[tobias]

Thanks for the feedback Tim!

[tobias]

Here is a concrete proposal for discussion, along with the work required to implement it. Any and all feedback is welcome.

---

Implementation of this proposal would put Clojars in a place where:

• all new groups are domain-based & verified
• all new libraries created in existing domain-based groups would require the group to be verified
• no new libraries can be created in non-verified groups
• no non-domain-based groups can be verified
• existing libraries can still have releases without group verification

1. Groups

We have a few different classes of groups when it comes to verification requirements and methods.

A. org.clojars.<username> groups

These are already verified, since they can only be created by the user they name. That user can then delegate access to that group to anyone, the same as any other group. Historically, these groups have been used for non-canonical forks/playgrounds. We currently create the group for a user when the user account is created.

B. net.clojars.<username> groups

These don't exist currently, but we own the clojars.net domain, so can support these as groups verified in the same manner as the org.clojars.* groups. This provides a group that doesn't have the baggage of being seen as fork/playground groups. This will require small changes to the upload and new user creation logic to create them. [#780][#781]

C. com.github.<gh-user-or-org-name> and io.github.<gh-user-or-org-name> groups

Clojars supports logging in via GitHub, so it should be straightforward to use that as a verification mechanism for user-based naming (https://github.com/tobias/, for example); the Clojars user that owns that GitHub account would need to log in to Clojars via GitHub auth, and would need to be a member of the Clojars group. If both are true, we could mark the group as verified. If the group does not exist, we would create it. [#784]

Existing libraries with github-based groups can have new versions deployed without verification. Creating new libraries under an existing group (same group name, new artifact name) won't be allowed if the group isn't verified.

i. Verifying GitHub org groups

We will need a different verification system when the GitHub name corresponds to a GitHub organization (https://github.com/clojars/, for example). These will be automated manually by Clojars admins until a better verification method emerges. Users will need to file an issue with Clojars to verify.

One approach is to have users create a repo with a special name to prove organization ownership (this is the method OSSRH uses).

D. com.gitlab.<gl-user-or-org-name> groups

These groups will work exactly like the GitHub groups, except Clojars doesn't yet support logging in via GitLab. [#786][#787]

E. Other reverse domain groups

i. Existing groups

Existing libraries with reverse domain groups (org.tcrawley, for example) can have new versions deployed without verification. Creating new libraries under an existing group (smae group name, new artifact name) won't be allowed if the group isn't verified.

ii. New groups

New groups must be verified before any libraries can be deployed to the group.

iii. Domain verification

We would primarly verify domains via DNS: this could be done by providing a string value to add as a TXT record for the domain. Verifying a domain or sub-domain would allow groups using that domain/sub-domain or any of its sub-domains. For example, verifying tcrawley.org would verify org.tcrawley, org.tcrawley.foo, org.tcrawley.foo.bar, etc. as groups. Verifying ham.tcrawley.org would verify org.tcrawley.ham, org.tcrawley.ham.biscuit, etc. as groups, but not org.tcrawley.

DNS verification will initially be done manually. Users will file issues on the Clojars repo to initiate the process. This may be automated in the future if it becomes burdensome or there is enough community interest to implement it.

F. Non-domain groups

i. Existing groups

Deploys of new versions of existing libraries in existing non-domain groups will be allowed to continue. This will prevent disruption of authors doing ongoing releases, and will prevent the confusion/large effort that would come from a forced group rename. Creating new libraries under an existing group (same group name, new artifact name) won't be allowed if the group isn't verified. Verification of existing groups will be a manual, case-by-case process, and will likely be based on trademark or uniqueness of the name. For example, we would likely verify https://clojars.org/groups/clj-commons but not https://clojars.org/groups/progress. Users will file issues on the Clojars repo to initiate the process. Creating new libraries under an existing group (same group name, new artifact name) will not be allowed. Verification of existing groups will not be allowed; authors will need to use a verified, domain-based group for new libraries.

ii. New groups

No new groups that aren't reverse-domain based can be created. This will require a small change to the upload logic to prevent creation of the group [#790]. We may relax this if we get community feedback, but only if we can come up with a mechanism for verifying these names.

2. Other work

• All verified groups would show a verified badge in the UI and would have a verified: true property in the API [Add verified badge to UI #785][Add verified flag to API and stat downloads #783]
• We may need to implement an audit log within the application so users can know why their deploys are being rejected. We have relied on the HTTP status message to relay that in the past, but the current version of lein (and possibly other tools) uses a version of wagon-http that won't display the status message. Regardless, an audit log would be generally useful. [Add deploy audit log #789]
• Documentation on deploying in the Clojars wiki and in Leiningen's tutorial will need to be updated [Update clojars and leiningen deploy documentation #792]
• This change will need to be communicated with the community in advance

3. Concerns

• There will be confusion among users who are trying to deploy after this goes out, especially if they haven't seen the communications about it in advance and/or won't be able to see the status message notifications about why deployments are failing. They will need help on the Clojurians slack, Clojureverse, the mailing list, and possibly other places.
• The burden of doing manual verifications is unknown; the Clojars admins may ask for volunteers to help with that process if the burden is high.

Edit History:

• 2020-02-23 9:54 UTC - numbered headers to ease discussion
• 2020-02-24 0:17 UTC - changed non-domain groups to not be verified in 1.F.i
• 2020-02-24 0:30 UTC - clarify goals in first para
• 2020-02-24 0:45 UTC - removed option of verifying new non-domain-based groups in 1.F.ii
• 2020-02-24 1:00 UTC - added links to implementation issues
• 2020-20-28 14:20 UTC - added io.github.* groups

[seancorfield]

Presumably because they verify via something in the web page, rather than via authorization through GitHub login?

[puredanger]

FYI, github is soon going to decommission gh-pages access from github.com (only github.io will be supported). No idea if that plays into anything, but verifying something on a page via github.com will likely no longer work in the near future (if that's how it did).

[seancorfield]

"verifying something on a page via github.com will likely no longer work" -- that could well be behind Maven's decision but it sounds like the Clojars team are looking to verify ownership of com.github.<username> via GitHub auth to Clojars?

Also @puredanger since you're here, your blog post about the CLI exec stuff talks about deprecating unqualified libs and says:

In cases where you have a lib with no domain name or trademark, you can use a third party source of identity (like github) in combination with an id you control on that site, so your lib id would be github-yourname/yourlib. Using a dashed name is preferred over a dotted name as that could imply a library owned by github.

I'm not sure if that suggested group ID pattern is anywhere else (on clojure.org for example) but that gets linked a lot when discussion about the unqualified lib warning comes up so, once Clojars has settled on an official pattern, we might want to do a pass over stuff and update such recommendations.

Edit to add: yes, that suggestion is in both the guide and the reference for deps/CLI on clojure.org.

[tobias]

Regarding com.github., maven has thus far encouraged io.github.. Might be nice to align with them on that.

Thanks for bringing that up @SevereOverfl0w. I saw that as well in the OSSRH docs, but when looking through approval requests on Jira, I saw mostly com.github.. I ran a query on the groups table for Clojars, and it looks like we currently have 47 com.github. groups and 18 io.github. ones. I'm fine with providing both - I don't think there is any harm in that.

[puredanger]

Re com.github vs io.github ... I think io.github should be the preferred form for this. Maven Central does verification of github accounts by asking you to create a repo with a requested name which then shows up at https://myorg.github.io/name. This kind of verification will soon no longer work for github.com domains and thus that will likely become the dominant form in Maven Central. (I realize that if you are using login verification clojars could check either way.)

@seancorfield I will be updating that guidance, thanks

[tobias]

This plan has been announced - let's move the conversation to the discussion set up for feedback at clojars/administration#2. I'm locking this discussion in favor of that one.