Pre-PLIP: autoinclude zcml, new style

[mauritsvanrees]

Copied from the Alpine City Strategic Sprint 2020 doc:

Plone on pip

pip install Plone will work with Plone 5.2.1. But z3c.autoinclude can fail when add-ons use includeDependencies in zcml. And buildout should get an option/extension to not install any Python packages anymore. Or we switch away from buildout to something else (Ansible?), but there are still all kinds of nice buildout recipes.
Integrating and finishing https://github.com/datakurre/plonectl
Some initial work by Six Feet Up https://github.com/sixfeetup/dietplonedocker
Good example is the warehouse project how to deploy.
Ansible role should be updated too, to be able to run without buildout. Maybe we could create an even lighter version of ansible.plone_server role, which does just the Plone setup, nothing more.

Who:

• Maurits van Rees
• Maik Derstappen
• Jens Klein
• Thomas Schorr
• Calvin Hendryx-Parker (Remote)

Prediscussion for PLIP:

Discourage use of z3c.autoinclude: Problem here is:

• pip installed packages can not be translated to dottednames
• it tries to translate a egg/wheel name as in PyPI to a dottedname
• the code does rely on internals how eggs are installed.
• this is slow and not easy to fix, if at all.

We want pip installed packages automatically picked up at Zope startup
Idea:

• introduce an entry point where we have a dottednames and filenames to load:

[zope.autoincludezcml]
package = collective.mypackage
load = configure.zcml,overrides.zcml 

• this would need to introduce a new package to be created.
• Maurits wonders if it would mostly work already with the current entry points. You get the entrypoints by calling pkg_resources.iter_entry_points(group="z3c.autoinclude.plugin") and then for each you can get entrypoint.dist, which seems to give enough info about the package.

Maybe try this out in a branch of CMFPlone, because that is where z3c.autoinclude is currently loaded, for configure.zcml and overrides.zcml.

[mauritsvanrees]

In coredev 5.2 with bin/zopepy:

>>> import pkg_resources
>>> from pprint import pprint as pp
>>> pp([(ep.dist.project_name, ep.dist.key) for ep in list(pkg_resources.iter_entry_points(group="z3c.autoinclude.plugin"))])
[('plone.app.upgrade', 'plone.app.upgrade'),
 ('plone.restapi', 'plone.restapi'),
 ('mockup', 'mockup'),
 ('plone.app.caching', 'plone.app.caching'),
 ('plone.app.contentlisting', 'plone.app.contentlisting'),
 ('plone.app.contenttypes', 'plone.app.contenttypes'),
 ('plone.app.dexterity', 'plone.app.dexterity'),
 ('plone.app.discussion', 'plone.app.discussion'),
 ('plone.app.event', 'plone.app.event'),
 ('plone.app.intid', 'plone.app.intid'),
 ('plone.app.iterate', 'plone.app.iterate'),
 ('plone.app.linkintegrity', 'plone.app.linkintegrity'),
 ('plone.app.lockingbehavior', 'plone.app.lockingbehavior'),
 ('plone.app.multilingual', 'plone.app.multilingual'),
 ('plone.app.querystring', 'plone.app.querystring'),
 ('plone.app.theming', 'plone.app.theming'),
 ('plone.app.users', 'plone.app.users'),
 ('plone.app.versioningbehavior', 'plone.app.versioningbehavior'),
 ('plone.app.widgets', 'plone.app.widgets'),
 ('plone.batching', 'plone.batching'),
 ('plone.formwidget.namedfile', 'plone.formwidget.namedfile'),
 ('plone.outputfilters', 'plone.outputfilters'),
 ('plone.portlet.static', 'plone.portlet.static'),
 ('plone.resource', 'plone.resource'),
 ('plone.rest', 'plone.rest'),
 ('plone.staticresources', 'plone.staticresources'),
 ('plone.stringinterp', 'plone.stringinterp'),
 ('plone.subrequest', 'plone.subrequest'),
 ('plonetheme.barceloneta', 'plonetheme.barceloneta'),
 ('Products.CMFDiffTool', 'products.cmfdifftool'),
 ('Products.CMFEditions', 'products.cmfeditions'),
 ('archetypes.multilingual', 'archetypes.multilingual'),
 ('archetypes.schemaextender', 'archetypes.schemaextender'),
 ('plone.app.blob', 'plone.app.blob'),
 ('plone.app.collection', 'plone.app.collection'),
 ('plone.app.imaging', 'plone.app.imaging'),
 ('plone.app.referenceablebehavior', 'plone.app.referenceablebehavior'),
 ('plone.formwidget.recurrence', 'plone.formwidget.recurrence'),
 ('collective.MockMailHost', 'collective.mockmailhost'),
 ('z3c.formwidget.query', 'z3c.formwidget.query')]

[mauritsvanrees]

For fun, this is the list of all packages of which the configure.zcml is loaded, some explicitly by the configure.zcml in Products.CMFPlone (explicit.txt below), others by z3c.autoinclude (auto.txt).
So:

• the ones with a minus-sign are explicitly loaded by CMFPlone
• the ones with a plus-sign are implicitly loaded by includePlugins
• the ones with no sign are both explicitly and implicitly loaded, so we could actually remove the autoinclude snippet from their setup.py. Most of those are packages that started life outside of the core, but are now in.

$ colordiff -U 30 explicit.txt auto.txt 
--- explicit.txt	2020-02-24 21:59:46.000000000 +0100
+++ auto.txt	2020-02-24 21:59:02.000000000 +0100
@@ -1,36 +1,40 @@
-Products.CMFCore
-Products.GenericSetup
-borg.localrole
+Products.CMFDiffTool
+Products.CMFEditions
+archetypes.multilingual
+archetypes.schemaextender
+collective.MockMailHost
 mockup
-plone.app.content
-plone.app.contentmenu
-plone.app.contentrules
+plone.app.blob
+plone.app.caching
+plone.app.collection
+plone.app.contentlisting
 plone.app.contenttypes
-plone.app.customerize
+plone.app.dexterity
 plone.app.discussion
-plone.app.i18n
-plone.app.layout
+plone.app.event
+plone.app.imaging
+plone.app.intid
+plone.app.iterate
 plone.app.linkintegrity
-plone.app.locales
+plone.app.lockingbehavior
 plone.app.multilingual
-plone.app.portlets
-plone.app.redirector
-plone.app.registry
+plone.app.querystring
+plone.app.referenceablebehavior
 plone.app.theming
+plone.app.upgrade
 plone.app.users
-plone.app.uuid
-plone.app.viewletmanager
-plone.app.vocabularies
-plone.app.workflow
+plone.app.versioningbehavior
+plone.app.widgets
 plone.batching
-plone.browserlayer
-plone.indexer
-plone.memoize
+plone.formwidget.namedfile
+plone.formwidget.recurrence
 plone.outputfilters
-plone.portlet.collection
 plone.portlet.static
-plone.protect
-plone.session
+plone.resource
+plone.rest
+plone.restapi
 plone.staticresources
-plone.theme
-zope.app.locales
+plone.stringinterp
+plone.subrequest
+plonetheme.barceloneta
+z3c.formwidget.query

[mauritsvanrees]

See this thread on Twitter started by @jensens with an analysis of Plone startup time:

#Plone Startup time (lots addons)
16.29s (Zope2/Startup/serve.py)
with biggest part
11.09s includePluginsDirective (z3c/autoinclude/zcml.py)
with
8.53s find_packages (z3c/autoinclude/utils.py)
mainly file system-operations on a NVMe SSD

Lets get rid of z3c.autoinclude

[jensens]

pp([(ep.dist.project_name, ep.dist.key) for ep in list(pkg_resources.iter_entry_points(group="z3c.autoinclude.plugin"))])

You get here dotted paths because in lots o cases projectname == dottedpath. But this is not mandatory, the dist name can be different from the dottedpath or it may contain several paths (look at the Zope package or Pillow)

[mauritsvanrees]

I was looking for a way to print which packages actually get auto included. I decided to put this in z3c.autoinclude itself. Set an environment variable Z3C_AUTOINCLUDE_DEBUG, startup Zope, and it prints the info in a way that you can copy-paste into zcml. That helps for migrating to explicitly including the zcml.

See zopefoundation/z3c.autoinclude#13

[mauritsvanrees]

I wanted to try something, to see what kind of information we can get with pkg_resources, and if that would be enough for our use cases.

$ python3.7 -mvenv plonepip
$ cd plonepip
$ . bin/activate
$ pip install -c https://dist.plone.org/release/5.2.2-pending/constraints.txt Plone
...
$ python
>>> import pkg_resources
>>> pkg_resources.working_set
<pkg_resources.WorkingSet object at 0x10bed5090>
>>> len(list(pkg_resources.working_set))
244
>>> list(pkg_resources.working_set)[0]
Zope2 4.0 (/Users/maurits/tmp/plonepip/lib/python3.7/site-packages)
>>> list(pkg_resources.working_set)[0].__class__
<class 'pkg_resources.DistInfoDistribution'>
>>> for dist in pkg_resources.working_set:
...     entry_map = pkg_resources.get_entry_map(dist)
...     if "z3c.autoinclude.plugin" in entry_map:
...         print("{}: {}".format(dist, entry_map["z3c.autoinclude.plugin"]))
... 
z3c.formwidget.query 0.17: {'target': EntryPoint.parse('target = plone')}
Products.CMFEditions 3.3.4: {'target': EntryPoint.parse('target = plone')}
Products.CMFDiffTool 3.3.1: {'target': EntryPoint.parse('target = plone')}
plonetheme.barceloneta 2.1.8: {'target': EntryPoint.parse('target = plone')}
plone.subrequest 1.9.2: {'target': EntryPoint.parse('target = plone')}
plone.stringinterp 1.3.2: {'target': EntryPoint.parse('target = plone')}
plone.staticresources 1.3.1: {'target': EntryPoint.parse('target = plone')}
plone.restapi 6.13.7: {'target': EntryPoint.parse('target = plone')}
plone.rest 1.6.1: {'target': EntryPoint.parse('target = plone')}
plone.resource 2.1.2: {'target': EntryPoint.parse('target = plone')}
plone.portlet.static 3.1.4: {'target': EntryPoint.parse('target = plone')}
plone.outputfilters 4.0.1: {'target': EntryPoint.parse('target = plone')}
plone.formwidget.recurrence 2.1.4: {'target': EntryPoint.parse('target = plone')}
plone.formwidget.namedfile 2.1.0: {'target': EntryPoint.parse('target = plone')}
plone.batching 1.1.6: {'target': EntryPoint.parse('target = plone')}
plone.app.widgets 3.0.4: {'target': EntryPoint.parse('target = plone')}
plone.app.versioningbehavior 1.4.0: {'target': EntryPoint.parse('target = plone')}
plone.app.users 2.6.5: {'target': EntryPoint.parse('target = plone')}
plone.app.upgrade 2.0.33: {'target': EntryPoint.parse('target = plone')}
plone.app.theming 4.1.2: {'target': EntryPoint.parse('target = plone')}
plone.app.querystring 1.4.14: {'target': EntryPoint.parse('target = plone')}
plone.app.multilingual 5.6.1: {'target': EntryPoint.parse('target = plone')}
plone.app.lockingbehavior 1.0.7: {'target': EntryPoint.parse('target = plone')}
plone.app.linkintegrity 3.3.13: {'target': EntryPoint.parse('target = plone')}
plone.app.iterate 3.3.14: {'target': EntryPoint.parse('target = plone')}
plone.app.intid 1.1.4: {'target': EntryPoint.parse('target = plone')}
plone.app.event 3.2.7: {'target': EntryPoint.parse('target = plone')}
plone.app.discussion 3.4.2: {'target': EntryPoint.parse('target = plone')}
plone.app.dexterity 2.6.5: {'target': EntryPoint.parse('target = plone')}
plone.app.contenttypes 2.1.9: {'target': EntryPoint.parse('target = plone')}
plone.app.contentlisting 2.0.2: {'target': EntryPoint.parse('target = plone')}
plone.app.caching 2.0.6: {'target': EntryPoint.parse('target = plone')}
mockup 3.2.1: {'target': EntryPoint.parse('target = mockup')}

Get the interesting ones:

>>> dists = [dist for dist in pkg_resources.working_set if "z3c.autoinclude.plugin" in pkg_resources.get_entry_map(dist)]
>>> len(dists)
33
>>> dist = dists[0]
>>> dist
z3c.formwidget.query 0.17 (/Users/maurits/tmp/plonepip/lib/python3.7/site-packages)

Show attributes and simple callables of a distribution:

>>> def dist_info(dist):
...     for attr in dir(dist):
...         if not attr.startswith("_"):
...             attrib = getattr(dist, attr)
...             if callable(attrib):
...                 try:
...                     attrib = attrib()
...                 except:
...                     continue
...             print("{}: {}".format(attr, attrib))
>>> dist_info(dist)
EQEQ: re.compile('([\\(,])\\s*(\\d.*?)\\s*([,\\)])')
PKG_INFO: METADATA
activate: None
as_requirement: z3c.formwidget.query==0.17
check_version_conflict: None
clone: z3c.formwidget.query 0.17
egg_info: /Users/maurits/tmp/plonepip/lib/python3.7/site-packages/z3c.formwidget.query-0.17.dist-info
egg_name: z3c.formwidget.query-0.17-py3.7
extras: ['test']
get_entry_map: {'z3c.autoinclude.plugin': {'target': EntryPoint.parse('target = plone')}}
has_version: True
hashcmp: (<Version('0.17')>, -1, 'z3c.formwidget.query', '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages', '', '')
key: z3c.formwidget.query
loader: None
location: /Users/maurits/tmp/plonepip/lib/python3.7/site-packages
module_path: /Users/maurits/tmp/plonepip/lib/python3.7/site-packages
parsed_version: 0.17
platform: None
precedence: -1
project_name: z3c.formwidget.query
py_version: None
requires: [Requirement.parse('zope.schema'), Requirement.parse('setuptools'), Requirement.parse('zope.i18nmessageid'), Requirement.parse('z3c.form>=3.2.10'), Requirement.parse('zope.component'), Requirement.parse('zope.interface')]
version: 0.17

Can we import these by project_name?

>>> import importlib
>>> for dist in dists:
...    importlib.import_module(dist.project_name)
... 
<module 'z3c.formwidget.query' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/z3c/formwidget/query/__init__.py'>
<module 'Products.CMFEditions' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/Products/CMFEditions/__init__.py'>
<module 'Products.CMFDiffTool' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/Products/CMFDiffTool/__init__.py'>
<module 'plonetheme.barceloneta' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plonetheme/barceloneta/__init__.py'>
<module 'plone.subrequest' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/subrequest/__init__.py'>
<module 'plone.stringinterp' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/stringinterp/__init__.py'>
<module 'plone.staticresources' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/staticresources/__init__.py'>
<module 'plone.restapi' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/restapi/__init__.py'>
<module 'plone.rest' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/rest/__init__.py'>
<module 'plone.resource' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/resource/__init__.py'>
<module 'plone.portlet.static' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/portlet/static/__init__.py'>
<module 'plone.outputfilters' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/outputfilters/__init__.py'>
<module 'plone.formwidget.recurrence' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/formwidget/recurrence/__init__.py'>
<module 'plone.formwidget.namedfile' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/formwidget/namedfile/__init__.py'>
<module 'plone.batching' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/batching/__init__.py'>
<module 'plone.app.widgets' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/widgets/__init__.py'>
<module 'plone.app.versioningbehavior' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/versioningbehavior/__init__.py'>
<module 'plone.app.users' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/users/__init__.py'>
<module 'plone.app.upgrade' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/upgrade/__init__.py'>
<module 'plone.app.theming' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/theming/__init__.py'>
<module 'plone.app.querystring' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/querystring/__init__.py'>
<module 'plone.app.multilingual' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/multilingual/__init__.py'>
<module 'plone.app.lockingbehavior' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/lockingbehavior/__init__.py'>
<module 'plone.app.linkintegrity' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/linkintegrity/__init__.py'>
<module 'plone.app.iterate' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/iterate/__init__.py'>
<module 'plone.app.intid' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/intid/__init__.py'>
<module 'plone.app.event' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/event/__init__.py'>
<module 'plone.app.discussion' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/discussion/__init__.py'>
<module 'plone.app.dexterity' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/dexterity/__init__.py'>
<module 'plone.app.contenttypes' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/contenttypes/__init__.py'>
<module 'plone.app.contentlisting' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/contentlisting/__init__.py'>
<module 'plone.app.caching' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/plone/app/caching/__init__.py'>
<module 'mockup' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/mockup/__init__.py'>

So project_name works, at least for these.
And key?

>>> for dist in dists:
...     importlib.import_module(dist.key)
... 
<module 'z3c.formwidget.query' from '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages/z3c/formwidget/query/__init__.py'>
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/Users/maurits/.pyenv/versions/3.7.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'products'
>>> dist
Products.CMFEditions 3.3.4 (/Users/maurits/tmp/plonepip/lib/python3.7/site-packages)
>>> dist.key
'products.cmfeditions'
>>> dist.project_name
'Products.CMFEditions'

So importing dist.key does not work, but importing dist.project_name does. I don't know if that is true in all cases.
As Jens says above:

You get here dotted paths because in lots o cases projectname == dottedpath. But this is not mandatory, the dist name can be different from the dottedpath or it may contain several paths (look at the Zope package or Pillow)

Let's look at the Zope dist:

>>> dist = [dist for dist in pkg_resources.working_set if dist.project_name == 'Zope'][0]
>>> dist_info(dist)
EQEQ: re.compile('([\\(,])\\s*(\\d.*?)\\s*([,\\)])')
PKG_INFO: METADATA
activate: None
as_requirement: Zope==4.5
check_version_conflict: None
clone: Zope 4.5
egg_info: /Users/maurits/tmp/plonepip/lib/python3.7/site-packages/Zope-4.5.dist-info
egg_name: Zope-4.5-py3.7
extras: ['docs', 'wsgi']
get_entry_map: {'console_scripts': {'addzope2user': EntryPoint.parse('addzope2user = Zope2.utilities.adduser:main'), 'mkwsgiinstance': EntryPoint.parse('mkwsgiinstance = Zope2.utilities.mkwsgiinstance:main'), 'runwsgi': EntryPoint.parse('runwsgi = Zope2.Startup.serve:main'), 'zconsole': EntryPoint.parse('zconsole = Zope2.utilities.zconsole:main')}, 'paste.app_factory': {'main': EntryPoint.parse('main = Zope2.Startup.run:make_wsgi_app')}, 'paste.filter_app_factory': {'httpexceptions': EntryPoint.parse('httpexceptions = ZPublisher.httpexceptions:main')}, 'zodbupdate': {'renames': EntryPoint.parse('renames = OFS:zodbupdate_rename_dict')}, 'zodbupdate.decode': {'decodes': EntryPoint.parse('decodes = OFS:zodbupdate_decode_dict')}}
has_version: True
hashcmp: (<Version('4.5')>, -1, 'zope', '/Users/maurits/tmp/plonepip/lib/python3.7/site-packages', '', '')
key: zope
loader: None
location: /Users/maurits/tmp/plonepip/lib/python3.7/site-packages
module_path: /Users/maurits/tmp/plonepip/lib/python3.7/site-packages
parsed_version: 4.5
platform: None
precedence: -1
project_name: Zope
py_version: None
requires: [Requirement.parse('transaction>=2.4'), Requirement.parse('zope.pagetemplate>=4.0.2'), Requirement.parse('zope.traversing'), Requirement.parse('ZConfig>=2.9.2'), Requirement.parse('zope.contenttype'), Requirement.parse('zope.container'), Requirement.parse('zope.event'), Requirement.parse('ExtensionClass'), Requirement.parse('PasteDeploy'), Requirement.parse('z3c.pt'), Requirement.parse('six'), Requirement.parse('zope.configuration'), Requirement.parse('zope.security'), Requirement.parse('zope.interface>=3.8'), Requirement.parse('AccessControl>=4.2'), Requirement.parse('waitress'), Requirement.parse('zope.deferredimport'), Requirement.parse('zope.browserpage>=4.4.0.dev0'), Requirement.parse('zope.contentprovider'), Requirement.parse('zope.tales>=5.0.2'), Requirement.parse('zope.publisher'), Requirement.parse('zope.testing'), Requirement.parse('setuptools>=36.2'), Requirement.parse('zope.size'), Requirement.parse('zope.testbrowser'), Requirement.parse('RestrictedPython'), Requirement.parse('Chameleon>=3.7.0'), Requirement.parse('zope.ptresource'), Requirement.parse('zope.location'), Requirement.parse('zope.tal'), Requirement.parse('Persistence'), Requirement.parse('zope.lifecycleevent'), Requirement.parse('zope.schema'), Requirement.parse('zope.site'), Requirement.parse('zope.exceptions'), Requirement.parse('DocumentTemplate>=3.0b9'), Requirement.parse('Acquisition'), Requirement.parse('DateTime'), Requirement.parse('zope.component'), Requirement.parse('zope.browser'), Requirement.parse('zope.viewlet'), Requirement.parse('zope.proxy'), Requirement.parse('zope.sequencesort'), Requirement.parse('zope.processlifetime'), Requirement.parse('zExceptions>=3.4'), Requirement.parse('zope.i18n[zcml]'), Requirement.parse('zope.i18nmessageid'), Requirement.parse('MultiMapping'), Requirement.parse('zope.browserresource>=3.11'), Requirement.parse('BTrees'), Requirement.parse('ZODB'), Requirement.parse('zope.globalrequest'), Requirement.parse('zope.browsermenu')]
version: 4.5
>>> importlib.import_module(dist.project_name)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/maurits/.pyenv/versions/3.7.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'Zope'

So importing by project name indeed does not work for all distributions.

>>> for dist in pkg_resources.working_set:
...     try:
...         ignored = importlib.import_module(dist.project_name)
...     except ImportError:
...         print("Failed importing by project name: {}".format(dist.project_name))
... 
Failed importing by project name: Zope
Failed importing by project name: ZODB3
Failed importing by project name: WSGIProxy2
Failed importing by project name: WebTest
Failed importing by project name: WebOb
Failed importing by project name: Unidecode
Failed importing by project name: python-gettext
Failed importing by project name: python-dateutil
Failed importing by project name: pyScss
Failed importing by project name: PyJWT
Failed importing by project name: Plone
Failed importing by project name: Pillow
Failed importing by project name: PasteDeploy
Failed importing by project name: Markdown
Failed importing by project name: importlib-metadata
Failed importing by project name: Chameleon
Failed importing by project name: beautifulsoup4
Failed importing by project name: attrs

That is 18 packages, most of which do not have any zcml.

So in a z3c.autoinclude alternative it might be an option to require that the project name is importable.

[mauritsvanrees]

I just keep hoping that, given a distribution, pkg_resources or some other tool will give us what we need...

[mauritsvanrees]

BTW, it would also be useful if an entry point can signal that the Python code of a package needs to be loaded at startup, even when it has no zcml.
Currently, if you add a package in the instance eggs, its code does not get loaded. You can force load by putting it in the zcml instance option, or it happens when the package declares a z3c.autoinclude entrypoint. I am not sure from the top of my head if you get an error then when it has no actual zcml.

The only way to make this happen when no zcml is involved, is to put the package in the special Products namespace. This is what we do for Plone hotfixes: latest hotfix is called Products.PloneHotfix20200121.

So what I am hoping we can do with an entry point, using a variant of the notation from the initial comment:

# Import the package by name, using its dist.project_name:
[zope.autoinclude]

# Import the package by the given package name:
[zope.autoinclude]
package = collective.mypackage

# For importing multiple packages, like in Zope:
[zope.autoinclude]
package = package1,package2,package3

# Import the package and load zcml:
[zope.autoinclude]
package = collective.mypackage
zcml = configure.zcml,overrides.zcml 

# When loading multiple packages, maybe we could support importing a different zcml per package,
# but importing multiple zcmls in one package then gets difficult to express:
[zope.autoinclude]
package = package1:configure.zcml,package2:overrides.zcml

Note that currently you need to add a target:

[z3c.autoinclude.plugin]
target = plone

There is code in Plone to only load the zcml of entry points that have target plone. So we may need to add a target as well in the new situation. Theoretically Zope could load zcml targeted at zope, or a plain CMF site could load zcml targeted at cmf. But the only target that I see other than plone, is mockup, which is in mockup itself so I wonder if that is a mistake.

Also note that, as far as I know, only Plone uses z3c.autoinclude, so maybe the target is overkill. And we could make a function that looks for entry points that either explicitly have target = plone or have no target.

Enough brain dump for today. :-)

[mauritsvanrees]

Okay, I have some initial code for a possible plone.autoinclude, very much Work In Progress:

https://github.com/plone/plone.autoinclude

[mauritsvanrees]

I improved some more. If anyone wants to try it out (please not on production...) I have updated the README. Especially see the section on installing with pip.

Status:

• It is for Python 3.6+, I test on 3.8.
• Works in a buildout for the core Plone packages that have the z3c.autoinclude entry point (most should not have this, but that is beside the point). Needs some changes in CMFPlone or additional zcml in the buildout so z3c.autoinclude is not used but only our code is used.
• Works in a pip-installed Plone and bin/mkwsgiinstance for the core Plone packages that have the z3c.autoinclude entry point.
• There are some important TODO items.

I am also experimenting with tox, not yet for testing, but for QA (lint), isort/black, release. See tox.ini. I am a bit inspired by https://github.com/zopefoundation/meta.

Note that there is no master branch, but only a main branch. GitHub recommends it and has it in the instructions when you create a new repository. Let's see.

[mauritsvanrees]

See CMPlone branch plone-autoinclude for the changes that would be needed to make the switch from z3c.autoinclude to plone.autoinclude.

[mauritsvanrees]

A lot of progress was made by @tschorr and me during the 'not an Alpine City Sprint' sprint. Mostly tests. With tox you can run them all locally. Takes less than 30 seconds for me when run in parallel (tox -p auto). GitHub Actions is setup for this as well. We have several test packages in the repository. Some tox tests install them with pip, others in a buildout. Both work. Tested on Python 3.6-3.9 plus PyPy3.

Biggest change to the actual code in plone.autoinclude is that we now support an own EntryPoint plone.autoinclude.plugin. I had some ideas for that in a comment above. But entry points are less flexible than I had hoped, so those ideas are for the most part not possible. See comments in the load_own_packages function. The summary is that entry points can only have one option: one key-value pair.

Let me give some examples of entry points in setup.py.

An empty EntryPoint is completely ignored, you cannot find it with pkg_resources.iter_entry_points.

# ignored
[plone.autoinclude.plugin]

So you must pass an option. And it must have a value, otherwise you get an error when pip or buildout installs it:

# error
[plone.autoinclude.plugin]
target =

You can pass a target, like z3c.autoinclude supports:

[plone.autoinclude.plugin]
target = plone

Or when your package is called A and it has module B, you can specify a module name:

[plone.autoinclude.plugin]
module = B

In this case you do not specify a target. In our code this is no problem: when we look for all entry points with a specific target (plone) and no target is set, we still include it.

In fact you cannot specify both a target and a module if you wanted to.
One of the two is ignored by pkg_resources and is invisible to us:

[plone.autoinclude.plugin]
target = plone
# ignored:
module = B

TODO:

• Write more unit tests.
• Add more test packages for corner cases.
• Write documentation.
• Release plone.autoinclude so it can be field tested. It needs Python 3. Should work fine on both Plone 5.2 and 6.
• Make PR for CMFPlone 6 to start using it. There is a branch but it is based on 5.2. Changes are small.

[jensens]

Hmmm. What about making module = .. mandatory? Otherwise it is very confusing IMO.

[tschorr]

In fact you cannot specify both a target and a module if you wanted to.
One of the two is ignored by pkg_resources and is invisible to us:

[plone.autoinclude.plugin]
target = plone
# ignored:
module = B

I wondered about this because you can e.g. specify multiple console script entry points. I've played with a different way of reading the entry maps in plone/plone.autoinclude#1 and I can read both target and module with my variant. I tried to keep the current semantics to keep the tests green.

[tschorr]

On second thought, instead of using pkg_resources for iterating over entry points, we could use importlib.metadata.entry_points(). That would remove setuptools from the list of requirements.

... but that API is only available since Python 3.8.7 :-(

[mauritsvanrees]

The way @tschorr reads the entry maps works, so we could indeed use both a target and module. I have merged it. We need a test package using that, but can be done later.

[mauritsvanrees]

I have released 1.0.0a1 of plone.autoinclude.
I think it is ready for inclusion in core Plone 6, and I will make a proper PLIP for it.
We may want to have this battle-tested more before including it, but there is 100 percent test coverage.

[mauritsvanrees]

I have released version 1.0.0a2, which I tested in a customer project.

I will close this 'pre-PLIP' in favour of a proper PLIP: #3339.