Document the requisite internal structure of module files for Flit building

[tmo1]

I was inspired by @takluyver's graciousness in his responses to #664 to report my new user experience with Flit.

Background: I have considerable (hobbyist) experience coding (scripting) in Python (almost ten years of Advent of Code!), but none at all in packaging Python code. Upon seeing Flit included in the tool recommendations of the Python Packaging documentation ("minimal and opinionated" sounded about right to me in this context), I decided to give it a whirl, only to encounter a seemingly bizarre error: Flit was apparently trying to run my script under its own name (and terminating abnormally since the script wasn't being provided its proper arguments).

After a bunch more reading, I concluded that Flit's build process includes the importing of the provided modules, and that Flit (like most Python packaging tools?) apparently expects its provided modules' code to be esconced inside functions (with the possible inclusion of an if __name__ == "__main__:" clause), and thus importing the modules shouldn't actually run the code. But I humbly suggest that this may not be quite so obvious to those new to Python packaging, and perhaps Flit should explain more clearly just what sort of code structure it expects provided modules to have. (I did read the "Scripts section" section, but even this leaves unstated the basic requirement that executable code be esconced within functions. Alternatively, if I've badly misunderstood what's going on here, I'll be happy to be set straight.)

Thank you for this tool, and I hope to successfully package my first Python project with it soon!

[takluyver]

Thanks, it is helpful to see pain points like this for new users.

The key bit of context that I'm not sure anyone says explicitly is that Python packaging is basically all designed around packaging libraries. A scripts is defined as a reference to a function in a library. If you look at installed scripts (pip, flit, black, ipython, ...), you'll see they're all based on a common template to import and call a function:

#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

Even pip, which explicitly says that you shouldn't use it as a library, is still structured as a library with a main function.

(This is 99% true: the packaging standards do allow packages to install scripts without using an importable main function, and you can still do that using setuptools. But this option is hardly ever used, it doesn't work well with Windows, and the setuptools docs discourage it.)

The extra detail for Flit is how it gets the version number from your code. It tries to find a __version__ = "1.2.3" statement in your code without executing it, but if that doesn't work, it will try executing (importing) the module to get that. That's a confusing mess, and I wish I hadn't designed it that way.

I'll look at how this can be clearer, either in Flit's docs and/or in the packaging user guide.

[tmo1]

The key bit of context that I'm not sure anyone says explicitly is that Python packaging is basically all designed around packaging libraries. A scripts is defined as a reference to a function in a library.

Yes, this is what I eventually concluded based on a lot more reading. I'd suggest again that this ought to be made a lot clearer for new users like me :)

The extra detail for Flit is how it gets the version number from your code. It tries to find a version = "1.2.3" statement in your code without executing it, but if that doesn't work, it will try executing (importing) the module to get that. That's a confusing mess, and I wish I hadn't designed it that way.

This is what I was still pretty confused about - why was Flit actually importing my code? But I'm still not sure I understand: I did put a __version__ = "0.1.0" in my code (in an __init__.py file, as per a suggestion I saw somewhere), so why wasn't Flit finding it without the need to import all the code?

Thanks much for the explanation, and for all your work on this tool!

[takluyver]

I forgot, it also looks for a module level docstring to use the first line as the short description for the package. If it doesn't get both that and the version, it falls back to importing.

If you had a docstring as well, it's possible that the automatic fallback is hiding some bug. The function that tries to get the version & docstring without executing code is here:

flit/flit_core/flit_core/common.py

Lines 140 to 172 in 432042e

	def get_docstring_and_version_via_ast(target):
	"""
	Return a tuple like (docstring, version) for the given module,
	extracted by parsing its AST.
	"""
	version = None
	for target_path in target.version_files:
	# read as bytes to enable custom encodings
	with target_path.open('rb') as f:
	node = ast.parse(f.read())
	for child in node.body:
	if sys.version_info >= (3, 8):
	target_type = ast.Constant
	else:
	target_type = ast.Str
	# Only use the version from the given module if it's a simple
	# string assignment to __version__
	is_version_str = (
	isinstance(child, ast.Assign)
	and any(
	isinstance(target, ast.Name)
	and target.id == "__version__"
	for target in child.targets
	)
	and isinstance(child.value, target_type)
	)
	if is_version_str:
	if sys.version_info >= (3, 8):
	version = child.value.value
	else:
	version = child.value.s
	break
	return ast.get_docstring(node), version

[tmo1]

Thank you!

The problem was that I was putting the module docstring and the __version__ into __init__.py, and assuming that even when pointing Flit at my_module.py, it would still look at the adjacent __init__.py. Once I put them into my_module.py, Flit was indeed able to find them and did not try to import the module, and the build succeeded.

I suppose that Flit will only look at __init__.py when pointed at the containing directory? Again, I suppose this is all obvious to those more experienced with Python packaging, but I found these introductory statements in the documentation somewhat vague:

Say you’re writing a module foobar — either as a single file foobar.py, or as a directory — and you want to distribute it.

Make sure that foobar’s docstring starts with a one-line summary of what the module is, and that it has a version:

How is Flit used with a single file foobar.py, and how is it used with a directory? I think concrete examples would help.
(FWIW, I find the Python official documentation and other online resources incredibly confusing regarding the package / module terminolopy and distinction.)

Another problem: when I set readme to something in the module directory or below, the build works, but when I set it to something in the module's parent directory, the build terminates with this traceback:

Traceback (most recent call last):
  File "/usr/bin/flit", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/usr/lib/python3/dist-packages/flit/__init__.py", line 191, in main
    main(args.ini_file, formats=set(args.format or []),
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         gen_setup_py=gen_setup_py(), use_vcs=sdist_use_vcs())
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/flit/build.py", line 51, in main
    with unpacked_tarball(sdist_file) as tmpdir:
         ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/usr/lib/python3.13/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/flit/build.py", line 26, in unpacked_tarball
    assert len(files) == 1, files
           ^^^^^^^^^^^^^^^
AssertionError: ['foobar-0.1.0', 'README.md']

Apparently there's a requirement that referenced files not be above the module in the filesystem? I suggest that this be documented and / or that Flit fail more gracefully here.

Thanks again for bearing with me.