Python package with expanded testing (#278)

* Code reorganization to address #273

* Fixed issues with renaming config_file as registry

* Removed a print statement left in by mistake

* On our way to more robust testing

* Added nocolor option to 'git branch list'

* It's enough to chdir to the fixture dir at the beginning of a test

* Added 'mepo branch' tests

* Added tests for mepo tag, fetch, pull, pull-all

* multiprocessing library supports the context management protocol

* Added tests for mepo push, diff, whereis and reset

* component.py - using Python3's relative import

* Suppressing test outputs to terminal

* Turning off testing 'mepo reset' and checking why 'mepo diff' is failing

Maybe 'mepo reset' is causing issues with testing 'mepo tag create'

* For testing 'mepo diff', ignore the last line '---'

* Ignore the last line of both stdout and saved outputs

* Escaping the string message of 'git tag create'

* Create regular (non-annotated) tag instead

* Testing 'mepo reset' is back on - this was not the issue

* Code from src/mepo/command/command.py included in src/mepo/__main__.py

* Small edits

* Updated GEOSfvdycore version for testing

* Minor quote changes

* Using SimpleNamespace to construct arguments to mepo commands

* Multiprocessing lib supports context manager protocol

* Removed tests/input/args.py - using SimpleNamespace now

* Updated README and CHANGELOG

* Moved duplicate code to get GIT_EDITOR, in commit.py and tag_create.py, to git.py

* Lint code as part of GitHub actions workflow

only for the case where OS is ubuntu-latest and Python version is 3.9

* Added __init__.py in cmdline, command and utilities directories

* Linting updates

1. Linting is a separate job now
2. Disabling some error codes, for now
3. Always return a zero status code

* Hardcoding Python version

* Renaming

* Adding ability to create a mepo binary that can uploaded as an artifact of a Github actions workflow. Pyinstaller requires that the top level script not use relative imports

* Added fix for reading mepo1 state

* Added a mepo command, update-state, to permanently update mepo1 state to mepo2

* Formatting changes to update-state.py

* Slightly different printable representation of component

* Added an optional argument one-per-line for 'list'

* component.py::MepoComponent - renamed to_dict to to_registry_format. Added to_dict

* Fixed test test_list

* Fixed a bug in update-state - need to switch to the fixture dir first

* Implemented a context manager for os.chdir

* Reformatted using black

* Separate GitHub workflows for running linter and formatter

* Added black to requirements

* Using poetry to manage dependencies and packaging

* Added creation of a top level script to build

* Added pylint as (poetry) dependency

* Added pylint to requirements.txt

* Updated pyproject.toml

* Small changes to the test script for consistency in mepo commands

* Enabled previously disabled Pylint (C)onvention and (E)rror codes

* Added directory containing mepo to PYTHONPATH

* Using contextlib.redirect_stdout instead

* Can't switch to contextlib.chdir as this exists only in 3.11+

* Run 'black --check' instead. Non-zero return code indicates that some files need to be formatted

* Formatted using black

* Config file is now called registry

* Moved fixture/component validation from MepoComponent to Registry

* Reformat using black

* Switched from pkl state file to json

* print() has a flush argument

* Trivial changes

* If mepo1 style state, print warning to run 'mepo update-state'. Updated update-state code.

* Removing hashes from requirements.txt

* Fixed bug in string formatting

* Update comment in registry.py

* Store full path to the fixture dir in state file

* Revert "Store full path to the fixture dir in state file". We want relocability.

This reverts commit 1bc8330.

* Local path saved to state file is relative to fixture directory

.github/workflows/run-formatter.yaml

@@ -0,0 +1,23 @@
+name: Run formatter
+
+on: [push]
+
+jobs:
+  format:
+    runs-on: ubuntu-latest
+    name: Format code
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python 3.9
+      uses: actions/setup-python@v5
+      with:
+        python-version: 3.9
+        cache: 'pip'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r requirements.txt
+      timeout-minutes: 5
+    - name: Run black
+      run: black --check .
+      timeout-minutes: 5

.github/workflows/run-linter.yaml

@@ -0,0 +1,23 @@
+name: Run linter
+
+on: [push]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    name: Lint code
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python 3.9
+      uses: actions/setup-python@v5
+      with:
+        python-version: 3.9
+        cache: 'pip'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r requirements.txt
+      timeout-minutes: 5
+    - name: Run pylint
+      run: pylint --exit-zero src/mepo
+      timeout-minutes: 5

.github/workflows/mepo.yaml → .github/workflows/run-tests.yaml

@@ -1,4 +1,4 @@
-name: Unit testing of mepo
+name: Run tests
 
 on: [push]
 
@@ -26,6 +26,8 @@ jobs:
         pip install -r requirements.txt
       timeout-minutes: 5
 
-    - name: Run unit tests
-      run: python3 tests/test_mepo_commands.py -v
+    - name: Run tests
+      run: |
+        export PYTHONPATH=$(pwd)/src:$PYTHONPATH
+        python tests/test_mepo_commands.py -v
       timeout-minutes: 5

CHANGELOG.md

@@ -16,11 +16,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Changed
 
 - Converted `mepo` to a Python project via the following renaming
-  -- `mepo.d` -> `src/mepo`
-  -- `mepo.d/utest` -> `tests`
-  -- `doc` --> `docs`
+  -- Added `src/mepo/__init__.py`
+  -- Renamed `mepo.d` -> `src/mepo`
+  -- Renamed `mepo.d/utest` -> `tests`
+  -- Renamed `doc` --> `docs`
+  -- A `mepo` config file is now called a `mepo` registry
+  -- More code reorganization
+
 - Helper script `mepo`, used for development, moved to the `bin` directory.
 
+- Expanded testing to cover more `mepo` commands
+
 ## [1.52.0] - 2024-01-10
 
 ### Added

README.md

@@ -1,6 +1,6 @@
 # mepo [![Actions Status](https://github.com/pchakraborty/mepo/workflows/Unit%20testing%20of%20mepo/badge.svg)](https://github.com/pchakraborty/mepo/actions) [![DOI](https://zenodo.org/badge/215067850.svg)](https://zenodo.org/badge/latestdoi/215067850)
 
-`mepo` is a tool, written in Python3 (3.6.0+), to manage (m)ultiple git r(epo)sitories, by attempting to create an illusion of a 'single repository' for multi-repository projects. Please see the [Wiki](../../wiki) for examples of `mepo` workflows.
+`mepo` is a tool, written in Python3 (3.9.0+), to manage (m)ultiple git r(epo)sitories, by attempting to create an illusion of a 'single repository' for multi-repository projects. Please see the [Wiki](../../wiki) for examples of `mepo` workflows.
 
 ## Commands
 

bin/mepo

@@ -8,10 +8,10 @@ import traceback
 if sys.version_info < (3, 9, 0):
     sys.exit('ERROR: Python version needs to be >= 3.9.0')
 
-# Add direactory containing mepo to path
-SRC_D = os.path.join(os.path.dirname(os.path.realpath(__file__)), '..', 'src')
+# Add directory containing mepo to path
+SRC_D = os.path.join(os.path.dirname(os.path.realpath(__file__)), "..", "src")
 sys.path.insert(0, SRC_D)
 
 if __name__ == '__main__':
-    from mepo.main import main
+    from mepo.__main__ import main
     main()

docs/make_md_docs.py

@@ -2,60 +2,52 @@
 
 import os
 import io
+import glob
 from mdutils.mdutils import MdUtils
 import subprocess as sp
 
-preamble='''
+preamble = """
 mepo provides many different commands for working with a multi-repository fixture.
-'''
+"""
 
 # Assume this script is in mepo/doc. Then we need to get to the mepo/mepo.d/command directory
 doc_dir_path = os.path.dirname(os.path.realpath(__file__))
 # Then we need to get to the mepo/mepo.d/command directory. First the "main" dir
 main_dir_path = os.path.dirname(doc_dir_path)
-# Now add 'src/mepo'
-mepod_dir_path = os.path.join(main_dir_path, 'src', 'mepo')
-# And then 'command'
-command_dir_path = os.path.join(mepod_dir_path, 'command')
+# Now add "src/mepo"
+mepod_dir_path = os.path.join(main_dir_path, "src", "mepo")
+# And then "command"
+command_dir_path = os.path.join(mepod_dir_path, "command")
 
-mepo_command_path = os.path.join(main_dir_path, 'bin', 'mepo')
+mepo_command_path = os.path.join(main_dir_path, "bin", "mepo")
 
-def get_command_list(directory):
-    # Walk the tree
-    roots = [x[0] for x in os.walk(directory)]
-
-    # Now remove "." from the list
-    roots = roots[1:]
-
-    # Just get the relative paths
-    rel_roots = [os.path.relpath(x,directory) for x in roots]
-
-    # Now exclude __pycache__
-    command_dirs = [x for x in rel_roots if '__pycache__' not in x]
 
-    # Convert slashes to spaces
-    all_commands = [x.replace('/',' ') for x in command_dirs]
+def get_command_list(directory):
+    # Get all commands
+    all_commands_py = glob.glob(os.path.join(directory, "*.py"))
+    all_commands = [os.path.basename(x).replace(".py", "") for x in all_commands_py]
+    all_commands.remove("command")  # manually remove
 
     # Now let's find the commands that have subcommands
-    ## First we get commands with spaces
-    commands_with_spaces = [x for x in all_commands if ' ' in x]
-    ## Now let's just get the first elements
-    temp = [x.split()[0] for x in commands_with_spaces]
-    ## Get the uniques
-    commands_with_subcommands = list(set(temp))
-
-    # Now remove those from our list
-    all_useful_commands = [x for x in all_commands if x not in commands_with_subcommands]
+    ## First we get commands with underscore
+    ## Then replace underscore with a space
+    commands_with_underscore = [x for x in all_commands if "_" in x]
+    commands_with_subcommands = [x.replace("_", " ") for x in commands_with_underscore]
+    all_useful_commands = [x for x in all_commands if x not in commands_with_underscore]
+    all_useful_commands += commands_with_subcommands
 
     return sorted(all_useful_commands)
 
+
 def create_markdown_from_usage(command, mdFile):
-    cmd = [mepo_command_path, command, '--help']
+    cmd = [mepo_command_path, command, "--help"]
 
     # Some commands have spaces, so we need to break it up again
-    cmd = ' '.join(cmd).split()
+    cmd = " ".join(cmd).split()
 
-    result = sp.run(cmd, capture_output=True, universal_newlines=True, env={'COLUMNS':'256'})
+    result = sp.run(
+        cmd, capture_output=True, universal_newlines=True, env={"COLUMNS": "256"}
+    )
     output = result.stdout
 
     output_list = output.split("\n")
@@ -66,25 +58,32 @@ def create_markdown_from_usage(command, mdFile):
 
     # Usage
     usage = output_list[0]
-    usage = usage.replace('usage: ','')
+    usage = usage.replace("usage: ", "")
     mdFile.new_header(level=3, title="Usage")
     mdFile.insert_code(usage)
 
-    positional_arguments = output.partition('positional arguments:\n')[2].partition('\n\n')[0]
+    positional_arguments = output.partition("positional arguments:\n")[2].partition(
+        "\n\n"
+    )[0]
     if positional_arguments:
         mdFile.new_header(level=3, title="Positional Arguments")
         mdFile.insert_code(positional_arguments)
 
-    optional_arguments = output.partition('optional arguments:\n')[2].partition('\n\n')[0]
+    optional_arguments = output.partition("optional arguments:\n")[2].partition("\n\n")[
+        0
+    ]
     # Remove extra blank lines
-    optional_arguments = os.linesep.join([s for s in optional_arguments.splitlines() if s])
+    optional_arguments = os.linesep.join(
+        [s for s in optional_arguments.splitlines() if s]
+    )
     if optional_arguments:
         mdFile.new_header(level=3, title="Optional Arguments")
         mdFile.insert_code(optional_arguments)
 
+
 if __name__ == "__main__":
 
-    doc_file = 'Mepo-Commands.md'
+    doc_file = "Mepo-Commands.md"
     mdFile = MdUtils(file_name=doc_file)
 
     mdFile.new_header(level=1, title="Overview")
@@ -94,8 +93,9 @@ def create_markdown_from_usage(command, mdFile):
     command_list = get_command_list(command_dir_path)
     for command in command_list:
         mdFile.new_header(level=2, title=command)
-        create_markdown_from_usage(command,mdFile)
+        print(f"mepo command: {command}")
+        create_markdown_from_usage(command, mdFile)
 
-    mdFile.new_table_of_contents(table_title='Table of Contents', depth=2)
+    mdFile.new_table_of_contents(table_title="Table of Contents", depth=2)
     mdFile.create_md_file()
-    print(f'Generated {doc_file}.')
+    print(f"Generated {doc_file}.")

mepo.spec

@@ -0,0 +1,51 @@
+# -*- mode: python ; coding: utf-8 -*-
+
+import os
+import glob
+
+cmd_dir = os.path.join(SPECPATH, 'src/mepo/command')
+cmd_list = [os.path.basename(x).split('.')[0] for x in glob.glob(os.path.join(cmd_dir, '*.py'))]
+hidden_imports = [f'mepo.command.{x}' for x in cmd_list if '_' not in x] # exclude subcommands
+print(f'hidden_imports: {hidden_imports}')
+
+a = Analysis(
+    ['src/mepo/__main__.py'],
+    pathex=[],
+    binaries=[],
+    datas=[],
+    hiddenimports=hidden_imports,
+    hookspath=[],
+    hooksconfig={},
+    runtime_hooks=[],
+    excludes=[],
+    noarchive=False,
+    optimize=0,
+)
+pyz = PYZ(a.pure)
+
+exe = EXE(
+    pyz,
+    a.scripts,
+    [],
+    exclude_binaries=True,
+    name='mepo',
+    debug=False,
+    bootloader_ignore_signals=False,
+    strip=False,
+    upx=True,
+    console=True,
+    disable_windowed_traceback=False,
+    argv_emulation=False,
+    target_arch=None,
+    codesign_identity=None,
+    entitlements_file=None,
+)
+coll = COLLECT(
+    exe,
+    a.binaries,
+    a.datas,
+    strip=False,
+    upx=True,
+    upx_exclude=[],
+    name='mepo',
+)