feat(author-jinja): load jinja extensions from plugins (#1710)

* Implement new helpers as jinja filters loaded via an extension

Signed-off-by: Ryan Ahearn <[email protected]>

* auto-load plugins with jinja extensions

Signed-off-by: Ryan Ahearn <[email protected]>

* Refactor jinja organization for ease of reuse

Signed-off-by: Ryan Ahearn <[email protected]>

* Document plugins including jinja extensions

Signed-off-by: Ryan Ahearn <[email protected]>

* Rename first_array_entry filter for clarity

fix some other random typos

Signed-off-by: Ryan Ahearn <[email protected]>

* Update api docs

Signed-off-by: Ryan Ahearn <[email protected]>

* Add docs for new built-in jinja filters

Signed-off-by: Ryan Ahearn <[email protected]>

* Correct the copyright line for new files

Signed-off-by: Ryan Ahearn <[email protected]>

* Remove inherited dangling comment

Signed-off-by: Ryan Ahearn <[email protected]>

---------

Signed-off-by: Ryan Ahearn <[email protected]>

docs/api_reference/trestle.core.jinja.base.md

@@ -0,0 +1,2 @@
+::: trestle.core.jinja.base
+handler: python

docs/api_reference/trestle.core.jinja.ext.md

@@ -0,0 +1,2 @@
+::: trestle.core.jinja.ext
+handler: python

docs/api_reference/trestle.core.jinja.filters.md

@@ -0,0 +1,2 @@
+::: trestle.core.jinja.filters
+handler: python

docs/api_reference/trestle.core.jinja.tags.md

@@ -0,0 +1,2 @@
+::: trestle.core.jinja.tags
+handler: python

docs/api_reference/trestle.core.plugins.md

@@ -0,0 +1,2 @@
+::: trestle.core.plugins
+handler: python

docs/contributing/plugins.md

@@ -13,17 +13,22 @@ The plugin project should be organized as shown below.
 ```text
 compliance-trestle-fedramp
 ├── trestle_fedramp
-│   ├── __init.py__
+│   ├── __init__.py
 │   ├── commands
-|   |   ├── __init.py__
+|   |   ├── __init__.py
 |   |   ├── validate.py
+|   ├── jinja_ext
+|   |   ├── __init__.py
+|   |   ├── filters.py
 │   ├── <other source files or folder>
 ├── <other files or folder>
 ```
 
 Trestle uses a naming convention to discover the top-level module of the plugin projects. It expects the top-level module to be named `trestle_{plugin_name}`. This covention must be followed by plugins to be discoverable by trestle. In the above example, the top-level module is named as `trestle_fedramp` so that it can be autmatically discovered by trestle. All the python source files should be created inside this module (folder).
 
-The top-evel module should contain a `commands` directory where all the plugin command files should be stored. Each command should have its own python file. In the above exaample, `validate.py` file conatins one command for this plugin. Other python files or folders should be created in the top-level module folder, outside the `commands` folder. This helps in keeping the commands separate and in their discovery by trestle.
+To add commands to the CLI interface, the top-level module should contain a `commands` directory where all the plugin command files should be stored. Each command should have its own python file. In the above example, `validate.py` file contains one command for this plugin. Other python files or folders should be created in the top-level module folder, outside the `commands` folder. This helps in keeping the commands separate and in their discovery by trestle.
+
+To add jinja extensions available during `trestle author jinja`, the top-level module should contain a `jinja_ext` directory where all extension files should be stored. Each extension should have its own python file. In the above example, `filters.py` file contains a single extension class, which may define many filters or custom tags. Supporting code should be created in the top-level module folder, outside the `jinja_ext` folder. This helps in keeping the extensions separate and in their discovery by trestle.
 
 ## Command Creation
 
@@ -61,3 +66,28 @@ There should be a command class for example, `ValidateCmd` which should either e
 The docstring of the command class is used as the help message for the command. Input arguments to the command should be specified in `_init_arguments` method as shown above. The acutal code of the command is contained in`_run` method. This method is called by ilcli when the command is excuted on the commandline. The command arguments can be accessed from the `args` input parameter as shown above. The command should return `0` in case of successful execution, or any number greater than 0 in case of failure. Please see `trestle.core.commands.common.return_codes.CmdReturnCodes` class for specific return codes in case of failure.
 
 The command class should conatin the `name` field which should be set to the desired command name. In the above example, the command is called `fedramp-validate`. This name is automatically added to the list of sub-command names of trestle during the plugin discovery process. This command can then be invoked as `trestle {name}` from the commandline e.g., `trestle fedramp-validate`. Any input parameters to the command can also be passed on the commandline after the command name.
+
+## Jinja Extension Creation
+
+The plugin extension should be created as shown in the below code snippet.
+
+```python
+from jinja2 import Environment
+from trestle.core.jinja.base import TrestleJinjaExtension
+
+def _mark_tktk(value: str) -> str:
+    """Mark a value with TKTK to easily find it for future revision."""
+    return f'TKTK {value} TKTK'
+
+
+class Filters(TrestleJinjaExtension):
+    def __init__(self, environment: Environment) -> None:
+        super(Filters, self).__init__(environment)
+
+        environment.filters['tktk'] = _mark_tktk
+
+```
+
+There should be an extension class, for example `Filters` that must extend from `TrestleJinjaExtension` or `jinja2.ext.Extention`. The `__init__` method must call init for its superclass. Beyond that, any behavior for standard [jinja2 custom extensions](https://jinja.palletsprojects.com/en/3.1.x/extensions/#module-jinja2.ext) is supported.
+
+Examples for implementing extensions can be found at `trestle/core/jinja/tags.py` and `trestle/core/jinja/filters.py`

docs/trestle_author_jinja.md

@@ -113,7 +113,7 @@ Trestle provides custom jinja tags for use specifically with markdown: `mdsectio
    1. `{% md_clean_include 'path_to_file.md' heading_level=2 %}`
    1. The heading level argument adjusts to (based on the number of hashes) the most significant heading in the document, if headings exist.
 
-`mdsection_include` is similar to the native `md_clean_include` except that.:
+`mdsection_include` is similar to `md_clean_include` except that:
 
 1. `mdsection_include` requires an second positional argument which is the title of a heading, from a markdown file, which you want the content from.
 
@@ -129,6 +129,23 @@ Trestle provides custom jinja tags for use specifically with markdown: `mdsectio
 1. `format` where a python [datetime strftime format string](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) is provided to format the output. E.g. `{% md_datestamp format='%B %d, %Y' %}` results in `December 28, 2021` being inserted.
 1. `newline` is a boolean to control the addition of a double newline after the inserted date string. For example `{% md_datestamp newline=false %}` inserts a date in the default format, without additional newlines.
 
+## Custom Jinja filters.
+
+Trestle provides custom jinja filters for help processing SSP data.
+
+- `as_list` will return the passed value, or an empty list if `None` is passed in.
+  - Example: `{% for party in ssp.metadata.parties | as_list %}`
+- `get_default` operates the same as the built-in Jinja `default` filter, with the optional second parameter set to `True`
+  - Example: `{{ control_interface.get_prop(user, 'user-property') | get_default('[Property]') }}`
+- `first_or_none` will return the first element of a list, or `None` if the list is empty or `None` itself.
+  - Example: `{% set address = party.addresses | first_or_none %}`
+- `get_party` will return the `Party` found in `ssp.metadata.parties` for a given `uuid`
+  - Example: `{% set organization = party.member_of_organizations | first_or_none | get_party(ssp) %}`
+- `parties_for_role` will yield individual `Party` entries when given a list of `ResponsibleParty` and a `role-id`
+  - Example: `{% for party in ssp.metadata.responsible_parties | parties_for_role("prepared-by", ssp) %}`
+- `diagram_href` will return the `Link.href` where `Link.rel == 'diagram'` when given a `Diagram` object
+  - Example: `![{{diagram.caption}}]({{ diagram | diagram_href }})`
+
 ## Generate controls as individual markdown pages.
 
 Trestle's Jinja functionality allows its users to generate individual markdown pages for each control from a resolved profile catalog. Such functionality can be used later on to pack individual pages into docs of various formats.
@@ -144,17 +161,17 @@ To achieve that, we can create a simple Jinja template that would be used to gen
 {{ control_writer.write_control_with_sections(
    control,
    profile,
-   group_title, 
-   ['statement', 'objective', 'expected_evidence', 'implementation_guidance', 'table_of_parameters'], 
+   group_title,
+   ['statement', 'objective', 'expected_evidence', 'implementation_guidance', 'table_of_parameters'],
    {
       'statement':'Control Statement',
-      'objective':'Control Objective', 
-      'expected_evidence':'Expected Evidence', 
-      'implementation_guidance':'Implementation Guidance', 
+      'objective':'Control Objective',
+      'expected_evidence':'Expected Evidence',
+      'implementation_guidance':'Implementation Guidance',
       'table_of_parameters':'Control Parameters'
    }
-   ) 
-   
+   )
+
    | safe
 }}
 ```

mkdocs.yml

@@ -132,7 +132,11 @@ nav:
       - duplicates_validator: api_reference/trestle.core.duplicates_validator.md
       - generators: api_reference/trestle.core.generators.md
       - generic_oscal: api_reference/trestle.core.generic_oscal.md
-      - jinja: api_reference/trestle.core.jinja.md
+      - jinja:
+        - base: api_reference/trestle.core.jinja.base.md
+        - ext: api_reference/trestle.core.jinja.ext.md
+        - filters: api_reference/trestle.core.jinja.filters.md
+        - tags: api_reference/trestle.core.jinja.tags.md
       - links_validator: api_reference/trestle.core.links_validator.md
       - markdown:
         - base_markdown_node: api_reference/trestle.core.markdown.base_markdown_node.md
@@ -152,6 +156,7 @@ nav:
       - object_factory: api_reference/trestle.core.object_factory.md
       - parser: api_reference/trestle.core.parser.md
       - pipeline: api_reference/trestle.core.pipeline.md
+      - plugins: api_reference/trestle.core.plugins.md
       - profile_resolver: api_reference/trestle.core.profile_resolver.md
       - refs_validator: api_reference/trestle.core.refs_validator.md
       - remote:

tests/conftest.py

@@ -32,8 +32,10 @@
 from trestle.cli import Trestle
 from trestle.common.err import TrestleError
 from trestle.oscal import catalog as cat
+from trestle.oscal.common import Party
 from trestle.oscal.component import ComponentDefinition, DefinedComponent
 from trestle.oscal.profile import Profile
+from trestle.oscal.ssp import SystemSecurityPlan
 
 TEST_CONFIG: dict = {}
 
@@ -190,6 +192,20 @@ def sample_catalog_subgroups():
     return catalog_obj
 
 
+@pytest.fixture(scope='function')
+def sample_party() -> Party:
+    """Return a valid Party object."""
+    return gens.generate_sample_model(Party, True, 3)
+
+
+@pytest.fixture(scope='function')
+def sample_system_security_plan(sample_party: Party) -> SystemSecurityPlan:
+    """Return a valid SSP object with some contents."""
+    ssp: SystemSecurityPlan = gens.generate_sample_model(SystemSecurityPlan, True, 2)
+    ssp.metadata.parties = [gens.generate_sample_model(Party, True, 3), sample_party]
+    return ssp
+
+
 @pytest.fixture(scope='function')
 def sample_component_definition():
     """Return a valid ComponentDefinition object with some contents."""

tests/trestle/core/jinja/filters_test.py

@@ -0,0 +1,69 @@
+# -*- mode:python; coding:utf-8 -*-
+
+# Copyright (c) 2024 The OSCAL Compass Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tests for trestle custom jinja filters functionality."""
+
+from typing import Any, List, Optional
+
+import pytest
+
+from trestle.core.jinja.filters import (diagram_href, first_or_none, get_party, parties_for_role)
+from trestle.oscal.common import Link, Party, ResponsibleParty
+from trestle.oscal.ssp import Diagram, SystemSecurityPlan
+
+
+@pytest.mark.parametrize(
+    'links,expected',
+    [
+        (
+            [
+                Link(rel='other', href='./path/to/local/thing'),
+                Link(rel='diagram', href='https://host.name/path/to/diagram.png')
+            ],
+            'https://host.name/path/to/diagram.png'
+        ), ([Link(rel='other', href='./path/to/local/file')], ''), ([], ''), (None, '')
+    ]
+)
+def test_diagram_href(links: Optional[List[Link]], expected: str) -> None:
+    """Test retrieving the link href for rel='diagram'."""
+    diagram = Diagram(uuid='26c1c7df-fb67-45ba-b60f-35d8b5c1d1dc', links=links)
+    assert diagram_href(diagram) == expected
+
+
+@pytest.mark.parametrize('actual,expected', [[['ok'], 'ok'], ([], None), (None, None)])
+def test_first_or_none(actual: Optional[List[Any]], expected: Optional[Any]) -> None:
+    """Test behavior of retrieving the first element or None for empty or missing list."""
+    assert first_or_none(actual) == expected
+
+
+def test_get_party(sample_system_security_plan: SystemSecurityPlan, sample_party: Party) -> None:
+    """Test behavior of retrieving a ssp.metadata.parties entry by UUID."""
+    assert get_party(sample_party.uuid, ssp=sample_system_security_plan) == sample_party
+
+
+def test_parties_for_role(sample_system_security_plan: SystemSecurityPlan, sample_party: Party) -> None:
+    """Test behavior of retrieving all parties for a given role-id."""
+    sample_system_security_plan.metadata.responsible_parties = [
+        ResponsibleParty(role_id='pytest-tester', party_uuids=[sample_party.uuid])
+    ]
+    result = list(
+        parties_for_role(
+            sample_system_security_plan.metadata.responsible_parties,
+            role_id='pytest-tester',
+            ssp=sample_system_security_plan
+        )
+    )
+    assert len(result) == 1
+    assert result[0] == sample_party

tests/trestle/core/jinja_test.py → tests/trestle/core/jinja/tags_test.py

@@ -22,7 +22,7 @@
 
 import pytest
 
-import trestle.core.jinja as tres_jinja
+import trestle.core.jinja.tags as tres_jinja
 from trestle.core.markdown import markdown_const
 
 JINJA_MD = 'jinja_markdown_include'

trestle/cli.py

@@ -14,11 +14,8 @@
 # limitations under the License.
 """Starting point for the Trestle CLI."""
 
-import importlib
-import inspect
 import logging
 import pathlib
-import pkgutil
 
 from trestle.common import const, log
 from trestle.core.commands.assemble import AssembleCmd
@@ -38,6 +35,7 @@
 from trestle.core.commands.task import TaskCmd
 from trestle.core.commands.validate import ValidateCmd
 from trestle.core.commands.version import VersionCmd
+from trestle.core.plugins import discovered_plugins
 
 logger = logging.getLogger('trestle')
 
@@ -63,29 +61,14 @@ class Trestle(CommandBase):
         VersionCmd
     ]
 
-    discovered_plugins = {
-        name: importlib.import_module(name)
-        for finder,
-        name,
-        ispkg in pkgutil.iter_modules()
-        if name.startswith('trestle_')
-    }
-
-    logger.debug(discovered_plugins)
     # This block is uncovered as trestle cannot find plugins in it's unit tests - it is the base module.
-    for plugin, value in discovered_plugins.items():  # pragma: nocover
-        for _, module, _ in pkgutil.iter_modules([pathlib.Path(value.__path__[0], 'commands')]):
-            logger.debug(module)
-            command_module = importlib.import_module(f'{plugin}.commands.{module}')
-            clsmembers = inspect.getmembers(command_module, inspect.isclass)
-            logger.debug(clsmembers)
-            for _, cmd_cls in clsmembers:
-                # add commands (derived from CommandPlusDocs or CommandBase) to subcommands list
-                if issubclass(cmd_cls, CommandBase):
-                    # don't add CommandPlusDocs or CommandBase
-                    if cmd_cls is not CommandPlusDocs and cmd_cls is not CommandBase:
-                        subcommands.append(cmd_cls)
-                        logger.info(f'{cmd_cls} added to subcommands from plugin {plugin}')
+    for plugin, cmd_cls in discovered_plugins('commands'):  # pragma: nocover
+        # add commands (derived from CommandPlusDocs or CommandBase) to subcommands list
+        if issubclass(cmd_cls, CommandBase):
+            # don't add CommandPlusDocs or CommandBase
+            if cmd_cls is not CommandPlusDocs and cmd_cls is not CommandBase:
+                subcommands.append(cmd_cls)
+                logger.info(f'{cmd_cls} added to subcommands from plugin {plugin}')
 
     def _init_arguments(self) -> None:
         self.add_argument('-v', '--verbose', help=const.DISPLAY_VERBOSE_OUTPUT, action='count', default=0)

trestle/core/commands/author/jinja.py

@@ -35,7 +35,7 @@
 from trestle.core.commands.common.return_codes import CmdReturnCodes
 from trestle.core.control_interface import ControlInterface, ParameterRep
 from trestle.core.docs_control_writer import DocsControlWriter
-from trestle.core.jinja import MDCleanInclude, MDDatestamp, MDSectionInclude
+from trestle.core.jinja.ext import extensions
 from trestle.core.profile_resolver import ProfileResolver
 from trestle.core.ssp_io import SSPMarkdownWriter
 from trestle.oscal.profile import Profile
@@ -191,10 +191,7 @@ def jinja_ify(
         """Run jinja over an input file with additional booleans."""
         template_folder = pathlib.Path.cwd()
         jinja_env = Environment(
-            loader=FileSystemLoader(template_folder),
-            extensions=[MDSectionInclude, MDCleanInclude, MDDatestamp],
-            trim_blocks=True,
-            autoescape=True
+            loader=FileSystemLoader(template_folder), extensions=extensions(), trim_blocks=True, autoescape=True
         )
         template = jinja_env.get_template(str(r_input_file))
         # create boolean dict
@@ -283,7 +280,7 @@ def jinja_multiple_md(
 
                 jinja_env = Environment(
                     loader=FileSystemLoader(template_folder),
-                    extensions=[MDSectionInclude, MDCleanInclude, MDDatestamp],
+                    extensions=extensions(),
                     trim_blocks=True,
                     autoescape=True
                 )
@@ -315,7 +312,7 @@ def render_template(template: Template, lut: Dict[str, Any], template_folder: pa
             dict_loader = DictLoader({str(random_name): new_output})
             jinja_env = Environment(
                 loader=ChoiceLoader([dict_loader, FileSystemLoader(template_folder)]),
-                extensions=[MDCleanInclude, MDSectionInclude, MDDatestamp],
+                extensions=extensions(),
                 autoescape=True,
                 trim_blocks=True
             )

trestle/core/jinja/__init__.py

@@ -0,0 +1,16 @@
+# -*- mode:python; coding:utf-8 -*-
+
+# Copyright (c) 2024 The OSCAL Compass Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Trestle core.jinja functionality."""