Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify plugin conventions as part of the build #5191

Open
dlvenable opened this issue Nov 15, 2024 · 0 comments
Open

Verify plugin conventions as part of the build #5191

dlvenable opened this issue Nov 15, 2024 · 0 comments
Assignees
Labels
maintenance Issues to help maintain the project, such as improving builds, testing, etc.

Comments

@dlvenable
Copy link
Member

Is your feature request related to a problem? Please describe.

We have a number of inconsistent names in our configurations. And recently we added a plugin name with hyphens instead of underscores (#5138).

Describe the solution you'd like

I'd like to have automation in Data Prepper that verifies that plugin names meet our conventions.

I think this could be done by creating a testing project that has built-in unit tests for scanning for plugins and verifying the naming convention. Additionally, we can have this added automatically to all plugin projects in Data Prepper.

To accomplish this, we could have a new project:

data-prepper-plugin-test

We apply it to all plugins by adding something like the following to data-prepper-plugins/build.gradle

subprojects {
apply plugin: 'data-prepper.publish'
group = 'org.opensearch.dataprepper.plugins'
}

Update the above to the following.

subprojects {
    apply plugin: 'data-prepper.publish'
    group = 'org.opensearch.dataprepper.plugins'
    dependencies {
        implementation project(':data-prepper-plugin-test')
    }
}

At this point, the data-prepper-plugin-test will be part of every plugin project. Now, we need to have tests in there.

The project data-prepper-plugin-test can take a dependency on data-prepper-plugin-framework so that it can use ClasspathPluginProvider. We would also need to add a new method to ClasspathPluginProvider to get all plugin names found. It might look somewhat like the following.

Set<String> listPlugins() {
  if (nameToSupportedTypeToPluginType == null) {
    nameToSupportedTypeToPluginType = scanForPlugins();
  }
  return nameToSupportedTypeToPluginType.keySet();
}

New we can write a unit test that list all plugins and verifies that they meet our naming conventions. Some things we should verify:

  • They should be all lowercase
  • They should have only _ as a special character

One additional change we'd need is some way to override the conventions. Some of our plugins already have some bad conventions. I propose that plugin authors can add a file to indicate ignores. This can go in src/test/resources/org.opensearch.dataprepper.plugin.test.conventions-configurations.yaml. It might look somewhat like the following:

overrides:
  plugin_names: 
    kinesis-data-streams:
      ignore_conventions: [require_underscores]

This would tell the framework to ignore the convention to require underscores for the kinesis-data-streams plugin. We would still perform the other validations on it.

Describe alternatives you've considered (Optional)

Alternative 1: We could create a Gradle plugin in buildSrc which looks for annotations of @DataPrepperPlugin and verifies the name. This would be a good way to verify the plugin names, but would be quite difficult to extend beyond the plugin names to use on the configurations themselves.

Alternative 2: We could try to use Checkstyle to verify the plugin names. This could be quite simple. But, the downside is that it would only cover the @DataPrepperPlugin annotation.

Additional context
Add any other context or screenshots about the feature request here.

@dlvenable dlvenable added enhancement New feature or request maintenance Issues to help maintain the project, such as improving builds, testing, etc. and removed untriaged enhancement New feature or request labels Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Issues to help maintain the project, such as improving builds, testing, etc.
Projects
Development

No branches or pull requests

2 participants