Mirror Plugin: Initial version #34
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Warning Rate Limit Exceeded@ikaneshiro has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 9 minutes and 52 seconds before requesting another review. How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. WalkthroughThe recent changes introduced a comprehensive suite of updates for a new plugin called Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review Status
Actionable comments generated: 25
Configuration used: CodeRabbit UI
Files ignored due to path filters (14)
charts/beskar-mirror/Chart.yamlis excluded by:!**/*.yamlcharts/beskar-mirror/templates/configmap.yamlis excluded by:!**/*.yamlcharts/beskar-mirror/templates/hpa.yamlis excluded by:!**/*.yamlcharts/beskar-mirror/templates/pvc.yamlis excluded by:!**/*.yamlcharts/beskar-mirror/templates/role.yamlis excluded by:!**/*.yamlcharts/beskar-mirror/templates/secret.yamlis excluded by:!**/*.yamlcharts/beskar-mirror/templates/service.yamlis excluded by:!**/*.yamlcharts/beskar-mirror/templates/serviceaccount.yamlis excluded by:!**/*.yamlcharts/beskar-mirror/templates/statefulset.yamlis excluded by:!**/*.yamlcharts/beskar-mirror/values.yamlis excluded by:!**/*.yamlgo.modis excluded by:!**/*.modgo.sumis excluded by:!**/*.suminternal/plugins/mirror/embedded/data.jsonis excluded by:!**/*.jsoninternal/plugins/mirror/pkg/config/default/beskar-mirror.yamlis excluded by:!**/*.yaml
Files selected for processing (34)
- build/mage/build.go (3 hunks)
- charts/beskar-mirror/.helmignore (1 hunks)
- charts/beskar-mirror/LICENSE (1 hunks)
- charts/beskar-mirror/templates/_helpers.tpl (1 hunks)
- cmd/beskar-mirror/main.go (1 hunks)
- internal/plugins/mirror/api.go (1 hunks)
- internal/plugins/mirror/embedded/router.rego (1 hunks)
- internal/plugins/mirror/pkg/config/beskar-mirror.go (1 hunks)
- internal/plugins/mirror/pkg/config/beskar-mirror_test.go (1 hunks)
- internal/plugins/mirror/pkg/index/embedded/index.html.tpl (1 hunks)
- internal/plugins/mirror/pkg/index/index.go (1 hunks)
- internal/plugins/mirror/pkg/index/index_test.go (1 hunks)
- internal/plugins/mirror/pkg/index/testdata/index.html (1 hunks)
- internal/plugins/mirror/pkg/mirrordb/log.go (1 hunks)
- internal/plugins/mirror/pkg/mirrordb/repository.go (1 hunks)
- internal/plugins/mirror/pkg/mirrordb/schema/log/000000_create_logs_table.sql (1 hunks)
- internal/plugins/mirror/pkg/mirrordb/schema/repository/000000_create_files_table.sql (1 hunks)
- internal/plugins/mirror/pkg/mirrordb/schema/status/000000_create_properties_table.sql (1 hunks)
- internal/plugins/mirror/pkg/mirrordb/schema/status/000001_create_events_table.sql (1 hunks)
- internal/plugins/mirror/pkg/mirrordb/schema/status/000002_create_reposync_table.sql (1 hunks)
- internal/plugins/mirror/pkg/mirrordb/status.go (1 hunks)
- internal/plugins/mirror/pkg/mirrorrepository/api.go (1 hunks)
- internal/plugins/mirror/pkg/mirrorrepository/database.go (1 hunks)
- internal/plugins/mirror/pkg/mirrorrepository/handler.go (1 hunks)
- internal/plugins/mirror/pkg/mirrorrepository/rsyncstorage.go (1 hunks)
- internal/plugins/mirror/pkg/mirrorrepository/sync.go (1 hunks)
- internal/plugins/mirror/plugin.go (1 hunks)
- internal/plugins/mirror/web.go (1 hunks)
- pkg/orasmirror/file.go (1 hunks)
- pkg/plugins/mirror/api/v1/api.go (1 hunks)
- pkg/plugins/mirror/api/v1/endpoint.go (1 hunks)
- pkg/plugins/mirror/api/v1/http.go (1 hunks)
- pkg/plugins/mirror/api/v1/http_client.go (1 hunks)
- pkg/plugins/mirror/api/v1/oas2.go (1 hunks)
Files skipped from review due to trivial changes (3)
- charts/beskar-mirror/.helmignore
- charts/beskar-mirror/LICENSE
- internal/plugins/mirror/pkg/index/testdata/index.html
Additional comments: 43
internal/plugins/mirror/pkg/mirrordb/schema/status/000001_create_events_table.sql (1)
- 1-4: The schema for creating the
eventstable looks good. It correctly defines anidas the primary key and apayloadto store the event data.internal/plugins/mirror/pkg/mirrordb/schema/log/000000_create_logs_table.sql (1)
- 1-6: The schema for creating the
logstable is correctly defined with appropriate fields for logging (id,level,date,message). The use ofAUTOINCREMENTfor theidfield ensures unique identifiers for each log entry.internal/plugins/mirror/pkg/mirrordb/schema/status/000000_create_properties_table.sql (2)
- 1-6: The schema for creating the
propertiestable is well-defined, including fields forid,created,mirror, andmirror_urls. This setup supports storing configuration properties for the mirror functionality.- 8-8: The initial
INSERTstatement ensures that there is a default row in thepropertiestable. This is a good practice for initializing default configurations.internal/plugins/mirror/pkg/mirrordb/schema/repository/000000_create_files_table.sql (2)
- 1-8: The schema for creating the
filestable is correctly structured, including fields for file metadata (tag,name,link,modified_time,mode,size). This setup supports storing information about files managed by the mirror plugin.- 10-10: Creating an index on the
namefield of thefilestable is a good practice for optimizing queries that search for files by name.internal/plugins/mirror/pkg/mirrordb/schema/status/000002_create_reposync_table.sql (2)
- 1-9: The schema for creating the
synctable is well-defined, including fields to track the synchronization process (id,syncing,start_time,end_time,total_files,synced_files,sync_error). This setup supports monitoring and managing synchronization tasks.- 11-11: The initial
INSERTstatement ensures that there is a default row in thesynctable. This is a good practice for initializing the synchronization status.internal/plugins/mirror/pkg/index/embedded/index.html.tpl (1)
- 1-33: The HTML template for generating directory indexes is well-structured, with placeholders for the current directory, previous directory, directories, and files. This setup supports dynamically generating browsable directory listings.
internal/plugins/mirror/pkg/index/index.go (1)
- 1-49: The Go code for generating directory indexes from the provided template is correctly implemented. The
Generatefunction and the data structures for directories and files (Directory,File,Config) are well-defined, supporting the dynamic generation of browsable directory listings.internal/plugins/mirror/pkg/mirrorrepository/sync.go (1)
- 12-63: The Go code for the repository synchronization process is well-structured, including comprehensive error handling, database updates, and rsync operations. The use of
deferfor cleanup and error handling is correctly implemented. However, ensure thatdbCtxis properly defined and passed to theupdateSyncDatabasefunction.internal/plugins/mirror/pkg/index/index_test.go (1)
- 16-61: The Go test code for the index generation function is correctly implemented, including setup for test data, calling the
Generatefunction, and performing assertions on the result. This test ensures that the index generation behaves as expected.internal/plugins/mirror/embedded/router.rego (1)
- 1-57: The Rego policy code for routing decisions is well-structured, including logic for handling blob URLs, redirecting based on repository patterns, and handling body-based routing. This setup supports flexible and dynamic routing decisions based on repository and filename.
cmd/beskar-mirror/main.go (1)
- 56-73: The Go code for the main entry point of the mirror plugin is correctly implemented, including command-line flag parsing, server setup, and handling of the
versionsubcommand. The structure and error handling are appropriate for initializing and running the plugin.internal/plugins/mirror/pkg/config/beskar-mirror_test.go (1)
- 12-50: The Go test code for parsing the Beskar Mirror configuration is well-implemented, including comprehensive assertions on various configuration fields. This test ensures that the configuration parsing behaves as expected and correctly initializes the configuration values.
internal/plugins/mirror/pkg/mirrordb/log.go (1)
- 1-117: The Go code for log database operations is well-structured, including functions for opening the log database, adding logs, and walking through logs. The use of embedded SQL schemas and the
WalkLogFunccallback pattern are correctly implemented. Ensure that theOpenandClosemethods are properly managing database connections.internal/plugins/mirror/pkg/mirrorrepository/database.go (1)
- 19-115: The Go code for database operations related to the mirror repository is correctly implemented, including functions for updating synchronization status, adding and removing files, and retrieving file information. The use of MD5 for generating file tags (
md5.Sum) should be reviewed for its suitability in this context, considering potential collision issues in large datasets.internal/plugins/mirror/pkg/config/beskar-mirror.go (1)
- 62-111: The Go code for parsing the Beskar Mirror configuration is well-structured, including logic for handling default configurations and custom directories. The use of a configuration parser with versioned parse information supports future configuration schema versions. Ensure that the default configuration (
defaultBeskarMirrorConfig) meets all necessary configuration requirements.internal/plugins/mirror/api.go (1)
- 14-82: The Go code for the mirror plugin API is correctly implemented, including functions for creating, deleting, updating, and syncing repositories, as well as retrieving repository properties and logs. The use of
checkRepositoryto validate repository names against a regex pattern is a good practice for ensuring input validity.internal/plugins/mirror/web.go (1)
- 18-105: The Go code for the web handler of the mirror plugin includes logic for resolving symlinks, handling file requests, and serving directory listings. The use of
resolveSymlinksto handle symlinked files and the integration with ORAS for serving files are correctly implemented. Ensure that the error handling and HTTP status codes used are appropriate for the various failure scenarios.pkg/plugins/mirror/api/v1/api.go (1)
- 1-115: The Go code for the API definitions of the mirror plugin is well-defined, including types and interfaces for managing mirror repositories, such as repository properties, logs, files, and sync status. The use of regular expressions for validating repository names (
RepositoryMatch) and the structured approach to defining API operations (Mirrorinterface) support a clear and extensible API design.pkg/orasmirror/file.go (1)
- 30-161: The Go code for ORAS mirror file operations, including functions for generating file references, pushing static files, and pulling mirror files, is correctly implemented. The use of MD5 for generating file tags should be reviewed for its suitability, considering potential collision issues. The
MirrorPullerimplementation and its integration with the ORAS library support efficient file handling in the mirror plugin.charts/beskar-mirror/templates/_helpers.tpl (1)
- 1-150: The Helm chart helper template for the Beskar Mirror plugin is well-structured, including definitions for common labels, service account names, environment variables, volume mounts, and volumes. The use of template functions (
define) for generating reusable snippets supports a DRY and maintainable Helm chart. Ensure that the environment variable names and volume configurations match the requirements of the Beskar Mirror plugin.pkg/plugins/mirror/api/v1/oas2.go (3)
- 1-1: Generated code disclaimer.
The file contains a clear disclaimer that it's generated code and should not be edited manually. This is a good practice.
- 13-28: Swagger version and metadata.
The Swagger version and metadata (title, version, description, license) are correctly defined and follow best practices for API documentation.
- 32-136: API paths and operations.
The API paths and operations are well-defined, following RESTful principles and correctly using HTTP methods for CRUD operations.
pkg/plugins/mirror/api/v1/http.go (2)
- 17-153: HTTP router setup for API endpoints.
The HTTP router setup for API endpoints is well-structured, using appropriate HTTP methods and paths for each operation. The use of middleware for validation and error handling is also correctly implemented.
- 156-298: Request decoding functions.
The request decoding functions are correctly implemented, using the provided codec for decoding and performing validation on the decoded request objects.
internal/plugins/mirror/pkg/mirrorrepository/handler.go (12)
- 55-60: The
QueueEventmethod is implemented correctly. It logs and enqueues events efficiently.- 63-77: The
getRepositoryDBmethod correctly ensures thread safety with mutex locks and lazy initialization. Good use of defer for unlocking.- 80-94: The
getStatusDBmethod follows best practices similar togetRepositoryDB, ensuring thread safety and lazy initialization.- 97-111: The
getLogDBmethod is consistent with the other database getter methods in its use of mutexes and lazy initialization. Well done.- 192-208: The
setMirrorURLsmethod correctly uses a mutex to ensure thread safety when setting mirror URLs. Good practice.- 211-215: The
getMirrorURLsmethod properly uses read locks for thread safety, which is efficient for concurrent reads.- 218-221: The
setMirrormethod correctly uses a mutex for thread safety. This is a good practice for concurrent access.- 224-227: The
setCreatedmethod follows best practices for thread safety with proper locking.- 230-234: The
isCreatedmethod efficiently uses read locks for thread-safe reads. Well implemented.- 237-241: The
getMirrormethod demonstrates good use of read locks for thread safety. This is efficient for concurrent reads.- 244-245: The
getSyncmethod correctly uses atomic operations for thread-safe access to the sync state. This is an efficient choice.- 248-250: The
setSyncmethod properly uses atomic operations to update the sync state safely. Good choice for concurrency.build/mage/build.go (3)
- 68-68: The constant
BeskarMirrorBinaryis correctly defined, aligning with the naming convention of existing binary constants.- 176-193: The configuration for
BeskarMirrorBinarywithin thebinariesmap includes correct paths for configuration files, API generation settings, and integration test environment variables. However, ensure that the specified paths and environment variables align with the actual directory structure and configuration requirements of thebeskar-mirrorplugin.- 236-236: The addition of
BeskarMirrorBinaryto thePluginsmethod correctly integrates the new plugin into the build process, ensuring it is built alongside other plugins.
- Syncronizes file repositories with rsync - Creates `index.html` files for each directory - Presents a web api for traversing or fetching content Signed-off-by: Ian Kaneshiro <[email protected]>
ikaneshiro
force-pushed
the
mirror-plugin
branch
from
2186463 to
9c8ef6f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
index.htmlfiles for each directorySummary by CodeRabbit
BeskarMirrorBinary, for enhanced configuration and integration capabilities.beskar-mirrorfor server and plugin services..helmignoreto improve package build process by excluding unnecessary files.