-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* improving README.md * Update dsh_sdk/README.md --------- Co-authored-by: Frank Hol <[email protected]> Co-authored-by: Frank Hol <[email protected]>
- Loading branch information
3 people
authored
Jan 13, 2025
1 parent
19403ab
commit e164f48
Showing
1 changed file
with
132 additions
and
55 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,107 +1,184 @@ | ||
|
|
||
| # dsh-sdk-platform-rs | ||
|
|
||
| [](https://github.com/kpn-dsh/dsh-sdk-platform-rs/actions/workflows/main.yaml) | ||
| [](https://codecov.io/gh/kpn-dsh/dsh-sdk-platform-rs) | ||
| [](https://deps.rs/repo/github/kpn-dsh/dsh-sdk-platform-rs) | ||
| [](https://opensource.org/licenses/Apache-2.0) | ||
|
|
||
| # NOTE | ||
| As this is a release candidate it may contain bugs and/or incomplete features and incorrect documentation and future updates may contain breaking changes. | ||
| A Rust SDK to interact with the DSH Platform. This library provides convenient building blocks for services that need to connect to DSH Kafka, fetch tokens for various protocols, manage Prometheus metrics, and more. | ||
|
|
||
| > **Note** | ||
| > This library (v0.5.x) is a _release candidate_. It may contain incomplete features and/or bugs. Future updates might introduce breaking changes. Please report any issues you find. | ||
| --- | ||
|
|
||
| Please report any issues you encounter. | ||
| ## Table of Contents | ||
|
|
||
| 1. [Migration Guide 0.4.X -> 0.5.X](#migration-guide-04x---05x) | ||
| 2. [Description](#description) | ||
| 3. [Usage](#usage) | ||
| 4. [Connecting to DSH](#connect-to-dsh) | ||
| 5. [Feature Flags](#feature-flags) | ||
| 6. [Environment Variables](#environment-variables) | ||
| 7. [Examples](#examples) | ||
| 8. [Changelog](#changelog) | ||
| 9. [Contributing](#contributing) | ||
| 10. [License](#license) | ||
| 11. [Security](#security) | ||
|
|
||
| --- | ||
|
|
||
| ## Migration guide 0.4.X -> 0.5.X | ||
| See [migration guide](https://github.com/kpn-dsh/dsh-sdk-platform-rs/wiki/Migration-guide-(v0.4.X-%E2%80%90--v0.5.X)) for more information on how to migrate from 0.4.X to 0.5.X. | ||
| ## Migration Guide 0.4.X -> 0.5.X | ||
|
|
||
| If you are migrating from `0.4.X` to `0.5.X`, please see the [migration guide](https://github.com/kpn-dsh/dsh-sdk-platform-rs/wiki/Migration-guide-(v0.4.X-%E2%80%90--v0.5.X)) for details on breaking changes and how to update your code accordingly. | ||
|
|
||
| --- | ||
|
|
||
| ## Description | ||
| This library can be used to interact with the DSH Platform. It is intended to be used as a base for services that will be used to interact with DSH. Features include: | ||
| - Connect to DSH Kafka (DSH, Kafka Proxy, VPN, System Space, Local) | ||
| - Bootstrap (fetch datastreams info and generate signed certificate) | ||
| - PKI Config Directory (for Kafka Proxy/VPN) | ||
| - Kafka config for DSH (incl. RDKafka) | ||
| - Management API Token Fetcher (to be used with [dsh_rest_api_client](https://crates.io/crates/dsh_rest_api_client)) | ||
| - Protocol Token Fetcher (MQTT and HTTP) | ||
| - Common utilities | ||
| - Prometheus Metrics (web server and re-export of metrics crate) | ||
| - Graceful shutdown | ||
| - Dead Letter Queue | ||
|
|
||
| The `dsh-sdk-platform-rs` library offers: | ||
|
|
||
| - **DSH Kafka Connectivity** | ||
| - Supports both direct DSH, Kafka Proxy, VPN, and local Kafka. | ||
| - Handles datastream information retrieval, certificate signing (bootstrap), and PKI configuration. | ||
|
|
||
| - **Token Fetchers** | ||
| - **Management API Token Fetcher**: For use with [`dsh_rest_api_client`](https://crates.io/crates/dsh_rest_api_client). | ||
| - **Protocol Token Fetcher**: Obtain tokens for MQTT and HTTP protocol adapters. | ||
|
|
||
| - **DSH Kafka Configuration** | ||
| - Trait for getting DSH Compatible Kafka Clients (DSH, Proxy, VPN and Local) | ||
| - **RDKafka** implementation | ||
|
|
||
| - **Common Utilities** | ||
| - Prometheus metrics (built-in HTTP server, plus re-export of the `metrics` crate). | ||
| - Tokio-based graceful shutdown handling. | ||
| - Dead Letter Queue (DLQ) functionality. | ||
|
|
||
| --- | ||
|
|
||
| ## Usage | ||
| To use this SDK with the default features in your project, add the following to your Cargo.toml file: | ||
|
|
||
|
|
||
| To get started, add the following to your `Cargo.toml`: | ||
|
|
||
| ```toml | ||
| [dependencies] | ||
| dsh_sdk = "0.5" | ||
| rdkafka = { version = "0.37", features = ["cmake-build", "ssl-vendored"] } | ||
| rdkafka = { version = "0.37", features = ["cmake-build", "ssl-vendored"] } | ||
| ``` | ||
| See [feature flags](#feature-flags) for more information on the available features. | ||
|
|
||
| To use this SDK in your project | ||
| > **Note** | ||
| > By default, this SDK enables several features (see [Feature Flags](#feature-flags)). If you do not need them all, you can disable default features to reduce compile times and dependencies. | ||
| ### Example | ||
|
|
||
| ```rust | ||
| use dsh_sdk::DshKafkaConfig; | ||
| use dsh_sdk::DshKafkaConfig; // Trait for applying DSH-specific configurations | ||
| use rdkafka::consumer::{Consumer, StreamConsumer}; | ||
| use rdkafka::ClientConfig; | ||
|
|
||
| #[tokio::main] | ||
| async fn main() -> Result<(), Box<dyn std::error::Error>>{ | ||
| // get a rdkafka consumer config for example | ||
| let consumer: StreamConsumer = ClientConfig::new().set_dsh_consumer_config().create()?; | ||
| async fn main() -> Result<(), Box<dyn std::error::Error>> { | ||
| // Configure an rdkafka consumer with DSH settings | ||
| let consumer: StreamConsumer = ClientConfig::new() | ||
| .set_dsh_consumer_config() | ||
| .create()?; | ||
|
|
||
| // Your application logic here | ||
|
|
||
| Ok(()) | ||
| } | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Connect to DSH | ||
| The SDK is compatible with running in a container on a DSH tenant, on DSH System Space, on a machine with Kafka Proxy/VPN or on a local machine to a local Kafka. | ||
| See [CONNECT_PROXY_VPN_LOCAL](CONNECT_PROXY_VPN_LOCAL.md) for more info. | ||
|
|
||
| ## Feature flags | ||
| See the [migration guide](https://github.com/kpn-dsh/dsh-sdk-platform-rs/wiki/Migration-guide-(v0.4.X-%E2%80%90--v0.5.X)) for more information on the changes in feature flags since the v0.5.X update. | ||
| This SDK accommodates multiple deployment environments: | ||
| - Running in a container on a DSH tenant | ||
| - Running in DSH System Space | ||
| - Running on a machine with Kafka Proxy/VPN | ||
| - Running locally with a local Kafka instance | ||
|
|
||
| For more information, see the [CONNECT_PROXY_VPN_LOCAL.md](CONNECT_PROXY_VPN_LOCAL.md) document. | ||
|
|
||
| The following features are available in this library and can be enabled/disabled in your Cargo.toml file: | ||
| --- | ||
|
|
||
| ## Feature Flags | ||
|
|
||
| | **feature** | **default** | **Description** | **Example** | | ||
| | --- |--- | --- | --- | | ||
| | `bootstrap` | ✓ | Certificate signing process and fetch datastreams properties | [Kafka](./examples/kafka_example.rs) / [Kafka Proxy](./examples/kafka_proxy.rs) | | ||
| | `kafka` | ✓ | Enable `DshKafkaConfig` trait and Config struct to connect to DSH | [Kafka](./examples/kafka_example.rs) / [Kafka Proxy](./examples/kafka_proxy.rs) | | ||
| | `rdkafka-config` | ✓ | Enable `DshKafkaConfig` implementation for RDKafka | [Kafka](./examples/kafka_example.rs) / [Kafka Proxy](./examples/kafka_proxy.rs) | | ||
| | `schema-store` | ✗ | Interact with DSH Schema Store | [Schema Store API](./examples/schema_store_api.rs) | | ||
| | `protocol-token-fetcher` | ✗ | Fetch tokens to use DSH Protocol adapters (MQTT and HTTP) | [Token fetcher](./examples/protocol_token_fetcher.rs) / [with specific claims](./examples/protocol_token_fetcher_specific_claims.rs) | | ||
| | `management-api-token-fetcher` | ✗ | Fetch tokens to use DSH Management API | [ Token fetcher](./examples/management_api_token_fetcher.rs) | | ||
| | `metrics` | ✗ | Enable prometheus metrics including http server | [Expose metrics](./examples/expose_metrics.rs)| | ||
| | `graceful-shutdown` | ✗ | Tokio based graceful shutdown handler | [Graceful shutdown](./examples/graceful_shutdown.rs) | | ||
| | `dlq` | ✗ | Dead Letter Queue implementation | [Full implementation example](./examples/dlq_implementation.rs) | | ||
| > **Important** | ||
| > The feature flags have changed since the `v0.5.X` update. Check the [migration guide](https://github.com/kpn-dsh/dsh-sdk-platform-rs/wiki/Migration-guide-(v0.4.X-%E2%80%90--v0.5.X)) for details. | ||
| See the [api documentation](https://docs.rs/dsh_sdk/latest/dsh_sdk/) for more information on how to use these features. | ||
| Below is an overview of the available features: | ||
|
|
||
| If you would like to use specific features, you can specify them in your Cargo.toml file. This can save compile time and dependencies. | ||
| For example, if you only want to use the Management API token fetcher feature, add the following to your Cargo.toml file: | ||
| | **feature** | **default** | **Description** | **Example** | | ||
| |--------------------------------|-------------|-------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `bootstrap` | ✓ | Certificate signing process and fetch datastreams properties | [Kafka](./examples/kafka_example.rs) / [Kafka Proxy](./examples/kafka_proxy.rs) | | ||
| | `kafka` | ✓ | Enable `DshKafkaConfig` trait and Config struct to connect to DSH | [Kafka](./examples/kafka_example.rs) / [Kafka Proxy](./examples/kafka_proxy.rs) | | ||
| | `rdkafka-config` | ✓ | Enable `DshKafkaConfig` implementation for RDKafka | [Kafka](./examples/kafka_example.rs) / [Kafka Proxy](./examples/kafka_proxy.rs) | | ||
| | `schema-store` | ✗ | Interact with DSH Schema Store | [Schema Store API](./examples/schema_store_api.rs) | | ||
| | `protocol-token-fetcher` | ✗ | Fetch tokens to use DSH Protocol adapters (MQTT and HTTP) | [Token fetcher](./examples/protocol_token_fetcher.rs) / [with specific claims](./examples/protocol_token_fetcher_specific_claims.rs) | | ||
| | `management-api-token-fetcher` | ✗ | Fetch tokens to use DSH Management API | [ Token fetcher](./examples/management_api_token_fetcher.rs) | | ||
| | `metrics` | ✗ | Enable prometheus metrics including http server | [Expose metrics](./examples/expose_metrics.rs) | | ||
| | `graceful-shutdown` | ✗ | Tokio based graceful shutdown handler | [Graceful shutdown](./examples/graceful_shutdown.rs) | | ||
| | `dlq` | ✗ | Dead Letter Queue implementation | [Full implementation example](./examples/dlq_implementation.rs) | | ||
| ### Selecting Features | ||
|
|
||
| To pick only the features you need, disable the default features and enable specific ones. For instance, if you only want the Management API Token Fetcher: | ||
|
|
||
| ```toml | ||
| [dependencies] | ||
| dsh_sdk = { version = "0.5", default-features = false, features = ["management-api-token-fetcher"] } | ||
| ``` | ||
|
|
||
| ## Environment variables | ||
| The SDK checks environment variables to change configuration for connnecting to DSH. | ||
| See [ENV_VARIABLES.md](ENV_VARIABLES.md) which . | ||
| --- | ||
|
|
||
| ## Environment Variables | ||
|
|
||
| This SDK uses certain environment variables to configure connections to DSH. For a full list of supported variables and their usage, see [ENV_VARIABLES.md](ENV_VARIABLES.md). | ||
|
|
||
| --- | ||
|
|
||
| ## Examples | ||
| See folder [dsh_sdk/examples](./examples/) for simple examples on how to use the SDK. | ||
|
|
||
| ### Full service example | ||
| See folder [example_dsh_service](../example_dsh_service/) for a full service, including how to build the Rust project and post it to Harbor. See [readme](../example_dsh_service/README.md) for more information. | ||
| You can find simple usage examples in the [`examples/` directory](./examples/). | ||
|
|
||
| ### Full Service Example | ||
|
|
||
| A more complete example is provided in the [`example_dsh_service/`](../example_dsh_service/) directory, showcasing: | ||
|
|
||
| - How to build the Rust project | ||
| - How to package and push it to Harbor | ||
| - An end-to-end setup of a DSH service | ||
|
|
||
| See the [README](../example_dsh_service/README.md) in that directory for more information. | ||
|
|
||
| --- | ||
|
|
||
| ## Changelog | ||
| See [CHANGELOG.md](CHANGELOG.md) for all changes per version. | ||
|
|
||
| All changes per version are documented in [CHANGELOG.md](CHANGELOG.md). | ||
|
|
||
| --- | ||
|
|
||
| ## Contributing | ||
| See [CONTRIBUTING.md](../CONTRIBUTING.md) for more information on how to contribute to this project. | ||
|
|
||
| Contributions are welcome! For details on how to help improve this project, please see [CONTRIBUTING.md](../CONTRIBUTING.md). | ||
|
|
||
| --- | ||
|
|
||
| ## License | ||
| See [LICENSE](../LICENSE) for more information on the license for this project. | ||
|
|
||
| This project is licensed under the [Apache License 2.0](../LICENSE). | ||
|
|
||
| --- | ||
|
|
||
| ## Security | ||
| See [SECURITY.md](../SECURITY.md) for more information on the security policy for this project. | ||
|
|
||
| For information about the security policy of this project, including how to report vulnerabilities, see [SECURITY.md](../SECURITY.md). | ||
|
|
||
| --- | ||
| _Copyright (c) Koninklijke KPN N.V._ | ||
|
|
||
| © Koninklijke KPN N.V. | ||
|
|