Skip to content

Commit

Permalink
NOISSUE - Sync Env Veriables With Docker Deployment (absmach#1841)
Browse files Browse the repository at this point in the history 
* Initial Commit: Sync Env Veriables With Docker Deployment

Signed-off-by: rodneyosodo <[email protected]>

* Sync Env Vars With Master

Signed-off-by: rodneyosodo <[email protected]>

* Remove Altprefix

Signed-off-by: rodneyosodo <[email protected]>

* Rename HttpPort to HTTPPort

Signed-off-by: rodneyosodo <[email protected]>

* Fix envPrefixDB After Rebase

Signed-off-by: rodneyosodo <[email protected]>

* Remove Server Parse

Signed-off-by: rodneyosodo <[email protected]>

* Add Provision For TLS on CoAP

Signed-off-by: rodneyosodo <[email protected]>

* Fix Exit After Defer

Signed-off-by: rodneyosodo <[email protected]>

* Remove Unused Function

Signed-off-by: rodneyosodo <[email protected]>

* Document Undocumentated Env Variables

Signed-off-by: rodneyosodo <[email protected]>

---------

Signed-off-by: rodneyosodo <[email protected]>
Co-authored-by: Drasko DRASKOVIC <[email protected]>
  • Loading branch information
@rodneyosodo @drasko
rodneyosodo and drasko authored Jul 31, 2023
1 parent b7b14cc commit 3e7bac4
Show file tree
Hide file tree
Showing 85 changed files with 2,009 additions and 1,572 deletions.
109 changes: 57 additions & 52 deletions bootstrap/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

New devices need to be configured properly and connected to the Mainflux. Bootstrap service is used in order to accomplish that. This service provides the following features:

1) Creating new Mainflux Things
2) Providing basic configuration for the newly created Things
3) Enabling/disabling Things
1. Creating new Mainflux Things
2. Providing basic configuration for the newly created Things
3. Enabling/disabling Things

Pre-provisioning a new Thing is as simple as sending Configuration data to the Bootstrap service. Once the Thing is online, it sends a request for initial config to Bootstrap service. Bootstrap service provides an API for enabling and disabling Things. Only enabled Things can exchange messages over Mainflux. Bootstrapping does not implicitly enable Things, it has to be done manually.

Expand All @@ -14,18 +14,18 @@ In order to bootstrap successfully, the Thing needs to send bootstrapping reques

Thing Configuration consists of two logical parts: the custom configuration that can be interpreted by the Thing itself and Mainflux-related configuration. Mainflux config contains:

1) corresponding Mainflux Thing ID
2) corresponding Mainflux Thing key
3) list of the Mainflux channels the Thing is connected to
1. corresponding Mainflux Thing ID
2. corresponding Mainflux Thing key
3. list of the Mainflux channels the Thing is connected to

>Note: list of channels contains IDs of the Mainflux channels. These channels are _pre-provisioned_ on the Mainflux side and, unlike corresponding Mainflux Thing, Bootstrap service is not able to create Mainflux Channels.
> Note: list of channels contains IDs of the Mainflux channels. These channels are _pre-provisioned_ on the Mainflux side and, unlike corresponding Mainflux Thing, Bootstrap service is not able to create Mainflux Channels.
Enabling and disabling Thing (adding Thing to/from whitelist) is as simple as connecting corresponding Mainflux Thing to the given list of Channels. Configuration keeps _state_ of the Thing:

| State | What it means |
|----------|--------------------------------------------------------|
| Inactive | Thing is created, but isn't enabled |
| Active | Thing is able to communicate using Mainflux |
| State | What it means |
| -------- | ------------------------------------------- |
| Inactive | Thing is created, but isn't enabled |
| Active | Thing is able to communicate using Mainflux |

Switching between states `Active` and `Inactive` enables and disables Thing, respectively.

Expand All @@ -35,40 +35,39 @@ Thing configuration also contains the so-called `external ID` and `external key`

The service is configured using the environment variables presented in the following table. Note that any unset variables will be replaced with their default values.

| Variable | Description | Default |
|-------------------------------|-------------------------------------------------------------------------|----------------------------------|
| MF_BOOTSTRAP_LOG_LEVEL | Log level for Bootstrap (debug, info, warn, error) | info |
| MF_BOOTSTRAP_DB_HOST | Database host address | localhost |
| MF_BOOTSTRAP_DB_PORT | Database host port | 5432 |
| MF_BOOTSTRAP_DB_USER | Database user | mainflux |
| MF_BOOTSTRAP_DB_PASS | Database password | mainflux |
| MF_BOOTSTRAP_DB | Name of the database used by the service | bootstrap |
| MF_BOOTSTRAP_DB_SSL_MODE | Database connection SSL mode (disable, require, verify-ca, verify-full) | disable |
| MF_BOOTSTRAP_DB_SSL_CERT | Path to the PEM encoded certificate file | |
| MF_BOOTSTRAP_DB_SSL_KEY | Path to the PEM encoded key file | |
| MF_BOOTSTRAP_DB_SSL_ROOT_CERT | Path to the PEM encoded root certificate file | |
| MF_BOOTSTRAP_ENCRYPT_KEY | Secret key for secure bootstrapping encryption | 12345678910111213141516171819202 |
| MF_BOOTSTRAP_CLIENT_TLS | Flag that indicates if TLS should be turned on | false |
| MF_BOOTSTRAP_CA_CERTS | Path to trusted CAs in PEM format | |
| MF_BOOTSTRAP_PORT | Bootstrap service HTTP port | 9013 |
| MF_BOOTSTRAP_SERVER_CERT | Path to server certificate in pem format | |
| MF_BOOTSTRAP_SERVER_KEY | Path to server key in pem format | |
| MF_THINGS_URL | Base url for Mainflux Things | http://localhost:9000 |
| MF_THINGS_ES_URL | Things service event source URL | localhost:6379 |
| MF_THINGS_ES_PASS | Things service event source password | |
| MF_THINGS_ES_DB | Things service event source database | 0 |
| MF_BOOTSTRAP_ES_URL | Bootstrap service event source URL | localhost:6379 |
| MF_BOOTSTRAP_ES_PASS | Bootstrap service event source password | |
| MF_BOOTSTRAP_ES_DB | Bootstrap service event source database | 0 |
| MF_BOOTSTRAP_EVENT_CONSUMER | Bootstrap service event source consumer name | bootstrap |
| MF_JAEGER_URL | Jaeger server URL | localhost:6831 |
| MF_AUTH_GRPC_URL | Users service gRPC URL | localhost:7001 |
| MF_AUTH_GRPC_TIMEOUT | Users service gRPC request timeout in seconds | 1s |
| MF_SEND_TELEMETRY | Send telemetry to mainflux call home server | true |
| Variable | Description | Default |
| ----------------------------- | ----------------------------------------------------------------------- | -------------------------------------------------- |
| MF_BOOTSTRAP_LOG_LEVEL | Log level for Bootstrap (debug, info, warn, error) | info |
| MF_BOOTSTRAP_DB_HOST | Database host address | localhost |
| MF_BOOTSTRAP_DB_PORT | Database host port | 5432 |
| MF_BOOTSTRAP_DB_USER | Database user | mainflux |
| MF_BOOTSTRAP_DB_PASS | Database password | mainflux |
| MF_BOOTSTRAP_DB_NAME | Name of the database used by the service | bootstrap |
| MF_BOOTSTRAP_DB_SSL_MODE | Database connection SSL mode (disable, require, verify-ca, verify-full) | disable |
| MF_BOOTSTRAP_DB_SSL_CERT | Path to the PEM encoded certificate file | |
| MF_BOOTSTRAP_DB_SSL_KEY | Path to the PEM encoded key file | |
| MF_BOOTSTRAP_DB_SSL_ROOT_CERT | Path to the PEM encoded root certificate file | |
| MF_BOOTSTRAP_ENCRYPT_KEY | Secret key for secure bootstrapping encryption | v7aT0HGxJxt2gULzr3RHwf4WIf6DusPphG5Ftm2bNCWD8mTpyr |
| MF_BOOTSTRAP_HTTP_HOST | Bootstrap service HTTP host | |
| MF_BOOTSTRAP_HTTP_PORT | Bootstrap service HTTP port | 9013 |
| MF_BOOTSTRAP_HTTP_SERVER_CERT | Path to server certificate in pem format | |
| MF_BOOTSTRAP_HTTP_SERVER_KEY | Path to server key in pem format | |
| MF_BOOTSTRAP_EVENT_CONSUMER | Bootstrap service event source consumer name | bootstrap |
| MF_BOOTSTRAP_ES_URL | Bootstrap service event source URL | localhost:6379 |
| MF_BOOTSTRAP_ES_PASS | Bootstrap service event source password | |
| MF_BOOTSTRAP_ES_DB | Bootstrap service event source database | 0 |
| MF_AUTH_GRPC_URL | Users service gRPC URL | localhost:7001 |
| MF_AUTH_GRPC_TIMEOUT | Users service gRPC request timeout in seconds | 1s |
| MF_AUTH_GRPC_CLIENT_TLS | Enable TLS for gRPC client | false |
| MF_AUTH_GRPC_CA_CERTS | CA certificates for gRPC client | |
| MF_THINGS_URL | Base url for Mainflux Things | http://localhost:9000 |
| MF_JAEGER_URL | Jaeger server URL | http://jaeger:14268/api/traces |
| MF_SEND_TELEMETRY | Send telemetry to mainflux call home server | true |
| MF_BOOTSTRAP_INSTANCE_ID | Bootstrap service instance ID | |

## Deployment

The service itself is distributed as Docker container. Check the [`boostrap`](https://github.com/mainflux/mainflux/blob/master/docker/addons/bootstrap/docker-compose.yml#L32-L56) service section in
The service itself is distributed as Docker container. Check the [`boostrap`](https://github.com/mainflux/mainflux/blob/master/docker/addons/bootstrap/docker-compose.yml#L32-L56) service section in
docker-compose to see how service is deployed.

To start the service outside of the container, execute the following shell script:
Expand All @@ -87,26 +86,32 @@ make install

# set the environment variables and run the service
MF_BOOTSTRAP_LOG_LEVEL=[Bootstrap log level] \
MF_BOOTSTRAP_ENCRYPT_KEY=[Hex-encoded encryption key used for secure bootstrap] \
MF_BOOTSTRAP_EVENT_CONSUMER=[Bootstrap service event source consumer name] \
MF_BOOTSTRAP_ES_URL=[Bootstrap service event source URL] \
MF_BOOTSTRAP_ES_PASS=[Bootstrap service event source password] \
MF_BOOTSTRAP_ES_DB=[Bootstrap service event source database] \
MF_BOOTSTRAP_HTTP_HOST=[Bootstrap service HTTP host] \
MF_BOOTSTRAP_HTTP_PORT=[Bootstrap service HTTP port] \
MF_BOOTSTRAP_HTTP_SERVER_CERT=[Path to HTTP server certificate in pem format] \
MF_BOOTSTRAP_HTTP_SERVER_KEY=[Path to HTTP server key in pem format] \
MF_BOOTSTRAP_DB_HOST=[Database host address] \
MF_BOOTSTRAP_DB_PORT=[Database host port] \
MF_BOOTSTRAP_DB_USER=[Database user] \
MF_BOOTSTRAP_DB_PASS=[Database password] \
MF_BOOTSTRAP_DB=[Name of the database used by the service] \
MF_BOOTSTRAP_DB_NAME=[Name of the database used by the service] \
MF_BOOTSTRAP_DB_SSL_MODE=[SSL mode to connect to the database with] \
MF_BOOTSTRAP_DB_SSL_CERT=[Path to the PEM encoded certificate file] \
MF_BOOTSTRAP_DB_SSL_KEY=[Path to the PEM encoded key file] \
MF_BOOTSTRAP_DB_SSL_ROOT_CERT=[Path to the PEM encoded root certificate file] \
MF_BOOTSTRAP_ENCRYPT_KEY=[Hex-encoded encryption key used for secure bootstrap] \
MF_BOOTSTRAP_CLIENT_TLS=[Boolean value to enable/disable client TLS] \
MF_BOOTSTRAP_CA_CERTS=[Path to trusted CAs in PEM format] \
MF_BOOTSTRAP_PORT=[Service HTTP port] \
MF_BOOTSTRAP_SERVER_CERT=[Path to server certificate] \
MF_BOOTSTRAP_SERVER_KEY=[Path to server key] \
MF_SDK_BASE_URL=[Base SDK URL for the Mainflux services] \
MF_SDK_THINGS_PREFIX=[SDK prefix for Things service] \
MF_JAEGER_URL=[Jaeger server URL] \
MF_AUTH_GRPC_URL=[Users service gRPC URL] \
MF_AUTH_GRPC_TIMEOUT=[Users service gRPC request timeout in seconds] \
MF_AUTH_GRPC_CLIENT_TLS=[Boolean value to enable/disable client TLS] \
MF_AUTH_GRPC_CA_CERT=[Path to trusted CAs in PEM format] \
MF_THINGS_URL=[Base url for Mainflux Things] \
MF_JAEGER_URL=[Jaeger server URL] \
MF_SEND_TELEMETRY=[Send telemetry to mainflux call home server] \
MF_BOOTSTRAP_INSTANCE_ID=[Bootstrap instance ID] \
$GOBIN/mainflux-bootstrap
```

Expand Down
4 changes: 4 additions & 0 deletions certs/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
# Certs Service

Issues certificates for things. `Certs` service can create certificates to be used when `Mainflux` is deployed to support mTLS.
Certificate service can create certificates in two modes:

1. Development mode - to be used when no PKI is deployed, this works similar to the [make thing_cert](../docker/ssl/Makefile)
2. PKI mode - certificates issued by PKI, when you deploy `Vault` as PKI certificate management `cert` service will proxy requests to `Vault` previously checking access rights and saving info on successfully created certificate.

## Development mode

If `MF_CERTS_VAULT_HOST` is empty than Development mode is on.

To issue a certificate:

```bash

TOK=`curl -s --insecure -S -X POST http://localhost/tokens -H 'Content-Type: application/json' -d '{"email":"[email protected]","password":"12345678"}' | jq -r '.token'`
Expand Down
50 changes: 26 additions & 24 deletions cmd/bootstrap/main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,12 @@ import (
"log"
"os"

"github.com/go-redis/redis/v8"
"github.com/jmoiron/sqlx"
chclient "github.com/mainflux/callhome/pkg/client"
"github.com/mainflux/mainflux"
"go.opentelemetry.io/otel/trace"

"github.com/mainflux/mainflux/bootstrap"
"github.com/mainflux/mainflux/bootstrap/api"
bootstrapPg "github.com/mainflux/mainflux/bootstrap/postgres"
rediscons "github.com/mainflux/mainflux/bootstrap/redis/consumer"
redisprod "github.com/mainflux/mainflux/bootstrap/redis/producer"
Expand All @@ -31,21 +33,17 @@ import (
mfsdk "github.com/mainflux/mainflux/pkg/sdk/go"
"github.com/mainflux/mainflux/pkg/uuid"
"github.com/mainflux/mainflux/users/policies"
"go.opentelemetry.io/otel/trace"
"golang.org/x/sync/errgroup"

"github.com/go-redis/redis/v8"
"github.com/jmoiron/sqlx"
"github.com/mainflux/mainflux/bootstrap"
"github.com/mainflux/mainflux/bootstrap/api"
)

const (
svcName = "bootstrap"
envPrefix = "MF_BOOTSTRAP_"
envPrefixDB = "MF_BOOTSTRAP_DB_"
envPrefixES = "MF_BOOTSTRAP_ES_"
envPrefixHttp = "MF_BOOTSTRAP_HTTP_"
envPrefixHTTP = "MF_BOOTSTRAP_HTTP_"
defDB = "bootstrap"
defSvcHttpPort = "9013"
defSvcHTTPPort = "9013"
)

type config struct {
Expand All @@ -72,23 +70,27 @@ func main() {
log.Fatalf("failed to init logger: %s", err)
}

var exitCode int
defer mflog.ExitWithError(&exitCode)

instanceID := cfg.InstanceID
if instanceID == "" {
instanceID, err = uuid.New().ID()
if err != nil {
log.Fatalf("Failed to generate instanceID: %s", err)
logger.Error(fmt.Sprintf("Failed to generate instanceID: %s", err))
exitCode = 1
return
}
}

// Create new postgres client
dbConfig := pgClient.Config{Name: defDB}

db, err := pgClient.SetupWithConfig(envPrefix, *bootstrapPg.Migration(), dbConfig)
db, err := pgClient.SetupWithConfig(envPrefixDB, *bootstrapPg.Migration(), dbConfig)
if err != nil {
logger.Fatal(err.Error())
logger.Error(err.Error())
exitCode = 1
return
}
var exitCode int
defer mflog.ExitWithError(&exitCode)
defer db.Close()

// Create new redis client for bootstrap event store
Expand All @@ -101,7 +103,7 @@ func main() {
defer esClient.Close()

// Create new auth grpc client api
auth, authHandler, err := authClient.Setup(envPrefix, svcName)
auth, authHandler, err := authClient.Setup(svcName)
if err != nil {
logger.Error(err.Error())
exitCode = 1
Expand All @@ -112,10 +114,12 @@ func main() {

tp, err := jaegerClient.NewProvider(svcName, cfg.JaegerURL, instanceID)
if err != nil {
logger.Fatal(fmt.Sprintf("failed to init Jaeger: %s", err))
logger.Error(fmt.Sprintf("failed to init Jaeger: %s", err))
exitCode = 1
return
}
defer func() {
if err := tp.Shutdown(context.Background()); err != nil {
if err := tp.Shutdown(ctx); err != nil {
logger.Error(fmt.Sprintf("error shutting down tracer provider: %v", err))
}
}()
Expand All @@ -125,8 +129,8 @@ func main() {
svc := newService(auth, db, tracer, logger, esClient, cfg)

// Create an new HTTP server
httpServerConfig := server.Config{Port: defSvcHttpPort}
if err := env.Parse(&httpServerConfig, env.Options{Prefix: envPrefixHttp, AltPrefix: envPrefix}); err != nil {
httpServerConfig := server.Config{Port: defSvcHTTPPort}
if err := env.Parse(&httpServerConfig, env.Options{Prefix: envPrefixHTTP}); err != nil {
logger.Error(fmt.Sprintf("failed to load %s HTTP server configuration : %s", svcName, err))
exitCode = 1
return
Expand All @@ -150,9 +154,7 @@ func main() {
// Subscribe to things event store
thingsESClient, err := redisClient.Setup(envPrefixES)
if err != nil {
logger.Error(err.Error())
exitCode = 1
return
logger.Fatal(err.Error())
}
defer thingsESClient.Close()

Expand Down
Loading

0 comments on commit 3e7bac4

Please sign in to comment.