Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support remote clusters using API keys #8089

Merged
merged 23 commits into from
Oct 31, 2024
Merged

Support remote clusters using API keys #8089

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
851219b
Remote Clusters using API Keys
barkbay Sep 24, 2024
fb34ee7
Cosmetic changes
barkbay Oct 9, 2024
ed29168
Publish the remote cluster Service to the client, not the Pod IP
barkbay Oct 10, 2024
4c2c6fe
Use existing headless service
barkbay Oct 10, 2024
8735592
[DOC] Update issuing node transport certificates with third-party tools
barkbay Oct 11, 2024
5ec1e8d
[E2E] Add end-to-end test
barkbay Oct 11, 2024
875c169
[E2E] Also attempt to search + more details about errors
barkbay Oct 11, 2024
c1223bd
Only delete the keystore which has been initially loaded
barkbay Oct 11, 2024
ab442d1
Add API keystore Secret expectations
barkbay Oct 11, 2024
2a8cd2d
Update from review
barkbay Oct 15, 2024
b6bf462
Add support for access.search.query
barkbay Oct 15, 2024
3225307
Add support for allow_restricted_indices
barkbay Oct 15, 2024
6576910
Fix unit tests
barkbay Oct 15, 2024
e9fd909
Apply Peter's suggestions
barkbay Oct 21, 2024
8c0a6fb
Add ForgetChangeFor
barkbay Oct 21, 2024
c42c3f1
Handle conflict
barkbay Oct 21, 2024
ba62848
Merge remote-tracking branch 'origin/main' into rcs2-pr
barkbay Oct 21, 2024
9af83ee
typos
barkbay Oct 21, 2024
823ee1e
Apply suggestions from code review
barkbay Oct 30, 2024
e4e053a
Update comments
barkbay Oct 30, 2024
c1dab18
Merge remote-tracking branch 'origin/main' into rcs2-pr
barkbay Oct 30, 2024
a35d056
make generate
barkbay Oct 30, 2024
610423a
Fix expected license
barkbay Oct 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pkg/apis/elasticsearch/v1/fields.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ const (

RemoteClusterEnabled = "remote_cluster_server.enabled"
RemoteClusterPublishHost = "remote_cluster.publish_host"
RemoteClusterBindHost = "remote_cluster.bind_host"
RemoteClusterHost = "remote_cluster.host"

NodeName = "node.name"

Expand Down
3 changes: 3 additions & 0 deletions pkg/controller/elasticsearch/certificates/transport/csr.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,10 @@ func buildGeneralNames(
// since these are the ones also used in the context of remote clusters access using API keys.
generalNames = append(
generalNames,
// Remote cluster headless service
certificates.GeneralName{DNSName: fmt.Sprintf("%s.%s.svc", esv1.RemoteClusterService(cluster.Name), cluster.Namespace)},
// Individual remote_cluster.publish_host is set to <pod name>.<statefulset headless service>.<namespace>.svc
certificates.GeneralName{DNSName: fmt.Sprintf("%s.%s.%s.svc", pod.Name, svcName, cluster.Namespace)},
)
}

Expand Down
8 changes: 0 additions & 8 deletions pkg/controller/elasticsearch/nodespec/podspec.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ import (
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/label"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/network"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/securitycontext"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/services"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/settings"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/stackmon"
esvolume "github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/volume"
Expand Down Expand Up @@ -135,13 +134,6 @@ func BuildPodTemplateSpec(
WithContainersSecurityContext(securitycontext.For(ver, enableReadOnlyRootFilesystem)).
WithPreStopHook(*NewPreStopHook())

if es.Spec.RemoteClusterServer.Enabled {
builder = builder.WithEnv(corev1.EnvVar{
Name: settings.EnvRemoteClusterService,
Value: services.RemoteClusterServiceName(es.Name),
})
}

builder, err = stackmon.WithMonitoring(ctx, client, builder, es)
if err != nil {
return corev1.PodTemplateSpec{}, err
Expand Down
25 changes: 18 additions & 7 deletions pkg/controller/elasticsearch/nodespec/statefulset.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (
sset "github.com/elastic/cloud-on-k8s/v2/pkg/controller/common/statefulset"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/label"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/network"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/services"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/settings"
es_sset "github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/sset"
esvolume "github.com/elastic/cloud-on-k8s/v2/pkg/controller/elasticsearch/volume"
Expand All @@ -33,6 +34,22 @@ func HeadlessServiceName(ssetName string) string {
// HeadlessService returns a headless service for the given StatefulSet
func HeadlessService(es *esv1.Elasticsearch, ssetName string) corev1.Service {
nsn := k8s.ExtractNamespacedName(es)
ports := []corev1.ServicePort{
{
Name: es.Spec.HTTP.Protocol(),
Protocol: corev1.ProtocolTCP,
Port: network.HTTPPort,
},
}
if es.Spec.RemoteClusterServer.Enabled {
ports = append(ports,
corev1.ServicePort{
Name: services.RemoteClusterServicePortName,
Protocol: corev1.ProtocolTCP,
Port: network.RemoteClusterPort,
},
)
}

return corev1.Service{
ObjectMeta: metav1.ObjectMeta{
Expand All @@ -44,13 +61,7 @@ func HeadlessService(es *esv1.Elasticsearch, ssetName string) corev1.Service {
Type: corev1.ServiceTypeClusterIP,
ClusterIP: corev1.ClusterIPNone,
Selector: label.NewStatefulSetLabels(nsn, ssetName),
Ports: []corev1.ServicePort{
{
Name: es.Spec.HTTP.Protocol(),
Protocol: corev1.ProtocolTCP,
Port: network.HTTPPort,
},
},
Ports: ports,
// allow nodes to discover themselves via DNS while they are booting up ie. are not ready yet
PublishNotReadyAddresses: true,
},
Expand Down
25 changes: 15 additions & 10 deletions pkg/controller/elasticsearch/services/services.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ import (

const (
globalServiceSuffix = ".svc"

RemoteClusterServicePortName = "rcs"
)

// TransportServiceName returns the name for the transport service associated to this cluster
Expand Down Expand Up @@ -153,25 +155,28 @@ func NewInternalService(es esv1.Elasticsearch) *corev1.Service {

// NewRemoteClusterService returns the service associated to the remote cluster service for the given cluster.
func NewRemoteClusterService(es esv1.Elasticsearch) *corev1.Service {
return &corev1.Service{
svc := &corev1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: RemoteClusterServiceName(es.Name),
Namespace: es.Namespace,
Labels: label.NewLabels(k8s.ExtractNamespacedName(&es)),
},
Spec: corev1.ServiceSpec{
Type: corev1.ServiceTypeClusterIP,
Ports: []corev1.ServicePort{
{
Name: "rcs",
Protocol: corev1.ProtocolTCP,
Port: network.RemoteClusterPort,
},
},
Type: corev1.ServiceTypeClusterIP,
Selector: label.NewLabels(k8s.ExtractNamespacedName(&es)),
PublishNotReadyAddresses: false,
PublishNotReadyAddresses: true,
ClusterIP: "None",
},
}
labels := label.NewLabels(k8s.ExtractNamespacedName(&es))
ports := []corev1.ServicePort{
{
Name: RemoteClusterServicePortName,
Protocol: corev1.ProtocolTCP,
Port: network.RemoteClusterPort,
},
}
return defaults.SetServiceDefaults(svc, labels, labels, ports)
}

type urlProvider struct {
Expand Down
1 change: 0 additions & 1 deletion pkg/controller/elasticsearch/settings/environment.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ const (
EnvProbeUsername = "PROBE_USERNAME"
EnvReadinessProbeProtocol = "READINESS_PROBE_PROTOCOL"
HeadlessServiceName = "HEADLESS_SERVICE_NAME"
EnvRemoteClusterService = "REMOTE_CLUSTER_SERVICE"

// These are injected as env var into the ES pod at runtime,
// to be referenced in ES configuration file
Expand Down
7 changes: 2 additions & 5 deletions pkg/controller/elasticsearch/settings/merged_config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,8 @@ func baseConfig(clusterName string, ver version.Version, ipFamily corev1.IPFamil

if remoteClusterServerEnabled {
cfg[esv1.RemoteClusterEnabled] = "true"
// This is required when each Pod server certificates do not contain the Pod's IP address, which might be the case
// if the certificates are generated by the cert-manager CSI driver for example. Using this parameter the remote client
// cluster is going to try to connect to the remote cluster service using the Service and each specific Pod IP.
cfg[esv1.RemoteClusterPublishHost] = "${" + EnvRemoteClusterService + "}.${" + EnvNamespace + "}.svc"
cfg[esv1.RemoteClusterBindHost] = "0.0.0.0"
cfg[esv1.RemoteClusterPublishHost] = "${" + EnvPodName + "}.${" + HeadlessServiceName + "}.${" + EnvNamespace + "}.svc"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default the published host is the Pod's IP address. While that IP address is automatically added in the ECK managed transport certificate it is not possible to include it when using the cert-manager (cert-manager/csi-driver#17).

That's why I decided to use the Pod hostname as available through the existing headless Service (so it can be resolved by other Pods).

It still has the downside that when using cert-manager CSI driver, the csi.cert-manager.io/dns-names is now a bit involved, something along the lines of:

            - name: transport-certs
              csi:
                driver: csi.cert-manager.io
                readOnly: true
                volumeAttributes:
                  csi.cert-manager.io/issuer-name: ca-cluster-issuer
                  csi.cert-manager.io/issuer-kind: ClusterIssuer
                  csi.cert-manager.io/dns-names: "${POD_NAME}.${POD_NAMESPACE}.svc.cluster.local,${POD_NAME}.<cluster-name>-es-<nodeset-name>.${POD_NAMESPACE}.svc,<cluster-name>-es-remote-cluster.${POD_NAMESPACE}.svc"
  • ${POD_NAME}.${POD_NAMESPACE}.svc.cluster.local is the existing, recommended DNS name, from our documentation (I guess it only works because of verification_mode: certificate in the transport configuration)
  • ${POD_NAME}.<cluster-name>-es-<nodeset-name>.${POD_NAMESPACE}.svc is to match the published host.
  • <cluster-name>-es-remote-cluster.${POD_NAMESPACE}.svc is to match the remote cluster service.

(I think an alternative would be to try to use verification_mode: certificate for the remote cluster server, this is something I wanted to avoid)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine

cfg[esv1.RemoteClusterHost] = "0"
}

// seed hosts setting name changed starting ES 7.X
Expand Down
2 changes: 1 addition & 1 deletion pkg/controller/elasticsearch/settings/merged_config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ func TestNewMergedESConfig(t *testing.T) {
// Remote cluster server configuration.
require.Equal(t, 1, len(cfg.HasKeys([]string{"remote_cluster_server.enabled"})))
require.Equal(t, 1, len(cfg.HasKeys([]string{"remote_cluster.publish_host"})))
require.Equal(t, 1, len(cfg.HasKeys([]string{"remote_cluster.bind_host"})))
require.Equal(t, 1, len(cfg.HasKeys([]string{"remote_cluster.host"})))
// Remote cluster server TLS configuration.
require.Equal(t, 1, len(cfg.HasKeys([]string{"xpack.security.remote_cluster_server.ssl.key"})))
require.Equal(t, 1, len(cfg.HasKeys([]string{"xpack.security.remote_cluster_server.ssl.certificate"})))
Expand Down