Merge collab-galley to master (istio#15891)

* Introduce galley/pkg/config/resource (istio#14725)

* Add pkg/config/resource

* Fix lint issues.

* Add pkg/config/collection (istio#14749)

* Add pkg/config/collection

* Remove commented out code.

* Make go linter happy.

* Fix CR comments.

* Delete runaway file.

* Add pkg/config/event (istio#14825)

* Add pkg/config/schema for Galley config metadata (istio#14852)

* Add pkg/config/schema

pkg/config/schema contains the main metadata parser/data model for Galley.
+ Adds the ast, model, parser &validator for the metadata model.
+ Adds codegeneration tools for generating code artifacts: mainly for pulling
in proto types, and having well-defined collection names.

* Add more schema tests.

* Fix lint errors.

* [collab-galley] Add basic kubernetes file processing utilities (istio#14965)

* Add pkg/config/util

* Linter fix.

* [collab-galley] Add pkg/config/meshcfg  (istio#15044)

* Add pkg/config/meshcfg

* Simplify concurrency model

* Move pkg/features/pilot to pilot/pkg/features.

This cleans up the /pkg package, in preparation of multi-repo.

* Add pkg/config/source/inmemory (istio#15100)

* [collab-galley] Add K8s core type model and in-memory support. (istio#15112)

* Add K8s core type model and in-memory support.

* Fix linter issues

* Accommodate CR feedback.

* [collab-galley] Add Filesystem source support. (istio#15163)

* Add Filesystem source support.

* Fix lint errors.

* [collab-galley] Introduce pkg/config/source/kube/apiserver (istio#15169)

* Introduce pkg/config/source/kube/apiserver

* Fix lint errors.

* Handle no-informer case.

* fix lint issues.

* Accommodate CR feedback.

* [collab-galley] Introduce the config processing engine (istio#15361)

* Introduce pkg/config/processing

* Remove monitoring code.

* Streamline session code.

* More Streamline session code.

* Fixup remaining state updates.

* Fix lint issues.

* Add comments to runtime for better explanation.

* Accommodate CR feedback.

* Add event.MergeSources.

* Add Source merging support.

* Fix lint error.

* Interim factory model.

* Fix lint error

* Rename combinedSources.

* Deflake TestRuntime_MeshEvent_WhileRunning

* Accommodate CR feedback.

* fix lint error

* fix test.

* Streamline logging scopes.

* Fix lint error.

* Fix lint error.

* Fix test.

* Fix test.

* fix lint issue

* [collab-galley] Complete the new config processor (istio#15580)

* Introducing pkg/config/processor

- Basic changes before landing the processor code.
- Add a event.NewFnTransform, that allows creating transformers from functions.
- Add various utility methods in eventing code.
- Add synchronization in Runtime to avoid missing initial events, due to timings.

* Add basic metadata.

* Add authpolicy and direct converters.

* Add Ingress transform

* Direct copy of transform code.

* Transform servieceentry code to the new model.

* Complete pkg/processor.

* Fix lint errors

* Fix lint errors & tests.

* Accommodate feedback.

* [collab-galley] Merge a few critical PRs from master to collab-galley (istio#15649)

* Refactor server.Args out to its own package. (istio#15202)

* Refactor server.Args out to its own package.

- Move all settings into Args struct.
- Make handling of defaults in command-line uniform.
- Minor code analysis fixes.

* Add missing edit.

* Fix lint errors

* fix server_test.go

* suppress lint check

* Fix imports.

* Add a process package to galley/pkg/server for tracking sub-components (istio#15203)

* Add a process package to pkg/server for tracking sub-component.

- process.Host is a basic container of multiple sub-components.
- process.Component is an interface to be implemented by sub-components.

* Fix lint errors.

* Accommodate CR feedback.

* Componentize code by moving it into server/components. (istio#15227)

# Conflicts:
#	galley/pkg/server/server.go

* [collab-galley] Wire-up the new pipeline (flag protected, off by default) (istio#15651)

* Wire-up the new processing pipeline.

* Fix comment

* Remove dashboard entries for defunct metricx.

* Accommodate feedback.

* Accommodate feedback.

* Accommodate feedback.

* Accommodate feedback. (istio#15661)

*  [collab-galley] CRD Discovery support (istio#15652)

* Add CRD listening support to Galley API Server Source.

With CRD support, Galley will discover CRDs dynamically and will listen
to only available ones. If Galley detects a change to the CRDs, then it
will restart its config processing pipeline.

Also add missing metadata entries for legacy Mixer resources.

* Accommodate CR feedback.

* Fix a crash, where the wrong collection name is passed to a snapshot. (istio#15818)

* Fix lint issues.

* Fix Yaml lint issues.

bin/yamllintrules

@@ -12,6 +12,10 @@ rules:
   colons:
     ignore: |
       /galley/tools/gen-meta/metadata.yaml
+      /galley/pkg/config/processor/metadata/metadata.yaml
+      /galley/pkg/config/testing/basicmeta/basicmeta.yaml
+      /galley/pkg/config/testing/basicmeta/basicmeta2.yaml
+      /galley/pkg/config/testing/k8smeta/k8smeta.yaml
   commas: disable
   comments: disable
   comments-indentation: disable

galley/cmd/galley/cmd/server.go

@@ -150,6 +150,8 @@ func serverCmd() *cobra.Command {
 		serverArgs.SinkMeta, "Comma-separated list of key=values to attach as metadata to outgoing sink connections. Ex: 'key=value,key2=value2'")
 	serverCmd.PersistentFlags().BoolVar(&serverArgs.EnableServiceDiscovery, "enableServiceDiscovery", false,
 		"Enable service discovery processing in Galley")
+	serverCmd.PersistentFlags().BoolVar(&serverArgs.UseOldProcessor, "useOldProcessor", serverArgs.UseOldProcessor,
+		"Use the old processing pipeline for config processing")
 
 	// validation config
 	serverCmd.PersistentFlags().StringVar(&serverArgs.ValidationArgs.WebhookConfigFile,

galley/pkg/config/collection/instance.go

@@ -0,0 +1,117 @@
+// Copyright 2019 Istio Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package collection
+
+import (
+	"sync"
+
+	"istio.io/istio/galley/pkg/config/resource"
+)
+
+// ChangeNotifierFn is called when a collection instance changes.
+type ChangeNotifierFn func()
+
+// Instance is collection of resources, indexed by name.
+type Instance struct {
+	mu          sync.RWMutex // TODO: This lock will most likely cause contention. We should investigate whether removing it would help.
+	collection  Name
+	generation  int64
+	entries     map[resource.Name]*resource.Entry
+	copyOnWrite bool
+}
+
+// New returns a new collection.Instance
+func New(collection Name) *Instance {
+	return &Instance{
+		collection: collection,
+		entries:    make(map[resource.Name]*resource.Entry),
+	}
+}
+
+// Generation of the current state of the collection.Instance
+func (c *Instance) Generation() int64 {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	return c.generation
+}
+
+// Size returns the number of items in the set
+func (c *Instance) Size() int {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	return len(c.entries)
+}
+
+// ForEach executes the given function for each entry
+func (c *Instance) ForEach(fn func(e *resource.Entry)) {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	for _, e := range c.entries {
+		fn(e)
+	}
+}
+
+// Set an entry in the collection
+func (c *Instance) Set(r *resource.Entry) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.doCopyOnWrite()
+	c.generation++
+	c.entries[r.Metadata.Name] = r
+}
+
+// Remove an entry from the collection.
+func (c *Instance) Remove(n resource.Name) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.doCopyOnWrite()
+	c.generation++
+	delete(c.entries, n)
+}
+
+// Clear the contents of this instance.
+func (c *Instance) Clear() {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.doCopyOnWrite()
+	c.generation++
+	c.entries = make(map[resource.Name]*resource.Entry)
+}
+
+func (c *Instance) doCopyOnWrite() { // TODO: we should optimize copy-on write.
+	if !c.copyOnWrite {
+		return
+	}
+
+	m := make(map[resource.Name]*resource.Entry)
+	for k, v := range c.entries {
+		m[k] = v
+	}
+	c.entries = m
+	c.copyOnWrite = false
+}
+
+// Clone the instance
+func (c *Instance) Clone() *Instance {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.copyOnWrite = true
+	return &Instance{
+		collection:  c.collection,
+		generation:  c.generation,
+		entries:     c.entries,
+		copyOnWrite: true,
+	}
+}

galley/pkg/config/collection/instance_test.go

@@ -0,0 +1,109 @@
+// Copyright 2019 Istio Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package collection_test
+
+import (
+	"testing"
+
+	. "github.com/onsi/gomega"
+
+	"istio.io/istio/galley/pkg/config/collection"
+	"istio.io/istio/galley/pkg/config/resource"
+	"istio.io/istio/galley/pkg/config/testing/data"
+)
+
+func TestInstance_Basics(t *testing.T) {
+	g := NewGomegaWithT(t)
+
+	inst := collection.New(data.Collection1)
+
+	g.Expect(inst.Size()).To(Equal(0))
+
+	var fe []*resource.Entry
+	inst.ForEach(func(e *resource.Entry) {
+		fe = append(fe, e)
+	})
+	g.Expect(fe).To(HaveLen(0))
+
+	g.Expect(inst.Generation()).To(Equal(int64(0)))
+
+	inst.Set(data.EntryN1I1V2)
+	inst.Set(data.EntryN2I2V2)
+
+	g.Expect(inst.Size()).To(Equal(2))
+
+	fe = nil
+	inst.ForEach(func(e *resource.Entry) {
+		fe = append(fe, e)
+	})
+	g.Expect(fe).To(HaveLen(2))
+
+	g.Expect(inst.Generation()).To(Equal(int64(2)))
+
+	inst.Remove(data.EntryN1I1V1.Metadata.Name)
+
+	g.Expect(inst.Size()).To(Equal(1))
+
+	fe = nil
+	inst.ForEach(func(e *resource.Entry) {
+		fe = append(fe, e)
+	})
+	g.Expect(fe).To(HaveLen(1))
+
+	g.Expect(inst.Generation()).To(Equal(int64(3)))
+
+	inst.Clear()
+
+	fe = nil
+	inst.ForEach(func(e *resource.Entry) {
+		fe = append(fe, e)
+	})
+	g.Expect(fe).To(HaveLen(0))
+
+	g.Expect(inst.Generation()).To(Equal(int64(4)))
+	g.Expect(inst.Size()).To(Equal(0))
+
+}
+
+func TestInstance_Clone(t *testing.T) {
+	g := NewGomegaWithT(t)
+
+	inst := collection.New(data.Collection1)
+	inst.Set(data.EntryN1I1V1)
+	inst.Set(data.EntryN2I2V2)
+
+	inst2 := inst.Clone()
+
+	g.Expect(inst2.Size()).To(Equal(2))
+	g.Expect(inst2.Generation()).To(Equal(int64(2)))
+
+	var fe []*resource.Entry
+	inst2.ForEach(func(e *resource.Entry) {
+		fe = append(fe, e)
+	})
+	g.Expect(fe).To(HaveLen(2))
+
+	inst.Remove(data.EntryN1I1V1.Metadata.Name)
+
+	g.Expect(inst2.Size()).To(Equal(2))
+	g.Expect(inst2.Generation()).To(Equal(int64(2)))
+
+	fe = nil
+	inst2.ForEach(func(e *resource.Entry) {
+		fe = append(fe, e)
+	})
+
+	g.Expect(fe).To(HaveLen(2))
+}

galley/pkg/config/collection/name.go

@@ -0,0 +1,43 @@
+// Copyright 2019 Istio Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package collection
+
+import "regexp"
+
+// Name of a collection.
+type Name struct{ string }
+
+var validNameRegex = regexp.MustCompile(`^[a-zA-Z0-9_][a-zA-Z0-9_\.]*(/[a-zA-Z0-9_][a-zA-Z0-9_\.]*)*$`)
+
+// EmptyName is a sentinel value
+var EmptyName = Name{}
+
+// NewName returns a strongly typed collection. Panics if the name is not valid.
+func NewName(n string) Name {
+	if !IsValidName(n) {
+		panic("collection.NewName: invalid collection name: " + n)
+	}
+	return Name{n}
+}
+
+// String interface method implementation.
+func (t Name) String() string {
+	return t.string
+}
+
+// IsValidName returns true if the given collection is a valid name.
+func IsValidName(name string) bool {
+	return validNameRegex.Match([]byte(name))
+}

galley/pkg/config/collection/name_test.go

@@ -0,0 +1,85 @@
+// Copyright 2019 Istio Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package collection
+
+import (
+	"testing"
+
+	. "github.com/onsi/gomega"
+)
+
+func TestNewName(t *testing.T) {
+	g := NewGomegaWithT(t)
+
+	c := NewName("c1")
+	g.Expect(c.String()).To(Equal("c1"))
+}
+
+func TestNewName_Invalid(t *testing.T) {
+	g := NewGomegaWithT(t)
+	defer func() {
+		r := recover()
+		g.Expect(r).NotTo(BeNil())
+	}()
+
+	_ = NewName("/")
+}
+
+func TestName_String(t *testing.T) {
+	g := NewGomegaWithT(t)
+	c := NewName("c1")
+
+	g.Expect(c.String()).To(Equal("c1"))
+}
+
+func TestIsValidName_Valid(t *testing.T) {
+	data := []string{
+		"foo",
+		"9",
+		"b",
+		"a",
+		"_",
+		"a0_9",
+		"a0_9/fruj_",
+		"abc/def",
+	}
+
+	for _, d := range data {
+		t.Run(d, func(t *testing.T) {
+			g := NewGomegaWithT(t)
+			b := IsValidName(d)
+			g.Expect(b).To(BeTrue())
+		})
+	}
+}
+
+func TestIsValidName_Invalid(t *testing.T) {
+	data := []string{
+		"",
+		"/",
+		"/a",
+		"a/",
+		"$a/bc",
+		"z//a",
+	}
+
+	for _, d := range data {
+		t.Run(d, func(t *testing.T) {
+			g := NewGomegaWithT(t)
+			b := IsValidName(d)
+			g.Expect(b).To(BeFalse())
+		})
+	}
+}