Update CHEESEhub UI with latest additions (#1)

* Add version 2.0 of client-go * Stub pod and RC watching * Add kubernetes go client to KubeHelper * Get clean healthz result * Migrate more operations over to go client * Migrate GetLog to go client * Mostly complete conversion to go client * Initial fixes * Just a reformat * Removed unused PutDataset endpoint * Changed Kubernetes version to 1.5 * Reverted change to Stream() call * Migrate to mockable kubernetes Interface * Updated event handling to use go client approach * Added instructions to run apiserver out of cluster * Added support for in-cluster configuration * NDS-1172: Add -p flag to ndsctl to allow specification of password (nds-org#242) * Roll versions forward to 1.0.14 * NDS-1172: Add -p flag to ndsctl to allow specification of password * NDS-1177: Delete ingress rule before recreating it during stack modification * NDS-1219: If stack is starting, StartStack should simply return 202 again * NDS-1116: Adds oauth endpoint to apiserver (nds-org#246) * Added admin port * Added admin port * Added oauth registration method * Parameterized auth urls * NDS-1127 / NDS-229: Globus Auth support for UI, refactor webui Docker build, remove Bower (nds-org#243) * Roll versions forward to 1.0.14 * Sync.. starting to refactor webui Dockerfile(s) * NDS-1127 / NDS-229: Rework Docker/Grunt build process, migrate away from using Bower * Removed bower from package.json * Removed Bower from base image * Replace static file server rule for '/bower_components' with '/node_modules' * Fix last few dependency issues from migration * Fixed SwaggerUI view (again) and added a test case * Removed the 'HTTP Basic' from 'HTTP Basic Authentication' on the dashboard * Made the SwaggerUI test case actually useful * Fixed an issue scrolling Docker image tags into view (addSpec) * Fixed race condition in consolePage.get() * Fixed erroneous comments * Removed documentation about GIT_DROPIN feature * Remove debian backports apt repo from Dockerfile * Adjusted Docker image to read in SIGNIN_URL * Adjusted Signin/Signout paths on LandingPage/Navbar * Added banner / redirect to LoginPage / SignUpPage when OAuth is enabled * Update README.md * Create simple travis script to verify basics (nds-org#247) * Update glide repos to fix sirupsen repo name change * Changed default NDSLABS_APISERVER_SERVICE_PORT to 30001 * Updated dependencies and imports * Added better defaults for debugging * Updated README to reflect current dev practices * Added insecure postman collection * Addressed review feedack * Allow user to configure nodeSelector in Workbench apiserver * Update entrypoint.sh to include new config values nodeSelectorName / nodeSelectorValue * Added new config for pvcStorageClass, new functions for Creating/Deleting PVCs in kube.go * Create 'home' PVC when namespace is created, create vol/user PVCs when stack starts * Removed references to glusterfs / HomeVolume, replaced with HomePvcSuffix * Added a test case for a stack mounting a PVC * Fixed HTTPS simple stack chaining/test case * Added helpers to Create/Delete NetworkPolicy resources and check for their existence * Improve NetworkPolicy error-handling slightly * Updated PVC support to use SubPath from home PVC, instead of creating a new PVC every time * Added new config for pvcStorageClass, new functions for Creating/Deleting PVCs in kube.go * Create 'home' PVC when namespace is created, create vol/user PVCs when stack starts * Removed references to glusterfs / HomeVolume, replaced with HomePvcSuffix * Added a test case for a stack mounting a PVC * Fixed HTTPS simple stack chaining/test case * Updated PVC support to use SubPath from home PVC, instead of creating a new PVC every time * Removed VOLUME_* variables and added HOME_PVC_SUFFIX * Removed unused VolumeType * Added default nfs storage class in entrypoint * Added nginx ingress class for GKE * Added Sphinx config * Basic install instructions * Added the volume.beta.kubernetes.io/nfs-mount-path annotation to the PersistentVolumeClaim * Updated the base containers to Debian Stretch due to 404 errors when running apt-get on golang:1.8-jessie * Check for error instead of nil * Add support for more fine-grained container security capabilities * Expand full.json example to show securityContext * Fix merge conflict, further expand error-handling in StartStack * Read subdomain_prefix from env instead of 'www' * Removed last references to Privileged * Don't re-pull dependencies if the --cache flag is passed * Moved networkpolicy creation to stack (not service) level * Gofmt * Add check for existing network policy * Added collocate spec option to support pod affinity on host * Changed network policy pod selector to support access by ILB * Disabled network policies * Temporary fix to ensure spdy installed * Implemented exec using go client * Added QPS and Burst overrides * Analytics event tracking (nds-org#296) * Low-barrier platform instrumentation/tracking using Google Analytics events * Fixed logs/console/edit/help/config event tracking * Use Catalog as default page when first logging in (nds-org#313) * Update Dashboard/Applications page names throughout, highlight current page in navbar (nds-org#314) * Catalog: Remove help icon, revise text in help link on dropdown (nds-org#315) * Catalog: Hide 'Add' button after installing once, highlight the 'View' button (nds-org#316) * Added description to Dashboard page, changed red panels to grey (nds-org#317) * Applications: Added tooltips to all buttons and columns (nds-org#318) * Replace link icon with text, add margin (nds-org#319) * Replace link icon with text, add margin * Replace link icon with text, add margin * Hide advanced features behind config flags (nds-org#320) * Hide some of the more advanced features behind environment variables * Fix default login path * Show Console by default * Added showLogs and showConfig flags * Remove unused envvar * fix bad merge * Remove debug variable * Update gui/README.md to correct Swagger codegen Fixes nds-org#312 * Hide Terms&Conditions link on signup if no 'Acceptable Use Policy' link is provided (nds-org#321) * Address SGCI Phase 2 Feedback (nds-org#324) * Hide purple boxes on landing, consistent naming for signUp * Move "See Also" from Landing to the "Help" dropdown on the navbar * Adjust hint text, make list / card buttons slightly more obvious * Highlight "Add" button, show popup when copying to clipboard, hide source when not needed * Hide "Add Application" button on Dashboard view * Increase size of Loading/Refresh icon * Upgrade AngularJS to 1.8.0 (nds-org#322) * Roll AngularJS forward to 1.7.9, update return signature for all uses of .then() * Roll AngularJS forward to 1.8.0 * Bump lodash from 4.5.1 to 4.17.19 in /gui (nds-org#325) Bumps [lodash](https://github.com/lodash/lodash) from 4.5.1 to 4.17.19. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.5.1...4.17.19) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: BenGalewsky <[email protected]> Co-authored-by: Craig Willis <[email protected]> Co-authored-by: BenGalewsky <[email protected]> Co-authored-by: Ben Galewsky <[email protected]> Co-authored-by: Kevin Coakley <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
cheese-hub · Oct 8, 2020 · 46050ca · 46050ca
1 parent 1caefc5
commit 46050ca
Show file tree

Hide file tree

Showing 80 changed files with 14,301 additions and 8,442 deletions.
diff --git a/.gitignore b/.gitignore
@@ -12,5 +12,7 @@
 gui/tests/reports/*
 gui/shared/ndslabs-api.js
 gui/app/shared/ndslabs-api.js
+.idea
+apiserver/vendor
 gui/*.auth.json
 **/.protractor-fail-fast
diff --git a/.travis.yml b/.travis.yml
@@ -0,0 +1,5 @@
+# Run workbench in minikube
+sudo: required
+
+script:
+  - cd apiserver; docker build -t apiserver .
diff --git a/README.md b/README.md
@@ -8,7 +8,7 @@ This is the main repository for the [National Data Service](http://www.nationald
 
 Labs Workbench is an experimental space for evaluating, developing, and exploring interoperability between research data management services. This is achieved through the use of applications containerized with [Docker](https://www.docker.com/what-docker), [Kubernetes](http://kubernetes.io/docs/whatisk8s/) for container orchestration, deployed on an [OpenStack](https://www.openstack.org/) cluster.
 
-For more information, see the [architecture documentation](https://github.com/nds-org/ndslabs/tree/master/docs/architecture).
+For more information, see the [architecture documentation](https://opensource.ncsa.illinois.edu/confluence/display/NDS/Workbench+Architecture).
 
 On top of this foundation, NDS Labs provides a user interface, command line interface, and an API server used to manage the configuration and deployment of containerized services. This repository includes the following:
 

diff --git a/apiserver/Dockerfile b/apiserver/Dockerfile
@@ -1,12 +1,13 @@
-FROM golang:1.8-jessie as gobuild
+FROM golang:1.12-stretch as gobuild
 COPY . /go/src/github.com/ndslabs/apiserver
 
 RUN apt-get -qq update && \
     apt-get -qq install bash build-essential git gcc  && \
     go get github.com/Masterminds/glide  &&  \
+    go get github.com/docker/spdystream  &&  \
     cd /go/src/github.com/ndslabs/apiserver && ./build.sh docker
 
-FROM debian:jessie
+FROM debian:stretch
 COPY --from=gobuild /go/src/github.com/ndslabs/apiserver/build/bin/ndslabsctl-linux-amd64 /ndslabsctl/ndslabsctl-linux-amd64
 COPY --from=gobuild /go/src/github.com/ndslabs/apiserver/build/bin/ndslabsctl-darwin-amd64 /ndslabsctl/ndslabsctl-darwin-amd64
 COPY --from=gobuild /go/src/github.com/ndslabs/apiserver/build/bin/apiserver-linux-amd64 /usr/local/bin/apiserver

diff --git a/apiserver/README.md b/apiserver/README.md
@@ -1,81 +1,149 @@
 
 # NDS Labs API Server
 
-## Latest release
-The latest release has been pushed to Dockerhub ndslabs/apiserver:latest. You can run the server via Docker:
+This document describes how to run the Labs Workbench API server for
+development.
 
+
+## Install and Configure Minikube
+
+Install and start Minikube based on the [official documentation](https://kubernetes.io/docs/tasks/tools/install-minikube/) for your operating system.
+
+Start your minikube instance:
+```
+minikube start
 ```
-docker pull ndslabs/apiserver:latest
+
+Workbench relies on labeled nodes. Label the node:
+
+```
+kubectl label nodes minikube  ndslabs-role-compute=true
 ```
 
-## Running
+## Install and Configure Etcd
 
-Prerequisites:
-* Etcd
-* Kubernetes
+Workbench requires `etcd`. For development purposes you can install it
+locally via Docker:
 
 ```
-docker run -e ETCD_ADDR=localhost:4001 -e KUBERNETES_ADDR=http://localhost:8080 -e CORS_ORIGIN_ADDR="http://localhost" -e HOST_ADDR=PUBLIC_IP_ADDR -e SPEC_GIT_REPO=https://github.com/nds-org/ndslabs-specs -e SPEC_GIT_BRANCH=master
+docker run --rm -p 4001:4001 -d ndslabs/etcd:2.2.5  /usr/local/bin/etcd \
+         --bind-addr=0.0.0.0:4001 \
+         --advertise-client-urls=http://127.0.0.1:4001 
 ```
 
-Configuration options:
-* ETCD_ADDR: address of Etcd (defaults to localhost:4001)
-* KUBERNETES_ADDR: URL for Kubernetes API (Defaults to http://localhost:8080)
-* CORS_ORIGIN_ADDR: URL of GUI (defaults to http://localhost)
-* HOST_ADDR: Public IP address of host
-* SPEC_GIT_REPO: URL to spec repo (defaults to https://github.com/nds-org/ndslabs-specs)
-* SPEC_GIT_BRANCH: Git repository branch (defaults to master)
+## Setup your Go environment
 
-## Building 
+Install Go for development (assumes Mac OS)
 
-Prerequisites:
-* For now, running on OS X with Docker Toolbox
-* Go 1.5
+```
+brew install go --cross-compile-common
+```
 
-To simply compile the go code:
+Setup your paths:
 ```
-go build
+mkdir $HOME/go
 ```
 
-To build the apiserver binary for multiple architectures (output in build/bin/)
+Edit `$HOME/.bash_profile`:
 ```
-./build.sh
+export GOPATH=$HOME/go
+export PATH=$PATH:$GOPATH/bin
 ```
 
-To tag and push images to dockerhub:
+Source your profile:
+```
+. $HOME/.bash_profile
 ```
-./build.sh <dev, test, release>
+
+
+## Clone and build the API server
+
+Clone this repo:
 ```
+mkdir -p $GOPATH/src/github.com/
+cd $GOPATH/src/github.com/
+git clone https://github.com/nds-org/ndslabs 
+cd ndslabs/apiserver
+```	
 
-### Configuration
+Build local binaries:
+```
+./build.sh local
+```
 
-The apiserver.conf is the primary configuration file. If running the apiserver outside of Docker, you will need to manually configure the following settings:
+## Configure API server
 
+Clone the `ndslabs-specs` repo anywhere on your machine:
+```
+git clone https://github.com/nds-org/ndslabs-specs
 ```
-[Server]
-Port=30001
-Origin=http://<CORS origin host>
-VolDir=/home/core/apiserver/volumes
-Host=<your host ip>
-VolumeSource=local
-SpecsDir=<optional path to local specs directory>
 
-[Etcd]
-Address=localhost:4001
+Modify the `apiserver.json` to reflect your local configuration. At a minimum,
+set the `support.email` to your email address, `kubernetes.address` to 
+`https://minikube-ip:8443` and the `specs.path` to the location of the cloned repo above.
 
-[Kubernetes]
-Address=localhost:8080
 
+## Run the API server
+
+Assuming `minikube` and `etcd` are running, you can simply run the apiserver
+binary:
+
+```
+./build/bin/apiserver-darwin-amd64
+```
+
+## Running integration tests
+
+Install `newman` (may require `sudo`):
 ```
+npm install -g newman
+```
+
+To configure the postman environment, edit `postman/workbench.postman_environment.json`
+and set the `host` value to `localhost:30001` and `email` value to your email
+address.
 
-If VolumeSource is "local", a local directory is used for hostPath volumes in Kubernetes. 
+Because the API server is running insecure, you'll need to edit `postman/Workbench.postman_collection_local.json` 
+and replace all instances of `https` with `http`.
 
+Run the tests:
+```
+cd postman
+newman run --insecure  --environment=workbench.postman_environment.json --delay-request=1000 Workbench.postman_collection_local.json
+```
 
-### Running the server
+## Running as an external service
 
-The server requires etcd and Kubernetes. You can either run a separate etcd or use the Kubernetes instance.
+It is also possible to run the Workbench API server outside of the cluster. This has been tested with minikube via VirtualBox on MacOS.
+
+Create a file `external-apiserver.yaml` with the following. Note that 10.0.2.2 is the internal address of your minikube VM. You can confirm this via `minikube ssh` and `netstat -rn`:
+```
+kind: "Service"
+apiVersion: "v1"
+metadata:
+  name: "ndslabs-apiserver"
+spec:
+  ports:
+    - protocol: "TCP"
+      port: 30001
+      targetPort: 30001
+---
+kind: "Endpoints"
+apiVersion: "v1"
+metadata:
+  name: "ndslabs-apiserver"
+subsets:
+  - addresses:
+    - ip: "10.0.2.2"
+    ports:
+    - port: 30001
+```
 
-To run
+Delete the in-cluster API server and create this external service/endpoint:
 ```
-./apiserver -v <log verbosity 1-4> --conf <path to apiserver.conf> --passwd <admin password>
+kubectl delete svc ndslabs-apiserver
+kubectl create -f external-apiserver.yaml
 ```
+
+
+
diff --git a/apiserver/apiserver.json b/apiserver/apiserver.json
@@ -1,11 +1,11 @@
 {
     "port": "30001",
     "timeout": 600,
-    "requireApproval": false,
-    "domain" : "",
+    "requireApproval": true,
+    "domain" : "workbench.local",
     "ingress": "LoadBalancer",
     "supportEmail": "",
-    "homeVolume": "global",
+    "homePvcSuffix": "-home",
     "defaultLimits": {
         "cpuMax": 2000,
         "cpuDefault": 1000,
@@ -18,23 +18,22 @@
         "maxMessages": 100
     },
     "kubernetes": {
-        "address": "http://localhost:8080"
+        "address": "https://minikube-ip:8443",
+        "pvcStorageClass": "",
+        "nodeSelectorName": "",
+        "nodeSelectorValue": ""
     },
     "email": {
-        "host": "smtp.ncsa.uiuc.edu",
-        "port": 25,
+        "host": "",
+        "port": 0,
         "tls": true
     },
     "specs": {
-        "path": "/specs"
+        "path": "/path/to/ndslabs-specs"
     },
-    "volumes": [{
-        "name": "global",
-        "path": "/var/glfs/global",
-        "type": "gluster"
-    }],
+    "volumes": [],
 	"support": {
-		"email": "",
+		"email": "[email protected]",
 		"forum": "",
 		"chat": ""
 	}

diff --git a/apiserver/build.sh b/apiserver/build.sh
@@ -17,7 +17,13 @@ if [ "$1" == "local" ] || [ "$1" == "docker" ]; then
     echo "  BUILD_DATE = \"$BUILD_DATE\"" >> $VERSIONFILE
     echo ")" >> $VERSIONFILE
 
-    glide install --strip-vendor 
+    # Check for --cache flag
+    args="$@"
+    replaced="${@/--cache/}"
+    if [ "$args" == "$replaced" ]; then
+        echo "Fetching dependencies..."
+        glide install --strip-vendor
+    fi
 
     COVERPKG=./cmd/server,./pkg/crypto,./pkg/etcd,./pkg/config,./pkg/email,./pkg/events,./pkg/kube,./pkg/middleware,./pkg/types,./pkg/validate
 	if [ "$1" == "local" ]; then 
@@ -57,7 +63,6 @@ if [ "$1" == "local" ] || [ "$1" == "docker" ]; then
           GOOS=darwin GOARCH=amd64 go build -o build/bin/ndslabsctl-darwin-amd64 ./cmd/apictl
     fi
 
-    rm -r pkg/version
 elif [ "$1" == "clean" ]; then
 	rm -r build
 	rm -r vendor/github.com vendor/golang.org vendor/gopkg.in vendor/k8s.io