From 3174f6f784e743f091def0b19e97ea3d558b2c53 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Tue, 14 Nov 2023 17:29:33 +0000 Subject: [PATCH 01/22] add rockcraft.yaml, and readme for building rock --- .github/ISSUE_TEMPLATE/bug_report.md | 30 ++++ .gitignore | 1 + README.md | 46 +++++- licenses/LICENSE-rock | 202 +++++++++++++++++++++++++++ rockcraft.yaml | 29 ++++ 5 files changed, 307 insertions(+), 1 deletion(-) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md create mode 100644 .gitignore create mode 100644 licenses/LICENSE-rock create mode 100644 rockcraft.yaml diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000..6ecc03d --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,30 @@ +--- +name: Bug report +about: File a bug report +labels: bug + +--- + + + +## Steps to reproduce +1. + +## Expected behavior + + +## Actual behavior + + + +## Versions + + +Operating system: + +## Log output + + + +## Additional context + \ No newline at end of file diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..013c47c --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +*.rock \ No newline at end of file diff --git a/README.md b/README.md index 28da090..83ec26c 100644 --- a/README.md +++ b/README.md @@ -1 +1,45 @@ -# charmed-superset-rock \ No newline at end of file +# Charmed Apache Superset ROCK + +This repository contains the packaging metadata for creating a Charmed Superset ROCK. This ROCK image is based on the upstream [Apache Superset](https://downloads.apache.org/superset/) image. + +For more information on ROCKs, visit the [rockcraft Github](https://github.com/canonical/rockcraft). + +## Building the ROCK +The steps outlined below are based on the assumption that you are building the ROCK with the latest LTS of Ubuntu. +If you are using another version of Ubuntu or another operating system, the process may be different. +To avoid any issue with other operating systems you can simply build the image with [multipass](https://multipass.run/): +```bash +sudo snap install multipass +multipass launch 22.04 -n rock-dev +multipass shell rock-dev +``` + +### Clone Repository +```bash +git clone https://github.com/canonical/charmed-superset-rock.git +cd charmed-superset-rock +``` +### Installing Prerequisites +```bash +sudo snap install rockcraft --edge --classic +sudo snap install docker +sudo snap install lxd +sudo snap install skopeo --edge --devmode +``` +### Configuring Prerequisites +```bash +sudo usermod -aG docker $USER +sudo lxd init --auto +``` +*_NOTE:_* You will need to open a new shell for the group change to take effect (i.e. `su - $USER`) +### Packing and Running the ROCK +```bash +rockcraft pack +sudo skopeo --insecure-policy copy oci-archive:charmed-superset-rock_2.1.0_amd64.rock docker-daemon:/charmed-superset-rock:2.1.0 +docker run --rm -it charmed-superset-rock:2.1.0 +``` +## License +The Charmed Superset ROCK is free software, distributed under the Apache +Software License, version 2.0. See +[LICENSE](https://github.com/canonical/charmed-superset-rock/blob/main/LICENSE) +for more information. \ No newline at end of file diff --git a/licenses/LICENSE-rock b/licenses/LICENSE-rock new file mode 100644 index 0000000..7a4a3ea --- /dev/null +++ b/licenses/LICENSE-rock @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file diff --git a/rockcraft.yaml b/rockcraft.yaml new file mode 100644 index 0000000..d4e0df6 --- /dev/null +++ b/rockcraft.yaml @@ -0,0 +1,29 @@ +# Copyright 2023 Canonical Ltd. +# See LICENSE file for licensing details. +--- +name: charmed-superset-rock +base: ubuntu@22.04 +version: '2.1.0' # Semantic versioning for human readability +summary: Charmed Superset ROCK OCI +description: | + Superset is fast, lightweight, intuitive, and loaded with options + that make it easy for users of all skill sets to explore and + visualize their data. +license: Apache-2.0 + +platforms: + amd64: + +parts: + superset: + plugin: nil + source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz + build-packages: + - build-essential + - python3.10 + - python3.10-venv + stage-packages: + - python3-pip + - bash + - vim + - curl From 73cb77fe5c29003c1d071f28830334e7846dc5f4 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Wed, 15 Nov 2023 18:22:14 +0000 Subject: [PATCH 02/22] override-build --- rockcraft.yaml | 44 ++++++++++++++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 6 deletions(-) diff --git a/rockcraft.yaml b/rockcraft.yaml index d4e0df6..9c7d0a9 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -17,13 +17,45 @@ platforms: parts: superset: plugin: nil - source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz + source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz + # https://superset.apache.org/docs/installation/installing-superset-from-scratch/ build-packages: - build-essential - - python3.10 - - python3.10-venv + - wget + - libsasl2-dev + - libldap2-dev + - libssl-dev + - libffi-dev + - default-libmysqlclient-dev + build-environment: + - APP_HOME: "/app" + - SOURCE: "https://downloads.apache.org/superset/" + - VERSION: "2.1.0" stage-packages: + - python3.10 + - python3-venv - python3-pip - - bash - - vim - - curl + override-build: | + # make home directory + mkdir -p ${APP_HOME} + + # apt-get update + apt-get update; apt-get upgrade -y; apt-get autoremove --purge -y; apt-get clean -y + + # Download and unpack tar + wget "${SOURCE}/${VERSION}/apache-superset-${VERSION}-source.tar.gz" -O superset.tar.gz + tar -zxvf superset.tar.gz -C ${APP_HOME} --strip-components 1 + rm -rf superset.tar.gz + + # Install dependencies + pip install cython<3.0.0 && pip install --no-build-isolation pyyaml==5.4.1 + pip install --upgrade setuptools pip + cd ${APP_HOME} + pip install --no-cache-dir -r requirements/base.txt + # pip wheel --no-index -w wheelhouse apache-superset==2.1.0 + # pip install wheelhouse/apache_superset-2.1.0-py3-none-any.whl + + echo "2" > version.txt + + # Stage the Superset directories + cd / && cp -r app "${CRAFT_PART_INSTALL}/app" From 6aa01ca6dc904eba3f39f5aa7771ca4cbd42eb7c Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Thu, 16 Nov 2023 18:22:03 +0000 Subject: [PATCH 03/22] use python plugin --- rockcraft.yaml | 50 +++++++++++++++++++++++++++++--------------------- 1 file changed, 29 insertions(+), 21 deletions(-) diff --git a/rockcraft.yaml b/rockcraft.yaml index 9c7d0a9..4a9c2c4 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -14,48 +14,56 @@ license: Apache-2.0 platforms: amd64: +# services: +# superset-ui: +# override: replace +# summary: "superset ui service" +# startup: enabled +# command: /usr/bin/run-server.sh +# environment: +# PYTHONPATH: "/app/pythonpath" + parts: - superset: - plugin: nil + backend: + plugin: python source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz # https://superset.apache.org/docs/installation/installing-superset-from-scratch/ build-packages: - build-essential - wget - - libsasl2-dev - - libldap2-dev - - libssl-dev - - libffi-dev - - default-libmysqlclient-dev + # - libsasl2-dev + # - libldap2-dev + # - libssl-dev + # - libffi-dev + # - default-libmysqlclient-dev + stage-packages: + - python3.10-venv build-environment: - - APP_HOME: "/app" + - APP_HOME: "app" - SOURCE: "https://downloads.apache.org/superset/" - VERSION: "2.1.0" - stage-packages: - - python3.10 - - python3-venv - - python3-pip + - DIST_PACKAGES: "/usr/local/lib/python3.10/dist-packages/" override-build: | + craftctl default # make home directory mkdir -p ${APP_HOME} # apt-get update apt-get update; apt-get upgrade -y; apt-get autoremove --purge -y; apt-get clean -y + apt-get install python3-pip # Download and unpack tar wget "${SOURCE}/${VERSION}/apache-superset-${VERSION}-source.tar.gz" -O superset.tar.gz tar -zxvf superset.tar.gz -C ${APP_HOME} --strip-components 1 rm -rf superset.tar.gz + cp -r ${APP_HOME} "${CRAFT_PART_INSTALL}/${APP_HOME}" # Install dependencies - pip install cython<3.0.0 && pip install --no-build-isolation pyyaml==5.4.1 pip install --upgrade setuptools pip - cd ${APP_HOME} - pip install --no-cache-dir -r requirements/base.txt - # pip wheel --no-index -w wheelhouse apache-superset==2.1.0 - # pip install wheelhouse/apache_superset-2.1.0-py3-none-any.whl - - echo "2" > version.txt + pip install "cython<3.0.0" + pip install --no-build-isolation pyyaml==5.4.1 + cd ${APP_HOME} && pip install --no-cache -r requirements/base.txt + pip install Werkzeug==2.2.2 + pip install sqlparse==0.4.3 - # Stage the Superset directories - cd / && cp -r app "${CRAFT_PART_INSTALL}/app" + cp -r /usr/local/lib/python3.10/dist-packages/* $CRAFT_PRIME From 00b3ba8fd65e4c6ced3b3800a29a9c1f3c04e117 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Tue, 21 Nov 2023 16:18:56 +0000 Subject: [PATCH 04/22] add frontend and gunicorn-app --- rockcraft.yaml | 55 +++++++++++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 21 deletions(-) diff --git a/rockcraft.yaml b/rockcraft.yaml index 4a9c2c4..5e33667 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -1,6 +1,7 @@ # Copyright 2023 Canonical Ltd. # See LICENSE file for licensing details. --- +# https://github.com/apache/superset/blob/2.1.0/RELEASING/Dockerfile.from_local_tarball name: charmed-superset-rock base: ubuntu@22.04 version: '2.1.0' # Semantic versioning for human readability @@ -14,35 +15,20 @@ license: Apache-2.0 platforms: amd64: -# services: -# superset-ui: -# override: replace -# summary: "superset ui service" -# startup: enabled -# command: /usr/bin/run-server.sh -# environment: -# PYTHONPATH: "/app/pythonpath" - parts: backend: plugin: python source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz - # https://superset.apache.org/docs/installation/installing-superset-from-scratch/ build-packages: - build-essential - wget - # - libsasl2-dev - # - libldap2-dev - # - libssl-dev - # - libffi-dev - # - default-libmysqlclient-dev stage-packages: - python3.10-venv build-environment: - APP_HOME: "app" - SOURCE: "https://downloads.apache.org/superset/" - VERSION: "2.1.0" - - DIST_PACKAGES: "/usr/local/lib/python3.10/dist-packages/" + - DIST_PACKAGES: "usr/local/lib/python3.10/dist-packages/" override-build: | craftctl default # make home directory @@ -56,14 +42,41 @@ parts: wget "${SOURCE}/${VERSION}/apache-superset-${VERSION}-source.tar.gz" -O superset.tar.gz tar -zxvf superset.tar.gz -C ${APP_HOME} --strip-components 1 rm -rf superset.tar.gz - cp -r ${APP_HOME} "${CRAFT_PART_INSTALL}/${APP_HOME}" + frontend: + after: [backend] + plugin: nil + source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz + build-snaps: + - node/14/stable + override-build: | + mkdir -p /superset/static/assets/ + cd superset-frontend + npm install currencyformatter.js --save + npm ci + npm run build + rm -rf node_modules + + gunicorn-app: + after: [backend, frontend] + plugin: python + source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz + build-packages: + - build-essential + stage-packages: + - python3.10-venv + build-environment: + - APP_HOME: "app" + - DIST_PACKAGES: "usr/local/lib/python3.10/dist-packages/" + - PYTHONPATH: /app/superset + override-build: | # Install dependencies pip install --upgrade setuptools pip pip install "cython<3.0.0" pip install --no-build-isolation pyyaml==5.4.1 - cd ${APP_HOME} && pip install --no-cache -r requirements/base.txt - pip install Werkzeug==2.2.2 - pip install sqlparse==0.4.3 + pip install -r requirements/base.txt --target=/${DIST_PACKAGES} - cp -r /usr/local/lib/python3.10/dist-packages/* $CRAFT_PRIME + # Copy packages to PRIME + mkdir -p ${CRAFT_PRIME}/${DIST_PACKAGES} + cp -r /${DIST_PACKAGES}* $CRAFT_PRIME/${DIST_PACKAGES} + cp -r . "${CRAFT_PART_INSTALL}/${APP_HOME}" From 5c1a863a08294ac9b41750c5f5d8e82dd4927422 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Wed, 22 Nov 2023 12:12:30 +0000 Subject: [PATCH 05/22] add optional connectors and copy assets --- rockcraft.yaml | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/rockcraft.yaml b/rockcraft.yaml index 5e33667..292ceb4 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -1,7 +1,7 @@ # Copyright 2023 Canonical Ltd. # See LICENSE file for licensing details. --- -# https://github.com/apache/superset/blob/2.1.0/RELEASING/Dockerfile.from_local_tarball +# https://github.com/apache/superset/blob/master/Dockerfile name: charmed-superset-rock base: ubuntu@22.04 version: '2.1.0' # Semantic versioning for human readability @@ -49,13 +49,17 @@ parts: source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz build-snaps: - node/14/stable + build-environment: + - APP_HOME: "app" + - ASSETS_DIR: "lib/python3.10/site-packages/superset/static/assets/" override-build: | mkdir -p /superset/static/assets/ cd superset-frontend npm install currencyformatter.js --save npm ci - npm run build - rm -rf node_modules + npm run build -- --output-path=/dist + mkdir -p "${CRAFT_PART_INSTALL}/${ASSETS_DIR}" + cp -r /dist/* "${CRAFT_PART_INSTALL}/${ASSETS_DIR}" gunicorn-app: after: [backend, frontend] @@ -76,6 +80,15 @@ parts: pip install --no-build-isolation pyyaml==5.4.1 pip install -r requirements/base.txt --target=/${DIST_PACKAGES} + # install optional connectors + pip install psycopg2-binary + pip install SQLAlchemy + pip install Authlib + pip install elasticsearch-dbapi + + pip install -e . + flask fab babel-compile --target superset/translations + # Copy packages to PRIME mkdir -p ${CRAFT_PRIME}/${DIST_PACKAGES} cp -r /${DIST_PACKAGES}* $CRAFT_PRIME/${DIST_PACKAGES} From 10797893166c94e6b43dc7025efe824b5add10a5 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Wed, 22 Nov 2023 12:50:23 +0000 Subject: [PATCH 06/22] add versions for optional packages --- rockcraft.yaml | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/rockcraft.yaml b/rockcraft.yaml index 292ceb4..878366e 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -81,10 +81,11 @@ parts: pip install -r requirements/base.txt --target=/${DIST_PACKAGES} # install optional connectors - pip install psycopg2-binary - pip install SQLAlchemy - pip install Authlib - pip install elasticsearch-dbapi + pip install psycopg2-binary==2.9.9 + pip install Authlib==1.2.1 + pip install elasticsearch-dbapi==0.2.10 + pip install trino==0.327.0 + pip install pyhive==0.7.0 pip install -e . flask fab babel-compile --target superset/translations From b3987dbde4057c7d3bc0d9274697f37d984b95b9 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Thu, 23 Nov 2023 18:05:25 +0000 Subject: [PATCH 07/22] add ca-certificates overlay --- rockcraft.yaml | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/rockcraft.yaml b/rockcraft.yaml index 878366e..b6c66d8 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -72,25 +72,36 @@ parts: build-environment: - APP_HOME: "app" - DIST_PACKAGES: "usr/local/lib/python3.10/dist-packages/" - - PYTHONPATH: /app/superset override-build: | # Install dependencies pip install --upgrade setuptools pip pip install "cython<3.0.0" pip install --no-build-isolation pyyaml==5.4.1 pip install -r requirements/base.txt --target=/${DIST_PACKAGES} + pip install Pillow==10.1.0 + pip install requests==2.31.0 + pip install jmespath==1.0.1 - # install optional connectors + # Install optional connectors pip install psycopg2-binary==2.9.9 pip install Authlib==1.2.1 pip install elasticsearch-dbapi==0.2.10 pip install trino==0.327.0 pip install pyhive==0.7.0 + pip install SQLALchemy==2.0.23 + pip install thrift==0.16.0 + pip install sqlalchemy-redshift==0.8.1 + # Install Superset in editable mode pip install -e . - flask fab babel-compile --target superset/translations # Copy packages to PRIME mkdir -p ${CRAFT_PRIME}/${DIST_PACKAGES} cp -r /${DIST_PACKAGES}* $CRAFT_PRIME/${DIST_PACKAGES} cp -r . "${CRAFT_PART_INSTALL}/${APP_HOME}" + + overlay-pkgs: + plugin: nil + overlay-packages: + - ca-certificates + - vim From 92ab3c399e26a716af05f8a2f4f4672123f0f4bd Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Fri, 24 Nov 2023 11:14:29 +0000 Subject: [PATCH 08/22] Add startup files --- rockcraft.yaml | 24 +++++++-- startup-scripts/k8s-bootstrap.sh | 87 ++++++++++++++++++++++++++++++++ startup-scripts/run-server.sh | 36 +++++++++++++ 3 files changed, 143 insertions(+), 4 deletions(-) create mode 100755 startup-scripts/k8s-bootstrap.sh create mode 100755 startup-scripts/run-server.sh diff --git a/rockcraft.yaml b/rockcraft.yaml index b6c66d8..d0508d7 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -34,7 +34,7 @@ parts: # make home directory mkdir -p ${APP_HOME} - # apt-get update + # install pip apt-get update; apt-get upgrade -y; apt-get autoremove --purge -y; apt-get clean -y apt-get install python3-pip @@ -43,6 +43,13 @@ parts: tar -zxvf superset.tar.gz -C ${APP_HOME} --strip-components 1 rm -rf superset.tar.gz + local-files: + plugin: dump + source: ./startup-scripts + organize: + run-server.sh : app/k8s/run-server.sh + k8s-bootstrap.sh: app/k8s/k8s-bootstrap.sh + frontend: after: [backend] plugin: nil @@ -53,16 +60,23 @@ parts: - APP_HOME: "app" - ASSETS_DIR: "lib/python3.10/site-packages/superset/static/assets/" override-build: | + # Prepare for asset creation mkdir -p /superset/static/assets/ cd superset-frontend + + # Install missing dependency npm install currencyformatter.js --save + + # Install dependencies and build assets npm ci npm run build -- --output-path=/dist + + # Copy assets to CRAFT_PART_INSTALL mkdir -p "${CRAFT_PART_INSTALL}/${ASSETS_DIR}" cp -r /dist/* "${CRAFT_PART_INSTALL}/${ASSETS_DIR}" gunicorn-app: - after: [backend, frontend] + after: [backend, frontend, local-files] plugin: python source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz build-packages: @@ -78,17 +92,19 @@ parts: pip install "cython<3.0.0" pip install --no-build-isolation pyyaml==5.4.1 pip install -r requirements/base.txt --target=/${DIST_PACKAGES} + + # Required dependencies missing from Superset requirements.txt pip install Pillow==10.1.0 pip install requests==2.31.0 pip install jmespath==1.0.1 - # Install optional connectors + # Optional dependencies for Database connectors pip install psycopg2-binary==2.9.9 pip install Authlib==1.2.1 pip install elasticsearch-dbapi==0.2.10 pip install trino==0.327.0 pip install pyhive==0.7.0 - pip install SQLALchemy==2.0.23 + pip install SQLALchemy==1.4.36 pip install thrift==0.16.0 pip install sqlalchemy-redshift==0.8.1 diff --git a/startup-scripts/k8s-bootstrap.sh b/startup-scripts/k8s-bootstrap.sh new file mode 100755 index 0000000..cf56478 --- /dev/null +++ b/startup-scripts/k8s-bootstrap.sh @@ -0,0 +1,87 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +set -e + +# +STEP_CNT=4 + +echo_step() { +cat < Date: Sat, 25 Nov 2023 09:58:19 +0000 Subject: [PATCH 09/22] add ui service --- README.md | 2 +- rockcraft.yaml | 17 +++++++- startup-scripts/k8s-bootstrap.sh | 58 ++++++-------------------- startup-scripts/k8s-init.sh | 70 ++++++++++++++++++++++++++++++++ 4 files changed, 99 insertions(+), 48 deletions(-) create mode 100755 startup-scripts/k8s-init.sh diff --git a/README.md b/README.md index 83ec26c..2dc4274 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ sudo lxd init --auto ```bash rockcraft pack sudo skopeo --insecure-policy copy oci-archive:charmed-superset-rock_2.1.0_amd64.rock docker-daemon:/charmed-superset-rock:2.1.0 -docker run --rm -it charmed-superset-rock:2.1.0 +docker run -d --name superset-ui-services -p 8088:8088 charmed-superset-rock:2.1.0 --args superset-ui -g 'daemon off:' \; start superset-ui ``` ## License The Charmed Superset ROCK is free software, distributed under the Apache diff --git a/rockcraft.yaml b/rockcraft.yaml index d0508d7..1bde1a7 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -12,6 +12,18 @@ description: | visualize their data. license: Apache-2.0 +services: + superset-ui: + override: replace + summary: "superset-ui service" + startup: disabled + command: "/app/k8s/k8s-bootstrap.sh" + environment: + CHARM_FUNCTION: "app-gunicorn" + SUPERSET_SECRET_KEY: "supersetR0cks!" + ADMIN_PASSWORD: "admin" + SUPERSET_LOAD_EXAMPLES: True + platforms: amd64: @@ -34,7 +46,7 @@ parts: # make home directory mkdir -p ${APP_HOME} - # install pip + # Install pip apt-get update; apt-get upgrade -y; apt-get autoremove --purge -y; apt-get clean -y apt-get install python3-pip @@ -48,6 +60,7 @@ parts: source: ./startup-scripts organize: run-server.sh : app/k8s/run-server.sh + k8s-init.sh: app/k8s/k8s-init.sh k8s-bootstrap.sh: app/k8s/k8s-bootstrap.sh frontend: @@ -104,7 +117,7 @@ parts: pip install elasticsearch-dbapi==0.2.10 pip install trino==0.327.0 pip install pyhive==0.7.0 - pip install SQLALchemy==1.4.36 + pip install SQLALchemy==2.0.23 pip install thrift==0.16.0 pip install sqlalchemy-redshift==0.8.1 diff --git a/startup-scripts/k8s-bootstrap.sh b/startup-scripts/k8s-bootstrap.sh index cf56478..eeefd54 100755 --- a/startup-scripts/k8s-bootstrap.sh +++ b/startup-scripts/k8s-bootstrap.sh @@ -15,62 +15,30 @@ # See the License for the specific language governing permissions and # limitations under the License. # -set -e -# -STEP_CNT=4 - -echo_step() { -cat < Date: Sat, 25 Nov 2023 11:18:12 +0000 Subject: [PATCH 10/22] cleanup paths --- rockcraft.yaml | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/rockcraft.yaml b/rockcraft.yaml index 1bde1a7..d879e59 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -37,8 +37,8 @@ parts: stage-packages: - python3.10-venv build-environment: - - APP_HOME: "app" - - SOURCE: "https://downloads.apache.org/superset/" + - APP_HOME: "/app" + - SOURCE: "https://downloads.apache.org/superset" - VERSION: "2.1.0" - DIST_PACKAGES: "usr/local/lib/python3.10/dist-packages/" override-build: | @@ -46,10 +46,6 @@ parts: # make home directory mkdir -p ${APP_HOME} - # Install pip - apt-get update; apt-get upgrade -y; apt-get autoremove --purge -y; apt-get clean -y - apt-get install python3-pip - # Download and unpack tar wget "${SOURCE}/${VERSION}/apache-superset-${VERSION}-source.tar.gz" -O superset.tar.gz tar -zxvf superset.tar.gz -C ${APP_HOME} --strip-components 1 @@ -70,23 +66,21 @@ parts: build-snaps: - node/14/stable build-environment: - - APP_HOME: "app" - - ASSETS_DIR: "lib/python3.10/site-packages/superset/static/assets/" + - ASSETS_DIR: "/lib/python3.10/site-packages/superset/static/assets" override-build: | # Prepare for asset creation - mkdir -p /superset/static/assets/ cd superset-frontend - # Install missing dependency + # Install missing frontend dependency npm install currencyformatter.js --save - # Install dependencies and build assets + # Install frontend dependencies and build assets npm ci npm run build -- --output-path=/dist - # Copy assets to CRAFT_PART_INSTALL - mkdir -p "${CRAFT_PART_INSTALL}/${ASSETS_DIR}" - cp -r /dist/* "${CRAFT_PART_INSTALL}/${ASSETS_DIR}" + # Copy assets to CRAFT_PRIME + mkdir -p "${CRAFT_PRIME}${ASSETS_DIR}" + cp -r /dist/* "${CRAFT_PRIME}${ASSETS_DIR}" gunicorn-app: after: [backend, frontend, local-files] @@ -97,10 +91,10 @@ parts: stage-packages: - python3.10-venv build-environment: - - APP_HOME: "app" - - DIST_PACKAGES: "usr/local/lib/python3.10/dist-packages/" + - APP_HOME: "/app" + - DIST_PACKAGES: "/usr/local/lib/python3.10/dist-packages" override-build: | - # Install dependencies + # Install superset requirements pip install --upgrade setuptools pip pip install "cython<3.0.0" pip install --no-build-isolation pyyaml==5.4.1 @@ -125,9 +119,9 @@ parts: pip install -e . # Copy packages to PRIME - mkdir -p ${CRAFT_PRIME}/${DIST_PACKAGES} - cp -r /${DIST_PACKAGES}* $CRAFT_PRIME/${DIST_PACKAGES} - cp -r . "${CRAFT_PART_INSTALL}/${APP_HOME}" + mkdir -p ${CRAFT_PRIME}${DIST_PACKAGES} + cp -r ${DIST_PACKAGES}/* ${CRAFT_PRIME}${DIST_PACKAGES} + cp -r . ${CRAFT_PRIME}${APP_HOME} overlay-pkgs: plugin: nil From d0005af1bbf34523f09a0af295f228b10d55beab Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Sat, 25 Nov 2023 11:20:44 +0000 Subject: [PATCH 11/22] add build ci/cd workflow --- .github/workflows/build.yaml | 52 ++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 .github/workflows/build.yaml diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml new file mode 100644 index 0000000..fc21d2d --- /dev/null +++ b/.github/workflows/build.yaml @@ -0,0 +1,52 @@ +name: Build ROCK + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +on: + pull_request: + workflow_call: + +jobs: + lint: + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - name: Checkout repository + uses: actions/checkout@v3 + - name: Install tox + run: python3 -m pip install tox + - name: YAML Lint + run: tox -e lint + + build: + runs-on: ubuntu-latest + timeout-minutes: 15 + needs: + - lint + steps: + - name: Checkout repository + uses: actions/checkout@v3 + - name: Setup LXD + uses: canonical/setup-lxd@main + - name: Install dependencies + run: | + sudo snap install yq + sudo snap install rockcraft --classic --edge + - name: Build ROCK + run: | + app_version=$(yq '.version' rockcraft.yaml) + version=$(yq '(.version|split("-"))[0]' rockcraft.yaml) + base=$(yq '(.base|split("@"))[1]' rockcraft.yaml) + risk=edge + tag=${version}-${base}_${risk} + + sed -i "s/${app_version}/${tag}/g" rockcraft.yaml + rockcraft pack --verbose + + - name: Upload locally built ROCK artifact + uses: actions/upload-artifact@v3 + with: + name: charmed-superset-rock + path: "charmed-superset-rock_*.rock" \ No newline at end of file From 9a7aa8174cd71ca8da90e05d85ad849078e40b2e Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Sat, 25 Nov 2023 11:38:13 +0000 Subject: [PATCH 12/22] add tox.ini and do linting --- rockcraft.yaml | 18 +++++++++++------- tox.ini | 12 ++++++++++++ 2 files changed, 23 insertions(+), 7 deletions(-) create mode 100644 tox.ini diff --git a/rockcraft.yaml b/rockcraft.yaml index d879e59..490d0fa 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -8,7 +8,7 @@ version: '2.1.0' # Semantic versioning for human readability summary: Charmed Superset ROCK OCI description: | Superset is fast, lightweight, intuitive, and loaded with options - that make it easy for users of all skill sets to explore and + that make it easy for users of all skill sets to explore and visualize their data. license: Apache-2.0 @@ -30,7 +30,8 @@ platforms: parts: backend: plugin: python - source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz + source: | + https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz build-packages: - build-essential - wget @@ -47,7 +48,8 @@ parts: mkdir -p ${APP_HOME} # Download and unpack tar - wget "${SOURCE}/${VERSION}/apache-superset-${VERSION}-source.tar.gz" -O superset.tar.gz + wget "${SOURCE}/${VERSION}/apache-superset-${VERSION}-source.tar.gz" \ + `-O superset.tar.gz tar -zxvf superset.tar.gz -C ${APP_HOME} --strip-components 1 rm -rf superset.tar.gz @@ -55,14 +57,15 @@ parts: plugin: dump source: ./startup-scripts organize: - run-server.sh : app/k8s/run-server.sh + run-server.sh: app/k8s/run-server.sh k8s-init.sh: app/k8s/k8s-init.sh k8s-bootstrap.sh: app/k8s/k8s-bootstrap.sh frontend: after: [backend] plugin: nil - source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz + source: | + https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz build-snaps: - node/14/stable build-environment: @@ -85,7 +88,8 @@ parts: gunicorn-app: after: [backend, frontend, local-files] plugin: python - source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz + source: | + https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz build-packages: - build-essential stage-packages: @@ -96,7 +100,7 @@ parts: override-build: | # Install superset requirements pip install --upgrade setuptools pip - pip install "cython<3.0.0" + pip install "cython<3.0.0" pip install --no-build-isolation pyyaml==5.4.1 pip install -r requirements/base.txt --target=/${DIST_PACKAGES} diff --git a/tox.ini b/tox.ini new file mode 100644 index 0000000..5c5b624 --- /dev/null +++ b/tox.ini @@ -0,0 +1,12 @@ +[tox] +requires = + tox>=4 +env_list = lint + +[testenv:lint] +description = run linters +skip_install = true +deps = + yamllint +commands = + yamllint --no-warnings rockcraft.yaml \ No newline at end of file From a6be44052b3600b3524468234bcb8538237f78b7 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Mon, 27 Nov 2023 09:21:34 +0000 Subject: [PATCH 13/22] update build versioning --- .github/workflows/build.yaml | 4 ++-- rockcraft.yaml | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index fc21d2d..7d26a69 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -40,9 +40,9 @@ jobs: version=$(yq '(.version|split("-"))[0]' rockcraft.yaml) base=$(yq '(.base|split("@"))[1]' rockcraft.yaml) risk=edge - tag=${version}-${base}_${risk} + tag=${version}-${base}-${risk} - sed -i "s/${app_version}/${tag}/g" rockcraft.yaml + sed -i "0,/${app_version}/{s//${tag}/}" rockcraft.yaml rockcraft pack --verbose - name: Upload locally built ROCK artifact diff --git a/rockcraft.yaml b/rockcraft.yaml index 490d0fa..6faeb68 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -4,7 +4,7 @@ # https://github.com/apache/superset/blob/master/Dockerfile name: charmed-superset-rock base: ubuntu@22.04 -version: '2.1.0' # Semantic versioning for human readability +version: 2.1.0 summary: Charmed Superset ROCK OCI description: | Superset is fast, lightweight, intuitive, and loaded with options From 3cdba5d0a0bc3154d260924eb81bf4a6b60e3698 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Mon, 27 Nov 2023 10:22:03 +0000 Subject: [PATCH 14/22] source on 1 line with disabled linting --- rockcraft.yaml | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/rockcraft.yaml b/rockcraft.yaml index 6faeb68..3998aa0 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -30,8 +30,7 @@ platforms: parts: backend: plugin: python - source: | - https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz + source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz # yamllint disable-line build-packages: - build-essential - wget @@ -64,8 +63,7 @@ parts: frontend: after: [backend] plugin: nil - source: | - https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz + source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz # yamllint disable-line build-snaps: - node/14/stable build-environment: @@ -88,8 +86,7 @@ parts: gunicorn-app: after: [backend, frontend, local-files] plugin: python - source: | - https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz + source: https://downloads.apache.org/superset/2.1.0/apache-superset-2.1.0-source.tar.gz # yamllint disable-line build-packages: - build-essential stage-packages: From 63d41b51409d4434e9cffca03b4b58260af94548 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Mon, 27 Nov 2023 10:37:45 +0000 Subject: [PATCH 15/22] fix typo and update version name --- rockcraft.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rockcraft.yaml b/rockcraft.yaml index 3998aa0..a61e0c0 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -4,7 +4,7 @@ # https://github.com/apache/superset/blob/master/Dockerfile name: charmed-superset-rock base: ubuntu@22.04 -version: 2.1.0 +version: 2.1.0-22.04-edge summary: Charmed Superset ROCK OCI description: | Superset is fast, lightweight, intuitive, and loaded with options @@ -48,7 +48,7 @@ parts: # Download and unpack tar wget "${SOURCE}/${VERSION}/apache-superset-${VERSION}-source.tar.gz" \ - `-O superset.tar.gz + -O superset.tar.gz tar -zxvf superset.tar.gz -C ${APP_HOME} --strip-components 1 rm -rf superset.tar.gz From c2ae85c98f9f06708956ad846b210da421fdc99c Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Mon, 27 Nov 2023 10:56:09 +0000 Subject: [PATCH 16/22] add publish CI/CD workflow --- .github/workflows/publish.yaml | 55 ++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 .github/workflows/publish.yaml diff --git a/.github/workflows/publish.yaml b/.github/workflows/publish.yaml new file mode 100644 index 0000000..ea54d46 --- /dev/null +++ b/.github/workflows/publish.yaml @@ -0,0 +1,55 @@ +name: Publish ROCK +on: + push: + branches: + - main + +jobs: + build: + uses: ./.github/workflows/build.yaml + publish: + needs: build + runs-on: ubuntu-latest + timeout-minutes: 20 + permissions: + packages: write + steps: + - name: Checkout repository + uses: actions/checkout@v3 + - name: Setup Docker + run: | + sudo snap install docker + sudo addgroup --system docker; sudo adduser $USER docker + newgrp docker + sudo snap disable docker; sudo snap enable docker + - name: Install skopeo + run: | + sudo snap install --devmode --channel edge skopeo + - name: Install yq + run: | + sudo snap install yq + - uses: actions/download-artifact@v3 + with: + name: charmed-superset-rock + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + with: + registry: ghcr.io + username: ${{ secrets.GHCR_USER }} + password: ${{ secrets.GHCR_TOKEN }} + - name: Import and push to GHCR + run: | + app_version=$(yq '.version' rockcraft.yaml) + version=$(yq '(.version|split("-"))[0]' rockcraft.yaml) + base=$(yq '(.base|split("@"))[1]' rockcraft.yaml) + risk=edge + tag=${version}-${base}_${risk} + + sudo skopeo --insecure-policy copy \ + oci-archive:charmed-superset-rock_${tag}_amd64.rock \ + docker-daemon:ghcr.io/canonical/charmed-superset-rock:${tag} + + docker tag \ + ghcr.io/canonical/charmed-superset-rock:${tag} + + docker push ghcr.io/canonical/charmed-superset-rock:${tag} From 38e811940a868e797763fe5c3c6f803039c859b9 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Mon, 27 Nov 2023 11:14:02 +0000 Subject: [PATCH 17/22] increase timeout minutes --- .github/workflows/build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 7d26a69..0d53a21 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -22,7 +22,7 @@ jobs: build: runs-on: ubuntu-latest - timeout-minutes: 15 + timeout-minutes: 30 needs: - lint steps: From 1dbb534ccfc403cf0c33922ad79cd3a0c53b1eac Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Mon, 27 Nov 2023 13:07:27 +0000 Subject: [PATCH 18/22] EOF and add superset description --- .github/ISSUE_TEMPLATE/bug_report.md | 2 +- .github/workflows/build.yaml | 2 +- .github/workflows/publish.yaml | 2 +- .gitignore | 3 ++- README.md | 2 +- licenses/LICENSE-rock | 2 +- rockcraft.yaml | 6 +++--- 7 files changed, 10 insertions(+), 9 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 6ecc03d..c3bdc6c 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -27,4 +27,4 @@ Operating system: ## Additional context - \ No newline at end of file + diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 0d53a21..d5d3a0d 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -49,4 +49,4 @@ jobs: uses: actions/upload-artifact@v3 with: name: charmed-superset-rock - path: "charmed-superset-rock_*.rock" \ No newline at end of file + path: "charmed-superset-rock_*.rock" diff --git a/.github/workflows/publish.yaml b/.github/workflows/publish.yaml index ea54d46..a5a1ade 100644 --- a/.github/workflows/publish.yaml +++ b/.github/workflows/publish.yaml @@ -43,7 +43,7 @@ jobs: version=$(yq '(.version|split("-"))[0]' rockcraft.yaml) base=$(yq '(.base|split("@"))[1]' rockcraft.yaml) risk=edge - tag=${version}-${base}_${risk} + tag=${version}-${base}-${risk} sudo skopeo --insecure-policy copy \ oci-archive:charmed-superset-rock_${tag}_amd64.rock \ diff --git a/.gitignore b/.gitignore index 013c47c..0fd7c2c 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ -*.rock \ No newline at end of file +*.rock +.tox diff --git a/README.md b/README.md index 2dc4274..172e5dd 100644 --- a/README.md +++ b/README.md @@ -42,4 +42,4 @@ docker run -d --name superset-ui-services -p 8088:8088 charmed-superset-rock:2.1 The Charmed Superset ROCK is free software, distributed under the Apache Software License, version 2.0. See [LICENSE](https://github.com/canonical/charmed-superset-rock/blob/main/LICENSE) -for more information. \ No newline at end of file +for more information. diff --git a/licenses/LICENSE-rock b/licenses/LICENSE-rock index 7a4a3ea..d645695 100644 --- a/licenses/LICENSE-rock +++ b/licenses/LICENSE-rock @@ -199,4 +199,4 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and - limitations under the License. \ No newline at end of file + limitations under the License. diff --git a/rockcraft.yaml b/rockcraft.yaml index a61e0c0..c2aeb9c 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -7,9 +7,9 @@ base: ubuntu@22.04 version: 2.1.0-22.04-edge summary: Charmed Superset ROCK OCI description: | - Superset is fast, lightweight, intuitive, and loaded with options - that make it easy for users of all skill sets to explore and - visualize their data. + Superset is a business intelligence tool that is fast, lightweight, + intuitive, and loaded with options that make it easy for users + to explore and visualize their data. license: Apache-2.0 services: From b4f79c7a9bbf6f2207e436fd476e49c07562d362 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Tue, 28 Nov 2023 11:24:32 +0000 Subject: [PATCH 19/22] Update to fix multipass and add login details --- README.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 172e5dd..1aaf39e 100644 --- a/README.md +++ b/README.md @@ -10,34 +10,36 @@ If you are using another version of Ubuntu or another operating system, the proc To avoid any issue with other operating systems you can simply build the image with [multipass](https://multipass.run/): ```bash sudo snap install multipass -multipass launch 22.04 -n rock-dev +multipass launch -n rock-dev -m 8g -c 2 -d 20G multipass shell rock-dev -``` - +``` ### Clone Repository ```bash git clone https://github.com/canonical/charmed-superset-rock.git cd charmed-superset-rock ``` -### Installing Prerequisites +### Installing & Configuring Prerequisites ```bash sudo snap install rockcraft --edge --classic -sudo snap install docker -sudo snap install lxd sudo snap install skopeo --edge --devmode -``` -### Configuring Prerequisites -```bash -sudo usermod -aG docker $USER sudo lxd init --auto + +# Note: Docker must be installed after LXD is initialized due to firewall rules incompatibility. +sudo snap install docker +sudo groupadd docker +sudo usermod -aG docker $USER +newgrp docker ``` -*_NOTE:_* You will need to open a new shell for the group change to take effect (i.e. `su - $USER`) ### Packing and Running the ROCK ```bash rockcraft pack -sudo skopeo --insecure-policy copy oci-archive:charmed-superset-rock_2.1.0_amd64.rock docker-daemon:/charmed-superset-rock:2.1.0 +sudo skopeo --insecure-policy copy oci-archive:charmed-superset-rock_2.1.0_amd64.rock docker-daemon:charmed-superset-rock:2.1.0 docker run -d --name superset-ui-services -p 8088:8088 charmed-superset-rock:2.1.0 --args superset-ui -g 'daemon off:' \; start superset-ui ``` +### Login +After following the above instructions, navigate to `localhost:8088` and login with `admin`/`admin`. +Further information on connecting data sources and creating dashboards can be found at [superset.apache.org](https://superset.apache.org/docs/creating-charts-dashboards/creating-your-first-dashboard/). + ## License The Charmed Superset ROCK is free software, distributed under the Apache Software License, version 2.0. See From 7b180cb9776c7e419c19c37879d954b85cfb9a46 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Tue, 28 Nov 2023 13:03:19 +0000 Subject: [PATCH 20/22] update readme access to superset --- README.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1aaf39e..a7f64e6 100644 --- a/README.md +++ b/README.md @@ -33,11 +33,18 @@ newgrp docker ### Packing and Running the ROCK ```bash rockcraft pack -sudo skopeo --insecure-policy copy oci-archive:charmed-superset-rock_2.1.0_amd64.rock docker-daemon:charmed-superset-rock:2.1.0 +sudo skopeo --insecure-policy copy oci-archive:charmed-superset-rock_2.1.0-22.04-edge_amd64.rock docker-daemon:charmed-superset-rock:2.1.0 docker run -d --name superset-ui-services -p 8088:8088 charmed-superset-rock:2.1.0 --args superset-ui -g 'daemon off:' \; start superset-ui ``` ### Login -After following the above instructions, navigate to `localhost:8088` and login with `admin`/`admin`. +To access Superset, now exit the multipass instance run `multipass list` for the below output. +``` +Name State IPv4 Image +rock-dev Running 10.137.215.60 Ubuntu 22.04 LTS + 10.194.234.1 + 172.17.0.1 +``` +Navigate to `:8088` (in this case 10.137.215.60:8088) and login with `admin`/`admin`. Further information on connecting data sources and creating dashboards can be found at [superset.apache.org](https://superset.apache.org/docs/creating-charts-dashboards/creating-your-first-dashboard/). ## License From fda8aaf203fe902237819fa41268d7e0a1651350 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Tue, 28 Nov 2023 14:43:39 +0000 Subject: [PATCH 21/22] add sha check for superset tar --- rockcraft.yaml | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/rockcraft.yaml b/rockcraft.yaml index c2aeb9c..a9e7f6b 100644 --- a/rockcraft.yaml +++ b/rockcraft.yaml @@ -40,15 +40,25 @@ parts: - APP_HOME: "/app" - SOURCE: "https://downloads.apache.org/superset" - VERSION: "2.1.0" + - TAR_FILE: "apache-superset-2.1.0-source.tar.gz" - DIST_PACKAGES: "usr/local/lib/python3.10/dist-packages/" override-build: | craftctl default # make home directory mkdir -p ${APP_HOME} - # Download and unpack tar - wget "${SOURCE}/${VERSION}/apache-superset-${VERSION}-source.tar.gz" \ - -O superset.tar.gz + # Download the Superset tar + wget "${SOURCE}/${VERSION}/${TAR_FILE}" -O superset.tar.gz + + # Download the Superset sha512 + wget "${SOURCE}/${VERSION}/${TAR_FILE}.sha512" -O superset.tar.sha512 + + # Validate successful file download + sha_file=$(tail -n +2 superset.tar.sha512) + hash=$(echo $sha_file | tr -d '[:space:]' | tr '[:upper:]' '[:lower:]') + echo "$hash superset.tar.gz" | sha512sum --check + + # Unpack tar tar -zxvf superset.tar.gz -C ${APP_HOME} --strip-components 1 rm -rf superset.tar.gz From db634b91b1d369968ae1395e139c07bb70a83f16 Mon Sep 17 00:00:00 2001 From: Amber Charitos Date: Wed, 29 Nov 2023 15:50:42 +0000 Subject: [PATCH 22/22] disable and enable docker snap for docker group --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index a7f64e6..424d3ad 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,11 @@ sudo snap install docker sudo groupadd docker sudo usermod -aG docker $USER newgrp docker + +# Note: disabling and enabling docker snap is required to avoid sudo requirement. +# As described in https://github.com/docker-snap/docker-snap. +sudo snap disable docker +sudo snap enable docker ``` ### Packing and Running the ROCK ```bash