deployment variable in a (secure) environment variable #349
Conversation
The argument name --dkenv is short for deployment key environment variable, still considering better names for this.
Still need to fine tune how best to print the private key to screen (newlines etc). Next need to use the environment key in deploy.
Getting the indentation to look nice (while deliberately not indenting the private key to make life simple for copy-and-paste).
|
I wouldn't try to test locally. I can give you push access to push this up directly and you can test it on Travis. I'm not clear if that gives you access to upload deploy keys so if it doesn't just let me know and I'll upload it. |
|
Looks like some useful error from TravisCI as is. Trying the deploy stuff as a shell script (locally), turned up another issue. The branch is here in the short term: https://github.com/peterjc/biopython/tree/deploy Called this file #!/bin/bash
# Call ssh using our GitHub repository deploy key (set via -i)
# using -F to make sure this ignores ~/.ssh/config
ssh -i "$HOME/.biopython_doc_deploy.key" -F /dev/null -p 22 $*Called by: #!/bin/bash
set -euo pipefail
# Assumes being called from the Biopython repository's root folder,
# (i.e. a clone of https://github.com/biopython/biopython) as part
# of our continuous integration testing to save the compiled docs
# to https://github.com/biopython/docs
#
# In order to have write permissions, we put a private key into the
# TravisCI settings as a secure environment variable, and put the
# matching public key into the GitHub documentation repository's
# settings as a deploy key with write permissions.
DEST_SLUG=biopython/docs
DEST_DIR=${TRAVIS_TAG:-dev}
SOURCE_DIR=${TRAVIS_BUILD_DIR:-.}/Doc/api/_build/html
WORKING_DIR=/tmp/deploy_biopython_docs
# This will make git use our deployment key,
# using sed to restore line breaks
echo $DOC_KEY | sed -E -e 's/ RSA PRIVATE /private/g' | sed -E -e 's/[[:space:]]+/\n/g' | sed -E -e 's/private/ RSA PRIVATE /g' > $HOME/.biopython_doc_deploy.key
chmod 600 $HOME/.biopython_doc_deploy.key
export GIT_SSH=${TRAVIS_BUILD_DIR:-$PWD}/.github/ssh_via_deploy_key.sh
# Clone the destination under /tmp (public URL, no key needed)
rm -rf $WORKING_DIR
git clone https://github.com/$DEST_SLUG.git $WORKING_DIR
# Update the files (assumes nothing removed)
mkdir -p $WORKING_DIR/$DEST_DIR
cp -R $SOURCE_DIR/* $WORKING_DIR/$DEST_DIR/
pushd $WORKING_DIR
git checkout gh-pages
# Switch the git protocol to SSH based so we can use our key
git remote set-url origin --push [email protected]:$DEST_SLUG.git
git add $DEST_DIR/
git commit -m "Automated update" --author "TravisCI <[email protected]>"
git push origin gh-pages
popd
echo "Documentation deployed!"The key finding from these experiments is a little massaging of the public key is needed to restore the line lines - done in the above with sed, but would be clearer in Python as part of |
|
Another important lesson from trying this on TravisCI, Most likely we need to print the private key to screen escaped suitably for TravisCI to accept it (spaces and new lines), and unescape in the script. |
|
Can we base64 encode it? |
|
Actually the private key should already be base64 encoded. There's probably an issue with the header part of the file. This is what a private key that I just generated with So we should remove the |
|
I forgot that we use >>> doctr.local.generate_ssh_key()
(b'-----BEGIN PRIVATE KEY-----\nMIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC6rPXZjhn82O3+\nvWBF0mZocRTKAdOxRvdl8ezF+P2B7+tjr0uoMveTFDSLny6dIf9j+NLGAmCywtT6\n3+/KOBZXiMFIfACcepQt9CEUWJfPZSo6/u8cjGneuZHNiT9z9Jo/PbZDAeKPGbdq\nZgTtJxb01Lo51zae5Z+rqiW5HqAwK2wpV/uz8FGvcsRN4luTjrgagG8o8Eu4AMzP\nZo0NiwSn+jY9SyrJ4ynMUXdA4AyvulYPMzDZtayNyws+31M5PQYMWmuVbdJ2wSnz\nfyvqchUay0jAnXytJ87c4wKbjrMeaAuy1QfOkpw/vMXQn8xvAepIrXvbwA+6tGne\nldXnXTQgPNtaYZC5vc0PRZAAbsAHf/5qQjk1mut9v6meYUnRZgKE8qWp0Jwo6JFi\nXWQ+9+Ba/WcqqVwtAho7vydvxKM72oMVLRhg3psYWJduQfa+p6vs6DJUBBf4cTIA\nzXg+ZJ0jZJ+UX5gmitI4p2gdstzQX95SbvLCPwxElAAOnmH5rKPR32JK08RmtwBc\nPDabQs/u+DY1EC97nb2VjM4QOH6xfc7pyYI1d0x+rPqRh8TZrZCTnLa5i8F9lMVn\nLColU0KzUdRzuBHaTFeUOuVnpPJ7j/VEmA8hFP03dCLHRt36tI++xW89Gd+I7seX\nfl1u1k0KxBuI8EVyTF8sPTz/vjS3XQIDAQABAoICAQCrlyrRVHZ83aY+fzLMW284\n16EFYpKFJLdqJOvAunPECZX0ZoCD1n2N24fFQ4fkdgi8i06rJipehwqgpFFVLyMS\nSSlpiFpDe8VTTFFP03OP9uzPl+CQ/FyglzD3ng4OdyuFsCMmCsiHQ1s+WRJ0L3dB\ni3y2iCWz/1w6vka4l/ck7/UXN8GtD9z2Ced5s/T7eLev3JjRJ7hiJZIdnqVPapbY\nFP3gb4SgWMfmAIg+wPPIX96VUDe6Fu3K1HW80Ck+tuIlXsP/chiAgmQeZ6olccIG\nhA+WxeyBedMDZUPTW2M4Mul1862ea1Nmnw2yDAEtlLQXJChywWNz+jxKlq4tYpXy\n7jgg9V6wcI5FkduRIkhVPny1ZZw1kMqycl0R8VzkR0vEBioC8WVrZhkHAEswWEhS\nhtmQ2xgstlPDkHSImpZCqWOS3v0gDxv2+9Sw27nSAi9zNsZKcfp4roDvPK3+9l2m\nOR17u1Vj9OQpMsMQ7OQy9ZH4NIQEI3DV+nf2JS9b8wFhAOhgLgnMnF5Frdpmxtcm\nU0n+gXemEl+13k8FBx7Ub4beRC5Xo++AifOs+O5GSPS9fhpMmIWHIk/e0ZATABch\nOf0VhYpamj86rxhLCQwjkvMO8EqDqFBwsY4XZVT8VilNU/M58BLKgKWxODGFj6ED\n+SUrVY+N8Nz/i8+2jDrhPQKCAQEA2/jFKpZgl99J8KkjBfhXoBxF4HSPUrIQg4s1\nK0Ewg0PMNLvAbDfXV1xxJ+xLyXNaWQwELh3oIdAe8wt2ftcSS1V4xHemf7oysBB2\n4cd2lHJy/+TjOzjocmo2cMlfp0aRKFvgt0B67gNhW6hYCas8TpKQBrVaHRdBHs+m\n5+rVkX5KQzafIYiRJ9tBmKUwO3Rk2HGHYovyuDY4C+1sNfXE5MyIYaRLQLAkvDa4\nnj9Rq6z6fI5Yb2gwYFJ6/5nT9IchjwmEzGKqGeeG761Xp7v+tJTZ1ka37AVl0qTW\nHMOnAxgQm1AJdcdG4oyEjFkWbdDD4fXq7nTuQ1azVFE4J2E2zwKCAQEA2UAci5oF\n/shgyjUGBOMSzCTpi3z2W3cjajHqUyK6DREKC6Jrldjec7EIx68DdI7th+VnWlwW\n2/7RjqfKGxkfGj7IUUFMgTlBHEq4mVFRVKhqHFhdh1FBPnH+ATn8hgOp0CxGP4v/\nqO1lLXLHe/Kar2cK8zNlbHg5/u1i6PuC3K0WbM+8CdXDvn+qwl5oUQQjKs/GZoVp\nS+cKJtiT50DH2qoOx9U6vwsguNOgWyEZgQ3Miss0NdMb+D/FBDXD2WjWPQkJ+ruR\nFdWZwR/Jx7JpboDlrNJpCZsIevunYqnR+23U+ZxBTnH9dAJkuIrSFhMm6+YviSOg\nk/uBSnX/C/F6EwKCAQEAz8f37hdnnG2VeVc6tvvzQVETjEZtz25VfPv0uCv2uDdF\nYBZtV4uTxHiUhmKE4AAvOmfIVwt25uGhKnEMeBmNtU1CK0reIk5ubLLQqMpxrx1A\nlYjOP3Ws086SKA1/ZhGZMec/p7mnpMXao+qrZk6yQ4HbvAp32XzKzWDWRsEjBTCm\n00B4JgPLITvRhW+b1L1IOM9cU/Dfz7OfU1zsVzgUyQ6OULURREReHs8NqqUi7ygQ\n37DRxkJDV+jxOBlFBfjS8TrLjwgvpxJ0+lbhspY4rLjh366jMrWSjduYSEljq9+C\naEK8/NzEj2CuH6hTMF3/eaSCSsZ2/XKbKC0j/sasLwKCAQEAgidtspkpJFYp3prb\nq0vbNCCdJmtMMMn0lqem6f2xFyjxKr041UJjK06RowgP+uGyHqtqOvFW5KAKLfwK\nEif/wTqBymRjkDub7XY6l+fm4OAxCiBKkEo221FxyoxR5HwHXWdZArM+DJeE+TB9\noJ1c3N7P6ZoOFmkE3dycWFZuNQUhnTjrP70ok1VrGR10Q61F4F0wULV2uvmE1HcG\nTRI7aZ5eUoxFsLTa+sAWnuH6pJ1+wFwzQFfkttqFjxsi5Xpwd4qVxvheWIVqoxAH\nVDNoBMMGVn6MXSvbbcqconh5C7fmU1Cws22JWdohO4o3iPAablOugOuuRVn1QIXm\nseIOrwKCAQBlMAi+1WJI3aEqW3Ej3EcfP6Vp2Y1lu/p4Wj7PLoExxV9iRudU6RxQ\ndNX8JvubJUxaXGgSMgEFT5rV5q8G8Ig7tVHG0WjgTeiW9JN+i9woWygybMl7Kcyz\nAUVDQFxsC/bLAvk0bZ+PuFpmTG+rzIeEE1o8RqF9MOnffqeD0thfdt20djC4aLdp\nkI0Wl9FGpCmuLiioXC40WOLnB1sD7ykRr6VKzPxyukESgd1WQ97SlM7ol47idTKS\n+NmfeIj1zbp5cfaAq+uwh0HZ0lYJ7Jub44BJov+HLBBScTivKUIhRrkbIyzudFwZ\nmLoKXnxd/xtDY5UfWxNqpc2iVsdyYgqf\n-----END PRIVATE KEY-----\n', b'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC6rPXZjhn82O3+vWBF0mZocRTKAdOxRvdl8ezF+P2B7+tjr0uoMveTFDSLny6dIf9j+NLGAmCywtT63+/KOBZXiMFIfACcepQt9CEUWJfPZSo6/u8cjGneuZHNiT9z9Jo/PbZDAeKPGbdqZgTtJxb01Lo51zae5Z+rqiW5HqAwK2wpV/uz8FGvcsRN4luTjrgagG8o8Eu4AMzPZo0NiwSn+jY9SyrJ4ynMUXdA4AyvulYPMzDZtayNyws+31M5PQYMWmuVbdJ2wSnzfyvqchUay0jAnXytJ87c4wKbjrMeaAuy1QfOkpw/vMXQn8xvAepIrXvbwA+6tGneldXnXTQgPNtaYZC5vc0PRZAAbsAHf/5qQjk1mut9v6meYUnRZgKE8qWp0Jwo6JFiXWQ+9+Ba/WcqqVwtAho7vydvxKM72oMVLRhg3psYWJduQfa+p6vs6DJUBBf4cTIAzXg+ZJ0jZJ+UX5gmitI4p2gdstzQX95SbvLCPwxElAAOnmH5rKPR32JK08RmtwBcPDabQs/u+DY1EC97nb2VjM4QOH6xfc7pyYI1d0x+rPqRh8TZrZCTnLa5i8F9lMVnLColU0KzUdRzuBHaTFeUOuVnpPJ7j/VEmA8hFP03dCLHRt36tI++xW89Gd+I7seXfl1u1k0KxBuI8EVyTF8sPTz/vjS3XQ==') |
|
Or alternately we can change the encoding argument in |
| @@ -134,6 +135,10 @@ def get_parser(config=None): | |||
| if we do not appear to be on Travis.""") | |||
| deploy_parser_add_argument('deploy_directory', type=str, nargs='?', | |||
| help="""Directory to deploy the html documentation to on gh-pages.""") | |||
| deploy_parser_add_argument('--dkenv', type=str, metavar="ENVVAR", | |||
There was a problem hiding this comment.
We should be able to detect this automatically.
There was a problem hiding this comment.
Maybe, personally I think the interface is much simpler and more flexible if the user picks the environment variable name(s) themselves.
There was a problem hiding this comment.
We can make a flag if the user wants to change it, but there should be a default. It should be possible to just use the variable names we already use, and detect based on the content if it is pointing to a file or if it is a private key. Most users don't really care how doctr works. They just do whatever it says to do and if it pushes their stuff up to gh-pages they are happy.
| @@ -193,6 +198,11 @@ def get_parser(config=None): | |||
| configure_parser.set_defaults(func=configure) | |||
| configure_parser.add_argument('--force', action='store_true', help="""Run the configure command even | |||
| if we appear to be on Travis.""") | |||
| configure_parser.add_argument('--dkenv', type=str, metavar="ENVVAR", | |||
There was a problem hiding this comment.
We should use the existing environment variable names. If this works we should use it by default and have the old way behind a flag (or just remove it).
There was a problem hiding this comment.
If you would be happy making the deploy key in an environment variable the default, then maybe you are right - re-using the old environment variable names would make sense.
There was a problem hiding this comment.
Assuming there are no issues with it we should. We will have to continue to support the old way at least for deploy for existing repos. It will need a lot of testing to make sure, once we get it working.
|
Right now I have been using this in bash to recover the key file: # This will make git use our deployment key, using sed to restore line breaks.
# On TravisCI, seems must create the variable using '\ ' and '\n'
# The sed commands unescape these (but the new line one fails on macOS):
echo $DOC_KEY | sed "s/\\\ / /g" | sed 's/\\n/\n/g' > $HOME/.biopython_doc_deploy.key
chmod 600 $HOME/.biopython_doc_deploy.key
export GIT_SSH=${TRAVIS_BUILD_DIR:-$PWD}/.github/ssh_via_deploy_key.shNote it won't necessarily be |
|
The SO answer says PKCS#8 uses "BEGIN PRIVATE KEY", which is what we use: Line 328 in 0f19ff7 |
|
Yes, |
|
I don't know if we really need to support that, unless it is trivial to do so. |
|
Took a bit of poking to work this out, but while (Note - This was a temp key, created just for testing) It looks like the slashes are treated as line continuations for the export command, so by the time we come to use it while the spaces can be recovered, the newlines have become just Instead, seems must escape spaces as |
|
I tried finishing this over at #350. When I try to encrypt the private key, I get Traceback (most recent call last):
File "/Users/aaronmeurer/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Users/aaronmeurer/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line 616, in <module>
sys.exit(main())
File "/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line 613, in main
return process_args(get_parser(config=config))
File "/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line 270, in process_args
return args.func(args, parser)
File "/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line 512, in configure
travis_token=travis_token, **login_kwargs)
File "/Users/aaronmeurer/Documents/doctr/doctr/local.py", line 84, in encrypt_variable
return base64.b64encode(key.encrypt(variable, pad))
File "/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 448, in encrypt
return _enc_dec_rsa(self._backend, self, plaintext, padding)
File "/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 68, in _enc_dec_rsa
return _enc_dec_rsa_pkey_ctx(backend, key, data, padding_enum, padding)
File "/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 122, in _enc_dec_rsa_pkey_ctx
_handle_rsa_enc_dec_error(backend, key)
File "/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 136, in _handle_rsa_enc_dec_error
"Data too long for key size. Encrypt less data or use a "
ValueError: Data too long for key size. Encrypt less data or use a larger key size.This is probably the original reason why I didn't include the private key directly in the environment variable. |
|
See https://stackoverflow.com/questions/55155642/python-cryptography-data-too-long-for-key-size and https://stackoverflow.com/a/7146463/161801. I did some testing and you can only encrypt 501 bytes with the Travis public key. Any more than that and you get the error. I think if we want to do this, we need to do the double encryption like we currently do, and put the contents of what would be github_deploy_key.enc in an environment variable. This doesn't really save the user much. Instead of doing a checkout of a file in the repo they would add the contents of said file to their .travis.yml. |
This is to demonstrate that this doesn't actually work. See #349 (comment). The private key is too large to be encrypted with the Travis public key. Instead we have to do what we are currently doing. Create a symmetric encryption key to encrypt the actual SSH key, and then encrypt that symmetric key into the secure environment variable (and save the encrypted SSH private key to a file).
|
@asmeurer I'm not sure why you are trying to encrypt anything - the goal of what I'm doing is to put the (slash escaped) private key directly into the continuous integration system's moderately secure environment variables settings (something TravisCI offers, and hopefully others too). This is working for me now using a proof of principle bash script for deployment, see biopython/biopython#2172 |
|
I didn't try to use the Travis UI. I can't imagine it would work there when it doesn't work in the API, but maybe they do something different there. The encryption is what makes the key secret. It is encrypted with the public key for the repo. It's the same thing as the |
|
It does let me paste it in the UI. I haven't tested yet if it actually works. |
|
Re: https://docs.travis-ci.com/user/environment-variables/
For my use case I don't want special values in If you are willing to trust TravisCI, then you don't need to do anything extra to secure the private key. I'm OK with that, but maybe you're not? |
|
I guess we can use https://docs.travis-ci.com/api/#settings-environment-variables. I think this API didn't exist before. I still need to test that it actually works. |
|
Nice - if the script can set the secret TravisCI variable via the API, that's one less manual step. Note in the UI, you can't update an existing secret variable - you have to delete and re-add it with the new value. |
|
FYI, my standalone script version of this has been live and working for over a year now: Apologies that I have not got back to finishing this pull request for doctr. |
Unfinished attempt to implement #348, adds a new command line option
--dkenvshort for deploy key in environment variable.As far as I can tell,
doctr configure --dbenv DOC_KEY ...works nicely. Building on the existing code this generates a key, sends the public key to the destination GitHub repository, and prints the private key to the terminal with instructions to add it to the source repository's TravisCI config.However, something is breaking on the
doctr deploy --dbenv DOC_KEY ...side, possibly in how I have been attempting to test is locally, and I don't understand enough of the design to immediately see where.e.g. Something like this with
--forceand setting the$TRAVIS_REPO_SLUGvariable to attempt to run locally: