save wp-content data to a separate volume #16

afeld · 2017-08-29T18:05:23Z

Closes #7.

afeld · 2017-08-29T18:06:18Z

@Vermyndax It shows up! Now just need to figure out what hooks to use for mounting/unmounting.

$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk
└─xvda1 202:1    0   8G  0 part /
xvdf    202:80   0  10G  0 disk

Vermyndax · 2017-08-29T19:42:45Z

\o/

Ok, so when you initially create this EBS volume (in the setup of the infrastructure), you'd want to attach it to an EC2 instance and make the file system on it:

sudo mkfs -t ext4 /dev/xvdf

...this will make the filesystem.

Next you'll want to mount it at a mount point. If wordpress is installed in /var/www/html/wordpress, then you would normally do:

sudo mount /dev/xvdf /var/www/html/wordpress/wp-content

The wp-content directory will still exist on the root volume, but it will become invisible because of the mounting of the volume in that directory. That's normal and fine, since the server's immutable anyway. Now, anything written to /var/www/html/wordpress/wp-content will go on the attached EBS volume.

To make this EBS volume attached to the instance throughout it's lifespan, two things have to happen:

The AWS orchestrator must attach the volume somehow... either through the console or an API call. Terraform would be handling this. this means the volume must not be attached to any other instance, which is where the real gotcha is... when terraform tears down the instance, it needs to detach the volume but not delete it
The volume, once attached by AWS, will be visible to the operating system as /dev/whatever (likely xvdf)... so then the operating system must mount it in the folder like above. Assuming there was data on the volume, it would now come online in the wp-content folder. Theoretically.

To preserve the mount as a permanent part of the instance's life, you would edit /etc/fstab as per the AWS documentation, but that's where I'm not sure what the best approach is... would it be better to do it this way, or would it be better to have terraform handle the attachment/detachment and folder mounting as part of infrastructure provisioning and modification? I suspect it's the latter.

afeld · 2017-08-30T10:15:44Z

It works! 🎉 Test steps (which we should get into code at some point):

Run terraform apply.
Sign into the WordPress admin interface.
Click Media on the sidebar.
Upload something.
Run terraform taint aws_instance.wordpress.
Run terraform apply.
Reload the Media Library page.
Ensure the image still appears.

Pulled a few smaller pull requests out of this one, so will rebase as soon as those are in.

afeld · 2017-08-30T10:19:58Z

README.md

@@ -110,3 +110,13 @@ For initial or subsequent deployment:
    ```sh
    terraform apply
    ```
+
+Note that if the public IP address changes after you set up the site initially, you will need to [change the site URL](https://codex.wordpress.org/Changing_The_Site_URL#Changing_the_Site_URL) in WordPress.


I learned the hard way that WordPress saves the URL where it's accessed for the initial setup in the database, and therefore having the public IP change frequently was making parts of the site inaccessible. While a custom domain is what you'd actually use, an Elastic IP is good enough for now.

You may want to look at installing the wp-cli to make changes like this easier to do from the machine itself in case you can't get in to the web interface.

Good call! Added as follow-up task: #22.

Vermyndax

Just a little nit on the volume about setting the type.

Vermyndax · 2017-08-30T19:18:39Z

terraform/volume.tf

@@ -0,0 +1,21 @@
+resource "aws_ebs_volume" "wp_content" {
+  # TODO deletion protection


@afeld Just a nit... When a "type" property isn't specified in this resource, it's created as a "standard" volume. Default for creating a volume in the console is "GP2" (General Purpose SSD). I'd recommend adding:
type = "gp2"
...just to keep things standardized I guess ;)
Ref: aws_ebs_volume

Along with that, you'll want encryption at rest enabled as well.

Per https://help.ubuntu.com/community/WordPress, this is required to allow automatic updates.

WordPress needs its database to be updated anytime the hostname changes - this sidesteps the issue for the common case of the instance being replaced, by using an Elastic IP that will stay constant.

afeld · 2017-08-31T21:48:17Z

ansible/wordpress.yml

+        owner: "{{ apache_user }}"
+        group: "{{ apache_group }}"
+        state: directory
+        recurse: true


Not positive that we want to allow automatic updates...? That would mean WordPress itself wouldn't be deployed immutably. Happy to take the conversation to an issue.

If the upstream source that deploys Wordpress on the instance is in sync with the updates, it may not be too bad. Wordpress updates download into /wp-content/updates and then deploy into the main directory. Sometimes those updates contain schema updates for the database as well. Lots to consider.

So, the way that I did this for our modified version of cf-ex-wordpress (deploying in cloud.gov) was to do some json in a particular schema, and feed that to wp-cli.

Relevant python bits here: cloud-gov/cf-ex-wordpress@25be580#diff-81085e34fc3ba4cf38cba76d477219bdR59 . I'm sure that could be turned into some ansible code.

Let's worry about it separately, if that's ok with you both: #23.

Also, make it an SSD.

afeld · 2017-08-31T22:10:34Z

Rebased and addressed the comments - should be ready to go!

LinuxBozo · 2017-09-01T13:45:10Z

terraform/files/attach-data-volume.sh

+sudo mv $DEST $OLDDEST
+sudo mkdir -p $DEST
+
+echo "$devpath $DEST ext4 defaults,nofail,noatime,nodiratime,barrier=0,data=writeback 0 2" | sudo tee -a /etc/fstab > /dev/null


Should probably check that it doesn't exist before appending to fstab, yeah?

Will look into the best way to do that. This script got complex enough that I'm considering changing it to Ansible (as a follow-up), which takes care of checks like that.

(but let me know if you know of an example)

Should be able to do with an egrep and regex:

if ! egrep "^${devpath}" /etc/fstab; then echo "$devpath $DEST ... etc fi

afeld · 2017-09-01T20:43:38Z

Ready for another look!

LinuxBozo

Looks good.

afeld force-pushed the persistent-volume branch from cb5c5d3 to dee0c3a Compare August 30, 2017 09:17

afeld commented Aug 30, 2017

View reviewed changes

afeld mentioned this pull request Aug 30, 2017

ensure the WordPress Site URL stays up-to-date #22

Open

Vermyndax reviewed Aug 30, 2017

View reviewed changes

afeld added 7 commits August 31, 2017 17:42

add a volume for saving wp-content data

660e5bc

mount volume after it's attached

fedd17a

fix automatic mounting of volume

3f15530

change the wordpress files to be owned by the apache user

4ec1503

Per https://help.ubuntu.com/community/WordPress, this is required to allow automatic updates.

remove need for symlinking volume mounting

d820392

use an elastic IP for the EC2 instance

5ab523c

WordPress needs its database to be updated anytime the hostname changes - this sidesteps the issue for the common case of the instance being replaced, by using an Elastic IP that will stay constant.

move the volume configuration to its own file

d7a6f1c

afeld force-pushed the persistent-volume branch from 303da4d to d7a6f1c Compare August 31, 2017 21:45

afeld commented Aug 31, 2017

View reviewed changes

afeld added 2 commits August 31, 2017 17:59

encrypt the EBS volume

2119322

Also, make it an SSD.

prevent destruction of the EBS volume and the RDS instance

85292c6

afeld mentioned this pull request Aug 31, 2017

prevent updates to WordPress? #23

Open

LinuxBozo reviewed Sep 1, 2017

View reviewed changes

afeld changed the title ~~[WORK IN PROGRESS] save wp-content data to a separate volume~~ save wp-content data to a separate volume Sep 1, 2017

don't copy mount info into fstab if already present

b4806ce

LinuxBozo approved these changes Sep 1, 2017

View reviewed changes

afeld merged commit 03067f6 into master Sep 1, 2017

afeld deleted the persistent-volume branch September 1, 2017 20:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

save wp-content data to a separate volume #16

save wp-content data to a separate volume #16

afeld commented Aug 29, 2017

afeld commented Aug 29, 2017

Vermyndax commented Aug 29, 2017

afeld commented Aug 30, 2017

afeld Aug 30, 2017

LinuxBozo Aug 30, 2017

afeld Aug 30, 2017

Vermyndax left a comment

Vermyndax Aug 30, 2017

LinuxBozo Aug 30, 2017

afeld Aug 31, 2017

Vermyndax Aug 31, 2017

LinuxBozo Aug 31, 2017

afeld Aug 31, 2017

afeld commented Aug 31, 2017

LinuxBozo Sep 1, 2017

afeld Sep 1, 2017

afeld Sep 1, 2017

LinuxBozo Sep 1, 2017

afeld commented Sep 1, 2017

LinuxBozo left a comment

		@@ -0,0 +1,21 @@
		resource "aws_ebs_volume" "wp_content" {
		# TODO deletion protection

save wp-content data to a separate volume #16

save wp-content data to a separate volume #16

Conversation

afeld commented Aug 29, 2017

afeld commented Aug 29, 2017

Vermyndax commented Aug 29, 2017

afeld commented Aug 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vermyndax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afeld commented Aug 31, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afeld commented Sep 1, 2017

LinuxBozo left a comment

Choose a reason for hiding this comment