Merge branch 'fixes/missing_path_of_new_file' of https://github.com/b…

…itkeeper/pi-gen-utils

LICENSE

@@ -0,0 +1,25 @@
+BSD 2-Clause License
+
+Copyright (c) 2020, Francis Turner
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,102 @@
+# pi-gen-utils
+Utility scripts to make it easy to develop custom RPi images using the pi-gen system (https://github.com/RPi-Distro/pi-gen )
+
+Pi-gen is incredibly useful as way to create images with precisely the packages and features required for a task. However it is conceptually wrong to have the custom scripts developed in the pi-gen directory structure. Firstly this causes issues as and when pi-gen itself is updated and secondly it is hard to identify the changed files and therefore what changes have been made for your project. 
+
+With these utilities you can have your files in a single directory and have them copied into the pi-gen directory to create the image. The original files are then stored in a backup directory and are restored between builds so as to keep the pi-gen directory tree clean.
+
+In the single directory the relatively complex directory structure of pi-gen is maintained but the '/'s are replaced by '_'. If a file needs an underscore in the name then it should be doubled. e.g. stage3/SKIP_IMAGES is called stage3_SKIP__IMAGES
+
+e.g. a project that creates a raspbian-lite image with different base packages might have the following files in its directory:
+```
+config
+stage0_00-configure-apt_00-run.sh
+stage2_01-sys-tweaks_00-packages
+stage2_01-sys-tweaks_01-run.sh
+stage2_01-sys-tweaks_files_customfile.txt
+stage3_SKIP
+stage4_SKIP
+stage4_SKIP__IMAGES
+stage5_SKIP
+stage5_SKIP__IMAGES
+```
+these would then be copied to the pi-gen directory as follows:
+```
+config
+stage0/00-configure-apt/00-run.sh
+stage2/01-sys-tweaks/00-packages
+stage2/01-sys-tweaks/01-run.sh
+stage2/01-sys-tweaks/files/customfile.txt
+stage3/SKIP
+stage4/SKIP
+stage4/SKIP_IMAGES
+stage5/SKIP
+stage5/SKIP_IMAGES
+```
+## The scripts
+
+There are three scripts in this repo:
+
+ * getpigenfile.sh
+   copies a file from pi-gen to the current location removing everything before "stage" or "export" and replacing all the path '/'s with '_'s
+ * setuppigen.sh 
+   copy the changed files into the pi-gen directory in preparation for building a new image, also copy any files that will be overwritten into a bak subdirectory of the current directory
+ * restorepigen.sh
+   copy all the files in the bak directory back to pi-gen and delete any other files that were added
+
+### Environment variable 
+`setuppigen.sh` and `restorepigen.sh` can have the path to the pi-gen directory set in a PP evironment variable, they can also have this passed to them on the command line
+
+## Intended usage
+
+copy the threee scripts to /usr/local/bin
+
+```
+cd basedirectory
+git clone https://github.com/RPi-Distro/pi-gen.git
+```
+install the pi-gen dependencies (and if planning to run in docker also install docker and docker-compose) 
+```
+mkdir myproject
+cd myproject
+```
+create a file called config with contents similar to below (note you must have ENABLE_SSH=1) 
+```
+IMG_NAME=example
+FIRST_USER_NAME=example
+FIRST_USER_PASS=example
+TARGET_HOSTNAME=example
+ENABLE_SSH=1
+```
+copy the files across 
+```
+getpigenfile.sh ../pi-gen/stageX/0...
+```
+create new files with the appropriate names ( e.g. stage2_01-sys-tweaks_files_customfile.txt or stage3_SKIP__IMAGES )
+
+```
+setuppigen.sh
+cd ../pi-gen
+build.sh # or build-docker.sh
+cd -
+restorepigen.sh
+```
+
+Create a second project
+```
+mkdir myproject2
+cd myproject2
+```
+create config file 
+```
+IMG_NAME=example2
+FIRST_USER_NAME=user
+FIRST_USER_PASS=password
+TARGET_HOSTNAME=example
+ENABLE_SSH=1
+```
+copy the files across 
+```
+getpigenfile.sh ../pi-gen/stageX/0...
+```
+etc.

getpigenfile.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+#
+# pi-gen-utils project
+#
+# Copyright (c) 2020, Francis Turner
+# All rights reserved.
+#
+# Copy files from the various pi-gen directories to the current working directory
+#
+# Usage: getpigenfile.sh /path/to/pi-gen/stageA/YY/file /path/to/pi-gen/stageB/ZZ/file*
+#
+# Note there is very little error checking
+#
+
+for arg in "$@"
+do
+  if [ ! -e $arg ]
+  then
+    echo "WARNING: file $arg does not exist"
+  elif [ -d $arg ]
+  then
+    echo "WARNING: skipping directory $arg, use $arg/* to get files in it"
+  else
+    target=`echo $arg | sed -e 's!^.*/stage!stage!' -e 's!^.*/export!export!' -e 's!/!_!g'`
+    echo "$arg -> $target"
+    dname=`dirname $PP/$D`
+    mkdir -p $dname
+    cp $arg $target
+  fi
+done

restorepigen.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+#
+# pi-gen-utils project
+#
+# Copyright (c) 2020, Francis Turner
+# All rights reserved.
+#
+# Copy files from bak subdirectory back into the various pi-gen directories 
+# and delete any other files in the pi-gen directory tree that have 
+# equivalent files in this directory
+#
+# Usage: restorepigen.sh [pi-gen-dir]
+#
+# If a pigen directory is specified on the command line it is used, if not
+# then if the environment variable PP is set use that, otherwise assume
+# the pi-gen directory is at '../pi-gen'
+#
+# Note there is very little error checking 
+# 
+
+PP=${1-${PP-'../pi-gen'}}
+
+echo "pi-gen directory: $PP"
+
+if [ ! -d $PP ] ; then
+    echo Directory '"'$PP'"' does not exist, cannot restore files to it
+    exit 1
+fi
+
+for F in stage* export*; do
+	if [ ! -e $F ] ; then
+	    continue
+	fi
+	D=`echo $F | sed 's/_/\//g' | sed 's!//!_!'`
+	if [ -e bak/$F ]; then
+		echo "back $F exists, restoring"
+		cp -v bak/$F $PP/$D
+	else
+		echo "no backup $F exists, removing"
+		rm -i $PP/$D
+	fi
+done

setuppigen.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+#!/bin/bash
+#
+# pi-gen-utils project
+#
+# Copyright (c) 2020, Francis Turner
+# All rights reserved.
+#
+# Copy files of the from bak subdirectory back into the various pi-gen directories
+# and delete any other files in the pi-gen directory tree that have
+# equivalent files in this directory
+#
+# Usage: setuppigen.sh [pi-gen-dir]
+#
+# If a pigen directory is specified on the command line it is used, if not
+# then if the environment variable PP is set use that, otherwise assume
+# the pi-gen directory is at '../pi-gen'
+#
+# Note there is very little error checking
+#
+
+PP=${1-${PP-'../pi-gen'}}
+
+if [ ! -d $PP ] ; then
+    echo Directory '"'$PP'"' does not exist, cannot copy files to it
+    exit 1
+fi
+
+echo "pi-gen directory: $PP"
+
+if [ ! -e bak ] ; then
+	mkdir bak
+fi
+for F in stage* export*; do
+	if [ ! -e $F ] ; then
+	    continue
+	fi
+	D=`echo $F | sed 's/_/\//g' | sed 's!//!_!'`
+	if [ -e $PP/$D ] ; then
+		if [ -e bak/$F ]; then
+			echo "Backup $F already present, skipping"
+		else
+			echo "Backing up $D"
+			cp  $PP/$D bak/$F
+		fi
+	else
+		echo "New file $F, skipping backup"
+	fi
+ 	dname=`dirname $PP/$D`
+    mkdir -p $dname
+	cp $F $PP/$D
+done
+cp config $PP