merge sfs and save files into a directory and apply whiteout

[s243a]

I wrote a script that merges sfs files and a save file into a single directory then updates the directory based on the whiteout files in the save folder.

https://pastebin.com/pvSANipi

The script is preliminary and I haven't tested all use cases. I'll update this post later.

This could be used as an offline way of remastering a puppy. Perhaps though it isn't as robust an approach as constructing a layered file system.

Currently this post is just a placeholder. I'll write more later.

[foxpup]

Interesting tool s243a.

I looked to the script and did not understand it at all.
But I am almost noob scripting.

I thought it would be simple.
This is what I think:
- unsquash all the sfs to the same working directory (wd)
- copy all files from savefile to the same wd
- remove all files with the whiteout prefix (suffix?) and the corresponding files without the whiteout prefix in de wd
- mksquashfs the wd
Is this wrong? How?

[s243a]

Interesting tool s243a.

I looked to the script and did not understand it at all.
But I am almost noob scripting.

It's partly my fault also because I added some complexity that wasn't strictly required to get the job done.

I thought it would be simple.
This is what I think:
- unsquash all the sfs to the same working directory (wd)
- copy all files from savefile to the same wd

This is correct. However, rather than unsquashing the sfs file I mount each sfs file at "/mnt/wksfs". Here is some trimmed down code :

around line #240 (but trimmed down and simplified

Code: Select all

while read -r -u9 file1 ; do
    mount -v -t squashfs $file1 /mnt/wksfs 

   mv_or_copy_files /mnt/wksfs $target_dir

   umount /mnt/wksfs
done 9< <( find $sfs_dir -maxdepth 1 -type f -name "*.sfs" | \
              grep -v -e "^$sfs_dir"/'f.*' | \
              grep -v -e "^$sfs_dir"/'z.*' )

Code: Select all

mv_or_copy_files()                                                 #Line number 120
  while IFS=$'\0' read  -r -d $'\0' -u10 fm_filePath_prefixed ; do #Line number 141
    #
    mkdir -p `dirname $to_filePath_prefixed`                       #Line number 161
    cp -a -u  $fm_filePath_prefixed $to_filePath_prefixed          #Files are only copied if they are newer 
  done 10< <( find $s_folder_fullpath -type f -name "*" -print0 )  #Line number 165
}

The input to the above loops comes from the find command at the end of the function:
https://blog.famzah.net/2016/10/20/bash ... -commands/

In the first find example we have:
(Not valid code)

Code: Select all

find $sfs_dir -maxdepth 1 -type f -name "*.sfs" \ #Find all files the end in .sfs (search only one level deep.
grep -v -e "^$sfs_dir"/'f.*' #exclude files that start with "f" (These are firware)
grep -v -e "^$sfs_dir"/'z.*' #exclude files that start with "z" (these are kernal modules)

In the second example we're finding all files within a given directory

Code: Select all

find $s_folder_fullpath -type f -name "*" -print0 

* the -type f option means find only files (no directors). It also seems to include symlinks although this isn't clear from the manpage.
* the -print0 option means to use the "null character" as the field seperator. This can help with unusually file names, However, I don't think my code is perfectly robust this way.

Both loops are reading and writing from a file descriptor. At the end of the loop we have:

Code: Select all

done 10< <( command ) 

the file descriptor is "10" or more properly written as "&10" (but not here). The reason to use a file descriptor is so that within the while loop we can accept terminal input wihout interfering with the read at the top of the loop. Terminal input comes from standard input (i.e. &1). I use standard input for debugging purposes. For instance I can pause the script with:

Code: Select all

read -p "Press enter to continue" <&1 

and then read from the screen a given set of echo statements that I printed out.

At the top of the loop the file descriptor input is showon with the -u10 option.

- remove all files with the whiteout prefix (suffix?) and the corresponding files without the whiteout prefix in de wd

This is correct:

Code: Select all

apply_white_out_files(){ #Line number 171
  case
...
  *) #Line number 207
...
    fname=${wh_fname#.wh.} #https://lwn.net/Articles/324291/
    rm $to_dirPath_prefixed/$fname  
    rm $to_dirPath_prefixed/$wh_fname  
    esac
...
} 

except that there is also a type of whitout file that means that the current directory is opaque (in other words it blocks all lower layers from showing up in the directory. In this case we replace the directory with the directory from the save folder.

Code: Select all

  case $wh_fname in
 '.wh..wh..opq') #Around line #190
    rm -rf $to_dirPath_prefixed
    cp $fm_dirPath_prefixed $to_dirPath_prefixed
   rm $to_dirPath_prefixed/$wh_fname       

Finally there are two types of whitout files that I don't know what they do so I just delete them:

Code: Select all

     '.wh..wh.plnk'|'.wh..wh.aufs') #https://unix.stackexchange.com/questions/92287/aufs-whiteout-removal
...
       rm $to_dirPath_prefixed/$wh_fname
       ;;    

- mksquashfs the wd
Is this wrong? How?

You could do this if you want to just replace the main puppy sfs file. Or alternatively you could build a new ISO.

Edit:; see:make_ISO_Fm_slitaz-rootfs
https://pastebin.com/5BfvZKXy

Requires:
make-tazpup_functions.sh
https://pastebin.com/K7bgqLFE

[s243a]

Some use notes. The actual script starts at line #261. Everything above that defines the functions that will be used.

Code: Select all

#This directory is where the file descriptors will write (don't change). 
mkdir -p /tmp/mv_or_copy_files

#Remove file descriptors and mount points. This line probably isn't 
#necessary because we are trapping the errors and removing the file
#descriptors
remove_fille_descriptpors

#This statment must occur after the "remove_fille_descriptpors" statment. (will fix spelling error later) 
trap remove_fille_descriptpors EXIT SIGKILL SIGTERM

#This is the location where the sfs are stored. Currently files that start
#with z or f are excluded. If you want to change this remove the #statments (around line #254):
#grep -v -e "^$sfs_dir"/'f.*' | \
#grep -v -e "^$sfs_dir"/'z.*' 
#
#
sfs_dir=/mnt/sdb1/TazPup/64/a1
#This is the directory where the sfs file will be merged

#Used to define the $target_dir variable. See below. 
root=/initrd/mnt/dev_save/slacko64save

#This is the folder where the files will be merged into
target_dir=$root/tazpup-builder/slitaz-rootfs-new

#This is the location of the save folder. If you are using a save file then first mount it (I believe you
# can click on it and select view to do this) and the path of the mounted file should be located in 
#the /mnt folder. 
savedir=$sfs_dir/tazpupsave
 
#This is the function that merges the save file 
merge_sfs_and_save_fils $sfs_dir $target_dir
#This is the function that applies the whit out files and then removes the whiteout files. 
apply_white_out_files $savedir $target_dir

Note that if you have a sandbox (e.g. sandbox.sh in fatdog64), I recommend running the script in a sandbox since it contains "rm -rf" statements. I ran the script without a sandbox and had no issues but be warned. Also note that I haven't tested the merged file yet.

[s243a]

I posted this message but It doesn't seem to be sending:

Hi s243a,

I am very interested in the topic of your new thread and the image below shows my three relevant files in /root/Lab as:

If you downloaded it from pastbin first convert the line endings to unix style using dos2unix. Alternatively, you can copy the raw text from pastbin into geany and the line endings should be correct. Otherwise, geany has tools to convert the line endings.

(1) copy of sfs file (with a and z drv already merged)

If you want the kernal modules merged that's cool. Otherwise you can remove them by deleting the /lib/modules folder once extracted.

(2) your downloaded code (executable and named s243a)

(3) copy of my current save file (200 M).

I would like to try your code and then report the result to you by PM. If that is OK with you, any extra detailed instruction you can add would help me.

Cheers
Mike

I added some use notes at:
http://murga-linux.com/puppy/viewtopic.php?t=115640

If you want I can add some dialog menus so that you don't have to edit the code but I'm not sure the time-frame (maybe within a week).

[rufwoof]

Why not overlay (or aufs) the two (or more) folders into a top view ... and mksquashfs that top view?

[s243a]

Why not overlay (or aufs) the two (or more) folders into a top view ... and mksquashfs that top view?

I did note above that the approach, which you suggest, might be more robust. However, I'm not sure whether or not if would use any more ram.

One reason I didn't take that approach is I don't understand it yet. Another, reason is that I was thinking of adapting the scipt to move files rather than copy them. The idea is to strip out each package from an OS in order to create the underlying skeleton as part of a build system. I might try this for lucid puppy as an example since it is a pretty lightweight puppy and supports multi-session cd/dvd saves better than more recent versions.

That all said it is on my list to learn more about the approach that you suggest. However, taking an alternative approach let me learn a bit about white-out files

[rufwoof]

Quite simple. for example if you have a sfs 'my.sfs' and a changes folder 'changes' then to mount the two with a top layer of 'top' ...

Code: Select all

mkdir changes sfs top
mount -r -t squashfs ./my.sfs sfs # mount my.sfs to sfs folder mountpoint
mount -t aufs -o br=changes:sfs none top # aufs mount 

Something like mksquashfs top top.sfs ... then creates a sfs of the merged top folder. You could then close that down
umount aufs
umount sfs
... and copy the newly created top.sfs to my.sfs, clear out the changes folder ... and you've in effect remastered all of the prior changes into a new version of my.sfs

[foxpup]

@s243a
@rufwoof

Thank you both for your explications! Very instructive.
I will have to read and try this out.

Btw, s243a, I like the idea of stripping to the 'skeleton'/core and then rebuild with just what you need/want.
I wonder if it could be done already with pkg from sc0ttman?

[wiak]

I wrote a script that merges sfs files and a save file into a single directory then updates the directory based on the whiteout files in the save folder.

I like this kind of work, which is based on understanding of what happens in a system using some kind of overlay (in terms of whiteout files etc), but doesn't need using overlay in order to do the merging. Reminds me of tinycorelinux showing how to mount and merge squashed filesystems using symlinks rather than aufs or overlayfs. Of course, using overlay to get merged version of sfs and savefile also works - but nice to illustrate alternative methodologies.

wiak

[s243a]

Just as a note on this, it mostly works but I had some weird errors the last time I tried it that may or may not be related to my code. I do know that I need to take more care when creating directories to get the permissions correct. One way to do this is to use the cpio command.

This concept is temporarily on hold because of other things that I"m working.

I apologize for not responding to some PMs but my sent box is full and I haven't gotten around to moving the old messages to a suitable alternative location.