Fatdog64-802/801/800 Final [21 May 2019]

[stefan21]

Followed your advice and created a new savefile with fd801.

Looks to me as this did the trick.

Thank you for your help.

regards.
stefan

[rufwoof]

rufwoof, are you aware of Fatdog's multisession capability? It stores save sessions are SFS files on harddisk (and DVD too). You can then check the integrity of these save sessions.

Blanked a 8GB usb stick by using gparted to create a new partition table. Created a new ext3 format partition in that. Set the boot flag on.

Installed grub4dos to that, just ticking the legacy box only.

Clicked to open the fatdog iso I have stored on sda1 (HDD) and copied across the initrd, vmlinuz

Edited menu.lst on the usb (that at that time showed as sdb1) to contain

Code: Select all

title title Fatdog USB Multi-session 
  root (hd0,0)
  kernel /vmlinuz rootfstype=ramfs savefile=direct:multi:sda1::
  initrd /initrd

title Fatdog64 NO SAVEFILE
  root (hd0,0)
  kernel /vmlinuz rootfstype=ramfs savefile=none
  initrd /initrd

and booted the No Save choice. My BIOS is set to a boot order of CD/USB/HDD so with no CD inserted the usb booted straight away (previously I'd disabled uefi boot and set it to legacy/bios). When the usb is the first to boot, it becomes sda1 on my system with the internal HDD becoming sdb1

Created a save, opting for multi-session, saving to the usb, saved/rebooted ... now that's all working fine. Just like the USB was a multisession save booted CD/DVD. Once booted the usb stick can be pulled out and only needs to be reinserted prior to clicking the desktop Save Session icon. I've set EventManager to a save interval of 0 so that it only ever saves when that icon is clicked. The huge initrd (fd64.sfs internal to that), boots pretty quickly, and I'm using a usb2, so for a usb3 stick it could potentially be 10x quicker (have a usb3 slot, but no spare usb3 stick to hand to test that).

A number of saves later, with a number of multisession save files having been created I wondered if it was the same as a DVD where you can reduce those down in number. So booted, wiped all of the multi-session saves on the usb, and then clicked the Save Session icon and indeed it did save OK, back down to just the two (base and save multisession files) save files, which when booted had all prior changes across all multi-session saves saved in those two files.

The ability to boot from usb, with all system and save files on usb, and remove that usb once booted only re-inserting it again to boot or perform a save, makes that a very secure choice. Leaves the HDD available just for data, and if you store your data under a single folder at the root level, that you encrypt using fatdogs rox right menu click encryption option ... then so much the better.

Mostly I tend to just boot, use and shutdown without saving, so the next session is exactly the same i.e. not save any data within fatdog, just on a separate partition/disk. Having the base system and save area all in ram, with the usb detached ... is very comforting. And in Fatdog 8 (haven't upgraded to 8.1 yet myself), multi-session usb booting all seems to work/run well.

[Flash]

That's excellent news. I'll give it a try. Thanks.

[tallboy]

Exciting! I'll play!

[rufwoof]

Multi-session USB thread : http://murga-linux.com/puppy/viewtopic.php?t=116147

[rufwoof]

A note of caution if you do use Fatdog multi-session usb. There's been a long term Puppy issue with layering multiple sfs's, for as long as I can remember, where in some cases it messes up. Typically with dot folders. For instance I'm linking /root folders of .ssh, ..ssh, .calcurse and ..calcurse (I'm storing my ssh keys and calendar files inside a encrypted folder on my main HDD) to outside of Fatdog space (sym linking from usb based /root to hdd based versions of those folders).

After saves, reboots etc. those symlinks however restore, themselves, such that they're no longer sym linked out to the hdd based versions, but become actual physical folders under /root again.

That's not a Fatdog specific issue, although Fatdog is affected as I believe that Fatdog uses elements of woof and it's a Puppy/Woof issue. i.e. likely due to how woof's layering (cleaning/arranging .wh files) has a bug when it comes to dot folder names.

[rufwoof]

Configuration : usb grub4dos (legacy) frugal boot using multi-session saves back to usb

=====
Bug 1

Menu, System, Fatdog Help - Install to a flash drive (alternative method) [file uefi-flashdrive2.md]

The steps
Step 1
Before mounting: Make sure your flash drive has a FAT32 partition, and it is large enough (300M or more) to hold copies of the Fatdog files. Some UEFI firmware also accepts FAT16 but the official standard requires FAT32. Most flash drives today ship with FAT32 factory-formatted so you usually don't even need to do this, but it is always good to check and confirm.

... needs revising to 500MB.

For info (to report that gpt/uefi secure boot and multi-session usb works fine) ...

I followed the steps to create a GPT UEFI usb boot and used a 512MB sized fat partition, and all of the files copied to that just barely fitted.

After creating a second partition (ext3) to use as a multi-session save area on the USB it all booted/worked fine. UEFI is not a boot method that I am familiar with and I somewhat stumbled through getting that running (somewhat randomly selected the keys and choice of fatdog keys to get it to pass security as that seemed the most obvious, at least to me).

=====
Bug 2

At the very end of boot-options-linux.md help file section, the first link is a dead link

=====
Bug 3

pkeys=uk kernel boot parameter doesn't set the console (ctrl-alt-F3 for instance) keyboard to uk layout (remains at the default us layout). Creating a /etc/keymap (that otherwise doesn't exist) containing 'uk' fixes the console keyboard layout to be UK layout without having to specify a pkeys=uk boot parameter.

[fatdog]

Thanks rufwoof. Bug 1,2,3 will be fixed in next release.

PS: we have uploaded kernel 4.19.44 with zombieload mitigation. You can update your running kernel using fatdog-updater from Control Panel. Otherwise we will release 802 soon with this kernel version included, soon.

PS2: the mitigations slow down the CPU. If you don't like the mitigations (which are enabled by default), you can pass kernel parameter "mitigations=off" which disable __ALL__ mitigations including spectre etc and will restore the speed, but, you know, you will be __vulnerable__. You can also disable various parts of the mitigations which you think is irrelevant - https://www.kernel.org/doc/Documentatio ... meters.txt has all the documentations.

[rufwoof]

I'd appreciate consideration by the fatdog team for a change request.

Extension of save2session (multi-session saves) to include validating that the device is ready/available (and perhaps additional tests to ensure that there is sufficient space on that device for the save).

Currently for my own purposes I've hard coded near the top of /usr/sbin/save2session

Code: Select all

[ ! -d /mnt/sda1 ] && mkdir -p /mnt/sda1
mount /dev/sda1 /mnt/sda1
if [ ! -f /mnt/sda1/menu.lst ]; then
	umount /mnt/sda1 >/dev/null 2>&1
	xmessage "usb unavailable exiting" &
	exit
fi
umount /mnt/sda1

... which in my case ensures that the usb I use for booting/multi-saving is available. Otherwise the code as-is, if the usb isn't connected, sees no prior save and starts forming a totally new (first save) set. That code is obviously hard coded specific to my particular installation/setup, where the usb booted shows as device sda and includes my grub4dos bootloader menu.lst file.

Fatdogs usb booted multi-session save choice caters for that usb being unplugged after booting, which I do as a standard practice (I even have a notification message being run as part of ~/Startup to remind me to unplug the usb), and where save files are also stored on that usb - such that the usb needs to be re-attached prior to running a "Save Session". Separately to that I store my ssh keys in one encrypted folder (rox right click option) on HDD, and have another encrypted folder for my calcurse (diary), and yet another encrypted folder for DATA (everything else). Generally I boot/use and shutdown without saving, excepting if configuration changes are needed when I boot/re-configure/save/reboot. So I start with the exact same clean session at each reboot, including a 'brand new' browser session (which does mean having to store data, and browser bookmarks separately (I maintain a local html file of all my bookmarks)).

Running that way means there is no need for the Spectre etc. measures to be active, I can run with mitigations=off kernel boot parameter. In practice real world actual exploits of Spectre etc. are rare, but have high public awareness of existence. Greater real world risks tending to be less widely publicised. For instance Google's browsers

Code: Select all

Google on Tuesday updated Chrome to version 74, an update that patched 39 security vulnerabilities and added support for websites that want to honor users' requests to limit stomach-churning motion effects.

The search company paid out $26,837 in bug bounties to 17 researchers who reported some of the vulnerabilities quashed in Chrome 74. Five of the flaws were ranked "High," the second-most-serious category in Google's four-step rating system.

Such holes are far more likely to be actually exploited, and where exploits look to install a crackers control persistently (across reboots). Booting 'clean' each time with the boot/system code loaded from removable media (usb) mitigates such threats. Leaving just online accounts userid/passwords at risk. Generally to mitigate main concern online account userid/passwords, such as online banking, if you boot clean and go nowhere else before or after other than to your banks website and reboot again afterwards then that userid/password will be protected. For other web sites, such as murga-linux.com/puppy or wherever, you just have to accept as everyone does, that those userid/passwords are at risk of being accessed by others at any time (more often userid/passwords aren't obtained via cracking individual systems, but instead via cracking the actual web site and securing thousands or in some cases millions of userids/passwords).

Of all the Puppy's/derivatives, Fatdog is the king of multi-session (that enables booting and saving from/to removable, that is physically disconnected (isolated) once booted), but as per the above change request could be improved further still

[fatdog]

Fatdog64 802 is released. Please see first post.

[chiefengineer]

Let me share two things:

1)I was so happy with the performance of my Lenovo Ideapad with this release I bought second. I discovered hibernation couple with the fact that bluetooth is on the same chip disable the clicker (not the mouse) so I worked around those.

2)My surprise was the second machine I bought was Intel (not AMD) and the battery goes undetected (this is also true in Kali which I run). It could be a defect so I am testing this blindly...but it appears to be a known issue even with Windows.

I am running a traditional legacy bios boot which I dd'd to a 3.1 usb key
and created a third save partition. It flies.

I have also disable/re-enabled some cryptic thermal controls in the bios to no avail. Battery seems to work/recharge so far, including an unplug beep---just no detection...it appear the power button blinks when it is running low.

Any ideas?

[don570]

I installed fatdog64 802 ---> frugal install on windows XP ntfs disk
I found UUID with 'blkid' command so I could use grub4 menu.lst file
I saved to a folder rather than a file.

First report:
Blender works. I started SAMBA server with the control panel.

Then I went to /root/share and clicked on network icon.
I was able to connect with another linux computer using YASSM
__________________________________________________

[rufwoof]

Opened initrd and rebuilt both the kernel modules sfs and fd64 sfs within that using no compression (mksquashfs -noI -noX -noD -noF parameters).

Reformed the initrd (cpio) which was now around 1.4GB in size.

Applied xz -e9 --check=crc32 initrd ... i.e. extreme compression and the resulting initrd.xz was 333MB (compared to 412MB for the default fatdog 8.2 initrd). So alongside a 6MB, a frugal booted Fatdog 8.2 can be as little as 340MB.

However, it looks to me that no matter what compression is applied within fd64.sfs it still takes up the same amount of ram when actually loaded/booted. It's as though the sfs is read, extracted and then stored into ram using whatever compression the kernel is set to use for storing the contents in ram (inodes, dentry compression). Booting that extreme compressed initrd.xz and it initially copied into ram quicker (being smaller), but then stalls as its loaded into ram (being high compression that takes quite a while on my setup, around 75 seconds). So no real overall speed advantage, and uses a similar amount of ram overall. No real benefit excepting perhaps for the likes of PXE booting when the initial reading of the initrd might be very slow, and slow enough to offset that 75 seconds of reading into ram delay.

Delving deeper, and using the benchmarks stated in https://catchchallenger.first-world.inf ... LZ4_vs_LZO, for a compress once, decompress many situation such as frugal booting, the 'best' choice (when we care little about how long it takes to compress, and are seeking the fastest decompression), then from that link we can measure the function of compression ratio x decompression speed. Based on those figures, lz4 stands out considerably, followed by lzop (lzo) level 9 compression, followed by gzip. Compression ratio x decompression speed values of (lower = better)

70.5 lzma
29.52 lzop
64.2 gzip
10.68 lz4
78.96 xz

We also however need to consider the time to initially read/load that (bios level reading/copying tends to be relatively slow), where smaller = faster. On my kit for instance (usb frugal boot) bios reads the initrd at around a 18.5MB/sec rate. So a initrd with lz4 Xhc compressed kernel modules and fd64.sfs, that weighs in a 611MB for instance takes around 33 seconds to read/load. Whereas a 411MB initrd (as per the default fatdog 8.2) takes 22 seconds (but then takes around 8 times longer to read into ram than when using lz4). On other kit (reading from DVD), the BIOS load time might be considerably longer.

No overall conclusion, other than to me it looks like the better choices are to either use xz as per how Fatdog currently does, or lz4. For me, when using lz4 the desktop/system does appear to be snappier after initially being booted.

[williams2]

it still takes up the same amount of ram when actually loaded/booted. It's as though the sfs is read, extracted and then stored into ram

An sfs is like a zip or a tar.gz file. There are files compressed or not in the sfs. The sfs file is a certain size.
For example, my bionicpup64 sfs file is 331 MB. (right-clicking the sfs file in /initrd/mnt/tmpfs and clicking count)
The files inside the sfs file are 1355 M (right clicking /initrd/pup_ro2 and clicking count)
This is the size the files are when and if they are accessed / used. If you execute or read a file it will be decompressed automatically, ready to use. Files that are not executed or read stay compressed in the sfs file.

If the sfs file is copied to the ram drive, it will take up space in the ram drive. In my case, 331 MB. If it is on a hard drive or flash drive and not copied to the ram drive, it will not take up space in the ram drive.

So you can boot Puppy without copying the sfs to the ram drive, and you would save 331 MB. The sfs still is taking space in the hard drive or flash drive. It would be slower to access than if it were in the ram drive. The contents of the sfs file are not extracted unless you execute or read a file that is in the sfs. If you execute a script in the sfs, it will automatically be extracted and decompressed and copied from the sfs to a buffer in ram, and copied (decompressed) to a cache in ram and (decompressed) to somewhere in ram where it can be executed. Only files in the sfs that are actually executed and/or read will be copied from the sfs to ram.

That is, mounting an sfs does not copy anything to ram. There is some overhead, like a device buffer. Only files that are actually accessed will be copied to ram.

If you use rox or du to examine the size of the files in the sfs, it will show you the decompressed sizes. It will not decompress the files unless they are actually used (accessed).

[rufwoof]

Thanks williams2.

In my case, fd64.sfs inside of initrd, whether that fd64.sfs is not compressed, or highly compressed, it still results in the same amount of htop mem once the system has booted. That's with the ram boot option i.e. copied to ram. Which led me to conclude that the underline process involves reading the fd64.sfs from within init ram space, and installing it into user ram space and where its not a direct copy but a 'loading' (transformation) in how its stored in user ram space.

[williams2]

In my case, fd64.sfs inside of initrd

Shouldn't matter. Booting FatDog721 (I haven't installed 8 yet):

Code: Select all

# losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop1         0      0         0  0 /fd64.sfs (deleted)
/dev/loop2         0      0         0  0 /aufs/devsave/fd64save.ext4
/dev/loop0         0      0         1  1 /kernel-modules.sfs (deleted)
#

The layered aufs file system seems to be:

/kernel-modules.sfs (at the bottom)
/fd64.sfs
/aufs/devsave/fd64save.ext4 (at the top)

rox shows /aufs/pup_ro is 299 M (right click, properties)
rox shows the contents of pup_ro if they were all uncompressed is 1251 M (right click, count)

In this case, the sfs files in the initrd were mounted and then seem to have been deleted. Because the sfs files are mounted and therefore still in use, the name is removed from the directory and so is invisible, but the inode is still in the directory and the sfs files are still taking up space.

[rufwoof]

I'm booting multi-session, saving back to the same usb booted from (and with usb removed after bootup), and (Fatdog 8.2) losetup shows

Code: Select all

# losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                     DIO LOG-SEC
/dev/loop1         0      0         0  0 /fd64.sfs (deleted)             0     512
/dev/loop0         0      0         1  1 /kernel-modules.sfs (deleted)   0     512
# 

/aufs/pup_ro properties shows 339MB with right click count : Total: 1442 M (29670 files, 3749 directories)

In this case, the sfs files in the initrd were mounted and then seem to have been deleted. Because the sfs files are mounted and therefore still in use, the name is removed from the directory and so is invisible, but the inode is still in the directory and the sfs files are still taking up space.

Seemingly deleted to new processes, but still accessible to any child processes i.e. not released and still accessible until any process or its children close it - but deleted to any new process that tried to open it.

Confused as to why a non compressed fd64.sfs inside initrd that is over a GB in size, results in the same htop mem being shown as when a compressed fd64.sfs inside initrd.

[williams2]

why a non compressed fd64.sfs inside initrd that is over a GB in size, results in the same htop mem being shown as when a compressed fd64.sfs

du of files in the sfs file system will always show the uncompressed sizes. So it would not matter what compression is used, it will always show the same uncompressed sizes.

I don't know about htop. I would think it should show more free ram if the sfs files are compressed. I don't know why it wouldn't. I think "free" gives me a better idea of what is going on compared to htop.

Code: Select all

# free -wh
              total        used        free      shared     buffers       cache   available
Mem:           3.5G        512M        1.6G        755M        181M        1.2G        1.8G
Swap:            0B          0B          0B
# 

htop says I'm using 1.24 G

[rufwoof]

I have similar figures to yours for 'free'

htop shows a compound memory value, sum of used + buffers + cached, colour coded green, blue, orange respectively.


(clickable thumbnail)

Side note : see how in that screenshot I've moved the multi-sessions Save Session desktop icon to be next to the main drives icons (I used event manager to set the left offset to be deeper to leave space for the Save Session icon (bottom 34, left 54 in my case)). Quite a nice location for it IMO.

[williams2]

Maybe FatDog is making the ram drive (the rw file system) a fixed percentage of the total ram. In which case it would not matter what size the sfs files are.

Just a thought.