alternative puppy build system

[s243a]

https://www.dropbox.com/s/e4hgfctfpbd25 ... 6.iso?dl=0
*Not for use
**tested on qemu
...
- I installed mistfire's package manager but it isn't working write. Use the following command to start ig:

Code: Select all

ppm

- as before sc0ttmann's package manager is installed. It also comes with a gui but there are some issues with it. Perhaps due to my configuration.
- wbar is installed but not configured (it's a side panel)

I did not install petget before installing either mistfire's or sc0ttmann's package manager. This could be the cause of some issues. "X11 aps" was installed because the puppy startups scripts called xload. However, this isn't needed and we don't use the other X11 aps. Full Perl isn't required but the only reason I installed it is that I was troubleshooting the perl in urxvt. Future isos will likely not have full-perl.

So I noticed a few things. When I did the update repo on mistfires package manager, I get the following errors.

Code: Select all

cat: Packages-devuan--main: No such file or directory
cat: Packages-devuan--main: No such file or directory
cat: Packages-devuan--contrib: No such file or directory
cat: Packages-devuan--contrib: No such file or directory
cat: Packages-devuan--non-free: No such file or directory
cat: Packages-devuan--non-free: No such file or directory

The issue is the double dash at the end. This isn't how I named these files. I noticed that after installation these files they got moved to:

Code: Select all

/var/packages/repo 
#aka /root/.packages/repo

I'm not sure if this will break sc0tman's package so I might have to add a few symlinks.

Also, changed. It now looks like:

Code: Select all

#Automatically generated by sync-repo
PKG_DOCS_DISTRO_COMPAT='z|http://packages.devuan.org/merged/dists//main/binary-/Packages.xz|Packages-devuan--main z|http://packages.devuan.org/merged/dists//contrib/binary-/Packages.xz|Packages-devuan--contrib z|http://packages.devuan.org/merged/dists//non-free/binary-/Packages.xz|Packages-devuan--non-free'
REPOS_DISTRO_COMPAT='z|http://packages.devuan.org/merged|Packages-devuan--*'

The conversion is wrong. Notice that there are some double slashes in the url but besides the double slashes is might be okay. What seems to have been missing in the coversion process is ${DISTRO_COMPAT_VERSION}
See the orginal before conversion:
woof-next/woof-code/rootfs-packages/puppy_ppm_configs_devaun_ascii/var/packages/DISTRO_COMPAT_REPOS

The missing value is set in /etc/DISTRO_SPECS

Code: Select all

DISTRO_COMPAT_VERSION='ascii'

woof-distro/x86/tiny_devuan/ascii/fs_basesfs/etc/DISTRO_SPECS#L10

I also see that ${DBIN_ARCH} wasn't used in the conversion process...so this is another part of the url I need to fix. I also wonder if scottmann's PKG depends on this file and what issues these changes might cause.

Edit:1 the double dash in the file name is caused by a missing ${DISTRO_COMPAT_VERSION}

Edit: 2 I figured out that /var/packages/DISTRO_COMPAT_REPOS, is getting overwritten with the info in /var/packages/database/COMPAT_DB_REPO via the function /usr/bin/sync-repo.

So anyway, I'll now edit COMPAT_DB_REPO and see if mistfires package manager works for me.

[s243a]

I seem to be missing the following:

Code: Select all

/usr/local/petget/configure.sh: line 209: /var/local/petget/si_category: No such file or directory
/usr/local/petget/configure.sh: line 214: /var/local/petget/bb_category: No such file or directory
/usr/local/petget/configure.sh: line 218: /var/local/petget/ui_choice: No such file or directory
/usr/local/petget/configure.sh: line 226: /var/local/petget/nt_category: No such file or directory
/usr/local/petget/configure.sh: line 232: /var/local/petget/rd_category: No such file or directory
/usr/local/petget/configure.sh: line 237: /var/local/petget/nd_category: No such file or directory

although I don't think they are that important...but I'll look into it.

I'm also missing:

Code: Select all

/usr/local/bin/ppm: line 788: /usr/sbin/indexgen.sh: No such file or directory

which I can grap from x-slacko slim if it isn't in woof-CE (need to check)

and I get lots of errors that look like the following:

Code: Select all

tr: warning: an unescaped backslash at end of string is not portable

and I think this is concerning.

Anyway, I'll try to add the missing stuff tomorrow and see if I can get mistfires package manager to work.

I should also note that I'm missing some icons:

Code: Select all

/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:17: Unable to locate image file in pixmap_path: "x48.png"
/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:18: Unable to locate image file in pixmap_path: "pc48.png"
/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:19: Unable to locate image file in pixmap_path: "configuration48.png"
/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:20: Unable to locate image file in pixmap_path: "utility48.png"
/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:22: Unable to locate image file in pixmap_path: "paint48.png"
/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:23: Unable to locate image file in pixmap_path: "word48.png"
/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:24: Unable to locate image file in pixmap_path: "spread48.png"
/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:25: Unable to locate image file in pixmap_path: "date48.png"
/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:27: Unable to locate image file in pixmap_path: "www48.png"
/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:28: Unable to locate image file in pixmap_path: "multimedia48.png"
/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:29: Unable to locate image file in pixmap_path: "games48.png"
/tmp/petget-proc/puppy_package_manager/gtkrc_ppm:30: Unable to locate image file in pixmap_path: "pet48.png"

and I will fix this but it doesn't concern me much.

[greengeek]

Hi watchdog,

sorry - but i have not been able to follow the entirety of this thread - but I would like to make some comments about a build system for puppy:

There are many puppys - some of them are built according to Woof, or WoofCE, or just personalised puppys. However, for me the real question is how a user can take any puppy (ie the base puppy.sfs) and build their own best puppy from it??

I think the key is for a user to take a dev's puppy.sfs and boot it in a minimal way - then discard all the files they don't need for their specific hardware - then add their own personalisations as required.

I probably can't explain this well, but here are my thoughts:

Boot the current hardware (using base puppy.sfs as released by dev).
- Identify which files were critical to the first boot environment (specific firmware etc)
- Throw away files (firmware etc) not used / not required during this boot. (not sure how these can be identified)
- Throw away undesired software (Abiword etc)
- Save the current (minimal) boot environment (as sfs? basedrv.sfs?? strippedpup.sfs???) to ensure next boot is successful with minimal files and minimal overall size.
(This sfs becomes the "base" puppy sfs for next boot on this person's hardware)

Adjust the current environment to match personal requirements. (adjust timezone, connect wifi, etc)
- Save the personalised environment as an sfs.
(This personalised sfs must be allowed to be "top level" in future boots) (ie: "pdrv" ??)

Add extra pets as required:
- Save these extras as a separate sfs. ("adrv"? "ydrv"??). This should sit above the base puppy.sfs and below the personal sfs (pdrv) i think.

This process allows a "pristine" lightweight boot that matches the hardware, then allows that to be expanded with required personalisations ("pdrv"), then allows specific extras (word processor, browser etc) to be added last ("adrv, ydrv" etc)

So:

Top level: pdrv personalised sfs
Mid level: adrv (or ydrv??) extra software sfs
Bottom level: minimal puppy.sfs as released by a dev, but also trimmed by some process after the first boot.

This process should be made easy and straight forward for any user. The "puppy build" process for most users should not include the complexities of the basic puppy.sfs creation - just the tailoring of that basic puppy.sfs to best suit the customer needs.

[rufwoof]

Firmware and browser consumes a lot of space. For instance I have a 5MB vmlinux (with extensive busybox taken from EasyOS 1.0) and a 12 MB initrd (with everything in that i.e. no main sfs) that boots/runs fine - but tui not gui. Just hardwired eth0 support for my sky ethernet card). With that I can net connect and/or mount sfs's etc.

OpenBSD does similar, enough for hardwired net and keyboard support, cli, from where other things (firmware etc) are pulled in as required. 9MB type size.

If both firmware and browser are loadable (portable versions of browser for instance) 'external' modules, then the core can be very lean. Potentially, at least for hard wired, the loadable modules could even be remote, download as/when required - which conceptually could be very nice i.e. a single centrally maintained/updated main desktop system. But for wireless, I guess its better to have local versions.

[mikeslr]

...

I probably can't explain this well, but here are my thoughts:

Boot the current hardware (using base puppy.sfs as released by dev).
- Identify which files were critical to the first boot environment (specific firmware etc)
- Throw away files (firmware etc) not used / not required during this boot. (not sure how these can be identified)
- Throw away undesired software (Abiword etc)
- Save the current (minimal) boot environment (as sfs? basedrv.sfs?? strippedpup.sfs???) to ensure next boot is successful with minimal files and minimal overall size.
(This sfs becomes the "base" puppy sfs for next boot on this person's hardware)...

Unfortunately, in practice what appears to be the easy way, isn't.

It's the way I built Slacko 5.7.2CE. Built into every Puppy, AFAIK, is Menu>Setup>Remove Builtins. It is very thorough; way too thorough. When initiated it offers to remove hundreds of packages. The attached screenshot only shows those, alpha-numerically through 'b'. Some which can be safely removed --abiword, for example-- are obvious. Most are not. Well, are not if your a newbie and a package name doesn't provide a clue as to whether it is just some trinket or an essential core package. Consequently, the choice is either to (a) undertake the steep curve to learn what each package actually does; or (b) take the easy way --as I did-- and only remove those packages which could safely be removed.

The downside is that most likely there remain with in the base.sfs many unnecessary packages. The upside is that for the most part they take up little space.

Among those packages unnecessary for the "base" may be some which you may want to have in an adrv or extra sfs. Before removing these, you can install and run jpeps' gnewpet, http://murga-linux.com/puppy/viewtopic. ... 673#598673. PaDS, previously mentioned, can be employed to create SFSes. I may be wrong, but AFAIK each of the Remaster applications available on the Forum provides an option to only include the firmware (and drivers) needed by the computer on which the 'remaster' is built -- that is automatically exclude 'unnecessary' firmware and drivers.

Creating a Puppy that way is a time consuming and --especially regarding removables-- confusing process.

Far simpler for a newbies are the built-scripts: start with a minimum install of essentials, add what you desire.

[wanderer]

hi all

could the base core.gz of tinycore linux
be used to identify the base packages in puppy
it boots completely and has wired internet

and then a list of puppy base packages
could be available to use in woof-ce and woof-next

the nice thing about tinycore
is that it is modular and can be used as a teaching tool
to see what is needed for each level of build

wanderer

[s243a]

hi all

could the base core.gz of tinycore linux
be used to identify the base packages in puppy
it boots completely and has wired internet

and then a list of puppy base packages
could be available to use in woof-ce and woof-next

the nice thing about tinycore
is that it is modular and can be used as a teaching tool
to see what is needed for each level of build

wanderer

The problem is that many puppy tools (e.g. PKG) need either the full version of the utilities or Xorg dependent GUIs (e.g. yad, gtkdialog, xmessage, message, etc)

Puppy could be rewritten to use dialog and busybox instead but it would be a lot of work.

[wanderer]

could the core.gz
and just the added xorg components
be incorporated into the puppy build system as a base

then any xorg stuff would work
but it would be minimal and modular
and xorg could be removed if needed

just thinking

wanderer

[greengeek]

Unfortunately, in practice what appears to be the easy way, isn't.

Do you think there is any way for a running puppy to identify which files were used during the boot?

For example - if the "date of access" of an Intel wifi firmware file showed that this file had just been accessed during boot - and all the other wifi firmwares were untouched (older access date) - wouldn't that give an indication which fware file should be kept and which others could be stripped?

Of course if there is no record of which files were accessed during boot then my hopeful idea turns to dust...


Maybe the initrd could be written in such a way that it could build a text log record of files accessed during boot???

[s243a]

Unfortunately, in practice what appears to be the easy way, isn't.

Do you think there is any way for a running puppy to identify which files were used during the boot?

For example - if the "date of access" of an Intel wifi firmware file showed that this file had just been accessed during boot - and all the other wifi firmwares were untouched (older access date) - wouldn't that give an indication which fware file should be kept and which others could be stripped?

Of course if there is no record of which files were accessed during boot then my hopeful idea turns to dust...


Maybe the initrd could be written in such a way that it could build a text log record of files accessed during boot???

Just because something is used at boot doesn't mean that it is needed.

[wiak]

For example - if the "date of access" of an Intel wifi firmware file showed that this file had just been accessed during boot - and all the other wifi firmwares were untouched (older access date) - wouldn't that give an indication which fware file should be kept and which others could be stripped?

Yes, that would be nice, but unfortunately atime doesn't generally work in that way. By default modern kernel uses relatime so simply reading a file does not change its access time. Changing access time stamps on every file read would be very inefficient. It could be forced when a filesystem is being mounted by telling mount to use option strictatime. Best explanation with examples I've come across is here:

https://linuxfreelancer.com/linux-why-i ... time-atime

So generally speaking there probably is no easy way to tell which files have been accessed (if only read): logs like those from dmesg, or syslogd generaly, can be used for a lot of relevant info but no easy way to filter out the specific info wanted that I know of.

See here for some related discussion:
http://www.murga-linux.com/puppy/viewto ... 97#1022797

One other problem is that firmware/drivers are often identified by firmware=version but version is often not used in the actual file name. For example my laptop uses iwlwifi-5000-5.ucode firmware for wifi card, but the very useful command lshw (which you may need to install) indicates it by "iwlwifi firmware=8.83.5.1". So some googling tends to be required...

wiak

[mikeslr]

The computing environment has come a long way since Puppy's early days. We no longer have to ration every byte of RAM and fret over every megabyte of storage to be used.

The applications Remove Builtins offers to remove fall into three categories: (1) Those, such as Abiword, whose name is sufficient to tell a Remove-Builtin's user exactly what will be removed; (2) Infra-structure modules which are needed by some application the user may or may not want; and (3) those whose name differs from the name by which it appears on the Menu. As to the latter, although it can take some time, the name by which such application is listed by Remove Builtins can be found by opening the /usr/share/applications desktop files and noting what executable is called by Exec=.

When an application is selected using Remove Builtins, before it is executed a warning is given as to what other applications may be broken if the application is, in fact, removed. After removal, a notice is given as to what other applications may no longer be necessary. That notice may pertain to Category "2" above. I may be completely wrong, but I think Remove-Builtins is reading the contents of the files found at /root/.packages. And I don't know how complete or accurate either the warning nor the post-removal advice may be. A pfind for included text of /root/.packages may be both time-consuming and necessary to avoid unnecessary breakage.

That said, NOT REMOVING every possibly unnecessary application will not have a significant impact on the resources (RAM and Disk-space) required by the resulting Puppy; and more importantly, its responsiveness to the User's commands.

When creating a Puppy for one's own use on the same computer, I recommend nic007's nicOS-Remaster-Suite, http://murga-linux.com/puppy/viewtopic. ... 89#1001289. As you probably know, Remov-Builtins doesn't actually remove anything: it merely "whites-out" the link to the application's files on Puppy_Version_#.sfs. One of the three modules in nicOS-Remaster merely rebuilds a Puppy_Version_#.sfs which will NOT contain the "removed" applications, but will contain any additional applications you installed/Saved into the "old" Puppy_Version_#SAVE. If I recall correctly, it will also (optionally) create a new zdrv_Puppy_Version_#.sfs that will only contain the drivers and firmware used by the creating computer. It's GUI makes it easy to choose, only requires User input at the very beginning; and that module only takes a couple of minutes on a reasonably fast computer to generate a new "replacement" Puppy_Version_#.sfs.

All of which still remains less user/newby-friendly than a build script which will generate an OS containing essential core and infrastructures (per the creator of the script) and such additional applications as the User chooses to include.

[wanderer]

hi all

i am not a guru like you guys
so i feel i can speak for the non gurus

the problem with puppy for non gurus
is not the intricacies of high level computer science

but simple organization

puppy has been designed by and for gurus
and it has not been designed to be minimal and modular

so one is always fighting a losing battle
trying to make it into something that structurally it wasn't designed to do

i don't know why the puppy community has no problem
with other distros adopting puppy ideas
but refuses to adopt anything from other distros

say what you will
tinycore has solved all these problems
they have purposely made it modular and minimal from the start

why not just adopt what they have done
and use it in puppy

you can just add what you need
when you want to run apps
that need more dependencies

puppy is really all the fantastic apps that have been made for puppy
and the puppy community
it doesn't matter how the structure of the base is organized

i started this thread
to develop a build system that non gurus
could use understand and develop

and after a lot of analysis and experimentation
i have one
it has a tinycore base
but a puppy soul

i will of course continue to put in my annoying comments
but in my opinion this problem has been solved

the puppy community does have an alternative build system
that non gurus can use instead of woof-ce

you can call it anything you want
i just call it delta to distinguish it
both from old puppy and tinycore
since over time it will become its own thing

so here is my mantra

"new puppy forever"

regards

wanderer

[sc0ttman]

I see a lot of hankering for some new magic build system...

Yes woof is horrible in 100 ways, but super cool and easy in 100 others (not that I know much about it, but i've tried/failed with a few by now)..


So.... you could try the following as a much quicker route to what you want...

You could just use woof to build a puppy with very few packages installed by default.. and have large repos available by default - for lots of "extra" packages..

To make things "modular", you could have lots of extra SFS files available to install...

So.. You could use Pkg (or whatever) to create a bunch of combined SFS packages - gimp.sfs (gimp plus deps), internet.sfs (browser, FTP, etc), multimedia.sfs (kodi, VLC, etc) ... (or whatever you want to be "modular") ...

Then put those SFS files on the ISO that you create ... (or use Pkg to create a custom repo and put them there).

EDIT: you could also use Woofy (see link in my sig) to remove progs from a Puppy ISO and remaster it... or to add pkg to it...

[sc0ttman]

The problem is that many puppy tools (e.g. PKG) need either the full version of the utilities or Xorg dependent GUIs (e.g. yad, gtkdialog, xmessage, message, etc)

Pkg does not need any X dependent stuff... Only its optional frontends do...

As for removing core-utils/findutils/etc as dependencies, and getting Pkg to use only Busybox, I would think it's not that big of a job... basically you'd need to remove some long options from sort, grep, find, and maybe a few other things.. I made an issue here, so it doesn't get lost: https://gitlab.com/sc0ttj/Pkg/issues/64

But I don't have time right now... Sorry..

But if anyone (?) wants to pull down the Pkg repo from https://gitlab.com/sc0ttj/Pkg and then create a new branch called 'busybox_only' and try to make Pkg work using only busybox applets, then that would be great - I'd defo merge it in to the main branch

And if u read this mistfire, I will merge yours soon - it looks fine, cheers.

[s243a]

tiny_puduan_ascii-PreAlpha8.iso (213M)

It's bed time now so not much testing besides booting it up in qemu, opening the puppy package manger and running JRB's portable browser installer.

Both sc0ttmann's and mistfire's package managers should be working correctly [1] on this but I'll verify that tomorrow. It likely won't boot in virtualbox, and I haven't tried it on actually hardware yet.

I will probably start a new thread for this soon.

Notes
---------------
1 - I don't know if the gui for sc0ttmann's package manager will work on this iso but his package manager (i.e. PKG) should work from the command line on this iso.

[wanderer]

hi s243a

just downloaded and booted your new iso

really cool

a real puppy

great work

so this was built with your woof-next ?

you probably should start a new thread
so we can follow your work more easily

regards

wanderer

[s243a]

hi s243a

just downloaded and booted your new iso

really cool

a real puppy

great work

thankyou

so this was built with your woof-next ?

Yes but it isn't completly turnkey. To get the themes to work I had to run ptheme in chroot. Annoyingly this resets Xorg in the host.

**There may be other post build tweaks but I can't think of any at the moment.

you probably should start a new thread
so we can follow your work more easily

regards

wanderer

I will start a new thread once I test it on actual hardware, which I'll do shortly. For now, here is an update with pmount working:

tiny_puduan_ascii-PreAlpha9.iso

To get pmount working I had to create a symlink from /usr/sbin/gtkdialog to /usr/bin/gtkdialog. I also changed the .desktop file so that it points directly to /usr/sbin/pmount. This removes the conflict with "portable mount", which is something different than puppies pmount.

[wanderer]

hi s243a

just downloaded and booted your new iso

looks and works great
i like the color scheme
and the apps

pretty amazing it was built without woof-ce

once again fantastic work
i am following your project with awe

regards

wanderer