s243a's fork of woof-next
s243a's fork of woof-next
My primary differences in philosophy include:
1. allow fall-backs for build install methods
2. modularize the roots-skeleton
3. better legacy support (my claim on point #3 seems to have offended one of the woof-CE developers).
I would like to go into implementation details about these differences but I would like to more focus on what needs to be done going forward.
On boot I get the errr:
Code: Select all
/init: line 979: which : not found
WARNING: 'e2fsck' command not found
The line number is completely wrong. I think the following is the offending code:
Code: Select all
which "${fsck_app}" || {
echo "WARNING: '${fsck_app}' not found"
FSCKDPARTS="${FSCKDPARTS}${1}|"
return
}
The which utility is part of busybox. So the issue here is likely related to the creating of the symlinks but this will only work if busybox is compiled with said applet. Woof-next use to get the architecture specific files from the following two locations:
Code: Select all
INITRD_ARCH=${INITRD_ARCH:-$WOOFCE/woof-arch/x86/target/boot/initrd-tree0}
Code: Select all
cp -a $INITRD_ARCH/* $INITRD_CODE/* $initrdtmp
Here's an example of what architecture specific files are included in woof-next
/s243a/woof-CE/tree/woof-next/woof-arch/x86/target/boot/initrd-tree0/bin
Anyway, while this error appears minor sometimes fixing what appears to be minor error in the init systems can fix thing which one wouldn't expect to be related.
So at this point I'm wondering what changes related from the zwn branch should be included to fix this. I don't think that in the zwn branch anything comes up related to this change as all changes to the build-iso.sh file appear to be in the initial commit. However, if I follow jamesbond's changes since my fork I might be able to find some record of this on github.
I will note that this file has been significantly reduced in size when removed to the zwn branch. The variable INITRD_ARCH seems to no longer exist and I think the architecture specific components might be now generated dynamically rather than prebuilt:
Code: Select all
./build.sh -prebuilt -auto -arch ${WOOF_TARGETARCH:-default} ${INITRD_LANG} ${INITRD_KM}
And if my hunch is right, I don't know how much work will be required to incorporate this new dynamic approach into my fork but it looks like a good idea.
For consistancy I may copy this functionality but It kinds of goes against my mental roadmap. My version is to create a generic builder that can work with (or emluate) the functionality of many package managers rather than having seperate builders for each type of distribution (i.e. slackware and debian).
Anway, back to the issue at hand. I noticed that in package that if a package is already installed than in will be automatically deleted from the /root/pkg. This might cause some confusion if one wants to first download a package and then re-isntall it. There is an enviornmental varialbe that one can set as a workaround but this is counter-instuative.
The offending code is:
Code: Select all
# if AUTOCLEAN=true silently delete all pkgs in WORKDIR that are already installed
if [ "$AUTOCLEAN" = 'yes' -a "`HIDE_BUILTINS=true list_installed_pkgs`" != '' ]; then
clean_pkgs &>/dev/null
fi
/sc0ttj/Pkg/.../usr/sbin/pkg#L7603
I will make a fix for this shortly.
**Note that I'm discussing my fork of pkg so any comments may or may not apply to sc0ttman's version.
Here is my fix (see commit e518085 )s243a wrote:The next issue that I wanted to look at is related to my fork of sc0tmann's package. I'ave included sc0tmann's package as part of my woof-next fork so that I could use it as part of the build system. In contrast; the woof-CE provided an alternate way to add pet packages in the ZWN branch:
For consistancy I may copy this functionality but It kinds of goes against my mental roadmap. My version is to create a generic builder that can work with (or emluate) the functionality of many package managers rather than having seperate builders for each type of distribution (i.e. slackware and debian).
Anway, back to the issue at hand. I noticed that in package that if a package is already installed than in will be automatically deleted from the /root/pkg. This might cause some confusion if one wants to first download a package and then re-isntall it. There is an enviornmental varialbe that one can set as a workaround but this is counter-instuative.
The offending code is:/s243a/woof-CE/.../woof-code/rootfs-packages/PKG/usr/sbin/pkg#L7720Code: Select all
# if AUTOCLEAN=true silently delete all pkgs in WORKDIR that are already installed if [ "$AUTOCLEAN" = 'yes' -a "`HIDE_BUILTINS=true list_installed_pkgs`" != '' ]; then clean_pkgs &>/dev/null fi
/sc0ttj/Pkg/.../usr/sbin/pkg#L7603
I will make a fix for this shortly.
**Note that I'm discussing my fork of pkg so any comments may or may not apply to sc0ttman's version.
I add an override to the autoclean when the --download option is used:
Code: Select all
--download|-d|download|d)
AUTOCLEAN_override='no'
[ ! "$2" ] && pkg_download || for x in $opt; do [ "$x" -a "$x" != "-" ] && pkg_download "$x"; done;;
and then it is used in the following place:
Code: Select all
[ ! -z "$AUTOCLEAN_override" ] && AUTOCLEAN="$AUTOCLEAN_override"
# if AUTOCLEAN=true silently delete all pkgs in WORKDIR that are already installed
if [ "$AUTOCLEAN" = 'yes' -a "`HIDE_BUILTINS=true list_installed_pkgs`" != '' ];then
clean_pkgs &>/dev/null
fi
To recap, what this does is prevents previously installed packages from being deleted when downloaded via the --download option.
Curently, I have the following line in /etc/profile:
Code: Select all
export PERL5LIB="/usr/lib/urxvt:/etc/perl:/usr/local/lib/i386-linux-gnu/perl/5.24.1:/usr/local/share/perl/5.24.1:/usr/lib/i386-linux-gnu/perl5/5.24:/usr/share/perl5:/usr/lib/i386-linux-gnu/perl/5.24:/usr/share/perl/5.24:/usr/local/lib/site_perl:/usr/lib/i386-linux-gnu/perl-base"
Is "${PWD}" every wrong?
Code: Select all
CURDIR="${PWD}" # get current dir, before cd to WORKDIR
but is it? Probably but is PWD inherited by subprocesses or do we need to export it? What if we execute a command like follows:
Code: Select all
urxvt -e pkg -i pkgname
Here is the script which is called when a .pet file is clicked..
https://pastebin.com/EPgPujar
but
Code: Select all
pkg -i pkgname
WHen I finally figured out how to debug it, I noticed that I got
Code: Select all
CURDIR=/root
Code: Select all
tar $taropts "${CURDIR}/${PKGNAME}.tar.${TAREXT}" 2> $TMPDIR/$SELF-cp-errlog
This likely occured because sc0ttman borrowed some code from puppies package manager. Since I identified this assumption, I simply made the current directorary the directory where the package is located.
Code: Select all
if [ -f "$1" ];then
PKGFILE="$1"
CURDIR="$(dirname "$(realpath "$1")")"
PKGNAME=`basename "$1" .$pkg_ext`
else
The code in the above link doesn't quite match the above code (it's missing the "CURDIR=" line) because I have yet pushed this change to github.
I need a spot to document and fix issues [2] but feedback is certainly welcome, although it is quite early at this point [1]. I will also be referencing this thread from my tiny_puduan thread as a way to provide more details about changes and issues.nic007 wrote:What is this, a monologue? Are you looking for feedback from other users?
Notes
---------------------------
1. The build process isn't yet seemless enough where I want to promote my fork of woof-next for general use. The most sginificant issue here is that there are some key distro/release specific related files that I have to merge into the chroot folder before certain things are installed. For instance I need to merge DISTROSPECS
https://github.com/s243a/woof-CE/tree/woof-next/woof-distro/x86/tiny_devuan/ascii/fs_basesfs/etc
into the chroot folder prior to installing the puppy package mangager or sc0ttman's package manager (i.e. pkg). I haven't yet done this because I was focussing lately on fixing issues with tiny_devaun which was built via this woof-next fork.
2. I may eventually move the project to gitlab so that this fork can have it's own issue tracking. However, first I want to get the code working well enough and possibly offer some pattaches to the zwn branch of woof-CE.
Re: Is "${PWD}" every wrong?
This appears to be working correctly so I pushed it to github in commit db46b1a. The above pastebin link contains test code. The code on github has the debugging code commented out:s243a wrote:Here is something that looks safe/s243a/woof-CE/.../woof-code/rootfs-packages/PKG/usr/sbin/pkg#L39Code: Select all
CURDIR="${PWD}" # get current dir, before cd to WORKDIR
but is it? Probably but is PWD inherited by subprocesses or do we need to export it? What if we execute a command like follows:is PWD still inherated by the subprocess if we don't export it? These questions might sound strange but I was working on a script to install a package (via pkg) by clicking on it and the last command of the script worked if I called the command directly from the command line but when the script was invoked by clicking on a .pet file the same command that worked from the command line (last command executed in my script) didn't work even though said command worked when being used directly on the command line.Code: Select all
urxvt -e pkg -i pkgname
Here is the script which is called when a .pet file is clicked..
https://pastebin.com/EPgPujar
butdid not work when it was called from a script that was invoked by clicking on a .pet file in rox.Code: Select all
pkg -i pkgname
WHen I finally figured out how to debug it, I noticed that I gotrather than the directory in which I clicked on the script. Does this sound right? My solution wasn't to export either CURDIR or PWD since parts of the code seem to assume that the "dot pet" file is in the CURDIR. For exampleCode: Select all
CURDIR=/root
/s243a/woof-CE/../woof-code/rootfs-packages/PKG/usr/sbin/pkg#L4676Code: Select all
tar $taropts "${CURDIR}/${PKGNAME}.tar.${TAREXT}" 2> $TMPDIR/$SELF-cp-errlog
This likely occured because sc0ttman borrowed some code from puppies package manager. Since I identified this assumption, I simply made the current directorary the directory where the package is located.
/s243a/woof-CE/.../woof-code/rootfs-packages/PKG/usr/sbin/pkg#L4587Code: Select all
if [ -f "$1" ];then PKGFILE="$1" CURDIR="$(dirname "$(realpath "$1")")" PKGNAME=`basename "$1" .$pkg_ext` else
The code in the above link doesn't quite match the above code (it's missing the "CURDIR=" line) because I have yet pushed this change to github.
/s243a/woof-CE/blob/woof-next/woof-code/rootfs-packages/puppycore_noarch/usr/sbin/petget (on s243a's woof next fork)
One thing that this doesn't do yet is notify one when the installation has finished.
Edit: the related change to /usr/sbin/pkg is in commit
6910569
Here's the link that you want:recobayu wrote:Hi s243a,
I just look at github. The last commit I see is at 30 May 2019.
Is there a new you upload?
Thank you.
https://github.com/s243a/woof-CE/tree/woof-next
from this link you can click on the compare button to see all the commits. The most recent commit is at the bottom of the list.
Anyway, in my fork of package there was an issue with some built-in files being re-installed. This is likely also an issue in the original version of pkg due to the fact that concurrency is used without locking the file which gives the list of installed packages. Some related posts are:
1. identifies where installed packages are filtered out from the list of dependencies.
2. identify that concurrency is used due to wait loop in the get_deps function.
3. Identify where the forking takes place in the pkg_get() function via the background call to list_deps(). Also propose some improvement to the concurancy strategy and other issues.
Anyway, the issue where built-in files are re-installed is probably fixed but I haven't tested this yet. That said the improvements that I made to the concurrency strategy has significantly sped up pkg by ensuring that that the list of installed packages is only produced once by only creating a new list of installed files if it doesn't exist:
Code: Select all
[ ! -f $TMPDIR/installed_pkgs ] && list_all_installed_pkgs
We also make sure that the list is completely created before using it. This is accomplished with the following wait loop:
Code: Select all
while [ ! -f "${TMPDIR}/installed_pkgs_finished" ];do #TODO: s243a: consider adding readlink here
echo -n '.'
sleep 0.1
done
The finished flag is simply a symlink to the list of installed packages.
Code: Select all
list_all_installed_pkgs(){
...
ln -s $TMPDIR/installed_pkgs $TMPDIR/installed_pkgs_finished
The list of installed packages is only cleaned up once all instances of list_deps() exit:
Code: Select all
list_deps_count=0 #The number of concurrent instances of list_deps()
Code: Select all
6132 list_deps(){
6133 list_deps_count=$(( list_deps_count + 1 ))
...
6284 if [ $list_deps_count -eq 1 ]; then
6285 rm "${TMPDIR}/installed_pkg"* 2>/dev/null
6286 fi
** considering moving this cleanup to the end of pkg.
*** My last comment I erroneously had the "-eq 1" and also forgot a "$" sign in the variable name of the if statement.
I also gave each instance of list_deps() a unique id so that they can use unique temporary files. This keeps me from having to worry about locking the temporary files.
Code: Select all
dep_id=0 #
Code: Select all
6132 list_deps(){
...
6152 dep_id=$((dep_id +1))
6153 local ldepID=dep_id
Code: Select all
6132 list_deps(){
...
6180 echo "$deps" | tr ' ' '\n' | tr ',' '\n' | sort -u > $TMPDIR/all_deps_${ldepID}_0
...
6187 comm -23 ${TMPDIR}/all_deps_${ldepID}_0 ${TMPDIR}/installed_pkgs | sort -u | grep -v ^$ > ${TMPDIR}/all_deps_${ldepID}_
Anyway, ${ldepID} is used a fair number of places in the code. The above changes were made in commit 8760d9b8655c5845bba889026c57a0f085566674. The changes are a bit obscured because I added a space between the "semicolon" and the "then keyword" for each instance where there was no space. I thought that this might make the syntax highlighting parse better on pastebin. It didn't.
On a final note I made some logic changes surrounding the HIDE_BUILTINS environmental variable (see post) in both the function list_all_installed_pkgs() and also the function list_deps().
Edit: When writing this post, I noticed that some changes weren't made that I had intended to made. These remaining changes I pushed in commit b5d1483424615add2ab4715171dbaf996137ee75
Code: Select all
5984 list_deps(){
...
6064 for subdep in `sort -u $deps_list_file | grep -v ^$`
6065 do
...
6234 subdeps_entry="`get_deps_entry "$subdep"`"
6085 subdeps="${subdeps_entry/+/}" # remove first + at start of line
6086 subdeps="${subdeps//,+/ }" # remove others .. DEPS will now be just 'dep1 dep2 dep3'
6087 subdeps="${subdeps// / }" # remove double spaces
6089 if [ "$subdeps" != '' ] && [ "$subdeps" != ' ' ];then
6290 # remove everything after the + (if the + is followed by alphanumeric chars), then add to tmp files
6291 echo "$subdep" >> ${TMPDIR}DEP_DONE
6293 # create the next deps list file to parse, containing the deps of this $subdep
6294 subdeps_list="`echo "${subdeps}" | tr ' ' '\n' | grep -v ^$ | sed -e 's/++/+++/g' -e 's/+[a-z0-9].*//g' | sort -u`"
6295 echo "$subdeps_list" >> $next_deps_list_file
6296 fi
6297 done
Here is some output:
Code: Select all
+ subdeps_entry=+libc6,+libgcc1,+libstdc++6
+ subdeps=libc6,+libgcc1,+libstdc++6
+ subdeps='libc6 libgcc1 libstdc++6'
+ subdeps='libc6 libgcc1 libstdc++6'
+ '[' 'libc6 libgcc1 libstdc++6' '!=' '' ']'
+ '[' 'libc6 libgcc1 libstdc++6' '!=' ' ' ']'
+ echo libvpx4
++ echo 'libc6 libgcc1 libstdc++6'
++ tr ' ' '\n'
++ grep -v '^$'
++ sed -e s/++/+++/g -e 's/+[a-z0-9].*//g'
++ sort -u
+ subdeps_list='libc6
libgcc1
libstdc++'
+ echo 'libc6
libgcc1
libstdc++'
I will think about how I want to modify this code.
My idea is to first check if the package exsists in the repo database before filtering out everything after the "+". Here is some draft code (will edit post later):s243a wrote:The above modifications didn't fix the issue of libstdc++-5.0.6 being installed as a firefox-esr dependency when I already had libstdc++6 installed on my systems.
https://gitlab.com/sc0ttj/Pkg/blob/mast ... /pkg#L6094Code: Select all
5984 list_deps(){ ... 6064 for subdep in `sort -u $deps_list_file | grep -v ^$` 6065 do ... 6234 subdeps_entry="`get_deps_entry "$subdep"`" 6085 subdeps="${subdeps_entry/+/}" # remove first + at start of line 6086 subdeps="${subdeps//,+/ }" # remove others .. DEPS will now be just 'dep1 dep2 dep3' 6087 subdeps="${subdeps// / }" # remove double spaces 6089 if [ "$subdeps" != '' ] && [ "$subdeps" != ' ' ];then 6290 # remove everything after the + (if the + is followed by alphanumeric chars), then add to tmp files 6291 echo "$subdep" >> ${TMPDIR}DEP_DONE 6293 # create the next deps list file to parse, containing the deps of this $subdep 6294 subdeps_list="`echo "${subdeps}" | tr ' ' '\n' | grep -v ^$ | sed -e 's/++/+++/g' -e 's/+[a-z0-9].*//g' | sort -u`" 6295 echo "$subdeps_list" >> $next_deps_list_file 6296 fi 6297 done
Here is some output:
The sed statment seems proglematic as it converts libstdc++6 to libstdc++. While some linux distros might only allow one version of libstdc++ debian/devaun is set up so on could have coexisting versions of liberalies with different major versions.Code: Select all
+ subdeps_entry=+libc6,+libgcc1,+libstdc++6 + subdeps=libc6,+libgcc1,+libstdc++6 + subdeps='libc6 libgcc1 libstdc++6' + subdeps='libc6 libgcc1 libstdc++6' + '[' 'libc6 libgcc1 libstdc++6' '!=' '' ']' + '[' 'libc6 libgcc1 libstdc++6' '!=' ' ' ']' + echo libvpx4 ++ echo 'libc6 libgcc1 libstdc++6' ++ tr ' ' '\n' ++ grep -v '^$' ++ sed -e s/++/+++/g -e 's/+[a-z0-9].*//g' ++ sort -u + subdeps_list='libc6 libgcc1 libstdc++' + echo 'libc6 libgcc1 libstdc++'
I will think about how I want to modify this code.
Code: Select all
subdeps_entry="`get_deps_entry "$subdep"`"
subdeps="${subdeps_entry/+/}" # remove first + at start of line
subdeps="${subdeps//,+/ }" # remove others .. DEPS will now be just 'dep1 dep2 dep3'
subdeps="${subdeps// / }" # remove double spaces
if [ "$subdeps" != '' ] && [ "$subdeps" != ' ' ]; then
# remove everything after the + (if the + is followed by alphanumeric chars), then add to tmp files
echo "$subdep" >> ${TMPDIR}/DEP_DONE
subdep_arry=()
while read subdep2; do
if [ ! -z "$(grep -m1 "|$subdep2|" "${ALL_REPO_DB_PATHS[@]}")" ]; then
subdep_arry+=("$subdep2")
else
subdep_arry+=$(echo subdep2 | sed -e 's/++/+++/g' -e 's/+[a-z0-9].*//g')
fi
done < <( `echo "${subdeps}" | tr ' ' '\n' | grep -v ^$` )
echo "$subdeps_list" >> $next_deps_list_file
fi
EDIT: And I am totally opposed to using Bash (instead of Ash), so I'd only accept PRs that use Ash, with no arrays.. You've just done some good speed improvements with dep resolution I've been meaning to finally figure out (nice work!), but then you make it slower again with Bash...
Also, from a previous post:
Code: Select all
# sort and clean the search results
LANG=C cat $TMPDIR/pkglist | sort --field-separator='-' -k1,1df -k2gr -k3gr -k4gr | uniq > $TMPDIR/pkglist1
From busybox sort -h:
Code: Select all
-t CHAR Field separator