Puppy In-House Development

[Ibidem]

I just built the entire DSLR from scratch with GCC 4.8.x, while the compiler is replaced with a wrapper:

Code: Select all

#!/bin/sh

case "$@" in
        *-O[0-9]*)
                echo "`pwd` $@" >> /tmp/args.txt
                ;;
esac

exec /usr/bin/cc "$@"

I found out that several packages do not respect CFLAGS or add -O2 thanks to auto* crap. I was able to shave a few KB here and there.

wpa_supplicant may be one of the offenders here.
Speaking of which, what version do you use? 1.0 weighs a little shy of 800k here; 2.0-2.2 run 1.4 megs (CFLAGS=-Os -fomit-frame-pointer), and I got 2.2 down to 1.15 megs with the config in the tarball I attached previously (docs/configs from building bareroot).
But you may prefer using "openssl" (libressl for you).

With the gc-sections options, lok shrinks a bit.

Also: try sstrip from github.com/BR903/ELFkickers (subdirectory sstrip). I can usually knock about 1k off beyond what strip can do. But don't expect sstrip'd dynamic libs to be usable for linking or nm.

[Iguleder]

My wpa_supplicant 2.1 binary is 1.4 MB, with LibreSSL crypto, for x86_64.

EDIT: updated to 2.2 -

(trusty)dima@localhost:~/dslr$ du -h sysroot/bin/wpa*
76K sysroot/bin/wpa_cli
140K sysroot/bin/wpa_passphrase
1.4M sysroot/bin/wpa_supplicant

[Ibidem]

My wpa_supplicant 2.1 binary is 1.4 MB, with LibreSSL crypto, for x86_64.

EDIT: updated to 2.2 -

(trusty)dima@localhost:~/dslr$ du -h sysroot/bin/wpa*
76K sysroot/bin/wpa_cli
140K sysroot/bin/wpa_passphrase
1.4M sysroot/bin/wpa_supplicant

What CFLAGS and LDFLAGS does it get? (I'm thinking that's the right vicinity, but it sounds maybe a little large...on the other hand, it has libnl-tiny and libressl built in)
Does it work with WPA?

For small size, I think the "internal" stuff may be the best; I think it's libtommath+libtomcrypt+a TLS implementation; but you would need to disable one or two features...
wpa_sup.config in brbuild.tar.gz is an example of how you can configure wpa_supplicant to use this. (You might try grep -v '^#', just to filter out the noise.)

dropbear and uvlan (http://www.certainkey.com/uvlan/) use libtommath+libtomcrypt also; sqlite can use those for encryption.

[Iguleder]

Code: Select all

CFLAGS="-mtune=generic -Os -ffunction-sections -fdata-sections -fomit-frame-pointer -fPIC -pipe"
LDFLAGS="-fPIC -Wl,-gc-sections -Wl,--sort-common"

Works great with WPA2.

LibreSSL does't add much to size, but I don't remember how much exactly. I switched from the internal tomcrypt to LibreSSL to reduce the overall size of the distro.

[technosaurus]

I've been experimenting with profile guided optimization to improve performance of PDMP3 and got some pretty good results, so I was thinking of trying it on tinyxserver... Do you guys have any better ideas on a good sample profiling run (I was thinking normal jwm startup + playing video) ... I normally build it with unix domain sockets only (no tcp, so no need to profile for that) with the assumption that if someone wants to used tcp they can use socat with ssh/dropbear rather than opening up a potential security hole by default.

[Ibidem]

I've been experimenting with profile guided optimization to improve performance of PDMP3 and got some pretty good results, so I was thinking of trying it on tinyxserver... Do you guys have any better ideas on a good sample profiling run (I was thinking normal jwm startup + playing video) ... I normally build it with unix domain sockets only (no tcp, so no need to profile for that) with the assumption that if someone wants to used tcp they can use socat with ssh/dropbear rather than opening up a potential security hole by default.

OT: I never got PDMP3 to play what I threw at it (mostly sermons from http://www.truthforlife.org/).

Anyhow, make sure to use an image viewer, text editor, and at least one gtkdialog wizard.

[Iguleder]

<UEFi rants>

gnu-efi and elilo make me sick. The documentation says they used to be a single package, but got separated. It seems they share some parts and the build system is ugly as hell (e.g it forces the use of /usr/bin/ld sometimes, so cross-compilation is not an option).

On some of my machines, only with certain combinations of the compiler/host distro/linker/alignment of the stars the elilo EFI binary is corrupt and causes a black screen or an immediate reboot, both against my laptop and TianoCore on QEMU.

It's been this way for a long time, until I wrote a script that builds these horrible packages in a clean chroot environment. Now, without any changes to the building infrastructure or a reasonable explanation, the problem is back again and I'm lost.

</UEFi rants>

I'm experimenting with the kernel EFI stub feature - since we need an initramfs, I made it built-in. I want to boot the kernel directly, without a crappy boot loader that makes life a nightmare.

Theoretically, this makes the whole thing extremely elegant, since the entire kernel-dependent stuff are contained in one file, so swapping the kernel or providing flavors with different kernel versions is easy.

EDIT: the initramfs is built-in and it works great. However, booting the EFI stub results in a black screen under QEMU.

[technosaurus]

I experimented with uefi too, but took a break to consider setting up root=PARTUUID=... in the builtin command line. My notes say:

rebuilding the uncompressed kernel with "piggy" after sed-iting the builtin command line to replace XXXXXXXX-XXXX-XXXX-XXXXXXXX with the user's actual root partition using data from gfdisk (and some notes about the GPT partition table about how to possibly replace gfdisk).

I know that sounds "hacky" but there is no way to know what the user's GUUID will be when the kernel is built. The easiest way I could think to go around having to include a builtin initramfs (not easily upgradeable) or rebuild the entire kernel on the user end (takes too long and kernel source is too big) was to keep the raw kernel image and include the tiny tools used to compress and build it in the install iso (the smaller/harder way would be to un-build the working kernel, but it isn't as simple as I would like). This will allow under 1 second boots for traditional full installs but only tested with my minimal kernel config (<1Mb xz compressed with builtin initramfs including busybox, Xvesa/xfbdev, jwm and rxvt).

The basic install process is to dump the root filesytem to the desired partition; use gfdisk to get the root=PARTUUID= parameter and replace it in the raw kernel image using sed (or similar tool ... optionally add rootfstype= since linux doesn't bother to officially associate any partition types with a UUID - I think this should be a kernel bug report). Finalize the kernel build with lz4_hc compression (fastest boot speed) and install it to the EFI boot directory (oversimplification of a process that I cannot fully recall).

This reminded me about a couple other needed kernel modifications.
1. vga and video parameters could use a patch to automatically set the maximum resolution in order to make xfbdev and other framebuffer based tools in general more user friendly (this would allow a sane default built-in kernel command line that is portable across architectures) ... I think I posted an early rudimentary attempt here somewhere and will replace this line with a link if I find it.
2. zram and zswap pages are still de/re-compressed instead of copied even if both use the same compression type. (this could drastically improve performance in low ram systems)

[Iguleder]

Why patch the "video parameter? KMS already takes care of detecting the resolution.

[wyzguy]

KMS will automatically detect your native (highest) resolution.

Please search the net for this string (with the quotes).

"any vga= options in your bootloader"

You don't even have to click on the link to go to the website, just read the
highlighted phrase in the full sentence (which is shown) of the first item,
or any of the 9 returned items.

[Iguleder]

I'm building a 32-bit variant of DSLR through VirtualBox. So far, results look quite weird - there's a size difference of 4-8K between 32-bit and 64-bit packages.

Do you have any good explanation for this? I expected a 15%-20% reduction in size. I want to get DSLR's 32-bit edition smaller, somewhere around 70 to 75 MB (compared to the 64-bit edition, at 93 MB).

[Ibidem]

I'm building a 32-bit variant of DSLR through VirtualBox. So far, results look quite weird - there's a size difference of 4-8K between 32-bit and 64-bit packages.

Do you have any good explanation for this? I expected a 15%-20% reduction in size. I want to get DSLR's 32-bit edition smaller, somewhere around 70 to 75 MB (compared to the 64-bit edition, at 93 MB).

Is that installed size or compressed?
I thought that 10%-15% shrinkage (80-84 MB, if everything were code) was the high end of what you might expect.

Also, with 32-bit compatability, you should be able to chroot into a 32-bit environment, run setarch/linux32, and nothing can tell you're running 64-bit.

[Iguleder]

That's the installed size (binaries only).

EDIT: wow, it's simply horrible. The 32-bit ISO is smaller than the 64-bit one by just 1 MB.

[technosaurus]

I'm building a 32-bit variant of DSLR through VirtualBox. So far, results look quite weird - there's a size difference of 4-8K between 32-bit and 64-bit packages.

Do you have any good explanation for this? I expected a 15%-20% reduction in size. I want to get DSLR's 32-bit edition smaller, somewhere around 70 to 75 MB (compared to the 64-bit edition, at 93 MB).

If you removed -fPIC from your default CFLAGS that may have made a difference, its only needed for shared libs, not binaries (also -fpic is smaller)... Depending on the program, there could be a 49% size difference just because longs, pointers etc... that are not initialized to 0 which take twice as much space in a 64bit binary.... but if there are very few of those types you won't get much size difference at all.

Re: KMS
Not all framebuffer devices support it. Its easier to do one patch to make it friendlier for existing framebuffers than to add KMS to every driver in the kernel.


<sidebar>
Have you looked at the sources? Its appalling. 90% of them have direct cut 'n pastes of entire functions, many with little to no changes.
Here are 2 that other framebuffer drivers include if they don't have builtin versions of their own:
drivers/video/fbdev/core/cfbimgblit.c
drivers/video/fbdev/core/sysimgblit.c
Note the similarities and shared static const variables (some functions are exactly the same except white space). The Linux kernel is systemically plagued with this kind of garbage programming that defeats the whole benefit of using a monolithic kernel in the first place. I'd really like to see some of toybox's techniques applied to the kernel - I'm pretty sure it could result in nearly a 50% size reduction if not more (If you think I am talking out of my ass, just run uncrustify on the sources followed by duplo). Way too many common functions are included in the modules rather than being moved to lib/ (or they _are_ already in lib/ and just not used) and many large blocks of shared code could be converted to functions.
For that matter the video drivers are unnecessarily split into multiple different drivers (fb/video, drm, kms, ...) and could be unified. For the openGL stuff, there could be cfb*/sys* fallbacks that use Fabrice Bellard's tinyGL code for any unsupported functions (then even uvesafb could support openGL), but this could potentially break compatibility and thus may never be fixed in the vanilla kernel.
</sidebar>

[greengeek]

EDIT: wow, it's simply horrible. The 32-bit ISO is smaller than the 64-bit one by just 1 MB.

Given that a 64 bit instruction can obviously do twice as much work per cycle as a 32bit instruction then the 32-bit ISO should be twice the size of the 64-bit one.
Logical. Totally.

[technosaurus]

EDIT: wow, it's simply horrible. The 32-bit ISO is smaller than the 64-bit one by just 1 MB.

Given that a 64 bit instruction can obviously do twice as much work per cycle as a 32bit instruction then the 32-bit ISO should be twice the size of the 64-bit one.
Logical. Totally.

sarcasm aside, this can be true when optimizations are turned on ... there are many peephole optimizations that can convert several instructions into a single SSE* or other 64 bit instruction.

Currently these peephole optimizations are relatively few in number, but it is in the range of possibility to add a superoptimizer like STOKE into llvm's JIT compiler to continuously discover new ones using extra clock cycles in a parallel thread and store the results to a remote database for verification. This could be extremely beneficial to any program that uses this method for alphas, betas and release candidates so that the final version has the best (or at least better) optimizations. This is one of my possible candidates for a PhD thesis.

[Iguleder]

Just uploaded a fresh 32-bit build. It looks great - uses 24 MB of RAM under Qemu, with an emulated Pentium and 512 MB of RAM.

If you removed -fPIC from your default CFLAGS that may have made a difference, its only needed for shared libs, not binaries (also -fpic is smaller)... Depending on the program, there could be a 49% size difference just because longs, pointers etc... that are not initialized to 0 which take twice as much space in a 64bit binary.... but if there are very few of those types you won't get much size difference at all.

Interesting. I removed -fPIC - I'll try to rebuild both flavors whenever I find the time next week.

Not all framebuffer devices support it. Its easier to do one patch to make it friendlier for existing framebuffers than to add KMS to every driver in the kernel.

Of course it's easier, but that's a workaround, not a solution for this problem.

[technosaurus]

Of course it's easier, but that's a workaround, not a solution for this problem.

The whole KMS infrastructure was originally a workaround for specific vendor cards, so it makes sense that other vendors don't want to implement it themselves. I wonder if I could figure out a way to use the auto resolution code iff KMS is not supported.

The original thread is here:
http://www.murga-linux.com/puppy/viewtopic.php?t=78993

[Keef]

First attempt using 32 bit dslr-latest-i686-bios.
Tried booting from USB using isoboot. Got the following message:

Code: Select all

esas2r: driver will not be loaded because no ATTO esasr2 devices were found.
wdt: WDT status 255
eurotechwdt: timeout WDT timeout
eurotechwdt: initialising system reboot

Then it hangs.

Copied files to HDD and booted via Grub4Dos. Same result until adding acpi=off to kernel line - long pause after the message above though.
Touchpad works well, but USB mouse is very jerky.
Dillo launched but would not load any webpages (Lynx is OK). Tried the Dillo binary from PupNgo and that works correctly.

HP nc6120 Laptop
Chip description:
VGA compatible controller: Intel Corporation Mobile 915GM/GMS/910GML Express Graphics Controller (rev 03)
Display controller: Intel Corporation Mobile 915GM/GMS/910GML Express Graphics Controller (rev 03)
Processor: Intel(R) Pentium(R) M processor 2.00GHz
Broadcom Corporation NetXtreme BCM5705M_2 Gigabit Ethernet

[Iguleder]

The Dillo part annoys me. What do you mean by "would not load any webpages"?