Until a few years back most of my systems used Linux From Scratch. It's amazing for learning your system and how all its libraries integrate. I created my own boot scripts, my own pacman package manager repository, etc. One of the main reasons I stopped using LFS was the amount of work required for updates.
Firefox is bad to compile yourself, because it's slow and often requires a library version you don't have, such as a new ICU version. Updating ICU also requires you to build packages that depend on it: harfbuzz, ffmpeg, qt5, poppler and the list goes on.
Chromium takes it to a whole new level. While Linux packages typically have an automake flow (configure, make, make install) Chromium requires ninja, not just ninja but a specific version depending on the Chromium version. Then you need a ton of libraries with specific versions (or custom patches) or use the built-in versions provided by chromium, which may or may not compile at all. Then there is clang, while most software is compiler agnostic, Chromium does not compile with gcc. Of course, it needs a specific clang release, sometimes even a pre-release, or again a load of patches to compile. It seems the chromium build process is mostly built by engineers desperate to reinvent the wheel. While everything can work fine with sane Linux defaults (GCC, automake, system libraries) they convinced themselves their software is so complex and unique it requires a special treatment.
Chromium looks like a build system for a complex integrated project where the engineers have one priority: make the application work as best it can. You will quickly find that when you focus on that then making the build system convenient, especially convenient for distribution packagers with their myriad of own rules and modifications that they want to make on top of your system, is quite a big blocker to the primary goal. You can see similar things with applications like OBS, where they bundle modified libraries to fix problems and then distributions unbundle them and release broken packages, causing no end of grief for users and the developers.
The Chromium build process is just a consequence of the monorepo mindset. For Google, depending on specific versions of tools and dependencies is not a problem because all their engineers will automatically get those versions. For the Linux distribution model it's a pain.
> Linux defaults (GCC, automake, system libraries)
Last time I tried Gentoo desktop I ended up with 8 separate autotools versions, and gcc & glibc was not that far behind in count. And just like in https://xkcd.com/927/ every C developer tries to improve the situation by creating another build system.
gentoo's emerge is awesome. I keep a gentoo vm just to compile firefox, because it is impossible any other way. even though i don't use gentoo overall.
the comments trying to defens chrome (and old OBS) are completely corporate shills or missing the point. yeah obs at first did that because they had to, since it was such a small and niche project the devs had no pull upstream. after distros did all the unbundling job and helped them move patches upstream now every software benefit from those libraries fixes.
Google took chromium et al from open source and actively moved in the wrong direction, while having all the pull and influence like no open source project will ever have! its malice and/or incompetence. if you don't see that you're either shillings or lying to yourself. or have no knowledge of what open source is about.
There is not some arbitrary notion of wheel reinvention just for browsers. Calling browsers 'just browsers' is doing their feature set a huge disservice: they are bona fide operating systems or virtual machines, which have to load and run untrusted, unverified data and code within critical system libraries (many of which with kernel-mode access, such as GPU drivers, sound drivers, USB subsystems, and more), and do a huge amount of sandboxing to guard against escalations.
Within Google, a Chromium build uses remote build execution by default, upstream Clang by default (for the latest features and fixes, and Chromium developers regularly contribute to LLVM), and needs to cross-compile easily for at least a dozen different OS-architecture triples (Linux x86-64, Linux ARM, macOS, Windows x86-64, Windows on ARM, iOS, Android x86-64, Android ARM, Android TV, some BSDs, etc).
Everything you've listed—from ninja, to Clang, to libraries with custom patches, to Clang releases, including requiring pre-releases sometimes—is optimised for Google's scale, rather than the lone warrior developer. What you call 'sane Linux defaults' (especially the examples you have listed: GCC, automake, so-called 'system libraries') are highly antagonistic to all of the above.
They may make life easy for the so-called 'default case' of desktop Linux (arguable; Chromium was first released on Windows, and secondly since desktop Linux's most common C library, glibc has an unstable ABI, making back-portability impossible), but make it incredibly hard to get out a project of such immense complexity. All of Chromium's tooling and 'complexity' was developed in response to the 'sane' defaults being not so sane after all.
To get a GCC cross-compile for a given set of architecture triples, one needs a complete graph of GCC toolchains targeting every pair of host and target. On the other hand Clang is natively a cross-compiler; pass in `--sysroot` and the host compiler will redirect for the target instantly. Automake (and the autotools in general) is slow as molasses compared to ninja. It is based on computing and compilation from the 1970s, and their defaults probe for long-since-dead string comparison functions that are expected to work by default nowadays on most hosted systems. Not to mention it has an extraordinarily complicated configure step[1], and an equally appalling macro-based configure language.
Ninja is purely declarative, and is not even meant to be human-readable, as it is a back-end for a variety of build generator tools (CMake, XMake, build2, Bazel (Blaze internally at Google), Meson, probably more). Google have in fact rewritten Ninja to be a remote-build-capable tool from the outset[2], and they call it Siso. Siso has more than halved my Android build times even compared to Ninja because of its incredible use of RBE.
Frankly I think it should be evident why they have opted for 'rewriting the world' including vendoring and modifying system libraries, as well as including efforts like Skia, ANGLE, and more: directly relying on 'system libraries' is painful. I mentioned GPU drivers. Imagine rewriting the same code over and over to have precisely the same behaviour on D3D, Vulkan, OpenGL, Metal, and every single vendor GPU driver on every single system (and there are a lot more on mobile than on desktop: ARM Mali, Qualcomm Adreno, Samsung Xclipse, Imagination PowerVR, Huawei, MediaTek; this is not even close to exhaustive, just look at the Khronos vendor list).
When comments like yours are made I wonder if there is ever a consideration beyond 'it makes the developer's life hard'. As developers, our ultimate target audience are the users of our software. We should be willing to do anything come hell or high water, even making the dev-ex extraordinarily convoluted, if it means the user experience will be the smoothest and fastest possible. Sure, there is a balance, but I believe it should be strongly skewed towards the user story first.
>glibc has an unstable ABI, making back-portability impossible
Of all the libraries to call out for this, glibc is not the one. It's basically the only library on linux that actually has a strong commitment to providing a stable ABI, with its own symbol versioning system to allow it. It is a bit of a PITA to compile for an older version of that ABI, but it's very possible to be compatible with glibc across a wide range of versions and distributions (I don't know why there isn't a flag to do it: the most widely supported option is compiling on a system with the oldest glibc you want to support, and there's some header hacks which claim to be able to avoid that).
It's everything else necessary to interact with the system (especially anything desktop/multimedia) that's wildly unstable.
I singled it out because I've been badly burnt by it in the past, and I dislike how it works compared to Win32. Back-targeting old glibc from newer (a very common case because developers tend to have newer distros than their users), requires either changes in source code (not build code, which is where it ought to be) to explicitly pin versions, or the ridiculousness of firing up an entire Docker container just for old glibc. It has regularly caused issues for things like Steam on Linux.
If you build on random developer's machines rather than a unified build environment you've already failed at so many things that the glibc problem is certainly not your biggest concern.
Fair enough for poor Google, if only they did not strong-armed Chrome on everyone's computer by leveraging their default search engine/home page position, their ads and pop-under "Try Chrome" that were plastered everywhere 15 years ago or so.
It's relevant because Chrome market dominance still has a very strong influence on websites working (tested?) only with Chrome. Add to that Google's NIH syndrome and its "move fast" mantra at the cost of interoperability with the wider Linux community and then both things are relevant.
I tried Gentoo some 20 years ago or so for fun at home. I guess my naive thinking was that software specifically compiled for my hardware would run noticeably faster and "better".
But I didn't last for very long. I remember starting the install for a bigger app before going to bed, only to find the thing still building when I woke up. And as for the dream of more tailored builds, I never really noticed anything much different - I probably was too uneducated about build configurations, or - more likely - the differences just didn't matter too much.
Eventually, I switched to Ubuntu and never looked back.
So, I'm curious - is anyone using Gentoo as their daily driver, and what is the main incentive for you?
I'm using it as my main OS. I have it installed on 4 machines: a RK3288 SBC, a RK3588 desktop, a x86-64 desktop and a x86-64 laptop. Main incentive for me is customization.
- It makes it very very easy to patch any package before they're installed - just put the patches in /etc/portage/patches/<pkg name>/, and that includes the kernel too.
- I can exclude package features that I don't like. For example, my Firefox doesn't try/offer to turn on the wifi.
- I can also exclude entire packages that I don't like, because the entire dependency chains can be eliminated by disabling USE flags. For example, none of my systems have systemd or dbus. Since last year I don't even have openrc anymore (the other Gentoo service manager). One of them doesn't even have udev. I tweaked it so much, I'm not even sure I can still call it Gentoo.
Daily Gentoo desktop user since 2007. Compilations/ricing is one thing, the variety and flexibility of the portage and USE flag system is another. For me it just makes sense to stay with this proven distro. I use Ubuntu variants on my work PC and they just have all this crap that I don't have to tolerate on my gentoo installs.
The docs were great for learning. And it made it so easy to recompile the kernel because it was right in the initial setup and you just had to walk through it again.
My understanding is that compiler flags yielding speed boosts where an actual thing back in the day, but not anymore really.
Nowadays the speed come from other things. For example a few years ago i realised my firefox on linux was doing video encoding/decoding on cpu without using the intel gpu’s hardware encoder/decoder. I did not have to rebuild firefox from scratch but rather fiddle with configurations and environment variables to make sure Firefox wouldn’t assume the lowest common denominator.
That saved battery on my laptop and yielded better performance.
I used it on an off as my daily machine for around 15-20 years I think. It is fast in terms of execution-on-hardware speed, and genuinely puts you in control. For a certain kind of user this is more important than anything.
These days I'm on OSX mostly because of the frequency of business trips meant I needed something small and portable a few years ago, and now the hardware is so much better than what you can get for the PC world. Plus, I am hyper-conscious of my time and can't really afford to be fiddling with new system libraries, kernel configurations and so forth.
Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall, and after enough iterations I'd started to err to Ubuntu at some point. But it was by far the most fun of any Linux I used, save perhaps Slackware which was my first.
I think in a few years' time the next machine I build will be a non-Apple ARM custom portable, hopefully running Gentoo, with a lot of custom machining and hardware features.
> at some point the system state would break and you'd have to reinstall
My advice is to start using btrfs snapshots. Make a backup snapshot, run the updates, and if anything breaks, just "btrfs set-default" to the backup snapshot and everything is working again.
> Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall
Not at all. I have even recovered from bricking the package manager (my fault, these days there are protections in place against that). Worst case you manually extract working binaries and then clean up the mess. And I have never run into issues that needed such drastic measures without doing stupid things.
It is however not a distro for those unwilling to understand how it works.
> It is fast in terms of execution-on-hardware speed, and genuinely puts you in control.
After installing Gentoo I felt I should make some kind of change, and I wanted to replace the GNOME foot in the GNOME menu with an icon I found of the Gentoo "g" in a wizard hat.
I found documentation suggesting that the menu's icon was controlled by some setting somewhere, and I changed that setting to point to the wizard-hat icon.
The menu was unaffected. If I viewed the menu bar's properties, the menu was associated with the wizard-hat icon in the properties display, but it continued to show the foot icon in its name.
I had nothing else to go on and gave up. I'm still mad about that.
> Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall
Well, that did happen to me once. But it happened after I uninstalled glibc. I can't really blame Gentoo for that. I always saw it as positive that Gentoo allowed me to do something that ridiculous.
I've always used source-based packages, but chromium was the one I simply dropped in favour of Firefox.
On a 16c32t / 64G machine, compiling Firefox takes ~1h with LTO and PGO, while Chromium already took 5+h in 2019 without PGO. Then they removed jumbo-build because there were some issues and kept adding code and toolchain deps (jumbo-build is a build step that merges a few hundred source files before compiling to do the header processing once, shrinking the build from 5h+ to ~1-2h). Last time I checked the build had 70k+ build steps, with the final link step easily taking 10m+ with LLD. Answering some other comments: ccache doesn't help, because every version will have some minor header changes somewhere, so basically the entire cache gets invalidated even within the same major (e.g. 151.44.22.11 to 151.44.23.28).
Compiling Chromium took absolutely forever 5+ years ago. I think a whole system took like 20% of the time that Chromium would take to compile. I can't even imagine with rust extensions added on top now. Gentoo was so awesome though but I had to switch to debian because I would just play with my configuration all the time and break things. One day I hope to return to using it though or setup another machine to use it.
Chromium is a pyramid of components and the higher you go up, the longer the compilation of single file takes, because they depend on so many things. This makes caching difficult. A small change at the bottom can invalidate many files, especially the heavy ones at the top. The build system ninja already does caching, but it only offers limited benefits. Then there is so much activity that if you plan to do a weekly build, you can do it from scratch.
ccache is not very effective accross different versions of software as it only takes one small change in a header that's included everywhere to make the cache completely useless.
Am I the only one who noticed just how entitled this Žilka person is? If they are so good at maintenance maybe they should take it over and learn the hard way that things like this are not at all remotely simple to pull off and that maintainers are (usually) unpaid volunteers who have their own lives (and in some cases, they maintain many other packages).
Edit: I should note that it definitely looks like entitlement based on how the article presents it, and doesn't at all seem like this person is taking into consideration all the maintenance policies/rules/whatnot.
This is what we get when we go all in on a monolithic bloated platform, and then let its direction be primarily determined by one company. We'd be better off with a simpler, albeit less featureful, platform, or at least one that's more modular.
The main problems seem to be that it takes a long time to build all combinations of USE flags and that users complain that volunteers are not working fast enough, causing annoyance of the maintainers. Can you explain how AI would help here?
You still need the hardware resources to do builds. And thanks to Slop Generator providers gobbling up alllllllll the RAM and storage they can get their icky hands on, it simply practically infeasible for everyone not affiliated with a tech BigCo.
I tried some months ago, and it'll make a blank one fine, but if you need a complex configure or install phase, I felt I was better off doing it by hand.
Firefox is bad to compile yourself, because it's slow and often requires a library version you don't have, such as a new ICU version. Updating ICU also requires you to build packages that depend on it: harfbuzz, ffmpeg, qt5, poppler and the list goes on.
Chromium takes it to a whole new level. While Linux packages typically have an automake flow (configure, make, make install) Chromium requires ninja, not just ninja but a specific version depending on the Chromium version. Then you need a ton of libraries with specific versions (or custom patches) or use the built-in versions provided by chromium, which may or may not compile at all. Then there is clang, while most software is compiler agnostic, Chromium does not compile with gcc. Of course, it needs a specific clang release, sometimes even a pre-release, or again a load of patches to compile. It seems the chromium build process is mostly built by engineers desperate to reinvent the wheel. While everything can work fine with sane Linux defaults (GCC, automake, system libraries) they convinced themselves their software is so complex and unique it requires a special treatment.
Last time I tried Gentoo desktop I ended up with 8 separate autotools versions, and gcc & glibc was not that far behind in count. And just like in https://xkcd.com/927/ every C developer tries to improve the situation by creating another build system.
the comments trying to defens chrome (and old OBS) are completely corporate shills or missing the point. yeah obs at first did that because they had to, since it was such a small and niche project the devs had no pull upstream. after distros did all the unbundling job and helped them move patches upstream now every software benefit from those libraries fixes.
Google took chromium et al from open source and actively moved in the wrong direction, while having all the pull and influence like no open source project will ever have! its malice and/or incompetence. if you don't see that you're either shillings or lying to yourself. or have no knowledge of what open source is about.
Within Google, a Chromium build uses remote build execution by default, upstream Clang by default (for the latest features and fixes, and Chromium developers regularly contribute to LLVM), and needs to cross-compile easily for at least a dozen different OS-architecture triples (Linux x86-64, Linux ARM, macOS, Windows x86-64, Windows on ARM, iOS, Android x86-64, Android ARM, Android TV, some BSDs, etc).
Everything you've listed—from ninja, to Clang, to libraries with custom patches, to Clang releases, including requiring pre-releases sometimes—is optimised for Google's scale, rather than the lone warrior developer. What you call 'sane Linux defaults' (especially the examples you have listed: GCC, automake, so-called 'system libraries') are highly antagonistic to all of the above.
They may make life easy for the so-called 'default case' of desktop Linux (arguable; Chromium was first released on Windows, and secondly since desktop Linux's most common C library, glibc has an unstable ABI, making back-portability impossible), but make it incredibly hard to get out a project of such immense complexity. All of Chromium's tooling and 'complexity' was developed in response to the 'sane' defaults being not so sane after all.
To get a GCC cross-compile for a given set of architecture triples, one needs a complete graph of GCC toolchains targeting every pair of host and target. On the other hand Clang is natively a cross-compiler; pass in `--sysroot` and the host compiler will redirect for the target instantly. Automake (and the autotools in general) is slow as molasses compared to ninja. It is based on computing and compilation from the 1970s, and their defaults probe for long-since-dead string comparison functions that are expected to work by default nowadays on most hosted systems. Not to mention it has an extraordinarily complicated configure step[1], and an equally appalling macro-based configure language.
Ninja is purely declarative, and is not even meant to be human-readable, as it is a back-end for a variety of build generator tools (CMake, XMake, build2, Bazel (Blaze internally at Google), Meson, probably more). Google have in fact rewritten Ninja to be a remote-build-capable tool from the outset[2], and they call it Siso. Siso has more than halved my Android build times even compared to Ninja because of its incredible use of RBE.
Frankly I think it should be evident why they have opted for 'rewriting the world' including vendoring and modifying system libraries, as well as including efforts like Skia, ANGLE, and more: directly relying on 'system libraries' is painful. I mentioned GPU drivers. Imagine rewriting the same code over and over to have precisely the same behaviour on D3D, Vulkan, OpenGL, Metal, and every single vendor GPU driver on every single system (and there are a lot more on mobile than on desktop: ARM Mali, Qualcomm Adreno, Samsung Xclipse, Imagination PowerVR, Huawei, MediaTek; this is not even close to exhaustive, just look at the Khronos vendor list).
When comments like yours are made I wonder if there is ever a consideration beyond 'it makes the developer's life hard'. As developers, our ultimate target audience are the users of our software. We should be willing to do anything come hell or high water, even making the dev-ex extraordinarily convoluted, if it means the user experience will be the smoothest and fastest possible. Sure, there is a balance, but I believe it should be strongly skewed towards the user story first.
[1]: https://upload.wikimedia.org/wikipedia/commons/8/84/Autoconf...
[2]: https://chromium.googlesource.com/build/+/refs/heads/main/si...
Of all the libraries to call out for this, glibc is not the one. It's basically the only library on linux that actually has a strong commitment to providing a stable ABI, with its own symbol versioning system to allow it. It is a bit of a PITA to compile for an older version of that ABI, but it's very possible to be compatible with glibc across a wide range of versions and distributions (I don't know why there isn't a flag to do it: the most widely supported option is compiling on a system with the oldest glibc you want to support, and there's some header hacks which claim to be able to avoid that).
It's everything else necessary to interact with the system (especially anything desktop/multimedia) that's wildly unstable.
I singled it out because I've been badly burnt by it in the past, and I dislike how it works compared to Win32. Back-targeting old glibc from newer (a very common case because developers tend to have newer distros than their users), requires either changes in source code (not build code, which is where it ought to be) to explicitly pin versions, or the ridiculousness of firing up an entire Docker container just for old glibc. It has regularly caused issues for things like Steam on Linux.
But I didn't last for very long. I remember starting the install for a bigger app before going to bed, only to find the thing still building when I woke up. And as for the dream of more tailored builds, I never really noticed anything much different - I probably was too uneducated about build configurations, or - more likely - the differences just didn't matter too much.
Eventually, I switched to Ubuntu and never looked back.
So, I'm curious - is anyone using Gentoo as their daily driver, and what is the main incentive for you?
- It makes it very very easy to patch any package before they're installed - just put the patches in /etc/portage/patches/<pkg name>/, and that includes the kernel too.
- I can exclude package features that I don't like. For example, my Firefox doesn't try/offer to turn on the wifi.
- I can also exclude entire packages that I don't like, because the entire dependency chains can be eliminated by disabling USE flags. For example, none of my systems have systemd or dbus. Since last year I don't even have openrc anymore (the other Gentoo service manager). One of them doesn't even have udev. I tweaked it so much, I'm not even sure I can still call it Gentoo.
Nowadays the speed come from other things. For example a few years ago i realised my firefox on linux was doing video encoding/decoding on cpu without using the intel gpu’s hardware encoder/decoder. I did not have to rebuild firefox from scratch but rather fiddle with configurations and environment variables to make sure Firefox wouldn’t assume the lowest common denominator.
That saved battery on my laptop and yielded better performance.
These days I'm on OSX mostly because of the frequency of business trips meant I needed something small and portable a few years ago, and now the hardware is so much better than what you can get for the PC world. Plus, I am hyper-conscious of my time and can't really afford to be fiddling with new system libraries, kernel configurations and so forth.
Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall, and after enough iterations I'd started to err to Ubuntu at some point. But it was by far the most fun of any Linux I used, save perhaps Slackware which was my first.
I think in a few years' time the next machine I build will be a non-Apple ARM custom portable, hopefully running Gentoo, with a lot of custom machining and hardware features.
My advice is to start using btrfs snapshots. Make a backup snapshot, run the updates, and if anything breaks, just "btrfs set-default" to the backup snapshot and everything is working again.
Not at all. I have even recovered from bricking the package manager (my fault, these days there are protections in place against that). Worst case you manually extract working binaries and then clean up the mess. And I have never run into issues that needed such drastic measures without doing stupid things.
It is however not a distro for those unwilling to understand how it works.
After installing Gentoo I felt I should make some kind of change, and I wanted to replace the GNOME foot in the GNOME menu with an icon I found of the Gentoo "g" in a wizard hat.
I found documentation suggesting that the menu's icon was controlled by some setting somewhere, and I changed that setting to point to the wizard-hat icon.
The menu was unaffected. If I viewed the menu bar's properties, the menu was associated with the wizard-hat icon in the properties display, but it continued to show the foot icon in its name.
I had nothing else to go on and gave up. I'm still mad about that.
> Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall
Well, that did happen to me once. But it happened after I uninstalled glibc. I can't really blame Gentoo for that. I always saw it as positive that Gentoo allowed me to do something that ridiculous.
To the single guy maintaining it, and no offer of help. Crazy entitlement.
I've always used source-based packages, but chromium was the one I simply dropped in favour of Firefox.
On a 16c32t / 64G machine, compiling Firefox takes ~1h with LTO and PGO, while Chromium already took 5+h in 2019 without PGO. Then they removed jumbo-build because there were some issues and kept adding code and toolchain deps (jumbo-build is a build step that merges a few hundred source files before compiling to do the header processing once, shrinking the build from 5h+ to ~1-2h). Last time I checked the build had 70k+ build steps, with the final link step easily taking 10m+ with LLD. Answering some other comments: ccache doesn't help, because every version will have some minor header changes somewhere, so basically the entire cache gets invalidated even within the same major (e.g. 151.44.22.11 to 151.44.23.28).
Shared libraries breaking ABI could mean recompiling half your operating system
On the other hand, bundled libraries meant insanely long build times and a sense of impurity/uncleanliness
Unfortunately, it looks like projects like ABI tracker are long dead:
https://github.com/lvc/abi-tracker
When you build your own kernel you don't compile all the random crap like token ring support and 30 filesystems nobody uses.
Edit: I should note that it definitely looks like entitlement based on how the article presents it, and doesn't at all seem like this person is taking into consideration all the maintenance policies/rules/whatnot.
Pretty rich from a compile-from-source distribution
Problem is solved!