Why Arch Linux’s Kernel Upgrades Suck

Update Sept 04, 2011

As this still gets comments occasionally, I should clearly note that this is (and has been for some time) an accepted issue on Arch’s bug tracker. There is a basic accepted game plan for how it will be implemented, but is currently marked as low-priority. If you’re interested in the details look at: https://bugs.archlinux.org/task/16702#comment80122. Also, please do not comment on that bug report because it’s very close to being closed just to keep people from further commenting “new ‘solutions’ that we simply won’t implement.”

Now, Arch Linux and I have had our differences in the past. It’s handling of documentation for instance has always been a bit backwards IMO. What’s that? I really have to recompile every single program which I want the API documentation for? Can we get with the times already and make perhaps -doc packages by default? Worried about their being lots of -doc packages when you do something like pacman -Ss some-program? Then put them in a separate repo call documentation and have it disable by default! This isn’t a hard concept people.

But that’s not what I want to complain about today. What really annoys me about Arch Linux more than anything else is it’s brain deadly stupid handling of kernel upgrades. Let’s go through a typically kernel upgrade: 1) A new kernel version is out that Arch wants to upgrade to, GREAT! 2) pacman -S kernel26 3) Delete kernel, initrd, and modules for the currently installed kernel 4) Install the new kernel, modules, and initrd.

Ok, there should be an obvious problem here. #3 deletes the current (typically working) kernel and replaces it. Well what if the new kernel has a serious regression in it on your hardware, or worse yet, what if it doesn’t boot at all? You’re now stuck. You’ll may well have to pull out a livecd, and try compiling the old kernel version again to get back to a working system. There are lots of things that could be done to fix the issue, but the fact of the matter is, why delete a working kernel with one that may or may not work?

So, there’s a problem. What should the solution be? Well, how I would do it would be something like this: Every kernel version PKGBUILD will have a version number in the pkgname. So instead of it being just kernel26, it’ll now be kernel-2.6.31 for instance. kernel26 will now be basically a dummy package that only depends on kernel-2.6.30. All of a sudden, whenever there’s a new kernel version that the Arch package maintainers want to upgrade to, they’ll change the kernel-2.6.30 PKGBUILD’s name, and update kernel26’s dependency to the new kernel version.

In other words:

kernel26 depends on kernel-2.6.30

post upgrade:
kernel26 depends on kernel-2.6.31
no packages depend on kernel-2.6.30

Now, the only real issue with this system would be, what do we call vmlinuz26, System.map26, & kernel26.img be called and how should be updated? The first part could very simply just be versioned. kernel-2.6.30.img, vmlinuz-2.6.30, etc. This really shoudn’t be that hard, and I think would work quite well. Well, that could be a bit tricky, but really, it shouldn’t be that difficult in all honesty. We’ll take grub for instance. It’s very easy to just add new rules to grub and everything “just works.” So really, Arch could do like other distributions do (Debian I know does this, but IIRC others do too) and update grub automatically. For users that don’t like this option, it could be turned off in rc.conf.

Another option would be to still use version prefixed names for vmlinuz, but then have links that are just updated upon update (so vmlinuz26->vmlinuz-2.6.30 becomes vmlinuz26->vmlinuz-2.6.31) All of a sudden, things work pretty much how they are now, but you can still edit grub (either at boot time if the new kernel fails, or after the fact) to have your old kernel in there.

Now, with this in mind, the first option my be more appealing to the more liberal Arch crowd, but for the more die-hard (the one’s that chant KISS anytime you suggest a new feature for Arch) might not be as violently opposed to the second option. I also think most of the people in between wouldn’t really care too much either way, but I have a feeling more would probably favor the second option.

I decided to file the feature request on Arch’s FS:
You can comment there if you directly want to let the developers know how you feel, or here if you’re just wanting to tell me the idea sucks or if you like it, if you too have had problems with it, etc. In other words, I’d suggest commenting on the feature request if you have constructive input on how it may be implemented or why it shouldn’t, and here for anything else.

16 thoughts on “Why Arch Linux’s Kernel Upgrades Suck

  • Pingback: Harley Laue (losinggeneration) 's status on Friday, 16-Oct-09 19:36:41 UTC - Identi.ca

  • Aaron Griffin

    This is actually a fairly interesting idea, yet posting it in some corner of the intertubes isn’t going to make anyone take notice. Please add a feature request to the bug tracker (linked with my name).


  • losinggeneration Post author

    Please add a feature request to the bug tracker (linked with my name).

    I wanted to get a bit of a feel for how users general users feel before I pitch the idea as a feature request. That’s why I submitted it to the identi.ca crowd first. I did this mainly because it sounds good to me, but other Arch users may totally disagree on general principle of keeping things simple, because after all, it’s simple to just have one kernel26 package that’s updated.

  • Aaron Griffin

    Actually, this doesn’t sound that bad, really. It’s one package, just renamed every time, with a skeleton package to make upgrades seamless.

    One additional benefit is that I can upgrade, and if it doesn’t work, I can still load the previous kernel as it’s still there.

    The major disadvantage is that it’s now up to the user to remove old kernels periodically.

    And a lateral point: this actually won’t happen all that often. kernel-2.6.31 will exist in numerous upgrades until 2.6.32 comes out. So the maintenance cost of this technique will only appear every month or so…

  • TioDuke

    With the current schema, if anything goes wrong, you just:
    1- Boot using a live cd,
    2- chroot to your arch install,
    3- issue a pacman -Su /var/cache/pkgs/kernel-xx.xx.xx.pkg.tar.bz

    And you are back to the working kernel.

  • losinggeneration Post author

    @TioDuke: True, but that’s not as ideal. What if you were traveling while this happened. All of a sudden, you computer doesn’t boot, and you have no way to boot until you get back (just an example off the top of my head.) Being able to just change two lines in grub seems like a better option IMO.

  • Krayon

    The first thing I think of when reading this is a server you have in a data center. A lot of servers have the ability to access them over network, even during boot (BIOS etc) however they don’t, to my knowledge, generally include a robot arm to put a boot CD in 😛 (yes I know, you could have a USB device or a partition ready to alternatively boot to, but that’s not as funny a thought).

  • xlq

    Just make a copy of a kernel you know works to /boot/bzImage-works and add an entry to your bootloader for it. As long as it has a different filename, pacman won’t touch it. It might not be the latest, but you know it works!

    (Don’t forget initramfs and /lib/modules)

    • losinggeneration Post author

      Here are two quotes from one Arch devs via linked issue:
      “Comment by Aaron Griffin (phrakture) – Saturday, 17 October 2009, 11:39 GMT-4

      Additionally, lts is a different kernel. This mechanism allows for much easier and clearer upgrade paths and testing of new kernels, whereas the lts kernel follows a different path, and releases less often”

      “Comment by Thomas Bächler (brain0) – Saturday, 17 October 2009, 14:21 GMT-4
      lts is way too old, it is not even guaranteed to work right with Arch lon-term (for example, udev will soon break).”

      I should probably do a small edit to note that, while the original post is still relevant, the issue is being worked on (but is currently low priority.) There’s already a basic game plan for its implementation.

  • Lit1s

    # Configuration file. Number of old kernels to keep installed
    /etc/default/linux-kernel-auto-purge-olds -> keep_kernels=4


    new kernel -> install it with version number, and activate in grub. don’t erase nothing.

    number.actually.installed.kernels > /etc/default/linux-kernel-auto-purge-olds.keep_kernels
    purge oldest kernel

  • Andrew

    Yes, this is a problem, but it’s extremely easy to solve, just install linux-lts as a backup kernel and configure the boot loader to show it but not make it the default one.
    If the new kernel doesn’t work, boot with linux-lts and downgrade to the old linux package in the cache.

    And if your main kernel it’s the lts one then… maybe you can do the same thing in reverse.

  • An Innocent Bystander

    Does anyone have any temporary workarounds for this? The only solution I can think of is:
    1 Block kernel updates from official package.
    2 Have a daily/weekly script that looks for the current version of the linux package in pacman’s db. This will likely be run with the typical daily maintenance script that allready includes a pacman -Sy.
    3 Write a pkgbuild that creates a versioned kernel like what you suggest #AllanBrokeIt do using data from previous script. Should be straightforward to automate this as well.
    4 Write a pkgbuild that depends on your new pkgbuilds. Kept in a local repository.

    Does this make sense, or have I lost my marbles?

    • archuser

      i have /boot on a different partition–which is not automatically mounted when the system starts up– ie: the boot directory in the root folder will not effect that partition or my boot sequence. So after a kernel update, i mount my boot partition, copy over the files (keeping the old ones there as a backup)

      since a backup kernel is no good without modules, i run pacman -R –dbonly linux to ‘uninstall’ the current kernel before performing an update. this lets pacman think its uninstalled, but keeps all files in place (like the modules which are in a version named folder) and i remove the kernel and initrd from the folder ‘boot’ in the root partition not the boot partition [so pacman will not complain] then “pacman -S linux”

      Alternatively, you can simply install “linux-lts” as your backup, but i prefer to keep what i have installed as the backup when i install new–so its only 1 version behind, so i do the above mess instead of ‘keeping it simple’
      also, i have a usb drive that i can boot off of to fix it if i screw something up…

      it should be noted, that you do not have to let pacman manage your kernel, you are free to have other kernels in your boot partition and select any of them from grub/lilo/refind so you can simply ignore the kernel that pacman installs and compile a custom kernel at anytime, and use your kernel to boot. This is sometimes prefered anyway, since you can make a hardware specific kernel/modules/initrd for mostly bragging rights, and possibly an insignificantly second shorter boot time.


Speak Your Mind

Your email address will not be published. Required fields are marked *

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>