I'm in the process of testing several SMGL upgrade strategies. I'll outline the general strategies below, and provide updates as results come in. Please feel free to add to the discussion!

Basically, there are four update strategies I'm testing:

1) Single-command upgrade

Running sorcery -s is intended as a single pass SMGL upgrade. It is a wrapper around sorcery update, scribe update, sorcery queue and cast --queue, which update the sorcery scripts, update the grimoire, generate a list of spells that have newer versions in the updated grimoire vs installed (queue), and cast the queue. It is my impression that this particular update strategy is the largest obstacle to generating a current stable release of grimoire due to some of the major upgrades and ABI issues that have occurred since the last stable grimoire release in 2017. It is my opinion that this upgrade path is most suitable for regular daily/weekly upgrades and security upgrades--say via a cron job--rather than a substantial system wide update, especially if the target system has spells that are several update versions behind.

2) Targeted upgrade

In this strategy, each component is upgraded individually or in groups in a more targeted manner. sorcery update is run to update sorcery. scribe update is run to update grimoire. sorcery queue is run to generate a list of upgradeable spells. Running gaze version $(gaze -q installed | cut -d: -1 | sort | xargs) will provide a list of currently installed spell versions vs grimoire versions from which the user can determine which spells they would like to update. A recommended order of upgrades may be able to be identified--such as toolchain + critical tools (bash, tar, etc)-- to get the system to a base level after which sorcery -s can be run to update the remainder of the system.

3) Package style upgrade

In this strategy, spells are cast in a "DESTDIR" style using INSTALL_ROOT and STATE_ROOT. The spell is built but not installed to the target system. Later the spell can be installed using resurrect and the cache file created during the cast. The benefit of this approach is the package contents can be inspected before being installed on the system which could reduce the chances of borking the system with a full cast on a bad spell. Also, groups of packages that may create dependency or linking issues can be pre-built together and then later resurrected in a way that sidesteps those issues. One drawback is, if a spell that is to be cast depends on another spell that is to be cast, the cast will fail because the dependency spell will not have been installed on the target system.

4) Chroot upgrade

This strategy is similar to package style upgrade except the spells are cast in a chroot environment. This strategy could be particularly useful in a system that is several versions behind in updating and certain intermediate steps are required in a sorcery -s or targeted upgrade, such as an intermediary version of gcc needs to be installed to build the current grimoire version. Chroot upgrade works by installing the current chroot image and casting spells in the chroot using INSTALL_ROOT and STATE_ROOT. The caches can then be resurrected to the target system. An entire system should be able to be upgraded using this strategy without the need for any extra code in the grimoire to accommodate certain intermediary version requirements, dependency or linking issues. One caveat to this upgrade strategy is the resurrect may not work with the target system online. It may be necessary to boot into a live iso, mount the target system and resurrect the spells to the mounted system. In this case certain processing may be required on the cached packages, namely setting the correct values for INSTALL_ROOT and STATE_ROOT in the tablet roots files embedded in the cache file. This requires extracting each cache, running a sed command to set the correct INSTALL_ROOT and STATE_ROOT values, then recompressing the cache.

Final (for now) thought on upgrades:

One thing IMO that SMGL does backwards during upgrades is dispelling the old spell before installing the new spell. I believe a better approach is to install the new version of the spell over the top of the old spell then remove any unused files from the previous version. This same sentiment seems to be reflected by sobukus during the discussion of this bug report: https://bugs.sourcemage.org/show_bug.cgi?id=16100

This could be fairly easily implemented at the grimoire level, either a custom grimoire or for testing purposes, by bypassing the default_pre_install (remove from PRE_INSTALL file using sed or echo "true" > PRE_INSTALL if it doesn't exist) and adding a function call to FINAL that will read each line in the install log of the previous version, grep the line in the current version install log, and if no match is found rm the file.

lol well I already messed up as it's sorcery -s not sorcery -c

Upgrade Test Setup

Here is the setup I'm using for testing the upgrade strategies:

Hardware is a Dell E6510 I got off ebay for <$100. The 500GB hard drive is partitioned into 6 75G partitions so I can install up to six separate OSs. In this case sda5 has slackware for installing the chroots, sda6 has chroot .62 (SMGL stable) and sda7 has chroot .63 (SMGL test).

To set up SMGL stable and test, I extract the appropriate chroot tarball to the appropriate partition, mount the kernel file systems and chroot in. First thing we need is a kernel, so cast linux. After that completes I'll cast -r linux then get into the menuconfig to add graphics and network drivers (nouveau and iwlwifi module for firmware loading). Then cast wpa_supplicant so I can connect the wifi. After that is complete I'll run the cleanse suite to clean up the system, exit chroot, copy iwlwifi firmware from slackware to SMGL and reboot. I have grub installed on sda5 slackware so nothing needs to be done with the bootloader. I also generally use grub command line rather than a config file, so to boot SMGL stable it's >set root=(hd0,msdos6) >linux /boot/vmlinuz root=/dev/sda6 >boot. If everything goes smooth we get a login prompt.

...and I forgot to set a root password so back to slackware, chroot to sda6 and run passwd.

Ok now we're in. Setup wifi with wpa_passphrase ssid password > /etc/wpa_supplicant.conf, edit /etc/sysconfig/wpa_supplicant to set the proper interface, edit /etc/network/interfaces to setup ip addressing, then /etc/init.d/runlevels/%3/wpa_supplicant restart, ifup wlan0 to connect to the network and ping sourcemage.org to test.

Once the networking is setup we'll backup the system. Reboot to slackware, mount sda6 and tar -cJpf ~/smgl-stable-backup.tar.xz. Now that we can restore at anytime it's time to break SMGL.

Single-pass Upgrade Test

So the first test to run against SMGL stable is the single pass upgrade sorcery -s. First we need to update the grimoire to test so run scribe remove stable and scribe add test. Then run sorcery -s and sit back and watch the magic!

Except what happens isn't very magical. The first pass cast a few spells, failed a few more and dropped the majority. Since the system didn't seem broken I went ahead and ran cast --queue. This time it broke. A few spells cast successfully then linker errors on libgmp.so.3 file does not exist. Searching the system, libgmp.so.3 certainly does not exist but libgmp.so.10 does. So it appears the gmp upgrade broke the system when it upgraded from so.3 to so.10 and deleted so.3 which other programs are still using (not sure which ones and not interested in chasing that down).

One solution I can see for this is to bypass the dispel so that so.3 does not get removed. Then the dependent spells cast before gmp upgrade will still be linked to so.3 while spells cast after gmp upgrade will (should) be linked to so.10 and in the meantime binaries that need it will still have so.3 to link to. Another pass (sorcery -r) will need to be made to rebuild the spells before gmp upgrade to link to so.10. Then the files that are in the previous version gmp install log but not the new version install log will need to be removed. I'll call this a differential dispel.

And keep in mind, gmp breaking the system here was only a matter of timing. There are many many other library upgrades that will cause the same--or worse--issues. Fixing gmp will be like cutting the head off hydra; another one will grow in to take its place. Even if every library breakage were able to be accounted for, the system would likely need to be sorcery -r over and over again to straighten out the mountain of dependent dynamic linkages. Additionally, there is no guarantee that any given binary will actually link to the newer version of a library if both new and old libraries are present. To me it seems like a losing battle trying to fix this in the grimoire. It appears Gentoo uses a preserve-libs strategy to account for this, so maybe something like this could be implemented: https://wiki.gentoo.org/wiki/Preserve-libs

However, in my mind, the best way given the current code base and dev time-to-contribute to really solve this issue is via a well documented chroot upgrade strategy.

Create Packages

Creating packages is easy: cast -c <spell>. Assuming sorcery is configured to create cache files, casting spells will create a "package" file in the /var/cache/sorcery directory. These cache files are used to resurrect a spell and contain a compressed archive of all the files installed during the cast. To view the contents of the cache file simply extract it to a directory.

What "packages" do we need to create in order to upgrade our system? Well in a perfect world, we would run gaze installed > gaze_installed then do cast -c $(cat gaze_installed | xargs) in our chroot to build all the packages we need. However there is some "nuance" in the grimoire that prevents us from doing this. One of the bigger culprits is gnupg. gnupg is used to validate the signatures on summoned spells. When gnupg breaks you can no longer cast spells as they fail in the PRE_BUILD stage. Along the same lines is the sorcery-pubkeys spell that can break casting.

To get a list of currently installed spells run gaze installed | cut -d: -f1 | sort > gaze_installed. Remove the stable grimoire (scribe remove stable) and install the test grimoire (scribe add test). Now let's look at what spell versions have changed by running gaze version $(cat gaze_installed | xargs). This will give a table that shows the spell, installed version and grimoire version. From this you can make a better decision on what modifications and additions and subtractions need to be made to the gaze_installed file which will be used for casting the upgrade spells in the chroot. For example you'll see that gnupg has an installed version of 1.4.22 and a grimoire version of 0. We should modify this in the gaze_installed file from gnupg to gnupg-2.4. It may take a bit of trial and error to successfully cast all the upgrade spells.

Once the gaze_installed file is satisfactory, go ahead and enter the .63 chroot and run cast -c $(cat gaze_installed | xargs) to build the packages.

I just want to note at this point that, in my always humble opinion, this whole process of trying to upgrade a many year old stable system is both trying and tedious. In my mind it should not be accommodated for in the grimoire, nor supported by the Sourcemage dev team. Sure it's fun and challenging to try and get this to work, but in my mind it ends up creating a big mess in the grimoire, like having multiple versioned spells of gnupg and gcc to make it work, rather than a single gnupg or gcc spell that gets version updates. The best path forward in my opinion at this point is to rebuild and clone your system using the new .63 "test". The way I do this is I always create at least 2 root partitions. Current stable would be running in root1 and then I would clone a .63 system in root2. This allows me to get the root2 system up to snuff while still having the root1 system available if I need it. I can also use one or the other for a chroot host in case something breaks. Of course the back and forth booting from root1 to root2 would not be acceptable on a production system, but if you're running a production system with software as out-of-date as SMGL stable-especially one connected to the internet, well I don't know what to tell you there.