Patching Compute Instance Fleets with Oracle Cloud (OCI) OS Management

What you should know: I am currently working for Orbit Cloud Solutions as Cloud Advisor, but any posts on this blog reflect my own views and opinions only.

An annoying thing when you cannot have immutable infrastructure is that you need to patch and update your compute instances from time to time. In the worst case this needs to be done manually, a painful and error-prone process. Of course there are tools to do that in a more sensible way, one of them being the OS Management tooling that is available for free in Oracle Cloud.

This is working really well to keep you up to date with the latest patches. But if you want to do validation and testing of the patches and updates to be installed this is not supported out of the box. In their blog Oracle A-Team showed a sample implementation of what they call a patch train in Python. In my blog post i want to show you how to do similar tasks using the OCI CLI.

Patch Trains

The patch train idea can be easily described: Start with defining patch bundles, i.e. a set of exact packages including their version. Then apply the patch bundle to a testing environment and test it. If test shows good results, promote patch to next environment. And you do this in your testing environments until you are ready for applying the bundle to the production environment.

So let us start trying to implement this in an OCI tenancy.

Environment

Enabling OS Management

OS Management will not out of the box in your tenancy. You first need to create a dynamic group and set some policies. This requires some administration rights on tenancy level.

Reference/Template Instances

Once OS Management is enabled, it is time to start thinking about the way to define a target configuration of packages and versions. Probably the easiest way to do this is to prepare a reference or template instance that have the patches, updates or other packages installed that you plan to propagate to your environments.

Caveat: Custom Software Sources that are needed for implementing the strategy described in this post are currently only supported for Oracle Linux instances.

So just create a new compute instance that uses a supported Oracle Linux version that has OS Management enabled. This will be our template still having all the default software sources attached.

Custom Parent Software Source

The next thing we need to prepare is a custom parent software source. This will later have the custom software sources attached that can be applied to specifc managed instance groups.

oci os-management software-source create --compartment-id $COMPARTMENT_OCID --display-name $PARENT_SOURCE_NAME --arch-type X86_64

It seens that this currently needs to be done via CLI or SDK. At least i wasn’t able to figure out how to create a new empty parent source in the Console. But I always recommend to use scripts for repeatable administrative tasks anyway.

Creating Managed Instance Groups

Create a managed instance group for each stage you plan to include in your patching process. This can easily be done using Console or CLI.

oci os-management managed-instance-group create --compartment-id $COMPARTMENT_OCID --display-name $MIG_NAME

I recommend to create a special managed instance group for reference/template instances. These instances must keep on using the standard software sources. The activities described in the section on adding new instances to managed instance groups must not be performed on those reference/template instances.

Adding Managed Instances to Managed Instance Groups

All we got so far are empty managed instance groups. Now it is time to add some Oracle Linux managed instances to the managed instance groups.

oci os-management managed-instance-group attach --managed-instance-group-id $TARGET_GROUP_OCID  --managed-instance-id $INSTANCE_OCID

Then we need to detach the standard parent software source.

CURRENT_PARENT_SOURCE_OCID=`oci os-management managed-instance get --managed-instance-id $INSTANCE_OCID --query 'data."parent-software-source".id' | jq . -r`
oci os-management managed-instance detach-parent --managed-instance-id $INSTANCE_OCID --software-source-id $CURRENT_PARENT_SOURCE_OCID

And eventually attach the empty custom parent software source that was created before.

oci os-management managed-instance attach-parent --managed-instance-id $INSTANCE_OCID --software-source-id $TARGET_PARENT_SOURCE_OCID

If you check the available updates for one of the managed instances, you should see that there are neither available updates nor packages.

Defining the Target Configuration

The environment has been set up, so we now can create the first patch/package bundle.

Install Packages and Updates on Reference/Template Instance

One thing that I need to highlight when talking about creating a target configuration is that with your choice of packages and their versions you are limited to what is available in the Oracle Linux software sources that are provided in OCI. There currently is no way to add your own repository to use for distributing you own packages.

So install those packages using the OCI OS Management tools and turn on the engines for some smoke testing.

For smoke testing you can use a simple script that you execute using Oracle Cloud Agent Commands. I really recommend taking a look at this neat feature in OCI compute.

Extracting the Target Configuration

If everything works reasonably well, we can move on to extract the list of installed packages from the template instance to create the target configuration as a json document.

oci os-management managed-instance list-installed-packages --managed-instance-id $TEMPLATE_INSTANCE_OCID --all --query 'data [?"software-sources"] | [*].name'` > packages_target.json

This now is the full list of all installed packages on the template. Some special packages that are used for cloud-specific purposes will be omitted.

Creating the Custom Child Software Source

As a container for all the patches and packages we will create a custom child software source.

SOURCE_OCID=`oci os-management software-source create --compartment-id $COMPARTMENT_OCID --parent-id $PARENT_SOURCE_OCID --display-name $SOURCE_NAME --arch-type X86_64 --query "data.id" | jq . -r`

The custom child software source now is attached to the empty parent software source created initially.

Adding the Packages to the Custom Child Software Source

Now the packages that have been extracted into a json document before can be added to the custom child software source.

oci os-management software-source add-packages --software-source-id $SOURCE_OCID --package-names file://packages_target.json

Please make sure to always add the packages to the child software source, not the parent software source.

Preparing Stage Instances

To have the newly created child software source actually being used for a managed instance, it then needs to be attached to that instance first. Even though you can have multiple child software sources, I recommend to detach older child software sources from your instances. This makes it much clearer which patches and updates you expect to see installed on the managed instances.

So pick the managed instance group you want to apply he configuration to. Get all the instances in your group and pick one to start with.

oci os-management managed-instance-group get --managed-instance-group-id $TARGET_GROUP_OCID --query 'data."managed-instances"[].id' | jq .[] -r

First get the currently attached child software sources.

oci os-management managed-instance get --managed-instance-id $TARGET_INSTANCE --query 'data."child-software-sources"[].id' | jq .[] -r

For each source returned, repeat the following step to detach the source.

oci os-management managed-instance detach-child --managed-instance-id $TARGET_INSTANCE --software-source-id $CURRENT_SOURCE_OCID

Then attach the new child software source that has been created before.

oci os-management managed-instance attach-child --managed-instance-id $TARGET_INSTANCE --software-source-id $SOURCE_OCID

Doing this manually using the CLI for each managed instance in the managed instance group is not much fun, so either prepare a script for these tasks or use the Console.

Aligning the Fleet Configurations

Checking for Deviations

The first thing you probably want to do before running any update jobs is to find out if a specific instance does not comply with the target configuration that should be installed. As we can easily fetch the packages included in a certain software source and the packages installed on an instance this should not be that hard.

oci os-management software-source list-packages --software-source-id $TARGET_SOURCE_OCID --all --query 'data[].name' > packages_target.json

oci os-management managed-instance list-installed-packages --managed-instance-id $TARGET_INSTANCE --all --query 'data [?"software-sources"] | [*].name' > packages_installed.json

Using the always helpful tool jq we can extract packages not installed on the instance:

jq -n --argfile a packages_target.json  --argfile b packages_installed.json '{"target":$a, "instance":$b} | .target-.instance'

Or we can extract packages that are installed, but actually should not be there. Of course this will show the packages that have not been patched or updated up to the target level.

jq -n --argfile a packages_target.json  --argfile b packages_installed.json '{"target":$a, "instance":$b} | .instance-.target'

Installing Packages and Updates

Installation on Managed Instance Groups using Scheduled Jobs

Updating a managed instance group can be done by scheduling a job. This job then is executed once or on a regular schedule, for the current description i will focus on onetime update jobs only. All you basically need is pass compartment and target managed instance group to the following command. It then installs all updates available from the software sources attached to the managed instances.

oci os-management scheduled-job create --compartment-id $COMPARTMENT_OCID \
--display-name "${SOURCE_NAME} Update All" --operation-type UPDATEALL --update-type ALL \
--schedule-type ONETIME --time-next-execution `date -v+5M +%s` \
--managed-instance-groups "[{ \"displayName\": \"${TARGET_GROUP_NAME}\",  \"id\": \"${TARGET_GROUP_OCID}\"}]" \
--wait-for-state ACTIVE --wait-for-state FAILED --query 'data."lifecycle-state"'

The update-type parameter will enable a more targeted installation of updates, letting you specify one of the following types ALL, BUGFIX, ENHANCEMENT& SECURITY. You as well could use an operation-type of UPDATE only that will let you do installations of individually specified updates.

The general installation of packages on a managed instance group is pretty simple too. Create a json structure containing the list of packages to install and then schedule a job to install them. (Kudos to Oracle for having different json structures for the list of packages to add to a software source and to pass to a scheduled job.)

TARGET_PACKAGES=`oci os-management software-source list-packages --software-source-id $TARGET_SOURCE_OCID --all --query 'data[*].{name: name}'`

oci os-management scheduled-job create --compartment-id $COMPARTMENT_OCID \
--display-name "${SOURCE_NAME}" --operation-type INSTALL \
--schedule-type ONETIME --time-next-execution `date -v+10M +%s` \
--managed-instance-groups "[{ \"displayName\": \"${TARGET_GROUP_NAME}\",  \"id\": \"${TARGET_GROUP_OCID}\"}]" \
--package-names $TARGET_PACKAGES \
--wait-for-state ACTIVE --wait-for-state FAILED --query 'data."lifecycle-state"'

This code will create a job scheduled to run once in 10 minutes.

Installation on individual Managed Instances

While being much more tedious to do for whole managed instance groups, i found in my testing environment that using a more direct installation approach gave me much more immediate feedback if something went wrong. But for real world scenarios the best probably would be to stick to scheduled tasks.

First simply install all available updates. The only updates available will be those that have been installed on the reference/template instance.

oci os-management managed-instance install-all-updates --wait-for-state SUCCEEDED --wait-for-state FAILED --managed-instance-id $TARGET_INSTANCE --query 'data[].resources[].name'

Then install all packages that are missing from the target instance. Identify those packages as described above in the section on finding deviations.

oci os-management managed-instance install-package --package-name $PACKAGE --wait-for-state SUCCEEDED --wait-for-state FAILED  --managed-instance-id $TARGET_INSTANCE --query 'data.status'

Removing Packages

You can install packages using OS Management, but of course you can remove packages as well. I found that simply removing the packages that were identified when checking the deviations does not work really well as that will return some protected packages and in consequence the removal operation will fail. You should really hand-pick the packages to remove.

Moving through the Stages

Once the patches, updates and other packages are installed on the managed instances in your managed instance group, it is time to do your testing to validate that the new configuration does not impact the applications and components deployed on those instances. Depending on the maturity of your processes this can be done mostly automated up to the production stage. In the majority of cases you will have your typical stages moving from test environments over quality assurance environments up to the production environment.

Conclusion

OCI OS Management is a long needed solution for the common problem of patching fleets of compute instances without having to rely on a 3rd party toolset like Ansible. While not as complete and sophisticated as e.g. AWS Systems Manager I find the current implementation very sound and useful for these types of tasks.

One drawback is that currently Custom Software Sources are supported on Oracle Linux. For Windows you will only be able to install the latest patches available at the time you push the button – you cannot follow a multi-stage strategy of testing the patches in several environments before applying them to the production environment.

This leads to the next drawback. You cannot have your own custom package repositories used in custom software sources. So no way to distribute your own packages automatically via OCI OS Management.

Another drawback is that the available GUI in the Console is a bit rough around the edges. Doesn’t hurt that much though as i’d go for using CLI and/or scripts anyway.

So if you are aware of the limitations of OCI OS Management, i really believe it will be a great addition to your OCI toolbelt for installing packages and applying patches to your compute instances.