Fun with Functions - OCI Serverless Part 2: Go Functions and Events

What you should know: I am currently working for Orbit Cloud Solutions as Cloud Advisor, but any posts on this blog reflect my own views and opinions only.

This is the second part of my series covering aspects of Oracle Functions. To get an understanding of the environment we are working in, i recommend reading part 1 first if you haven’t done so already.

In this post i will cover a simple serverless function written in Go that generates thumbnails from images uploaded to an object storage bucket. We will use cloud events to trigger the execution of the function.

For all of this i prepared code and scripts to get everything running and published it on github.

Before proceeding with the hands-on example, you should have the following setup ready as described in part 1:

Environment variables TF_VAR_oci_compartment_ocid, TF_VAR_oci_region set (use your env.sh in infrastructure directory)
Environment variables from generated oci_env.sh set
fn CLI installed and context set (e.g. using fn_context.sh)
OCI CLI installed and configured
docker server running and successful docker login with your registry (OCI Docker Registry)
OCI environment as described in part 1: users & (dynamic) groups, policies, network & automation-app application

Building and Deploying the Function

OK, then we now can simply go to the subdirectory holding the event-create-thumb function code and simply deploy it to OCI:

cd functions/event-create-thumb
fn -v deploy -app automation-app

Note: If you get problems when building your own functions relying on newer modules or Go features, you might have to create your own docker build and runtime images. You can find a guide for this in my post on this topic.

Let’s take a look at the definition file func.yaml. I have added a config section containing a key-value pair of OCI_BUCKET_NAME to img. This config will allow to easily change the name of the object storage bucket.

schema_version: 20180708
name: event-create-thumb
version: 0.0.1
runtime: go
entrypoint: ./func
config:
  OCI_BUCKET_NAME: img

You can check yourself the config available to your function easily via CLI:

fn list config func automation-app event-create-thumb

The value will later be read in the Go function from an environment variable.

imgBucket = os.Getenv("OCI_BUCKET_NAME")

Gluing everything together using an Event Rule

Triggering the function after a new file has been uploaded to an object storage bucket can be achieved with a multitude of different techniques. Even though you could do it old school by having a cron job periodically check if a new file has appeared, we want to go the more elegant way by using events emitted by the bucket itself. For this to work, we need to enable the emitting of events for our bucket and if you have used my terraform scripts to set up, you should have everyting in place already.

resource "oci_objectstorage_bucket" "img_bucket" {
  compartment_id        = var.oci_compartment_ocid
  name                  = "img"
  namespace             = data.oci_objectstorage_namespace.os_ns.namespace
  object_events_enabled = true
  access_type           = "ObjectRead"
}

This now enables event rules to act on the events received from object storage. We simply set up a rule to execute a FaaS when this happens.

But as we do not want to deal with all events emitted by any object storage bucket in the tenancy, we have to consider some conditions for a rule handling those events:

{
    "eventType": ["com.oraclecloud.objectstorage.createobject",
        "com.oraclecloud.objectstorage.updateobject",
        "com.oraclecloud.objectstorage.deleteobject"
    ],
    "data": {
        "resourceName": ["*.jpg", "*.JPG", "*.png", "*.PNG", "*.gif", "*.GIF"],
        "additionalDetails": {
            "bucketName": ["img"]
        }
    }
}

So first of all, only handle createobject, updateobject and deleteobject events emitted from the bucket with name img. And then there is some very basic filter for image file extensions – so we want files with extensions indicating JPG, GIF or PNG format to be handled by our function.

Again, i put together a short script that will create the event rule process_images calling the event-create-thumb function for you. Simply run fn_event_rule.sh from the scripts directory.

./fn_event_rule.sh

The Go Function Code

Now let us take a deeper look into the function code. The code has 3 basic parts:

The main function is the starting point for all executions. Here just the dispatcher handler is registered.
The dispatcher handler will take input received, put it in a Go struct and then call the processing functions matching to the type of event (create, update or delete) that was triggering the call of the function.
The processing functions to handle the create/update and delete events. This is where the original image is read from object storage, converted to a thumbnail and written back. Or where any orphaned thumbnails get deleted.

Dispatcher: Working with OCI events format

A key element for the processing of the images is the data passed to the function from the event. OCI Events adhere to the Cloudevent format which defines some basic properties and an extension mechanism that is extensively used here. The data we are most intererested in will be found in that Data extension. For use in Go we will define a type that matches that format.

// EventsInput structure will match the OCI events format
type EventsInput struct {
    CloudEventsVersion string      `json:"cloudEventsVersion"`
    EventID            string      `json:"eventID"`
    EventType          string      `json:"eventType"`
    Source             string      `json:"source"`
    EventTypeVersion   string      `json:"eventTypeVersion"`
    EventTime          time.Time   `json:"eventTime"`
    SchemaURL          interface{} `json:"schemaURL"`
    ContentType        string      `json:"contentType"`
    Extensions         struct {
        CompartmentID string `json:"compartmentId"`
    } `json:"extensions"`
    Data struct {
        CompartmentID      string `json:"compartmentId"`
        CompartmentName    string `json:"compartmentName"`
        ResourceName       string `json:"resourceName"`
        ResourceID         string `json:"resourceId"`
        AvailabilityDomain string `json:"availabilityDomain"`
        FreeFormTags       struct {
            Department string `json:"Department"`
        } `json:"freeFormTags"`
        DefinedTags struct {
            Operations struct {
                CostCenter string `json:"CostCenter"`
            } `json:"Operations"`
        } `json:"definedTags"`
        AdditionalDetails struct {
            Namespace        string `json:"namespace"`
            PublicAccessType string `json:"publicAccessType"`
            ETag             string `json:"eTag"`
        } `json:"additionalDetails"`
    } `json:"data"`
}

This helps us to convert the input in as provided to the dispatcher to a matching Go structure we can actually work with more easily.

event := &EventsInput{}
json.NewDecoder(in).Decode(event)

The new event struct then will be used to determine the handler to use and also is passed on with the context.

switch event.EventType {
    case "com.oraclecloud.objectstorage.deleteobject":
        outMsg, err := handleDelete(ctx, event)
        if err != nil {
            log.Println(outMsg, err)
        }
    case "com.oraclecloud.objectstorage.createobject", "com.oraclecloud.objectstorage.updateobject":
        outMsg, err := handleCreateUpdate(ctx, imgType, event)
        if err != nil {
            log.Println(outMsg, err)
        }
    default:
        log.Fatalln("received unhandled event ", event.EventType)
    }

Processing Functions: Common Initializiation of Go OCI API

In both functions used to handle create, update or delete events there is some common code that first gets the configuration using resource principal. This is why we needed the dynamic groups to be defined during the setup described in part 1.

provider, err := auth.ResourcePrincipalConfigurationProvider()
if err != nil {
    log.Fatalln("Error: ", err)
}

We then can use the provider configuration to create an OCI object storage client which we use to fetch the namespace used as input for other functions.

osClient, err := objectstorage.NewObjectStorageClientWithConfigurationProvider(provider)
if err != nil {
    log.Println("Error: ", err)
    return
}

nsRequest := objectstorage.GetNamespaceRequest{}
nsResp, err := osClient.GetNamespace(ctx, nsRequest)
if err != nil {
    log.Println("Error: ", err)
    return
}

Processing Functions: Generate Thumbnail in Function handleCreateUpdate

When the function is triggered from an create or update event we use the object storage client to fetch the object from the bucket. Note that we use the resource name provided as input to the function now available as event.Data.ResourceName.

getReq := objectstorage.GetObjectRequest{
    NamespaceName: nsResp.Value,
    BucketName:    &imgBucket,
    ObjectName:    &event.Data.ResourceName,
}

getResp, err := osClient.GetObject(ctx, getReq)
if err != nil {
    log.Println("Error: ", err)
    return
}

We then convert the object using the github.com/disintegration/imaging module. First the image is decoded into common bitmap structure, then the image is resized to a width of 200 and finally encoded again into the image file format used by the original image.

orig, err := imaging.Decode(getResp.Content)
if err != nil {
    log.Println("Error: ", err)
    return
}

var b bytes.Buffer
imgWriter := bufio.NewWriter(&b)

thumb := imaging.Resize(orig, 200, 0, imaging.Lanczos)

if err = imaging.Encode(imgWriter, thumb, imgTypeMapped[extension]); err != nil {
    log.Println("Error: ", err)
    return
}
imgWriter.Flush()

Now that we got the thumbnail in the writer buffer, we write it as a new object to the bucket with a thumb/ prefix to the object name. Note that as there is no folder structure in an object storage bucket, we can use such a prefix to create some kind of structure to the consumer. So that you will have an URL like https://foo.com/thumb/badger.jpg for the thumbnail.

thumbName := "thumb/" + event.Data.ResourceName
thumbsize := int64(len(b.Bytes()))

request := objectstorage.PutObjectRequest{
    NamespaceName: nsResp.Value,
    BucketName:    &imgBucket,
    ObjectName:    &thumbName,
    ContentLength: &thumbsize,
    PutObjectBody: ioutil.NopCloser(&b),
    OpcMeta:       nil,
}
if _, err = osClient.PutObject(ctx, request); err != nil {
    log.Println(err)
    return
}

Processing Functions: Clean up deleted Images in Function handleDelete

And when the function is triggered via a delete event, we want to tidy up the bucket so that no orphaned thumbnail files remain after the original image has been deleted.

thumbName := "thumb/" + event.Data.ResourceName
deleteRequest := objectstorage.DeleteObjectRequest{
    NamespaceName: nsResp.Value,
    BucketName:    &imgBucket,
    ObjectName:    &thumbName,
}

if _, err = osClient.DeleteObject(ctx, deleteRequest); err != nil {
    log.Println("Error: ", err)
}

Testing everything together

So we got everything in place: a function to process and image, a bucket emitting events that are handled and a rule that triggers the function. Let us now do a quick check if everything is working as expected:

Upload a sample image and wait a few seconds. Then check the bucket contents, you should see the thumbnail appear after a while.

oci os object put --bucket-name img --file foo.jpg
oci os object list --bucket-name img

Delete sample image, but not thumbnails. Again, wait some time and check the bucket. The thumbnail should be removed.

oci os object delete --bucket-name img --object-name foo.jpg
oci os object list --bucket-name img

This concludes the second part of my series on serverless functions on OCI. We now got a working example but there is so much more interesting stuff to investigate. For example:

Add an API to write, update or delete images to the object storage bucket.
Do image analysis using Azure cognitive services and store the result in a backend (nosql) database.
Add an API to query images based on the results of the analysis

So there most likely will be more posts in this series:)

Fun with Functions – OCI Serverless Part 2: Go Functions and Events

Building and Deploying the Function

Gluing everything together using an Event Rule

The Go Function Code

Dispatcher: Working with OCI events format

Processing Functions: Common Initializiation of Go OCI API

Processing Functions: Generate Thumbnail in Function handleCreateUpdate

Processing Functions: Clean up deleted Images in Function handleDelete

Testing everything together

Like this: