Complex Amplify VTL resolver writing and testing

Let's start with a problem: you want to implement a profile editing feature, your app is a VueJS frontend and an Node AppSync GraphQL backend, and you database layer is a set of DynamoDB tables. As for the implementation, the AppSync resolvers are a mix of Javascript Lambdas and Apache VTL resolvers.

The profile editing requires dynamic data inputs: when editing a profile you can edit one feature, none of them, or all of them.

Here's a simple DynamoDB.UpdateItem command example.

{
    "version" : "2018-05-29",
    "operation" : "UpdateItem",
    "key": {
        "id" : $util.dynamodb.toDynamoDBJson($context.identity.username)
    },

    "update" : {
        "expression" : "set #name = :name, imgUrl = :imgUrl, bgImgUrl = :bgImgUrl, bio = :bio, #location = :location, website = :website, birthdate = :birthdate",
        "expressionNames" : {
           "#name" : "name",
           "#location": "location",
       },

       "expressionValues" : {
            ":name" : $util.dynamodb.toDynamoDBJson($context.arguments.newProfile.name),
            ":imgUrl" : $util.dynamodb.toDynamoDBJson($context.arguments.newProfile.imgUrl),
            ":bgImgUrl" : $util.dynamodb.toDynamoDBJson($context.arguments.newProfile.bgImgUrl),
            ":bio" : $util.dynamodb.toDynamoDBJson($context.arguments.newProfile.bio),
            ":location" : $util.dynamodb.toDynamoDBJson($context.arguments.newProfile.location),
            ":website" : $util.dynamodb.toDynamoDBJson($context.arguments.newProfile.website),
            ":birthdate" : $util.dynamodb.toDynamoDBJson($context.arguments.newProfile.birthdate),
        }

    },

    "condition" : {
        "expression" : "attribute_exists(id)"
    },
}

The AppSync GraphQL query looks a bit like this.

const query = `mutation editMyProfile($input: ProfileInput!) {
        editMyProfile(newProfile: $input) {
          ... myProfileFields

          tweets {
            nextToken
            tweets {
                ... on Tweet {
                    ... tweetFields
                }
            }
          }
        }
    }`

And the ProfileInput type has this shape.

input ProfileInput {
    name: String!
    imgUrl: AWSURL
    bgImgUrl: AWSURL
    bio: String
    location: String
    website: String
    birthdate: AWSDate
}

Within ProfileInput, only name is required, with the remaining values being optional.

In the VTL template we wrote the command as if that all the attributes exist at execution time. When they are missing DynamoDB will override those values with 'null', a significant issue that leads to data loss.

Best practice for DynamoDB updates

Updates is DynamoDB tend to require complex queries because they have a large amount of requirements, and come with the added risk of data loss.

Officially AWS recommends that an Update query handles these situations:

  • handles each argument to produce a list of required updates
  • skips over any guaranteed arguments
  • if an argument is null or empty (""), that argument is removed from the item
  • if an argument is missing from the item, that argument is added
  • if an argument exists on the item, that argument is set to the new value

Even though this is true for most DynamoDB update commands, the example below is an AppSync specific one.

These scenarios produce 3 sets of instruction lists around the directive verbs: remove, set, add.

The expression itself is empty and we then use the lists to populate the expression, expressionNames, and expressionValues accordingly.

AWS AppSync, Mapping Template Resource / DynamoDB.UpdateItem example

{
    "version" : "2017-02-28",

    "operation" : "UpdateItem",

    "key" : {
        "id" : $util.dynamodb.toDynamoDBJson($ctx.args.id)
    },

    ## Set up some space to keep track of things we're updating **
    #set( $expNames  = {} )
    #set( $expValues = {} )
    #set( $expSet = {} )
    #set( $expAdd = {} )
    #set( $expRemove = [] )

    #foreach( $entry in $context.arguments.entrySet() )
        #if( $entry.key != "name" )
            #if( (!$entry.value) && ("$!{entry.value}" == "") )
                ## If the argument is set to "null", then remove that attribute from the item in DynamoDB **

                #set( $discard = ${expRemove.add("#${entry.key}")} )
                $!{expNames.put("#${entry.key}", "$entry.key")}
            #else
                ## Otherwise set (or update) the attribute on the item in DynamoDB **

                $!{expSet.put("#${entry.key}", ":${entry.key}")}
                $!{expNames.put("#${entry.key}", "$entry.key")}

                #if( $entry.key == "ups" || $entry.key == "downs" )
                    $!{expValues.put(":${entry.key}", { "N" : $entry.value })}
                #else
                    $!{expValues.put(":${entry.key}", { "S" : "${entry.value}" })}
                #end
            #end
        #end
    #end

    ## Start building the update expression, starting with attributes we're going to SET **
    #set( $expression = "" )
    #if( !${expSet.isEmpty()} )
        #set( $expression = "SET" )
        #foreach( $entry in $expSet.entrySet() )
            #set( $expression = "${expression} ${entry.key} = ${entry.value}" )
            #if ( $foreach.hasNext )
                #set( $expression = "${expression}," )
            #end
        #end
    #end

    ## Continue building the update expression, adding attributes we're going to ADD **
    #if( !${expAdd.isEmpty()} )
        #set( $expression = "${expression} ADD" )
        #foreach( $entry in $expAdd.entrySet() )
            #set( $expression = "${expression} ${entry.key} ${entry.value}" )
            #if ( $foreach.hasNext )
                #set( $expression = "${expression}," )
            #end
        #end
    #end

    ## Continue building the update expression, adding attributes we're going to REMOVE **
    #if( !${expRemove.isEmpty()} )
        #set( $expression = "${expression} REMOVE" )

        #foreach( $entry in $expRemove )
            #set( $expression = "${expression} ${entry}" )
            #if ( $foreach.hasNext )
                #set( $expression = "${expression}," )
            #end
        #end
    #end

    ## Finally, write the update expression into the document, along with any expressionNames and expressionValues **
    "update" : {
        "expression" : "${expression}"
        #if( !${expNames.isEmpty()} )
            ,"expressionNames" : $utils.toJson($expNames)
        #end
        #if( !${expValues.isEmpty()} )
            ,"expressionValues" : $utils.toJson($expValues)
        #end
    },

    "condition" : {
        "expression"       : "version = :expectedVersion",
        "expressionValues" : {
            ":expectedVersion" : $util.dynamodb.toDynamoDBJson($ctx.args.expectedVersion)
        }
    }
}

Solution

For our app this means that whenever we pass a profile with an undefined value we will remove the existing values from the item.

Again we hit the data loss problem, however we can stop building the 'remove' instructions lists for undefined values.

#set( $expNames  = {} )
#set( $expValues = {} )
#set( $expSet = {} )
#set( $expRemove = [] )

#foreach( $entry in $context.arguments.newProfile.entrySet() )
    #if( $entry.key != "name" )
        #if( (!$entry.value) && ("$!{entry.value}" == "") )
            ## If the argument is set to null, then remove that attribute from the item in DynamoDB **

            #set( $discard = ${expRemove.add("#${entry.key}")} )
            $!{expNames.put("#${entry.key}", "$entry.key")}
        #else
            ## Otherwise set (or update) the attribute on the item in DynamoDB **

            $!{expSet.put("#${entry.key}", ":${entry.key}")}
            $!{expNames.put("#${entry.key}", "$entry.key")}
            $!{expValues.put(":${entry.key}", { 
                "S": "${entry.value}"
            })}
            
        #end
    #else
        $!{expSet.put("#${entry.key}", ":${entry.key}")}
        $!{expNames.put("#${entry.key}", "$entry.key")}
        $!{expValues.put(":${entry.key}", { 
            "S": "${entry.value}"
        })}
    #end
#end

## Start building the update expression, starting with attributes we are going to SET **

#set( $expression = "" )
#if( !${expSet.isEmpty()} )
    #set( $expression = "SET" )
    #foreach( $entry in $expSet.entrySet() )
        #set( $expression = "${expression} ${entry.key} = ${entry.value}" )
        #if ( $foreach.hasNext )
            #set( $expression = "${expression}," )
        #end
    #end
#end

## Continue building the update expression, adding attributes we are going to REMOVE **
#if( !${expRemove.isEmpty()} )
    #set( $expression = "${expression} REMOVE" )

    #foreach( $entry in $expRemove )
        #set( $expression = "${expression} ${entry}" )
        #if ( $foreach.hasNext )
            #set( $expression = "${expression}," )
        #end
    #end
#end

{
    "version": "2018-05-29",
    "operation": "UpdateItem",
    "key": {
        "id" : $util.dynamodb.toDynamoDBJson($context.identity.username)
    },
    "update": {
        "expression" : "${expression}"
        #if( !${expNames.isEmpty()} )
            ,"expressionNames" : $utils.toJson($expNames)
        #end
        #if( !${expValues.isEmpty()} )
            ,"expressionValues" : $utils.toJson($expValues)
        #end
    },
    "condition" : {
        "expression" : "attribute_exists(id)"
    }
}

We continue to use the 'remove' lists for incoming values set to null or "" eg. user wants to change their website to an empty string.

We no longer handle any 'add' instructions because can handle new properties with 'set'.

With these changes we can now support the dynamic ProfileInput object and correctly make the database changes.

Testing

The testing of AWS AppSync VTL templates generically involves:

  • creating a mock context object with relevant identity and arguments (payload)
  • loading the VTL template given the mock context
  • verifying the VTL template expansion

When it comes to the data verification we are looking to check:

  • expression shape
  • expressionNames presence and values
  • expressionValues shape, presence, and values

If payload newProfile contains name, bio, website

  • Then the template expression should feature name, bio, website
  • Where all params in the expression are under the SET verb
  • The expressionNames should contain #name, #bio, #website
  • The expressionNames should have corresponding values
  • The expressionValues should contain :name, :bio, :website
  • The expressionValues should have the corresponding values
  • The expressionValues are correctly mapped as DynamoDB string ("S")

If payload newProfile contains name, bio, and bio is null or ""

  • Then the template expression should feature name, bio
  • Where *name is under the SET verb
  • Where *bio is under the REMOVE verb

Notes

  1. ExpressionValues are strings

One crucial thing to note is the discrepancy between the ProfileInput types and the DynamoDB expressionValues.

Even though some values are of type AWSUrl or AWSDate, when given to the expressionValue they must be of type string ("S"), otherwise the query will fail.

Please note that this issue will trigger a false positive in testing because template itself is still value regardless of the correct value of expressionValues

  1. Template evaluation with the AppSync API

AppSync testing can be done by combining the "@aws-amplify/amplify-appsync-simulator" and "amplify-velocity-template" packages.

This method is prone to errors, and requires some output parsing of the templates after they've been loaded by the simulator eg. trimming training commas from the output.

A more reliable option is the EvaluateMappingTemplateCommand from "@aws-sdk/client-appsync". This uses an AppSync Client to evaluate any template string for validity, and it's a locally executing command.

AWS Amplify EvaluateMappingTemplateCommand documentation

  1. VTL variations

You can also try a simpler loop for building the expression.

#set($exp = "SET #name = :name")
#set($expNames = {"#name":"name"})
#set($expValues = {":name": {"S": $ctx.args.newProfile.name}})

#set($fields = ["imgUrl", "bgImgUrl", "bio", "location", "website", "birthdate"])

#foreach($field in $fields)
  #if($ctx.args.newProfile[$field] && "$!ctx.args.newProfile[$field]" != "")
    #set($exp = "$exp, #${field} = :${field}")
    $util.qr($expNames.put("#${field}", "$field"))
    $util.qr($expValues.put(":${field}", {"S": "$ctx.args.newProfile[$field]"}))
  #end
#end

{
    "version": "2018-05-29",
    "operation": "UpdateItem",
    "key": {
        "id": $util.dynamodb.toDynamoDBJson($ctx.identity.username)
    },
    "update": {
        "expression": "$exp",
        "expressionNames": $util.toJson($expNames),
        "expressionValues": $util.toJson($expValues)
    },
    "condition": {
        "expression": "attribute_exists(id)"
    }
}

This is shorter, but will not handle scenarios where users require the removal of values eg. when a user wants to set the their existing profile website to "".

Documentation

Besides the AWS resources and the VTL project docs, I found the JEdit plugin writing guide to be excellent. The JEdit plugins are written in Java, but make use of VTL and the explanations of the various VTL directives are very useful and concise.

On testing the amplify-velocity-template module test suite is a great resource for ideas on how to test more complex VTL templates.

AWS AppSync resolver mapping template reference for DynamoDB

The Apache Velocity Project - Developer reference

JEdit - Plug-in writing developer guide, VTL Directives

Amplify Velocity Template - test suite