Promised Dynamo Release

When I first started working with DynamoDB there were not many options for interacting with the database in Node.js outside the aws-sdk itself.  While the documentation for the SDK is notably thorough, I found the SDK to be difficult to work with along multiple dimensions.  

  • Difficult Data Structures - The data structures expected and exposed by the DynamoDB operations seemed cumbersome and non-intuitive.
  • Awkward Query / Update Syntax and Structure - The query syntax and structure needed to present property type information to Dynamo was difficult to understand and even more difficult to write correctly the first time.
  • Extremely Stringy - Strings are used for most operations exposed to end users.  Strings define query statements, update commands, various options, the table to be operated upon, etc.
  • Callbacks - Not necessarily difficult to work with, but I have become so accustom to working with promises that being forced to work with callbacks sets me back a moment or two.

Given this I started formulation of promised-dynamo, an open source wrapper for the aws-sdk DynamoDB driver.  This formulation was guided by the following criteria which I was looking for in a Dynamo driver.

  1. Consumption and exposition of common Javascript objects
  2. Usage of objects as query definitions / option definitions
  3. Fluent API around tables in use
  4. Promise based

Now there is a much more broad ecosystem of libraries supporting different DynamoDB operations but I have yet to find one which meets all of my criteria or serves all of the problem areas bulleted above.

Difficult Data Structures

It is hard to describe why the data structure is hard to work with without presentation of an example.  

{

  "familyName": {

  "S": "Johnson"

  },

  "givenName": {

  "S": "John"

  },

  "homeAddress": {

  "M": {

  "streetAddress": {

  "S": "123 Fake St."
  },

"state": {

  "S": "IL"

  },

"city": {

  "S": "Chicago"

  }

 }

},

"favoriteFoods": {

  "SS": ["Tacos", "Risotto", "Sushi"]

  }

}

                               fig 1

In the above we see data about John Johnson.  Notice that the property names are associated not with a value but with an object describing both the data type of the value and the value itself.  familyName for example is associated with an object having a single key “S”.  The “S” indicates that the value under that key, the actual value to associate with the familyName property, should be interpreted as a String.  This means that in order to access the value we need to know the name of the property as well as the type and explicitly spell that out in our code.

Ideally we would be working with the following object instead

{

  familyName: "Johnson",

  givenName: "John",

  homeAddress: {

      streetAddress: "123 Fake St.",

      state: "IL",

      city: "Chicago"

  },

  favoriteFoods: [

      "Tacos",

      "Risotto",

      "Sushi"

  ]

}

fig 2

All operations exposed on tables in promised-dynamo such as getItem and putItem take and produce objects in the form of the later figure, typical Javascript objects, making it easy to operate in this familiar territory.  promised-dynamo also exposes the mapping methods as DynamoDb.mapDynamoObjectToJavascriptObject and DynamoDb.mapJavascriptObjectToDynamoObject which are useful in cases such as operating on the payload of a DynamoDB Stream where data will be coming with DynamoDB semantics attached.

Awkward Query / Update Syntax and Structure

Again, an example of an Update will do well to illustrate the awkwardness involved with its issuance

 

{

"Key": {

"id": {

"S": "person1234"

}

},

"TableName": "promisedDynamoTest",

"UpdateExpression": "SET #A = :1 ADD favoriteFoods  :2",

"ExpressionAttributeValues": {

":1": {

"S": "Lorem ipsum dolor sit ..."

},

":2": {

"S": "Ramen"

}

},

"ExpressionAttributeNames": {

"#A": "about"

},

"ReturnConsumedCapacity": "INDEXES",

"ReturnValues": "ALL_NEW"

}

fig 3

The preceding, issued to the updateItem method of the aws-sdk, attempts to update the row identified by the key “person1234” by setting an about property to “Lorem ipsum dolor sit …” and adding the item “Ramen” to the favoriteFoods string set.  It is difficult, even looking at this object nicely formatted, to see exactly what is going on here largely due to the fact that the update itself is expressed using placeholders such as :1 and :2 which are defined lower in the ExpressionAttributeValues property.  Many of the expressions used in the aws-sdk follow this same format.

promised-dynamo defines a Javascript object structure for declaring update expressions and for declaring condition expressions in the case of a query, scan, conditional update, etc.  This allows the above to be expressed by passing the string key “person1234” and the object

{

  about: "Lorem ipsum dolor sit ...",

  ADD: {

      favoriteFoods: "Ramen"

  }

}

fig 4

to a table’s updateItem method.

Stringyness

The example illustrated in figure 3 already starts to indicate the stringyness of the operations on DynamoDB.  The update expression is a string with inferred mappings to the expression attribute values object issued against a table identified by its String name.  Writing this out by hand is rather error-prone and time consuming work, especially if you are new to working with the SDK.  

In addition to providing object structures for describing updates and query expressions as noted in the prior section, promised-dynamo takes at construction an array of table names to be used.  It then does the dirty work of looking up the table definition, understanding the structure of the table such as the keys and indices defined on it, and exposing all table operations via a property on the constructed driver.  Given that the above update boils down to the following code

var Dynamo = require( 'promised-dynamo' );



var dynamo = new Dynamo( {

  accessKeyId: "...",

  secretAccessKey: "...",

  region: "..."

}, [ "promisedDynamoTest" ] );



dynamo.promisedDynamoTest.updateItem(

 'person1234',

 {

   about: "Lorem ipsum dolor sit ...",

   ADD: {

      favoriteFoods: "Ramen"

   }

 },

 {

   returnConsumedCapacity: Dynamo.consumedCapacityOptions.INEXES,

   returnValues: Dynamo.valuesOptions.ALLNEW

 } )

 .then( function( result ) {

   …

 }

fig 5

As an alternative to an array of strings, the constructor may also be provided with an array of table alias descriptors

var dynamo = new Dynamo( {

  accessKeyId: "...",

  secretAccessKey: "...",

  region: "..."

}, [ { name: "promisedDynamoTest-" + environment, alias: "promisedDynamoTest" ] );

fig 6

This helps considerably in numerous cases such as the case where the full table name is based on environmental variables, allowing the table to be consistently referenced in the code base without having to litter your code with calls to look up the appropriate table name.  

Callbacks

As stated before and as seen in the prior example, all the operations exposed on tables by promised-dynamo are, as the name would suggest, promise based.  Q promises are used behind the scenes to wrap the various aws-sdk operations and the results which the promises resolve to are automatically mapped from objects housing DynamoDb property syntax to standard Javascript objects where appropriate.  For example, a simple get of the person worked with above would look as follows:

dynamo.promisedDynamoTest.getItem( ‘person1234’ )

   .then( function( person ) {

       //Javascript object representation of the identified person record

       console.log( person );  

   } )

   .catch( function( e ) {

       console.error( e );

   } );

fig 7

Alpha Release

promised-dynamo is being released in its current alpha state in the hopes that it is interesting and useful to the community and in the hopes that its feature-set can be fleshed out and increased guiding it into a beta state based on real world usage and needs.  Current documentation concerning the capabilities exposed exists at Github.  Please feel free to submit issues or pull requests to the github repo or to comment on the repo with any questions, requests, etc.

 

Github Repo: https://github.com/Citytechinc/promised-dynamo

NPM Entry: https://www.npmjs.com/package/promised-dynamo