Extending Backbone.js to Map Rough API Responses into Beautiful Client-Side Models

at May 2nd, 2012
Hello loyal readers of Groupon’s Dev Blog!  My name is Matthew and I’m going to be writing today’s feature on extending Backbone for awesome client-side model madness.  Here at Groupon I’m the front end lead on the Merchant Experience (Mx) Team.  Our team is responsible for, you guessed it, the experience a merchant has while working with Groupon. The main product of the Mx team is the Merchant Center — a website that allows the merchant to see everything about what they have done or could do with Groupon. The Merchant Center product has been expanding incredibly fast as we at Groupon offer more and more options for how to improve a merchant’s business.  With an eye towards making our application ready for our future plans, we recently redesigned the entire Merchant Center from the ground up.  As part of our redesign we made the major decision to have our entire application client-side rendered with all data retrieved via ajax calls to our internal web API.  The previous incarnation of the Merchant Center was a more traditional server-side rendered Rails app. We made this decision for a lot of different reasons both technological and organizational.  A few key advantages were:
  • Moving the application to use our API gives our product a service oriented architecture. SOA is a direction that Groupon as a company is headed in.
  • Having our website use the same APIs as our mobile clients (iOS devices, Android devices, etc..) eliminates redundancy and makes it easier to have similar feature-sets across platforms.
  • Going full client-side rendered allows us to make the app much more desktop app-like.  No more hard refreshes.  Lots more fluid UI/UE.
For this plan to succeed we needed to build a front-end with a true MVC architecture.  The first step in that process was to evaluate existing frameworks available and see what best fit our needs.  After careful deliberation we chose Backbone.  Backbone provides the structure necessary to make a great app and is thin and extensible enough for a developer to mold it however they wish.The first step in the actual coding of an MVC app is to build solid data models.  That step is the main topic of today’s dev blog. Our goal in defining data models was to make models that cleanly represented real-world data, that were easy for a developer to jump in and understand, and which played well with the areas of the app where they would be consumed.  Consumption areas include our view templates (written with Handlebars) and our charts (written with Raphael).  An example of a nice clean model for the basic information for a deal looks something like this:
Deal
-Title
-City
-Price
-Launched Date
-Expiration Date
-Number Sold
Anyone that wished to use this model could access the attributes easily.  Backbone supports accessing attributes through getter/setter methods or direct access to the model’s attribute hash.  In the ideal situation, To create a model and pull its data from a server requires only a few lines of code, where Deal is a Backbone model.
myDeal = new Deal();
myDeal.url = "<a href="http://server.com/myDeal">server.com/myDeal</a>";
myDeal.fetch();
Out of the box, Backbone takes the JSON response from the server at the specified url, parses it, and puts it into the attributes hash of the model.  To achieve the clean Deal model described above, the response from the server would need to look like this:
{
  title: "Deal Title",
  city: "San Francisco",
  price: "$10.00",
  launchedDate: "1/1/11",
  expirationDate: "1/1/12",
  numberSold: 100
}
Accessing attributes on this model is then clean and direct:
> myDeal.get("title")
"Deal Title"

> myDeal.attributes.title
"Deal Title"

> myDeal.get("price")
"$10.00"

> myDeal.attributes.price
"$10.00"
Unfortunately, this is where reality begins to differ from best-case scenarios and code is required to bridge the gap. The internal API we use already had many of the calls the Merchant Center needed and many different consumers already using them.  Like any good API it was written with flexibility in mind and not tailored to the specific needs of any one consumer.  An actual response to an API request for information about a deal looks more like this:
{
  deal: {
  division: {
    lng: -100,
    lat: 100,
    id: "SanFrancisco",
    timezoneOffsetInSeconds: 1000,
    name: "San Francisco"
  },
  placementPriority: "nearby",
  title: "Deal Title",
  option: {
    soldQuantity: 100,
    price: {
      currencyCode: "USD",
      formattedAmount: "10.00",
      amount: 1000
    },
    grid4ImageUrl: "<a href='http://groupon.com/image/imageLocation.jpg'>groupon.com/image/imageLocation.jpg</a>",
    startAt: "2011-01-01T01:01:01Z",
    expiresAt: "2011-01-01T01:01:01Z"
  },
  finePrint: "One Per Customer"
}
Making a basic Backbone model and hitting a server with the above response gives you a confusing model that is not easy to work with, contains data at various different depths, and contains information extraneous to our application’s needs.  Finding the data we need in this model is not clean and direct:
> myDeal.get("deal").title
"Deal Title"

> myDeal.attributes.deal.title
"Deal Title"

> myDeal.get("deal").option.price.formattedAmount
"$10.00"

> myDeal.attributes.deal.option.price.formattedAmount
"$10.00"
Using this kind of model in our application was not how we wanted to start our redesign.  We decided the best solution was to modify the way Backbone parsed responses from the server.  This would give us an opportunity to have the fetch process finish with a nicely formed model containing exactly what we needed no matter what the initial server response looked like. In building this solution we also had the opportunity to extend Backbone to meet an additional goal of ours.  We wanted the definition of each model to include an explicit declaration of what attributes are available on the model and what “type” each attribute is.  (Of course type is a fuzzy issue in JavaScript, but it’s fun to pretend!)  This type of attribute declaration is standard in most programming languages and allows developers the ability to glance quickly at the model and understand how it might be used.  As we showed earlier, the only definition required in a server-fetched Backbone Model is a resource URL.  While Backbone supports defining a defaults hash for a model, the hash does not contain any type information. With these goals in mind, we created an array of mapping hashes for each model.  Each hash contains the name of the attribute on the model, a mapping to where the data is found in the response from the server, and the data type of the attribute.  The mapping for the deal model is as follows:
mappings: [
  {name: "title", mapping: "deal.title", type: "string"},
  {name: "city", mapping: "deal.division.name", type: "string"},
  {name: "price", mapping: "deal.option.price.formattedAmount", type: "string"},
  {name: "launchedDate", mapping: "deal.startsAt", type: "date"},
  {name: "expirationDate", mapping: "deal.endsAt", type: "date"},
  {name: "numberSold", mapping: "deal.option.soldQuantity", type: "int"}
]
To utilize this mappings data we extended Backbone.Model with a base model from which all of our models extend.  This base model overrides Backbone’s parse method:
parse : function(resp, xhr) {
  if(this.mappings === null) {
    return resp;
  } else {
    var result = {};
    for( i in this.mappings ) {
      var attrMapping = this.mappings[i];
      var mapping = attrMapping.mapping;
      var val = null;
      try {
        var mapParts = mapping.split(".");
        val = resp;
        for (var i in mapParts) {
          val = val[mapParts[i]];
        }
      }
      catch(err) {
        if(err.name === "TypeError") {
          val = null;
        } else {
          throw(err);
        }
      }
      if ( val === null || val === undefined ) {
        result[attrMapping.name] = null
      } else {
        switch (attrMapping.type) {
          case "string":
            result[attrMapping.name] = String(val);
            break;
          case "date":
            result[attrMapping.name] = Date.parse(val);
            break;
          case "int":
            result[attrMapping.name] = parseInt(String(val).trim());
            break;
          default:
            result[attrMapping.name] = eval("resp." + attrMapping.mapping);
        }
        return result;
     }
   }
}
In the event no mappings array exists, the server response is passed along in the normal Backbone manner.  In the event it does exist, it is looped through.  For each mapping, a check is made if data exists at that location in the response.  If so, it is processed according to the type specified in the mapping.  If not, a null is placed in its location. The end result of this parse method is a results hash containing only the data specified in the mappings array.  Each bit of data is type-specific and available at the base level of the hash.  This results hash is passed on and Backbone creates the ideal model we desired all along.  Should the form of the API response change in the future, only the mappings information for the model needs to be updated to preserve the rest of our app. The parse method I’ve provided above is a basic example of our extension.  As we progressed in development we found we needed a few additional options for mapping server responses into useful data models.  For example, Groupon deals often have multiple options for purchase.  At a restaurant you might have a $5 for $10 option or a $10 for $20 option.  Ideally, our Deal model would contain an array of Option models.  To achieve this we created an array mappings type that included the target model class as an additional parameter.  Here is an an example of a basic Deal model that includes a mapping to an array of Option models:
mappings: [
  {name: "title", mapping: "announcementTitle", type: "string"},
  {name: "startAt", type: "date"},
  {name: "options", mapping: "options", type: "array", model: Groupon.MerchantCenter.Option}
]
Within our parse method’s handling of different mapping types we added a case for an array. When our parse method encounters this mapping it loops through the array in the server response, instantiating an appropriate model for each element and appending it to a temporary array.  Upon completion it appends this array to the results hash.
case "array":
  if (_.isArray(val)) {
    if (attrMapping.model) {
      var tmpArray = [];
      for( i in val) {
        var modelInst = new attrMapping.model();
        modelInst.fromJSON(val[i]);
        tmpArray.push(modelInst);
      }
      result[attrMapping.name] = tmpArray;
    } else {
      result[attrMapping.name] = val;
    }
  } else {
    var tmpArray = [];
    if ( attrMapping.model ) {
      var modelInst = new attrMapping.model();
      modelInst.fromJSON(val);
      tmpArray.push(modelInst);
    } else {
      tmpArray.push(val);
  }
  result[attrMapping.name] = tmpArray;
}
break;
In order for this logic to function we created a helper method on our base model that allows manual object creation from JSON:
fromJSON: function(input) {
  this.set(this.parse(input));
}
We also found we occasionally needed attributes on our model that were calculated from data contained in responses from the server.  To do this we created a postParseCalculation method on our base model.  Any model that wishes to calculate attributes simply overrides this method with the logic they desire.  Here is an example of where we needed an attribute that was the sum of two values returned from the server:
postParseCalculation: function(result) {
  result.actualSold = result.redeemed + result.unredeemed;
}
By performing this logic in the parse process our calculated attributes are functionally identical to attributes returned directly from the server.  This has meaningful ramifications when it comes to working with precise timing around change events on a model. As we progressed we found our parse extensions easy to modify to the various situations we encountered.  In the end we were able to create the solid, easy to work with data models we desired without having to change our API.  Along the way we created explicitly declared models and created a nice separation between the front-end app and the underlying API.  Below you will see source of our final parse method along with a sample mapping.  Please feel free to re-use or modify the code as you see fit.  All questions and comments are welcome as well. Source:
//Parse method
//This method is defined in our base model that extends Backbone.Model
//All of our models extend our base model.

parse : function(resp, xhr) {
  if(this.mappings===null){
    return resp;
  } else {
    var result = {};
    for( i in this.mappings ) {
      var attrMapping = this.mappings[i];
      var mapping = attrMapping.mapping ? attrMapping.mapping : attrMapping.name;
      var val = null;
      try {
        var mapParts = mapping.split(".");
        val = resp;
        for (var i in mapParts) {
          val = val[mapParts[i]];
        }
      }
      catch(err) {
        if(err.name === "TypeError") {
          val = null;
        } else {
          throw(err);
        }
      }
      if ( val === null || val === undefined ) {
        result[attrMapping.name] = null
      } else {
        switch (attrMapping.type) {
          case "string":
            result[attrMapping.name] = String(val);
            break;
          case "localDate": // Server returns UTC, convert to user's local timezone
            var date = Date.parse(val);
            date = new Date(date.getTime() - (date.getTimezoneOffset() * 60 * 1000));
            result[attrMapping.name] = date;
            break;
          case "date":
            result[attrMapping.name] = Date.parse(val);
            break;
          case "int":
            result[attrMapping.name] = parseInt(String(val).trim());
            break;
          case "float":
            result[attrMapping.name] = parseFloat(String(val).trim());
            break;
          case "array":
            if (_.isArray(val)) {
              if (attrMapping.model) {
                var tmpArray = [];
                for( i in val) {
                  var modelInst = new attrMapping.model();
                  modelInst.fromJSON(val[i]);
                  tmpArray.push(modelInst);
                }
                result[attrMapping.name] = tmpArray;
              } else {
                result[attrMapping.name] = val;
              }
            } else {
              var tmpArray = [];
              if ( attrMapping.model ) {
                var modelInst = new attrMapping.model();
                modelInst.fromJSON(val);
                tmpArray.push(modelInst);
              } else {
                tmpArray.push(val);
              }
              result[attrMapping.name] = tmpArray;
            }
            break;
          case "calculation":
            result[attrMapping.name] = this[attrMapping.calculationMethod](result);
            break;
          default:
            result[attrMapping.name] = eval("resp." + attrMapping.mapping)
        }
      }
    }
    this.postParseCalculation(result);
    return result;
  }
}

//fromJSON method required for the array type mapping parsing to function
//this method is defined on the base model

fromJSON: function(input){
  this.set(this.parse(input));
},

//Example Model Definition w/ Mapping

Groupon.MerchantCenter.DealSnapshot = Groupon.MerchantCenter.BaseModel.extend({

mappings: [
  {name: "title", mapping: "announcementTitle", type: "string"},
  {name: "fullTitle", mapping: "title", type: "string"},
  {name: "merchantName", mapping: "merchant.name", type: "string"},
  {name: "id", type: "string"},
  {name: "status", type: "string"},
  {name: "startAt", type: "date"},
  {name: "division", mapping: "division.name", type: "string"},
  {name: "sold", mapping: "soldQuantity", type: "int"},
  {name: "redeemed", mapping: "redeemedQuantity", type: "int"},
  {name: "unredeemed", mapping: "unredeemedQuantity", type: "int"},
  {name: "options", mapping: "options", type: "array", model: Groupon.MerchantCenter.Option}
]

});

No Tags


One thought on “Extending Backbone.js to Map Rough API Responses into Beautiful Client-Side Models

  1. [...] My question is, does Angular (or Restangular) provide a way to map model property names to custom ones like this in Backbone? [...]

    pingback by Is it possible to have a mapping for Angular model property names? | StackAnswer.com on February 14, 2014 at 8:01 am

Comments are closed.