End Point

News

Welcome to End Point's blog

Ongoing observations by End Point people.

jQuery Tips and an Ecommerce Demo

I've recently been jumping back and forth between YUI and jQuery on several different client projects. I prefer working with jQuery, but whenever I work with another framework, I realize what's out there and how I should continue to improve my skills in my preferred framework. I read up on jQuery tips and put together a summary of common tips I need to follow more explained here in an ecommerce demo.

The Setup

Before we get started, some notes on the ecommerce demo and performance testing:

  • The fake product images come from DryIcons
  • A JSON array contains product information (price, image, and title).
  • The code runs on a quick and dirty sinatra app.
  • console.time(‘task’) and console.timeEnd(‘task’) are used to debug task runtime for performance measurement
  • The performance numbers provided in the article were measured in Chrome (Ubuntu) where the average of 10 tests is reported. In all the tests, an additional for loop was added as part of the test to see measurable performance differences. See the notes at the bottom of the article on performance differences between Chrome and Firefox.
A screenshot from the demo app.

1. The first tip I came across was a recommendation to use a for loop instead of jQuery's each method. I start off with some un-optimized code to test that loops through our products JSON object and renders the images to the page. In this case, the use of the for loop instead of each doesn't give us a significant performance difference.

for(var k=0;k<10;k++) {
  $('div.products').html('');
  console.time('test ' + k);
  for(var j=0;j<10;j++) {
    $.each(products, function(index, e) {
      var ihtml = $('div.products').html();
      ihtml += '<a href="#">'
        + '<img class="product" src="/images/'
        + e.image + '" /></a>';
      $('div.products').html(ihtml);
    });
  }
  console.timeEnd('test ' + k);
}
for(var k=0;k<10;k++) {
  $('div.products').html('');
  console.time('test ' + k);
  for(var j=0;j<10;j++) {
    for(var i=0;i<products.length; i++) {
      var ihtml = $('div.products').html();
      ihtml += '<a href="#">'
        + '<img class="product" src="/images/'
        + products[i].image + '" /></a>';
      $('div.products').html(ihtml);
    }
  }
  console.timeEnd('test ' + k);
}
10 tests average: 505ms 10 tests average: 515ms


Products displayed with a for loop instead of jQuery's each method.

2. The next "I Can't Believe I'm Not Using this jQuery Technique" I found was a recommendation to use data tag. Although I've read about the data tag, I haven't worked with it enough to use it consistently. With this code, we assign product data (name, price) to each product link as it’s added to the DOM. Upon clicking a product, instead of traversing through the products array, we render the "featured_product" contents based on it's data. The data tag is recommended over assigning values to arbitrary HTML tag attributes such as assigning our product <a> and <img> the name and price values to title or alt attributes.

for(var i=0;i<products.length; i++) {
  var link = $(document.createElement('a'))
    .attr(‘href’, ‘#’)
    .html('<img class="product" src="/images/' + products[i].image + '" />')
    .data('name', products[i].name)
    .data('price', products[i].price);
  if(products[i].featured) {
    link.addClass('featured');
  }
  if(products[i].sale) {
    link.addClass('sale');
  }
  $('div.products').append(link);
};
$('div.products a').click(function() {
   $('div.featured_product')
    .html('<h1>' + $(this).data('name') + '</h1>');
  $(this).find('img').clone().appendTo('.featured_product');
});

jQuery's data tag is used to populate the right side of the page as a product is clicked.

3. The next tip I came across frequently was a recommendation to cache your selectors, shown in the example below. A selector $('div.products a.featured') is created and used when users like to view "Featured" Items. This gave me a 33% performance gain.

$('a.featured').click(function() {
  for(var k=0;k<10; k++) {
    console.time('test ' + k);
    for(var j=0;j<100;j++) {
      $('div.products a').css('background', '#FFF');
      $('div.products a.featured').css('background', '#999');
    }
    console.timeEnd('test ' + k);
  }
});
var all_products = $('div.products a');
var featured_products = $('div.products a.featured');
$('a.featured').click(function() {
  for(var k=0;k<10; k++) {
    console.time('test ' + k);
    for(var j=0;j<100;j++) {
      all_products.css('background', '#FFF');
      featured_products.css('background', '#999');
    }
    console.timeEnd('test ' + k);
  }
});
10 tests average: 31ms 10 tests average: 21ms


Multiple products added to test onclick for identifying "featured" products.

4. Another tip I came across was the recommendation on using context in jQuery selectors, described here in the jQuery documentation. In one example, I try updating the $('div.products') selector to set the context as $(‘div.wrapper’), but saw performance worsen here. In another example, I added "this" as a context for populating the featured product, but again saw performance worsen slightly here. In this case, performance gain will depend on your original selector, but it's worth testing.

for(var k=0;k<10;k++) {
  $('div.products').html('');
  console.time('test ' + k);
  for(var j=0;j<10;j++) {
    for(var i=0;i<products.length; i++) {
      ...
      $('div.products').append(link);
    }
  }
  console.timeEnd('test ' + k);
}
for(var k=0;k<10;k++) {
  $('div.products').html('');
  console.time('test ' + k);
  for(var j=0;j<10;j++) {
    for(var i=0;i<products.length; i++) {
      ...
      $('div.products', $('div.wrapper')).append(link);
    }
  }
  console.timeEnd('test ' + k);
}
10 tests average: 64ms 10 tests average: 76ms

5. Another common tip I came across that "I can't believe I don't follow" is to use an id selector instead of a class. I’ve admittedly read this several times, but again it's a practice that I sometimes forget about. In my ecommerce setup, I add a loop to add the products 10x to our DOM, and with a change of selector from 'div.products' to 'div#products', I saw a small performance improvement.

for(var k=0;k<10;k++) {
  $('div.products').html('');
  console.time('test ' + k);
  for(var j=0;j<10;j++) {
    for(var i=0;i<products.length; i++) {
       ...
       $('div.products').append(link);
    }
  }
  console.timeEnd('test ' + k);
}
for(var k=0;k<10;k++) {
  $('div.products').html('');
  console.time('test ' + k);
  for(var j=0;j<10;j++) {
    for(var i=0;i<products.length; i++) {
      ...
       $('div#products').append(link);
    }
  }
  console.timeEnd('test ' + k);
}
10 tests average: 64ms 10 tests average: 60ms

6. I found a recommendation to minimize the DOM minimally. This is a tip that I typically follow, but another one that's easily forgotten. In our ecommerce setup, I create a single point of appending to the div#products selector after generating my product links and data. This gave a ~25% performance gain.

for(var k=0;k<10;k++) {
  $('div.products').html('');
  console.time('test ' + k);
  for(var j=0;j<10;j++) {
    for(var i=0;i<products.length; i++) {
      ...
      $('div#products').append(link);
    }
  }
  console.timeEnd('test ' + k);
}
for(var k=0;k<10;k++) {
  $('div#products').html();
  console.time('test ' + k);
  var collection = $(document.createElement('div'));
  for(var j=0;j<10;j++) {
    for(var i=0;i<products.length; i++) {
      ...
      collection.append(link);
    }
  }
  $('div#products').append(collection);
  console.timeEnd('test ' + k);
}
10 tests average: 60ms 10 tests average: 45ms

7. Another tip I came across was to use event delegation in jQuery. The idea is that your event has more context to work with than general selectors. You can access the event and manipulate it’s parent. I found that $(e.target).parent() is the same as manipulating $(this) performance-wise. It’s likely that using one or the other is much faster than using a general DOM selector such as $(div.product' + id).

var featured_product = $('div#featured_product');
$('div#products a').click(function(e) {
  var product = $(e.target).parent();  // same as $(this)
  featured_product
    .html('<h1>' + product.data('name') + '</h1>')
    .append(product.find('img').clone());
});

8. One tip that I've never seen before is to use ".end()" in chaining. Instead of reselecting the $('div.products') region, I use "find()" to apply a css style, then traverse up the chain to find another set of products to apply a css style, and repeat. This change gave a small performance bump, but you might tend to use the cached selectors described in Tip #3 instead of the following.

$('a#special').click(function() {
  for(var k=0;k<10;k++) {
    console.time('test ' + k);
    for(var j=0;j<100; j++) {
      $('div#products').find('a').css('background', '#FFF');
      $('div#products').find('.featured').css('background', '#999');
      $('div#products').find('.sale').css('background', '#999');
    }
    console.timeEnd('test ' + k);
  }
});
$('a#special').click(function() {
  for(var k=0;k<10;k++) {
    console.time('test ' + k);
    for(var j=0;j<100; j++) {
      $('div#products')
      .find('a')
        .css('background', '#FFF')
      .end()
      .find('.featured')  
        .css('background', '#999')    
      .end()
      .find('sale')   
        .css('background', '#999');   
    }
    console.timeEnd('test ' + k);
  }
});
10 tests average: 47ms 10 tests average: 37ms


Featured and Sale products are highlighted here.

9. I found several examples of writing your own selectors and how easy it is! I wrote two selectors with the following to identify products under $20.00 and products over $1000.00.

$(function() {
   $('a#under20').click(function() {
    all_products.css('background', '#FFF');
    $('div#products a:under20').css('background', '#999');
  });
  $('a#over1000').click(function() {
    all_products.css('background', '#FFF');
    $('div#products a:over1000').css('background', '#999');
  });
});

$.extend($.expr[':'], {
  under20: function(a) {
    if($(a).data('price') && $(a).data('price') < 20) {
      return true;
    }
    return false;
  },
  over1000: function(a) {
    if($(a).data('price') && $(a).data('price') > 1000) {
      return true;
    }
    return false;
  }
});


Products priced over $1000 are highlighted with a custom selector.

10. I also found several examples of how to write your own chain methods. In this example, I create two chain methods to set the product background to white or grey and update my under20 & over1000 methods to use this new chain method. The nice thing about creating your own chain methods is that these methods can be easily modified in the future with minimal code changes because it follows the DRY principle. I'm not sure if it's intended, but the use of my custom chain method did not work with the end() chain method described in Tip #8.

$.fn.unhighlight_product = function() {
  return $(this).css('background', '#FFF');
}   
$.fn.highlght_product = function() {
  return $(this).css('background', '#999');
}

$(function() {
  $('a#under20').click(function() {
    all_products.unhighlight_product();
    $('div#products a:under20').highlight_product();
  });
  $('a#over1000').click(function() {
    all_products.unhighlight_product();
    $('div#products a:over1000').highlight_product();
  });
});


Products priced under $20 are highlighted with a custom selector and custom chain method.

Again, the product images for this demo app are from DryIcons.com and the final application code can be found here. The application was also deployed on Heroku to verify that the code works in IE [8], Chrome, and Firefox.

In Case You are Interested

During development of the demo, I found a significant performance differences between Chrome and Firefox. See the following tests:

  • Tip #1: 6200ms (FF) vs 515ms (Chrome)
  • Tip #3: 106ms (FF) vs 21ms (Chrome)
  • Tip #4: 258ms (FF) vs 60ms (Chrome)
  • Tip #6: 169ms (FF) vs 45ms (Chrome)
  • Tip #8: 154ms (FF) vs 37ms (Chrome)

Or in visual form:

DROID 2 review

I got a Motorola DROID 2 phone a couple of months ago and have assembled here my notes about how it's worked out so far. First, some background.

This is my second Android phone. My first was the Google Ion, basically the same as the HTC Magic. That was running standard Android 1.5 (Cupcake), while the DROID 2 runs Android 2.2 (Froyo) tweaked somewhat by Motorola. I've used several other Android phones belonging to friends and relatives.

Overall I like the Android operating system fairly well. Like everything, it can be improved. It's been advancing at a fairly quick pace. It's mostly free software. Too many phones are locked down and have to be broken into to change the operating system, but Android's still a freer computing environment than most phones have and I hope the situation will improve over time.

I take for granted much of Android's feature set: The excellent system-wide notification bar that many apps hook into and which is always easy to get to. The solid multitasking. Automatic screen adjustment for using the phone in landscape vs. portrait mode. The ability to mount the normal filesystem on the SD card from another computer via USB or by removing the SD card. The freedom to run any applications I want, downloaded from wherever, not just the Android Market.

For one of our clients, I'm currently developing an Android application in Java that uses geolocation, JSON web services, and Urban Airship's AirMail Push for push notifications. So I'm using the phone both as my main mobile phone and as an app developer.

Keyboard

This is the first phone I've had with a full slide-out QWERTY keyboard. I wasn't sure if I'd use it often or just stick with the touchscreen keyboard. So far I use the real keyboard most of the time. But it's nice to be able to use either the hard keyboard or touchscreen keyboard, whichever's more convenient.

I like the keyboard a lot, especially that its rows are offset like a normal keyboard and not a straight aligned grid:

The real keyboard makes using ssh (via the ConnectBot app) on the phone much easier than with the touchscreen keyboard, because the real keyboard doesn't use up any screen real estate.

The biggest annoyance for me has been the keyboard's microphone key which launches the voice commands function. I don't use voice commands, so this has never been what I've wanted and is highly annoying when accidentally pressed.

Third-party alternative touchscreen keyboards for European languages, Hebrew, Arabic, and others work pretty well too.

Touchscreen

I really miss having physical buttons for menu, home, back, and search, as I had on the HTC Magic:

On the DROID 2 they're touchscreen buttons and it's annoying and error-prone to have to look at the phone and carefully press one. Much better to have real buttons for those core functions used by all apps.

However, good riddance to the trackball.

The DROID 2 screen itself is higher resolution, bright, and responds well to touch, swipe, and multitouch, as you'd expect.

Battery

Battery life on the DROID 2 is not great, especially when using GPS heavily. But that's a problem on most phones.

I later got the optional extended-life battery. It isn't much bigger or heavier, and does add a lot more life to the phone. It wouldn't be necessary if I weren't doing heavy GPS & web service interaction with my own Android app under development.

Audio

The DROID 2 has a normal 1/8" stereo headphone jack, which is nice. The HTC Magic needed a special adapter to use normal headphones, which was a pain.

Audio playback is usually fine, though on a couple of occasions it's stuttered, presumably while contending with other apps for CPU. Audio playback when the screen is off doesn't drain the battery much at all.

Calling and Contacts

There have been a few isolated incidents, perhaps involving switching off Bluetooth, where I couldn't turn down the call volume in the middle of the call. That could've been either a generic Android problem, or the Motorola Android build; I'm not sure.

I didn't much like the special Motorola apps that shipped on the phone, but simply not using them has made them no problem.

An exception to that is the Contacts app that integrates with Facebook and Twitter. That's a lot more helpful than I imagined it would be, though when you have many contacts it can sometime slow down on opening an individual contact, when I imagine it tries to fetch the latest details from social networks.

The in-call screen's "Mute" button is small and way too close to the "End call" button:

That's led to me accidentally hanging up during conference calls a couple of times. The stock open source Android call screen is arguably less pretty, but equally-sized buttons makes it easier to use:

Because I'm using Verizon for the first time with this phone, I've been introduced to the sad problem of not being able to use voice and data at the same time on CDMA. I used to laugh about that when I was using AT&T because it seemed silly. Now I find it really does get in the way fairly often.

The normal Skype app doesn't allow using a Bluetooth headset or using the cell network for calls (in the US). The special Verizon Skype app burns voice minutes and won't let you use wifi for calling (in the US). The combination of proprietary software plus controlling phone carriers is bad for users! Because of the limitations I use Skype mostly for text chat on my phone.

GPS

As I mentioned, the GPS uses a lot of battery. But it works well. The GPS on my HTC Magic sometimes took a very long time to get a location fix, but on the DROID 2 it's fast and works very well. Given the location-aware app I'm working on, I've had more opportunity to notice this than I otherwise might have.

There was one funny problem with the GPS, though. Once when the battery got down to 5% or so, the GPS started to report hilariously wrong data to my app, and thus to the web service it reported to. For example, it reported every second that my phone, which was sitting on my nightstand, was traveling at 121 m/s (270 mph), and the point in latitude and longitude moved quickly. This continued even after I'd plugged the phone back in. I haven't seen it happen since, but it was strange.

Useful apps

I don't spend a lot of time trying out new apps, but I've found several to be very useful: K-9 Mail for multiple mail accounts (reviewed yesterday on Ars Technica), Yaaic (an IRC client), 3G Watchdog (bandwidth monitor), TweetsRide (Twitter client), Google Sky Map, and My Tracks. I haven't found a great music player yet, but the built-in Android Music app and Winamp are both fine.

Security

I've long used crypto filesystems on desktop and laptop computers, and feel the lack on Android. A phone needs a solid cryptofs option more than normal computers even, and I hope that'll be available soon.

The end (for now)

I'm probably forgetting lots of things, but this is already longer than I'd expected, so I'll stop now and put any afterthoughts in the comments. In short, I like the phone, but it's hard to avoid looking forward to a brighter future when it comes to mobile devices.

Ruby Ecommerce with Sinatra: Admin and Products

Last week, I wrote about creating a very simple ecommerce application on Ruby with Sinatra. This week, we continue on the yellow brick road of ecommerce development on Ruby with Sinatra.

yellow brick road
A yellow brick road.

Part 2: Basic Admin Authentication

After you've got a basic application running which accepts payment for a single product as described in the previous tutorial, the next step is to add admin authorization to allow lookup of completed orders. I found several great resources for this as well as a few Sinatra extensions that may be useful. For the first increment of implementation, I followed the instructions here, which uses Basic::Auth. The resulting code can be viewed here. I also introduce subclassing of Sinatra::Base, which allows us to keep our files a bit more modular and organized.

And if we add an "/admin" method to display orders, we can see our completed orders:


Completed orders.

Part 3: Introducing Products

Now, let's imagine an ecommerce store with different products! Whoa! For this increment, let's limit each order to one product. A migration and model definition is created to introduce products, which contains a name, description, and price. For this increment, product images match the product name and live ~/public/images. The orders table is modified to contain a reference to products.id. The orders model is updated to belong_to :products. Finally, the frontend authorization method is modified to use the order.product.price in the transaction.

# Products ActiveRecord migration
require 'lib/model/product'
class CreateProducts < ActiveRecord::Migration
  def self.up
    create_table :products do |t|
      t.string :name,
        :null => false
      t.decimal :price,
        :null => false
      t.string :description,
        :null => false
    end
  end

  def self.down
    drop_table :products
  end
end
 
# Products model class
class Product < ActiveRecord::Base
  validates_presence_of :name
  validates_presence_of :price
  validates_numericality_of :price
  validates_presence_of :description

  has_many :orders
end
# Order migration update
class CreateOrders < ActiveRecord::Migration
  def self.up
    create_table :orders do |t|
+      t.references :product,
+        :null => false
    end
  end

  def self.down
    drop_table :orders
  end
end
# Order model changes
class Order < ActiveRecord::Base
...
+ validates_presence_of :product_id
+
+  belongs_to :product
end
# in main checkout action
# Authorization amount update
- response = gateway.authorize(1000,
-   credit_card)
+ response = gateway.authorize(order.product.price*100,
+   credit_card)


Our new data model.

And let's use Sinatra's simple and powerful routing to build resource management functionality that allows our admin to list, create, update, and delete items, or in this case orders and products. Here's the sinatra code that accomplishes this basic resource management:

# List items
app.get '/admin/:type' do |type|
  require_administrative_privileges
  content_type :json

  begin
    klass = type.camelize.constantize
    objects = klass.all
    status 200
    objects.to_json
  rescue Exception => e
    halt 500, [e.message].to_json 
  end
end
# Delete item
app.delete '/admin/:type/:id' do |type, id|
  require_administrative_privileges
  content_type :json

  begin
    klass = type.camelize.constantize
    instance = klass.find(id)
    if instance.destroy
      status 200
    else
      status 400
      errors = instance.errors.full_messages
      [errors.first].to_json
    end
  rescue Exception => e
    halt 500, [e.message].to_json
  end
end
# Create new item
app.post '/admin/:type/new' do |type|
  require_administrative_privileges
  content_type :json
  input = json_to_hash(request.body.read.to_s)
 
  begin
    klass = type.camelize.constantize
    instance = klass.new(input)
    if instance.save
      status 200
      instance.to_json
    else
      status 400
      errors = instance.errors.full_messages
      [errors.first].to_json
    end
  rescue Exception => e
    halt 500, [e.message].to_json
  end
end
# Edit item
app.post '/admin/:type/:id' do |type, id|
  require_administrative_privileges
  content_type :json
  input = json_to_hash(request.body.read.to_s)
  
  begin
    klass = type.camelize.constantize
    instance = klass.find(id)
    if instance.update_attributes(input)
      status 200
      instance.to_json
    else
      status 400
      errors = instance.errors.full_messages
      [errors.first].to_json
    end
  rescue Exception => e
    halt 500, [e.message].to_json
  end
end

Note that in the code shown above, the request includes the class (product or order in this application), and the id of the item in some cases. The constantize method is used to get the class constant, and ActiveRecord methods are used to retrieve and edit, create, or delete the instance. This powerful routing now allows us to easily manage additional resources with minimal changes to our server-side code.

Next, I use jQuery to call these methods via AJAX, also in such a way that it'll be easy to manage new resources with minimal client side code. That base admin code can be found here. With this jQuery admin base, we now define our empty resource, content for displaying that resource, and content for editing that resource. Examples of this are shown below:

functions.product = {
  edit: function(product) {
    return '<h4>Editing Product: '
      + product.id
      + '</h4>'
      + '<p><label for="name">Name</label>'
      + '<input type="text" name="name" value="'
      + product.name
      + '" /></p>'
      + '<p><label for="price">Price</label>'
      + '<input type="text" name="price" value="'
      + parseFloat(product.price).toFixed(2)
      + '" /></p>'
      + '<p><label for="description">Description</label>'
      + '<textarea name="description">'
      + product.description
      + '</textarea></p>';
  },
  content: function(product) {
    var inner_html = '<h4>Product: '
      + product.id
      + '</h4>'
      + 'Name: '
      + product.name
      + '<br />Price: $'
      + parseFloat(product.price).toFixed(2)
      + '<br />Description: '
      + product.description
      + '<br />';
    return inner_html;
  },
  empty: function() {
    return { name: '',
      price: 0, 
      description: '' };  
  }
};


Product listing.

Creating a new product.

Editing an existing product.

functions.order = {
  edit: function(order) {
    return '<b>Order: '
      + order.id
      + '</b><br />'
      + '<input name="email" value="'
      + order.email
      + '" />'
      + ' – '
      ...
      //Order editing is limited
  },
  content: function(order) {
    return '<b>Order: '
      + order.id
      + '</b><br />'
      + order.email
      + ' – '
      + order.phone
      + '<br />'
      ...
  },
  empty: function() {
    return { 
      email: '',
      phone: '',
      ...
    };  
  }
};


For this example, we limit order editing to email and phone number changes.

With a final touch of frontend JavaScript and CSS changes, the following screenshots show the two customer-facing pages from our example store. Like the application described in the previous article, this ecommerce application is still fairly lightweight, but it now allows us to sell several products and manage our resources via the admin panel. Stay tuned for the next increment!

The cupcake images shown in this article are under the Creative Commons license and can be found here, here, and here. The code shown in this article can be found here (branches part2 and part3).

check_postgres without Nagios (Postgres checkpoints)

Version 2.16.0 of check_postgres, a monitoring tool for Postgres, was just released. We're still trying to keep a "release often" schedule, and hopefully this year will see many releases. In addition to a few minor bug fixes, we added a new check by Nicola Thauvin called hot_standby_delay, which, as you might have guessed from the name, calculates the streaming replication lag between a master server and one of the slaves connected to it. Obviously the servers must be running PostgreSQL 9.0 or better.

Another recently added feature (in version 2.15.0) was the simple addition of a --quiet flag. All this does is to prevent any normal output when an OK status is found. I wrote this because sometimes even Nagios is overkill. In the default mode (Nagios, the other major mode is MRTG), check_postgres will exit with one of four states, each with their own exit code: OK, WARNING, CRITICAL, or UNKNOWN. It also outputs a small message, per Nagios conventions, so a txn_idle action might exit with a value of 1 and output something similar to this:

POSTGRES_TXN_IDLE WARNING: (host:svr1) longest idle in txn: 4638s

I had a situation where I wanted to use the functionality of check_postgres (to examine the lag on a warm standby server), but did not want the overhead of adding it into Nagios, and just needed a quick email to be sent if there were any problems. Thus, the use of the quiet flag yielded a quick and cheap Nagios replacement using cron:

*/10 * * * * bin/check_postgres.pl --action=checkpoint -w 300 -c 600 --datadir=/dbdir --quiet

So every 10 minutes the script gathers the number of seconds since the last checkpoint was run. If that number is under five minutes (300 seconds), it exits silently. If it's over five minutes, it outputs something similar to this, which cron then sends in an email:

POSTGRES_CHECKPOINT CRITICAL:  Last checkpoint was 842 seconds ago

I'm not advocating replacing Nagios of course: there are many other good reasons to use Nagios instead of cron, but this worked well for the situation at hand. Other actions, feature requests, and patches for check_postgres are always welcome, either on the check_postgres bug tracker or the mailing list.

Zayda Internet joins End Point

End Point is pleased to announce the merging of web hosting firm Zayda Internet into our company as of December 2010.

Zayda Internet was a small hosting company that started in 1999 and focused on Linux web and email hosting and consulting for many small businesses and individuals, and a few larger sites including most prominently the busy community site OSNews.com.

We're happy to welcome all Zayda Internet customers into the End Point fold and look forward to working with them in the future.

Ecommerce on Sinatra: In a Jiffy

Several of us at End Point have been involved in a non-ecommerce project for one of our clients running on Ruby, Sinatra, Unicorn, using DataMapper, PostgreSQL, PostGIS, with heavy use of JavaScript (specifically YUI). Sinatra is a lightweight Ruby web framework – it's not in direct competition with Rails but it might be a better "tool" for lightweight applications. It's been a fun project to work with Sinatra, DataMapper, and YUI as I've been working traditionally focused on their respective related technologies (Rails, ActiveRecord, jQuery).

Out of curiosity, I wanted to see what it might take to implement a bare-bones ecommerce store using Sinatra. Here is a mini-tutorial to develop an ecommerce store using Sinatra.


A snapshot of our final working app.

Getting Started

I create a new directory for the project with the following directories:

sinatrashop/
  db/
    migrate/
  models/
  public/
    images/
     stylesheets/
  views/

Data Model

Now, let's look at the data model. Since this is a bare-bones store, I have one order model which contains all the order information including contact information and addresses. We're not storing the credit card in the database. Also, since this is a bare-bones app, we're going to go with one product with a set price and force the limitation that users only buy one at a time. I've also chosen to use ActiveRecord here since I'm still not sold on DataMapper, but another ORM can be used as well. Here is our model:

# sinatrashop/models/order.rb
class Order < ActiveRecord::Base
  validates_presence_of :email
  validates_presence_of :bill_firstname
  validates_presence_of :bill_lastname
  validates_presence_of :bill_address1
  validates_presence_of :bill_city
  validates_presence_of :bill_state
  validates_presence_of :bill_zipcode
  validates_presence_of :ship_firstname
  validates_presence_of :ship_lastname
  validates_presence_of :ship_address1
  validates_presence_of :ship_city
  validates_presence_of :ship_state
  validates_presence_of :ship_zipcode
  validates_presence_of :phone
  validates_format_of :email,
    :with => /\A([^@\s]+)@((?:[-a-z0-9]+\.)+[a-z]{2,})\Z/i,
    :on => :create
end

And here is our migration:

# sinatrashop/db/migrate/001_create_orders.rb
class CreateOrders < ActiveRecord::Migration
  def self.up
    create_table :orders do |t|
      t.string   :email, :null => false
      t.string   :bill_firstname, :null => false
      t.string   :bill_lastname, :null => false
      t.string   :bill_address1, :null => false
      t.string   :bill_address2
      t.string   :bill_city, :null => false
      t.integer  :bill_state, :null => false
      t.string   :bill_zipcode, :null => false
      t.string   :ship_firstname, :null => false
      t.string   :ship_lastname, :null => false
      t.string   :ship_address1, :null => false
      t.string   :ship_address2
      t.string   :ship_city, :null => false
      t.integer  :ship_state, :null => false
      t.string   :ship_zipcode, :null => false
      t.string   :phone, :null => false
      t.timestamps
    end
  end

  def self.down
    drop_table :orders
  end
end

I did some research here and created the Rakefile shown below to run the migrations. The Rakefile establishes a connection to a sqlite3 database and runs migrations in the db/migrate directory.

# sinatrashop/Rakefile
namespace :db do
  task :environment do
    require 'rubygems'
    require 'logger'
    require 'active_record'
    ActiveRecord::Base.establish_connection :adapter => 'sqlite3',
      :database => 'db/development.sqlite3.db'
  end

  desc "Migrate the database"
  task(:migrate => :environment) do
    ActiveRecord::Base.logger = Logger.new(STDOUT)
    ActiveRecord::Migration.verbose = true
    ActiveRecord::Migrator.migrate("db/migrate")
  end
end

Views

Now, let's think about the views we'll present to users. There are many template rendering options in Sinatra, but we'll go with erb and create an index.erb file. By default, Sinatra looks for views in the ROOT/views directory. This will be our only view and layout and below is a breakdown of what it will include:

# header information
<body>
# product information
# form for submission
# errors or success message
</body>

Obviously, there will be a lot more code here, but the view needs to show the basic product information, the form fields to collection information, and errors or a success message to handle the different use cases. See the code here to examine the contents.

Application Code

Next, let's take a look at the application code. This will be in sinatrashop/store.rb:

require 'sinatra'
require 'erb'
require 'active_record'
require 'configuration'
require 'models/order'

get '/' do
  erb :index
end

post '/' do
  erb :index
end

The application code handles two requests, a get and post to '/'. The get is a standard home page request. The post to '/' is the order submission. The post '/' action needs to save the order, establish a connection to the payment gateway, and authorize and capture the payment. If any of these actions fail, the order must not be saved to the database and errors must be presented to the user. Consider the following code, which uses ActiveRecord::Base.transaction method and will rollback the saved order if any part of the authorization fails. We also use ActiveMerchant here, which is an extraction from Shopify for payment gateway integration that can be used as a gem.

# sinatrashop/store.rb
post '/' do
  begin
    order = Order.new(params[:order])
    ActiveRecord::Base.transaction do
      if order.save
        params[:credit_card][:first_name] = params[:order][:bill_firstname]
        params[:credit_card][:last_name] = params[:order][:bill_lastname]
        credit_card = ActiveMerchant::Billing::CreditCard.new(params[:credit_card])
        if credit_card.valid?
           gateway = ActiveMerchant::Billing::AuthorizeNetGateway.new(settings.authorize_credentials)

           # Authorize for $10 dollars (1000 cents) 
           response = gateway.authorize(1000, credit_card)
           if response.success?
             order.update_attribute(:status, "complete")
             gateway.capture(1000, response.authorization)
             @success = true
           else
             raise Exception, response.message
           end
         else
           raise Exception, "Your credit card was not valid."
         end
       else
         raise Exception, 'Errors: ' + order.errors.full_messages.join(', ')
       end
     end
  rescue Exception => e
    @message = e.message 
  end
end

Configuration

You might notice above that there is a "settings" hash used in the payment gateway connection request. I create a configuration file which sets up some configuration variables in Sinatra's configure do block:

# sinatrashop/configuration.rb
require 'active_merchant'

configure do
  set :authorize_credentials => {
    :login => "LOGIN"
    :password => "PASSWORD"
  }
  ActiveRecord::Base.establish_connection(
    :adapter => 'sqlite3',
    :database =>  'db/development.sqlite3.db'
  )
  ActiveMerchant::Billing::Base.mode = :test
end

Testing

I wrote several tests to handle a few use cases. These can be examined here. The tests use Rack::Test and can be run with the command ruby -rubygems test_store.rb.

Infrastructure Concerns

Additional changes are required for running the application on the HTTP server of your choice. Additionally, the entire site should probably run in SSL, which would need configuration. Finally, sqlite may be replaced with a different database here with updates to the configuration.rb and Rakefile files. During development, I ran my app with the command ruby -rubygems store.rb.

Conclusion

Our bare-bones ecommerce app contains a single simple order model that contains all of our order information. There are only two actions defined our application code. There is one index view. Public assets are served from ROOT/public/. The Rakefile contains functionality for running migrations. There is no admin interface here. A site administrator needs to retrieve orders from the database for sending email notifications and fulfillment. In incremental development, this is the simplest setup to allow someone to collect money, but it requires quite a bit of manual management (emails, fulfillment, etc.). The code can be found here. Here are more screenshots of the working application:


Order errors!


Payment gateway errors.


A successful transaction.

DBD::Pg, UTF-8, and Postgres client_encoding

Photo by Roger Smith

I've been working on getting DBD::Pg to play nicely with UTF-8, as the current system is suboptimal at best. DBD::Pg is the Perl interface to Postgres, and is the glue code that takes the data from the database (via libpq) and gives it to your Perl program. However, not all data is created equal, and that's where the complications begin.

Currently, everything coming back from the database is, by default, treated as byte soup, meaning no conversion is done, and no strings are marked as utf8 (Perl strings are actually objects in which one of the attributes you can set is 'utf8'). If you want strings marked as utf8, you must currently set the pg_enable_utf8 attribute on the database handle like so:

$dbh->{pg_enable_utf8} = 1;

This causes DBD::Pg to scan incoming strings for high bits and mark the string as utf8 if it finds them. There are a few drawbacks to this system:

  • It does this for all databases, even SQL_ASCII!
  • It doesn't do this for everything, e.g. arrays, custom data types, xml.
  • It requires the user to remember to set pg_enable_utf8.
  • It adds overhead as we have to parse every single byte coming back from the database.

Here's one proposal for a new system. Feedback welcome, as this is a tricky thing to get right.

DBD::Pg will examine the client_encoding parameter, and see if it matches UTF8. If it does, then we can assume everything coming back to us from Postgres is UTF-8. Therefore, we'll simply flip the utf8 bit on for all strings. The one exception is bytea data, of course, which we'll read in and dequote into a non-utf8 string. Any non-UTF8 client_encodings (e.g. the monstrosity that is SQL_ASCII) will simply get back a byte soup, with no utf8 markings on our part.

The pg_enable_utf8 attribute will remain, so that applications that do their own decoding, or otherwise do not want the utf8 flag set, can forcibly disable it by setting pg_enable_utf8 to 0. Similarly, it can be forced on by setting pg_enable_utf8 to 1. The flag will always trump the client_encoding parameter.

A further complication is client_encoding: What if it defaults to something else? We can set it ourselves upon first connecting, and then if the program changes it after that point, it's on them to deal with the issues. (As DBD::Pg will still assume it is UTF-8, as we don't constantly recheck the parameter.)

Someone also raised the issue of marking ASCII-only strings as utf8. While technically this is not correct, it would be nice to avoid having to parse every single byte that comes out of the database to look for high bits. Hopefully, programs requesting data from a UTF-8 database will not be surprised when things come back marked as utf8.

Feel free to comment here or on the bug that started it all. Thanks also to David Christensen, who has given me great input on this topic.

SSH config wildcards and multiple Postgres servers per client

The SSH config file has some nice features that help me to keep my sanity among a wide variety of servers spread across many different clients. Nearly all of my Postgres work is done by using SSH to connect to remote client sites, so the ability to connect to the various servers easily and intuitively is important. I'll go over an example of how a ssh config file might progress as you deal with an ever‑expanding client.

Some quick background: the ssh config file is a per‑user configuration file for the SSH program. It typically exists as ~/.ssh/config. It has two main purposes: setting global configuration items (such as ForwardX11 no), and setting things on a host‑by‑host basis. We'll be focusing on the latter.

Inside the ssh config file, you can create Host sections which specify options that apply only to one or more matching hosts. The sections are applied if the host name you type in as the argument to the ssh command matches what is after the word "Host". As we'll see, this also allows for wildcards, which can be very useful.

I'm going to walk through a hypothetical client, Acme Corporation, and show how the ssh config can grow as the client does, until the final example mirrors an actual section of my ssh config section file.

So, you've just got a new Postgres client called Acme Corporation, and they are using Amazon Web Services (AWS) to host their server. We're coming in as the postgres user, and have our public ssh keys already in place inside ~postgres/.ssh/authorized_keys on their server. The hostname is ec2‑456‑55‑123‑45.compute‑1.amazonaws.com. So, generally, we would connect by running:

$ ssh postgres@ec2‑456‑55‑123‑45.compute‑1.amazonaws.com

That's a lot to type each time! We could create a bash alias to handle this, but it's better to use the ssh config file instead. We'll add this to the end of our ssh config:

##
## Client: Acme Corporation
##

Host  acmecorp
User postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com

Now we can simply use 'acmecorp' in place of that ugly string:

$ ssh acmecorp

Notice that we don't need to specify the user anymore: ssh config plugs that in for us. We can still override it if we need to connect as someone else:

$ ssh greg@acmecorp

The next week, Acme Corporation decides that rather than allow anyone to SSH to their servers, they will use iptables or something similar to restrict access to select known hosts. Because different people with different IPs at End Point may need to access Acme, and because we don't want to have Acme have to open a new hole each time we connect from a different place, we will connect from a shared company box. In this case, the box is vp.endpoint.com. Acme arranges to allow SSH from that box to their servers, and each End Point employee has a login on the vp.endpoint.com box. What we need to do now is create a SSH tunnel. Inside of the ssh config file, we add a new line to the entry for 'acmecorp':

Host  acmecorp
User  postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com
ProxyCommand  ssh -q greg@vp.endpoint.com nc -w 180 %h %p

Now, when we run this:

$ ssh acmecorp

...everything looks the same to us, but what we are really doing is connecting to vp.endpoint.com, running the nc (netcat) command, and then connecting to the amazonaws.com box over the new netcat connection. (The arguments to netcat specify that the connection should be closed if there is the connection goes away for 180 seconds, and the host and port should be echoed along). As far as amazonaws.com is concerned, we are connecting from vp.endpoint.com. As far as we are concerned, we are going directly to amazonaws.com. A nice side effect, and a big reason why we don't simply use bash aliases, is that the scp program will use these aliases as well. So we can now do something like this:

$ scp check_postgres.pl acmecorp:

This will copy the check_postgres.pl program from our computer to the Acme one, going through the tunnel at vp.endpoint.com.

Business has been good for Acme lately and they finally have conceded to your strong suggestion to set up a warm standby server (using Postgres' Point In Time Recovery system). This new server is located at ec2‑456‑55‑123‑99.compute‑1.amazonaws.com, and the internal host name they give it is maindb‑replica (the original box is known as maindb‑db). This new server requires another host entry to ssh config. Rather than copy over the same ProxyCommand, we'll refactor the information out into a separate host entry. What we end up with is this:

Host  acmetunnel
User  greg
Hostname  vp.endpoint.com

Host  acmedb
User  postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

Host  acmereplica
User  postgres
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

We also changed the name from acmecorp to just "acme" as that's enough to uniquely identify among our clients, and who wants to type more than they have to?

Next, the company adds a QA box they want End Point to help setup. This box, however, is *not* reachable from outside their network; it can be reached only from other hosts in their network. Luckily, we already have access to some of those. What we'll do is extend our tunnel by one more host, so that the path we travel from us to the Acme QA box is:

Local box → vp.endpoint.com → acreplica → acqa

Here's the section of the ssh config after we've added in the QA box:

Host  acmetunnel
User  greg
Hostname  vp.endpoint.com

Host  acmedb
User  postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

Host  acmereplica
User  postgres
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

Host  acmeqa
User  postgres
Hostname  qa
ProxyCommand  ssh -q acreplica nc -w 180 %h %p

Note that we don't need the full hostname at this point for the "acmeqa" Hostname, as we can simply say 'qa' and the acreplica box knows how to get there.

There is still some unwanted repetition in the file, so let's take advantage of the fact that the "Host" item inside the ssh config file will take wildcards as well. It's not really apparent until you use wildcards, but a ssh host can match more than one "Host" section in the ssh config file, and thus you can achieve a form of inheritance. (However, once something has been set, it cannot be changed, so you always want to set the more specific items first). Here's what the file looks like after adding a wildcard section:

Host  acme*
User  postgres
ProxyCommand  ssh -q greg@vp.endpoint.com nc -w 180 %h %p

Host  acmedb
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com

Host  acmereplica
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com

Host  acmeqa
User  root
Hostname  qa
ProxyCommand  ssh -q acreplica nc -w 180 %h %p

Notice that the file is now simplified quite a bit. If we run this command:

$ ssh acmereplica

...then the Host acme* section sets up both the User and the ProxyCommand. It then also matches on the Host acmereplica section and applies the Hostname there.

Note that we have removed the "acmetunnel" section. Now that all the ProxyCommands are in a single place, we can simply go back to the original ProxyCommand and specify the exact user and host.

All of the above presumes we want to login as the postgres user, but there are also times when we need to login as a different user (e.g. 'root'). We can again use wildcards, this time to match the end of the host, to specify which user we want. Anything ending in the letter "r" means we log in as user root, and anything ending in the letter "p" means we log in as user postgres. Our final ssh config section for Acme is now:

##
## Client: Acme Corporation
##

Host  acme*
ProxyCommand  ssh -q greg@vp.endpoint.com nc -w 180 %h %p
Host  acme*r
User  root
Host  acme*p
User  postgres

Host  acmedb*
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com

Host  acmereplica*
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com

Host  acmeqa*
Hostname  qa
ProxyCommand  ssh -q acreplica nc -w 180 %h %p

From this point on, if Acme decides to add a new server, adding it into our ssh config is as simple as adding two lines:

Host  acmedev*
Hostname  ec2-456-55-999-45.compute-1.amazonaws.com

This automatically sets up two hosts for us, "acmedevr" and "acmedevp". What if we leave out the ending "r" or "p" and just ssh to "acmedev"? Then we'll connect as the default user, or $ENV{USER} (in my case, "greg").

Have fun configuring your ssh config file, don't be afraid to leave lots of comments inside of it, and of course keep it in version control!

New Year Bug Bites

Happy New Year! And what would a new year be without a new year bug bite? This year we had one where figuring out the species wasn't easy.

On January 2nd one of our ecommerce clients reported that starting with the new year a number of customers weren't able to complete their web orders because of credit card security code failures. Looking in the Interchange server error logs we indeed found a significant spike in the number of CVV2 code verification failures (Payflow Pro gateway error code "114") starting January 1st.

We hadn't made any programming or configuration changes on the system in the recent days. We double-checked to make sure: nope, no code changes. So it had to be a New Year's bug and presumably something with the Payflow Pro gateway or banks further upstream. We checked error logs for other customers to see if they were being similarly impacted, but they weren't. Our client contacted PayPal (the vendor for Payflow Pro) and they reported there were no problems with their system. The failures must indeed be card failures or a problem with the website according to them. We further checked our code looking for what we could possibly have done that might be the cause, double-checking our Git repository (which showed no recent changes) and reexamining our checkout code for possible year-based logic flaws.

Our client's top-notch customer service group got on the phone with a customer who'd gotten a security code failure and got PayPal tech support on another line. The customer service rep tried to place the customer's order on the website using the customer's credit card info and once again got the CVV2 error. She then did the credit card transaction using the swipe machine in the office, and lo and behold the order went through! What was going on??!

It turned out that despite the Payflow Pro gateway returning CVV2 verification errors what was really happening was that the year of the credit card was coming into the Payflow Pro gateway as "2012"—not as "2011" as entered into the checkout form. We knew all along that it was possible that the 114 error code responses were possibly misleading because payment gateway error codes are notorious this way. (Payment gateways blame the banks, saying they can only pass along what the banks give them. Some banks' credit card validations don't actually even care about the years being correct, but just that they not be in the past; but I digress...)

Previously we'd reviewed the checkout pages and the dropdown menus to verify that the dropdown menus weren't off, but nevertheless it very much sounded like this rather stupid problem could very well be the culprit. So we checked and checked again. What we found is that sometimes on the checkout form the year dropdown menu was mangled such that the values associated with the displayed years were YYYY+1.

The oddly intermittent behavior of the problem, the process of elimination and the all around hair pulling this loss of business was causing made somebody in the marketing group at our client realize that they are in fact still running an Omniture Test & Target A/B test on the checkout pages that they thought had been discontinued. To quote David Christensen (thanks, David!): "The Omniture system works by replacing select content for randomly chosen users in an effort to track user behavior/response to proposed site changes. Alternate site content is created and dynamically replaced for these users as they use the site, such as the specific content on the checkout page in this instance."

We verified that the Omniture A/B test's JavaScript replacement code was alternately mangling and not mangling the year dropdown on the checkout form as mentioned. Our client took out the A/B test and the "security code errors" dropped back to a normal low level.

This was a difficult and expensive problem—not only was there business lost because of the problem, but there were a lot of resources put into troubleshooting it. We've come away from this episode with some lessons learned and with plenty of food for thought. I'll leave it to commentators to opine away on this, including the End Point folks who scratched this itch: David Christensen, Jeff Boes, Mark Johnson, and Jon Jensen.

jQuery code for making a block level element clickable while maintaining left/middle/right clicking

While working on a recent redesign for a client we needed the ability to click on a div and have it function as a link to a product page. The initial implementation used the jQuery plugin BigTarget.js. The plugin searches within the div for a link and when the div is clicked changes the location to the link that is found. This plugin worked fine and was fairly easy to setup, however there was one drawback that we found once it was released in the wild. Most people expect to be able to right click, middle click, shift click, and control click to get the context-sensitive menu, open in a new window, or open in a new tab.

Enter Superlink.js, a jQuery plugin that uses a cool trick of moving the location of a link to the location of the mouse within the block level element (such as a div, li, tr, td, etc.). With this implementation the mouse is actually over a link so that the various ways of clicking function as expected. Initially I started moving the clickable area within a table, as shown in the example, but then quickly realized there was no reason this shouldn't work within a div or li. One other neat thing with this plugin is that the events attached to the block will continue to function, they are passed to the event handlers for the link being moved within the block.

Version Control Visualization and End Point in Open Source

Over the weekend, I discovered an open source tool for version control visualization, Gource. I decided to put together a few videos to showcase End Point's involvement in several open source projects.

Here's a quick legend to help understand the videos below:

The branches and nodes correlate to directories and files, respectively. In the case of the image to the left, the repository has a main directory with several files and three directories. One of the child directories has one file and the other two have multiple files.
A big dot represents a person, and a flash connecting the person and a file signifies a commit.
White + blue dots represent current End Point employees.
White + grey dots represent former End Point employees.
White dots represent other people, out there!

The Videos

Interchange from endpoint on Vimeo.

pgsi from endpoint on Vimeo.

Spree from endpoint on Vimeo.

Bucardo from endpoint on Vimeo.

One of the articles that references Gource suggests that the videos can be used to visualize and analyze the community involvement of a project (open source or not). One might also be able to qualitatively analyze the stability of project file architecture from a video, but this won't reveal anything definitive about the code stability since external factors can influence file structure. For example, since I am intimately familiar with the progress of Spree, I can identify when Spree transitioned to Rails 3 in the video, which required reorganization of the Spree core functionality (read more about this here and here).

In the case of this article, I wanted to highlight End Point's involvement in a few open source projects where we've had various levels of involvement. We've contributed to Interchange since 2000. We've been involved in Spree less lately, but had more presence in early 2009. In the smaller projects Bucardo and pgsi, End Point employees have worked on a team to be the primary contributors to the projects in addition to a few external contributors. Open source is important to End Point, and it's great to see our presence demonstrated in these cute videos.