Welcome to End Point’s blog

Ongoing observations by End Point people.

Thursday, May 24, 2012

Simple Pagination with AJAX

Posted by Jeff Boes

Here's a common problem: you have a set of results you want to display (search results, or products in a category) and you want to paginate them in a way that doesn't submit and re-display your results page every time. AJAX is a clear winner in this; I'll outline a very simple, introductory approach for carrying this off.

(I'm assuming that the reader has some modest familiarity with Javascript and jQuery, but no great expertise. My solutions below will tend toward the “Cargo Cult” programming model, so that you can cut and paste, tweak, and go, but with enough “how and why” sprinkled in so you will come away knowing enough to extend the solution as needed.)

Firstly, you have to have the server-side processing in place to serve up paginated results in a way you can use. We'll assume that you can write or adapt your current results source to produce this for a given URL and parameters:

/search?param1=123&param2=ABC&sort=colA,colB&offset=0&size=24

That URL offers a state-less way to retrieve a slice of results: in this case, it corresponds to a query something like:

SELECT … FROM … WHERE param1='123' AND param2='ABC'
ORDER BY colA,colB OFFSET 0 LIMIT 24

You can see that this will generate a slice of 0-24 results; changing “offset” will get other slices, which is the foundation of our ability to “page” the results.

The code behind “/search” should return a JSON structure suitable for your needs. My usual approach is to assemble what I want in Perl, then pass it through JSON::to_json:

my $results = perform_search(...);
my $json = JSON::to_json($results);

Don't forget to include an appropriate document header:

Content-type: application/json

Now we need a Javascript function to retrieve a slice; I'll use jQuery here as it's my preferred solution (and because I'm not at all fluent in non-jQuery approaches!).

function(){
  $.ajax({ url: '/search', data: { …, offset: $offset, limit: 24 },
…}

You'll need to keep track of (or calculate) the offset within your page. My approach is to drop each result into a div or similar HTML construction, then I can count them on the fly:

var $offset = $('div.search_result').length;

For the “data” passed in the AJAX call above, you need to assemble your query parameters, most likely from form elements on the page. (Newbie note: you can put <input> and <select> elements in the page without a surrounding <form>, as jQuery doesn't care -- you aren't going to submit a form, but construct a URL that looks like it came from a form.) Here's one useful model to follow:

var $data = { offset: $offset, limit: 24 };
$.each(['param1', 'param2'], function(ix, val) {
  $data[val] = $('input[name=' + val + '], select[name=' + val + ']').val();
};
and then:
$.ajax({ url: '/search', data: $data, … });

Now we need something to handle the returned data. Within the ajax() call, we reference (or construct) a function that takes at least one argument:

function(results) { … }

“results” is of course the JSON structure you built up in the “/search” response. Here, we'll assume that you are just sending back an array of objects:

[ { col1: 'val1', col2: 'val2', col3: 'val3' }, { col1: 'val4', col2: 'val5', col3: 'val6' } ]
would represent two rows of three columns each. We can now process these:
$.each(results, function(ix, val){
  var new_result = $('div.search_result').first().clone();
  $(new_result).find('span.col1').html(val.col1);
  $(new_result).find('span.col2').html(val.col2);
  $(new_result).find('span.col3').html(val.col3);
  $('div.search_result').last().append(new_result);
};

The entire working example can be seen in action here.

In place of an actual database query or search engine, I have a simple PHP program that sends back a chunk of simulated rows. A few other notes and finesses:

  • In the HTML document, I have a template for the "search_result" DIV that is hidden. I can style this any way I like, then I clone it for each returned result row. Note that it's initially hidden, so after appending a new clone to the page, I have to "show()" it.
  • I do some very simple arrangement of the results by inserting a hard break after every four results. You could do much fancier arrangements: assigning CSS classes based on whether the new result is at an edge of the grid, for instance.
  • Error handling in this example is very rudimentary.

Here's a screenshot of what you might expect, with Firebug showing the returned JSON object.


Monday, May 21, 2012

Liquid Galaxy at Doodle 4 Google

Posted by Jandro

Last week I went to Google’s New York Office on 8th Ave with Ben, intern Ben, and hired hand Linton. For those who have not experienced this wonderful place, Google’s building takes up an entire city block, is very colorful, and is probably one of the coolest places I have ever been to in the Big Apple. Walking through the huge building is an experience in itself, with people riding Razor Scooters by you as you pass by street signs marking different areas in the office (It was explained to me that each floor is themed after a different place in the city. For example, the 10th floor, the main floor we were working on, is based on Queens). And of course they have the best break rooms. Free food EVERYWHERE!

Also they have ball pits. You know you are awesome when you have ball pits.

Anyway, the reason we were at Google in the first place was to move the Liquid Galaxy on the 10th floor down to the 5th floor. It was great to see how many people came up to us and told how much they enjoyed using the system, and they all wanted to know when and if it would ever be back. Moving the Liquid Galaxy went smoothly, and setting it back up on the 5th floor (at the “Water Tower”) went even smoother, if possible. With everyone’s help we were able to get into the office and out in about four and a half hours with the Liquid Galaxy all set up, cleaned off, and in good working condition.

The move was made for the Doodle 4 Google event, an awards ceremony where 50 student finalists (Kindergarten to 12th grade, one from each state) are awarded for their interpretation of the Google homepage logo. The theme this year was “If I could go back in time, I would…” The national finalist won a $30,000 college scholarship, $50,000 for their school, a Chromebook, a Wacom Digital Design tablet, and their design printed on a t-shirt, though all finalists received some sort of scholarship and a t-shirt printed with their design. Also, if you checked Google's homepage last Friday (2012-05-18), you would have seen the winner’s drawing in place of the normal logo. The winner, a 2nd grader from Wisconsin, visited the golden era of pirates in his drawing. I did not know it at the time, but he was also the first kid to wander over to the Liquid Galaxy to find out what it was. He was a little intimidated at first, but after some friendly encouragement he decided to check it out. It turned out later that we would have to take him to the side and explain that he could not hit the top of the 3D Space Navigator and yell “SMASH” while other people were using the Liquid Galaxy (and completely disorienting all the surrounding parents in the process).

National winner’s drawing

Regardless, staffing the booth and talking to people was a great experience. Nearly everyone at the event (including the staff!) wanted to come over and fly around, and I heard many funny stories from children and adults alike as people went to different places that meant something to them. However, as an anonymous Internet user once said: “It’s amazing… with Google Earth you can fly anywhere in the world, but all anyone wants to see is their house,” and this was certainly true at Doodle 4 Google.

Linton and Jandro with some people checking out the LG

Doodle 4 Google lasted from 12:30 to 2:30, and after the event ended, Ben, Ben, Linton and I packed up the Liquid Galaxy once more and brought it back upstairs to the 10th floor. The second packing up and re-installation went even better than the first day, and Ben took the opportunity to show us some Linux System Administration and how to operate the Switched PDU (Power Distribution Unit) which controls the outlets that provide power to the computers on the Liquid Galaxy.

The two days at Google were very busy, but helping out at Google's office was a great experience, and one which I hope to enjoy again sometime in the future.

Website Performance Boot Camp at UTOSC 2012

Posted by Jon Jensen

I’ll keep brief my last post about this year’s Utah Open Source Conference.

I was asked to give on both day one and day two a talk called “Website Performance Boot Camp” which carried this brief description:

What’s the difference between a snappy website and a sloth that you turn away from in frustration? A lot of little things, usually. It’s rarely worth doing 100% of the optimization you could do, but getting 75% of the way isn’t hard if you know where to look.

We’ll look at HTTP caching, compression, proxying, CDNs, CSS sprites, minification, and more, how to troubleshoot, and what’s best to leave alone when you have limited time or tolerance for risk.

Here is the video recording of the first time I presented the talk. (The technician noted its audio was “a little hot”.)

Use this Website Performance Boot Camp direct YouTube video link if the embedded video doesn’t work for you.

The slides for this Website Performance Boot Camp presentation are available.

Thanks again to the conference organizers and the other speakers and sponsors, and the nice venue Utah Valley University, for making it a great conference!

Friday, May 18, 2012

UTOSC 2012 talks of interest

Posted by Jon Jensen

It's been two weeks now since the Utah Open Source Conference for 2012. My fellow End Pointers wrote previously about it: Josh Ausborne about the mini Liquid Galaxy we set up there for everyone to play with, and Josh Tolley with a write-up of his talks on database constraints and KML for geographic mapping markup.

There were a lot of interesting talks planned, and I could only attend some of them. I really enjoyed these:

  • Rob Taylor on AngularJS
  • Brandon Johnson on Red Hat's virtualization with oVirt, Spacewalk, Katello, and Aeolus
  • Clint Savage about RPM packaging with Mock & Koji
  • Daniel Evans on testing web applications with Capybara, embedded WebKit, and Selenium (which End Pointer Mike Farmer wrote about here back in December)
  • Aaron Toponce on breaking full-disk encryption (I missed this talk, but learned about it from Aaron in the hallway track and his slides afterwards)
  • Matt Harrison's tutorial Hands-on intermediate Python, covering doctest, function parameters and introspection, closures, function and class decorators, and more.

I gave a talk on GNU Screen vs. tmux, which was fun (and ends with a live demo that predictably fell apart, and audience questions that you can't hear on the recording). Here's the video:

Follow this Direct YouTube link in case the embedded version doesn't work for you. And here are the presentation slides.

I have a bit more to cover from the conference later!

Thursday, May 17, 2012

Keeping Your Apps Neat & Tidy With RequireJS

Posted by Greg Davidson

RequireJS is a very handy tool for loading files and modules in JavaScript. A short time ago I used it to add a feature to Whiskey Militia that promoted a new section of the site. By developing the feature as a RequireJS module, I was able to keep all of its JavaScript, HTML and CSS files neatly organized. Another benefit to this approach was the ability to turn the new feature "on" or "off" on the site by editing a single line of code. In this post I'll run through a similar example to demonstrate how you could use RequireJS to improve your next project.

File Structure

The following is the file structure I used for this project:
├── index.html
└── scripts
    ├── main.js
    ├── my
    │   ├── module.js
    │   ├── styles.css
    │   └── template.html
    ├── require-jquery.js
    ├── requirejs.mustache.js
    └── text.js

The dependencies included RequireJS bundled together with jQuery, mustache.js for templates and the RequireJS text plugin to include my HTML template file.

Configuration

RequireJS is included in the page with a script tag and the data-main attribute is used to specify additional files to load. In this case "scripts/main" tells RequireJS to load the main.js file that resides in the scripts directory. Require will load the specified files asynchronously. This is what index.html looks like:

<!DOCTYPE html>
<html>
<head>
<title>RequireJS Example</title>
</head>
<body>
<h1>RequireJS Example</h1>
<!-- This is a special version of jQuery with RequireJS built-in -->
<script data-main="scripts/main" src="scripts/require-jquery.js"></script>
</body>
</html>

I was a little skeptical of this approach working on older versions of Internet Explorer so I tested it quickly with IE6 and confirmed that it did indeed work just fine.

Creating a Module

With this in place, we can create our module. The module definition begins with an array of dependencies:

define([
  "require",
  "jquery",
  "requirejs.mustache",
  "text!my/template.html"
  ],

This module depends on require, jQuery, mustache, and our mustache template. Next is the function declaration where our module's code will live. The arguments specified allow us to map variable names to the dependencies listed earlier:

  function(require, $, mustache, html) { ... }

In this case we're mapping the $ to jQuery, mustache to requirejs.mustache and, html to our template file.

Inside the module we're using Require's .toUrl() function to grab a URL for our stylesheet. While it is possible to load CSS files asynchronously just like the other dependencies, there are some issues that arise that are specific to CSS files. For our purposes it will be safer to just add a <link> element to the document like so:

  var cssUrl = require.toUrl("./styles.css");
  $('head').append($('',
    { rel: "stylesheet", media: "all", type: "text/css", href: cssUrl }));

Next, we define a view with some data for our Mustache template and render it.

  var view = {
    products: [
      { name: "Apples", price: 1.29, unit: 'lb' },
      { name: "Oranges", price: 1.49, unit: 'lb'},
      { name: "Kiwis", price: 0.33, unit: 'each' }
    ],
    soldByPound: function(){
      return (this['unit'] === 'lb') ? true : false;
    },
    soldByEach: function() {
      return (this['unit'] === 'each') ? true : false;
    }
  }

  // render the Mustache template
  var output = mustache.render(html, view);

  // append to the HTML document
  $('body').append(output);
});

The Template

I really like this approach because it allows me to keep my HTML, CSS and JavaScript separate and also lets me write my templates in HTML instead of long, messy JavaScript strings. This is what our template looks like:

<ul class="hot-products">
{{#products}}
<li class="product">
{{name}}: ${{price}} {{#soldByEach}}each{{/soldByEach}}{{#soldByPound}}per lb{{/soldByPound}}
</li>
{{/products}}
</ul>

Including the Module

To include our new module in the page, we simply add it to our main.js file:

require(["jquery", "my/module"], function($, module) {
    // jQuery and my/module have been loaded.
    $(function() {

    });
});

When we view our page, we see that the template was was rendered and appended to the document:
Require rendered

Optimizing Your Code With The r.js Optimizer

One disadvantage of keeping everything separate and using modules in this way is that it adds to the number of HTTP requests on the page. We can combat this by using the the RequireJS Optimizer. The r.js script can be used a part of a build process and runs on both node.js and Rhino. The Optimizer script can minify some or all of your dependencies with UglifyJS or Google's Closure Compiler and will concatenate everything into a single JavaScript file to improve performance. By following the documentation I was able to create a simple build script for my project and build the project with the following command:

node ../../r.js -o app.build.js

This executes the app.build.js script with Node. We can compare the development and built versions of the project with the Network tab in Chrome's excellent Developer Tools.

Development Version:
Webapp devel

Optimized with the RequireJS r.js optmizer:
Webapp built

It's great to be able to go from 8 HTTP requests and 360 KB in development mode to 4 HTTP requests and ~118 KB after by running a simple command with Node! I hope this post has been helpful and that you'll check out RequireJS on your next project.

 

Wednesday, May 16, 2012

Vim - working with encryption

Posted by Terry Grant

On occasion I have to work with encrypted files for work or personal use. I am partial to a Linux environment and I prefer Vim as my text editor, even when I am only reading a file. Vim supports quite a few different ways of interfacing with external encryption packages. I only use two of those variations as described below.

Vim comes packaged with a default encryption mechanism referred to as VimCrypt in the documentation. I typically use this functionality as a temporary solution in a situation where my GPG is not immediately available, like a remote system that is not mine.

Using Vim's default VimCrypt feature

Creating a new encrypted file or open a plain text file you wish to encrypt:
vim -x 

This will create a new file if it does not exist or open an existing file and then prompt you for a password. This password is then used as the key to encrypt and decrypt the specified file. Upon saving and exiting this file, it will be saved in this encrypted format using your crypt key.

You can also save and encrypt an open file you are currently working on like so. Please note this is a capital X:

:X 
This will also ask you for a password to encrypt the file.

Reasons I usually don't use this option:
  • Vim uses a weak encryption method by default. Vim encrypts the file using an encryption method 'zip', the same encryption algorithm that is used by Pkzip (known to be flawed). You can set the default encryption to use the more secure 'blowfish' cipher. For more information see the documentation.
    Set the cryptmethod to use the blowfish cipher
    :setlocal cm=blowfish 
    
    Documentation on Vim Encryption
    :h :X
    
  • Typically uses swap files that can compromise the security of the encrypted file. You can turn this off by using the 'n' flag.
    vim -xn <filename>
    

Integrating with GPG

In order to seamlessly integrate with GPG encrypted files you will need to add the following to your .vimrc file

" Transparent editing of gpg encrypted files.
" By Wouter Hanegraaff
augroup encrypted
  au!

  " First make sure nothing is written to ~/.viminfo while editing
  " an encrypted file.
  autocmd BufReadPre,FileReadPre *.gpg set viminfo=
  " We don't want a swap file, as it writes unencrypted data to disk
  autocmd BufReadPre,FileReadPre *.gpg set noswapfile

  " Switch to binary mode to read the encrypted file
  autocmd BufReadPre,FileReadPre *.gpg set bin
  autocmd BufReadPre,FileReadPre *.gpg let ch_save = &ch|set ch=2
  " (If you use tcsh, you may need to alter this line.)
  autocmd BufReadPost,FileReadPost *.gpg '[,']!gpg --decrypt 2> /dev/null

  " Switch to normal mode for editing
  autocmd BufReadPost,FileReadPost *.gpg set nobin
  autocmd BufReadPost,FileReadPost *.gpg let &ch = ch_save|unlet ch_save
  autocmd BufReadPost,FileReadPost *.gpg execute ":doautocmd BufReadPost " . expand("%:r")

  " Convert all text to encrypted text before writing
  " (If you use tcsh, you may need to alter this line.)
  autocmd BufWritePre,FileWritePre *.gpg '[,']!gpg --default-recipient-self -ae 2>/dev/null
  " Undo the encryption so we are back in the normal text, directly
  " after the file has been written.
  autocmd BufWritePost,FileWritePost *.gpg u
augroup END
Source: Vim Wiki - Encryption

This works by detecting the extension on the files you are opening with Vim. This allows you to open, edit, and save files as if they were plain text in a seamless fashion.

Now you can create a new GPG encrypted file or edit an existing GPG encrypted doing this:

vim <filename>.gpg

This should prompt you for GPG password either with a GUI window or command line depending on your environment's configuration. If the file did not exist Vim creates one and when you save it will encrypt when you write and quit.

Another thing I like to do is save a file that I have already decrypted, but want to save in plain text. This can be done by simply opening the encrypted GPG file as seen above and change the extension when saving. Simply save like so:

:w <newfilename>.txt
Any extension other than .gpg will save your file as plain text.

Reason to use GPG
  • Much safer encryption as it uses GPG
  • No swap files thanks to this line in .vimrc autocmd BufReadPre,FileReadPre *.gpg set noswapfile

Of course when using Vim there are many features and many different ways of doing this. This is simply how I use Vim to easily work with encrypted files in my daily life.

For more information on Vim and external encryption programs please see and making use of GPG:



Tuesday, May 15, 2012

Points of Interest

Posted by Brian Buchalter

It's been a fairly straight forward week at work, but I have stumbled a few interesting finds along the way this week.

Vim Adventures

Finally! A game based approach to learning Vim keyboard commands. I was hoping someone would do this. It's just getting started (only two levels) and sadly, it looks like it'll be charging money to unlock higher levels. However, some things are worth paying for. I've found just playing the first two levels a few times have helped retrain my brain to not take my fingers off the home row. It's still quite buggy and seems to only work in Chrome. I found several times I needed to close all my Chrome windows after playing. Also, incognito mode seems to help with the bugs, as it disables all extensions you may have installed.

MySQL query comments in Rails

Ever wanted to know where that slow query was being called from? Well, if you're using MySQL with your Rails 2.3.x or 3.x.x app, you can get debug information about what controller's action made the call. Check out 37Signals new marginalia gem.

How to use EC2 as a web proxy

Kevin Burke provides a very detailed HOWTO article for working around restrictions you may experience in the course of an Internet based life. Pretty amazing what Amazon's free usage tier puts out there; of course it's only free for 12 months.

Include PIDs in your Logs

For many Rails developers we get comfortable looking at development log files. Sometimes when I have to investigate a customer issue on a production server using logs, I wished I had the level of detail the development logger had. While that's a wish, I'm finding it mandatory to include PID numbers in my production logs. In production systems with multiple requests being handled simultaneously, Rails logs start to become unusable. It's not clear which log lines are from which requests. Adding a PID in front of the time stamp can help untangle the mess. Here are some example approaches to this for Rails 3.x.x and Rails 2.3.x. Also, if you're really a log-lover and manage a lot of servers, check out Papertrail, it looks very impressive for $7/mo.

Spectrum Shortages - Why it's happening and what can be done

Telecom is not an area I have much familiarity with, but I found this article to be an interesting read. For example did you know that that largest owner of spectrum licenses are "under-capitalized or unwilling to build out networks" to use the spectrum? So while AT&T and Verizon struggle to meet the iPhone 4S's data demands (twice as much as iPhone 4!), "there are some companies that have spectrum, but they're struggling financially. Or they aren't quite sure what to do with the spectrum. And others that have the money and business model, but need the spectrum." It seems the way out of the mess is 4G, offering to improve the efficiency of spectrum use by 700 percent.