News

Welcome to End Point’s blog

Ongoing observations by End Point people

Highlighting Search Pattern Matches in Vim

Vim’s hlsearch option is a commonly-used way to enable visual feedback when searching for patterns in a Vim buffer. When highlighting of search matches is enabled (via :set hlsearch), Vim will add a colored background to all text matching the current search.

Search highlighting is useful but it’s not obvious how to turn off the highlighting when you’re no longer concerned with the results of your last search. This is particularly true if you didn’t enable the hlsearch setting yourself but inherited it from a prebuilt package like Janus or copied someone else’s .vimrc file.

One commonly-used way to clear the highlighting is to search for a garbage string by doing something like /asdfkj in normal mode. This method will clear the search highlights but has the undesired side effect of altering your search history.

A better way to disable search highlighting temporarily is with the :nohlsearch command (which can be abbreviated to :noh). This will clear the highlights temporarily, but they’ll be turned back on the next time you perform a search. Also, you can use the n/N keys to resume your previous search, which isn’t possible if you use the above method of searching for a garbage string.

For more information on highlighting search matches in Vim, check out the Highlight all search pattern matches entry on the Vim Tips Wiki.

Developer Specific Configuration in Ruby on Rails

Here's a quick tip on how to set Rails configurations that you might want for yourself but not for the rest of the team.

I find the default Rails log too noisy when I'm developing because it gives me more information than what I generally need. 90% of the time I only want to see what route is being hit for a request, what controller action responded to the route, and what parameters are being passed to the action. Finding this info with the default Rails log means wading through a jungle of SQL statements, and other things that I'm not interested in. Fortunately, Rails makes it easy to change log levels and the one I prefer is log level "info".

Setting this up however presents a new problem in that I recognize I'm deviating from what's conventional in the Rails world and I only want this configuration for myself and not anyone else working on the project. The typical way to change the log level would be to add a line to the environments/development.rb:
config.log_level = :info

If I do this and then commit the change I've now forced my own eccentricities on everyone else. What I could do instead is simply not commit it but then I create noise in my git workflow by having this unstaged change always sitting in my workspace and if I don't like noisy logs, I don't like dirty git workspaces even more. The solution I've come up with is to create a special directory to hold all my custom configurations and then have git ignore that directory.

  1. Create a directory with a specific name and purpose, I use config/initializers/locals.
  2. Add an entry to .gitignore

    locals/

  3. Add any configurations you want. In my case I created config/initializers/locals/log_level.rb which has the code that will change the log level at start up:

    Rails.logger.level = LOGGER::INFO

As a bonus you can add a "locals" directory anywhere in the application tree where it might be useful, and it will always be ignored. Perhaps you might stick one in app/models/locals where you can add decorators and objects that serve no other purpose than to aid in your local development.

Increasing MySQL 5.5 max_connections on RHEL 5

Busy database-backed websites often hit scalability limits in the database first. In tuning MySQL, one of the first things to look at is the max_connections parameter, which is often too low. (Of course another thing to look at is appropriate fragment caching in your app server, HTTP object caching in your web server, and a CDN in front of it all.)

When using MySQL 5.5 from Oracle's RPMs through cPanel (MySQL55-server-5.5.32-1.cp1136) on RHEL 5.10 x86_64, there is an interesting problem if you try to increase the max_connections setting beyond 214 in /etc/my.cnf. It will silently be ignored, and the limit remains 214:

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 214   |
+-----------------+-------+
1 row in set (0.00 sec)

The problem is that the maximum number of open files allowed is too small, by default 1024, to increase max_connections beyond 214.

There are plenty of online guides that explain how to handle this, including increasing the kernel fs.file-max setting, which may be necessary by editing /etc/sysctl.conf, in this example to double the default:

fs.file-max = 2459688

Then run sysctl -p to make the change take immediate effect. (It'll remain after reboot too.)

There are also many guides that say you need to change /etc/security/limits.conf along these lines:

mysql           soft    nofile         4096
mysql           hard    nofile         4096

However, the /etc/security/limits.conf change does not actually work when mysqld is started via the init script in /etc/init.d/mysql or via service mysql restart.

With standard Red Hat mysql-server (5.1) package that provides /etc/init.d/mysqld (not /etc/init.d/mysql as the Oracle and Percona versions do), you could create a file /etc/sysconfig/mysqld containing ulimit -n 4096 and that setting will take effect for each restart of the MySQL daemon.

But the ulimit -n setting hacked into the init script or put into /etc/sysconfig/mysqld isn't really needed after all, because you can simply set open_files_limit in /etc/my.cnf:

[mysqld]
open_files_limit = 8192
max_connections = 1000
# etc.

... and mysqld_safe will increase the ulimit on its own before invoking the actual mysqld daemon.

After service mysql restart you can verify the new open file limit in the running process, like this:

# cat /var/lib/mysql/*.pid
30697
# ps auxww | grep 30697
mysql    30697 97.8  9.8 6031872 1212224 pts/1 Sl   13:09   3:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/some.hostname.err --open-files-limit=8192 --pid-file=/var/lib/mysql/some.hostname.pid
# cat /proc/30697/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             96086                96086                processes
Max open files            8192                 8192                 files
Max locked memory         32768                32768                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       96086                96086                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0

And the running MySQL server will reveal the desired max_connections setting stuck this time:

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 1000  |
+-----------------+-------+
1 row in set (0.00 sec)

The relevant code in /usr/bin/mysqld_safe is here:

if test -w / -o "$USER" = "root"
then
  # ... [snip] ...
  if test -n "$open_files"
  then
    ulimit -n $open_files
  fi
fi

if test -n "$open_files"
then
  append_arg_to_args "--open-files-limit=$open_files"
fi

I have found that some newer versions of either MySQL55-server or cPanel or some intersection of the two has made manually specifying a higher open_files_limit in /etc/my.cnf no longer necessary, although it does not do any harm.

But in conclusion, if you find yourself hitting the mysterious max_connections = 214 limit, just add the appropriately-sized open_files_limit to the [mysqld] section of /etc/my.cnf and restart the server with service mysql restart, and your problem should be solved!

Building ImageMagick on RHEL/CentOS 6 with Perl 5.18.1

This is a quick tip for anyone in the exact same situation I was recently, and everyone else can probably just skip it!

RHEL 6 and CentOS 6 and other derivatives come with ImageMagick-6.5.4.7-6.el6_2.x86_64, which is a bit dated but still reasonable. They also come with Perl 5.10.1, which has grown very old. We wanted to use the latest version of Perl (5.18.1) with plenv, but the latest version of the Perl libraries for ImageMagick (PerlMagick) does not work with the older ImageMagick 6.5.4.7.

The first task, then, was to locate the matching older version of PerlMagick from BackPAN, the archive of historical CPAN modules: http://backpan.perl.org/authors/id/J/JC/JCRISTY/PerlMagick-6.54.tar.gz, and try to build that.

However, that fails to build without applying a patch to make it compatible with newer versions of Perl. The patch is available from http://trac.imagemagick.org/changeset?format=diff&new=4950, or you can just create a file called typemap in the root of the unpacked directory, with one line:

Image::Magick T_PTROBJ

Then build, test, and install as usual. That's it.

Setting a server role in Salt (comparing Puppet and Salt)

There are many ways to solve a given problem, and this is no truer than with configuration management. Salt (http://www.saltstack.com) is a fairly new tool in the configuration management arena joining the ranks of Puppet, Chef, and others. It has quickly gained and continues to grow in popularity, boasting its scalable architecture and speed. And so with multiple tools and multiple ways to use each tool, it can get a little tricky to know how best to solve your problem.

Recently I've been working with a client to convert their configuration management from Puppet to Salt. This involved reviewing their Puppet configs and designs and more-or-less mapping them to the equivalent for Salt. Most features do convert pretty easily. However, we did run into something that didn't at first- assigning a role to a server.

We wanted to preserve the "feeling" of the configs where possible. In Puppet they had developed and used a convention for using some custom variables in their configs to assign an "environment" and a "role" for each server. These variables were assigned in the node and role manifests. But in Salt we struggled to find a similar way to do that, but here is what we learned.

In Puppet, once a server's "role" and "environment" variables were set, then they could be used in other manifest files to select the proper source for a given config file like so:

    file    {
        "/etc/rsyslog.conf":
            source  =>
            [
                "puppet:///rsyslog/rsyslog.conf.$hostname", 
                "puppet:///rsyslog/rsyslog.conf.$system_environment-$system_role", 
                "puppet:///rsyslog/rsyslog.conf.$system_role", 
                "puppet:///rsyslog/rsyslog.conf.$system_environment", 
                "puppet:///rsyslog/rsyslog.conf"
            ],
            ensure  => present,
            owner   => "root",
            group   => "root",
            mode    => "644"
    }

Puppet will search the list of source files in order and use the first one that exists. For example, if $hostname = 'myniftyhostname' and $system_environment = 'qa' and $system_role = 'sessiondb', then it will use rsyslog.conf.myniftyhostname if it exists on the Puppet master, or if not then use rsyslog.conf.qa-sessiondb if it exists, or if not then rsyslog.conf.sessiondb if it exists, or if not then rsyslog.conf.qa if it exists, or if not then rsyslog.conf.

In Salt, environment is built into the top.sls file, where you match your servers to their respective state file(s), and can be used within state files as {{ env }}. Salt also allows for multiple sources for a managed file to be listed in order and it will use the first one that exists in the same way as Puppet. We were nearly there; however, setting the server role variable was not as straight forward in Salt.

We first looked at using Jinja variables (which is the default templating system for Salt), but soon found that setting a Jinja variable in one state file does not carry over to another state file. Jinja variables remain only in the scope of the file they were created in, at least in Salt.

The next thing we looked at was using Pillar, which is a way to set custom variables from the Salt master to given hosts (or minions). Pillar uses a structure very similar to Salt's top.sls structure- matching a host with its state files. But since the hostnames for this client vary considerably and don't lend themselves to pattern matching easily, this would be cumbersome to manage both the state top.sls file and the Pillar top.sls file and keep them in sync. It would require basically duplicating the list of hosts in two files, which could get out of sync over time.

We asked the salt community on #salt on Freenode.net how they might solve this problem, and the recommended answer was to set a custom grain. Grains are a set of properties for a given host, collected from the host itself- such as, hostname, cpu architecture, cpu model, kernel version, total ram, etc. There are multiple ways to set custom grains, but after some digging we found how to set them from within a state file. This meant that we could do something like this in a "role" state file:

# sessiondb role
# {{ salt['grains.setval']('server_role','sessiondb') }}

include:
  - common
  - postgres

And then within the common/init.sls and postgres/init.sls state files we could use that server_role custom grain in selecting the right source file, like this:

/etc/rsyslog.conf:
  file.managed:
    - source:
      - salt://rsyslog/files/rsyslog.conf.{{ grains['host'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}-{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}
      - salt://rsyslog/files/rsyslog.conf
    - mode: 644
    - user: root
    - group: root

This got us to our desired config structure. But like I said earlier, there are probably many ways to handle this type of problem. This may not even be the best way to handle server roles and environments in Salt, if we were more willing to change the "feeling" of the configs. But given the requirements and feedback form our client, this worked fine.

Database federation performance showdown

Flickr user garryknight

The PostgreSQL Foreign Data Wrapper has gotten a fair bit of attention since its release in PostgreSQL version 9.3. Although it does much the same thing the dblink contrib module has long done, it is simpler to implement for most tasks and reuses the same foreign data wrapper infrastructure employed by several other contrib modules. It allows users to "federate" distinct PostgreSQL databases; that is, it allows them to work in combination as though they were one database. This topic of database federation has interested me for some time -- I wrote about it a couple years ago -- and when postgres_fdw came out I wanted to see how it compared to the solution I used back then.

First, some background. The key sticking point of database federation that I'm focused on is transaction management. Transactions group a series of steps, so either they all complete in proper sequence, or none of them does. While lots of databases, and other technologies like messaging servers, can handle transactions that involve only one service (one database or one messaging server instance, for example), federation aims to allow transactions to span multiple services. If, for instance, given a transaction involving multiple databases, one database fails to commit, all the other databases in the transaction roll back automatically. See my post linked above for a more detailed example and implementation details. In that post I talked about the Bitronix transaction manager, whose job is to coordinate the different databases and other services in a transaction, and make sure they all commit or roll back correctly, even in the face of system failures and other misbehavior. There are other standalone transaction managers available. I used Bitronix simply because a knowledgeable friend recommended it, and it proved sufficient for the testing I had in mind.

So much for introduction. I wanted to see how Bitronix compared to postgres_fdw, and to get started I took the simple sequence of queries used by default by pgbench, and created a test database with pgbench, and then made three identical copies of it (named, of course, athos, porthos, aramis, and dartagnan -- I wasn't energetic enough to include the apostrophe in the name of the fourth database). The plan was to federate athos and porthos with Bitronix, and aramis and dartagnan with postgres_fdw. More precisely, the pgbench test schema consists of a small set of tables representing a simple banking scenario. In its default benchmark, pgbench selects from, inserts into, and updates these tables with a few simple queries, shown below. Like pgbench, my test script replaces identifiers starting with a ":" character with values selected randomly for each iteration.

UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);

I decided to configure my test as though pgbench's "accounts" table was in one database, and the "tellers", "branches", and "history" tables were in another. For the Bitronix test I can simply connect to both databases and ignore the tables that aren't applicable, but for testing postgres_fdw I need to set up dartagnan's pgbench_accounts table as a foreign table in the aramis database, like this:

aramis=# drop table pgbench_accounts;
DROP TABLE
aramis=# create server dartagnan foreign data wrapper postgres_fdw options (dbname 'dartagnan');
CREATE SERVER
aramis=# create user mapping for josh server dartagnan options (user 'josh');
CREATE USER MAPPING
aramis=# create foreign table pgbench_accounts (aid integer not null, bid integer, abalance integer, filler character(84)) server dartagnan;
CREATE FOREIGN TABLE

The test script I wrote has two modes: Bitronix mode, and postgres_fdw mode. For each, it repeats the pgbench test queries a fixed number of times, grouping a certain number of these iterations into a single transaction. It then changes the number of iterations per transaction, and repeats the test. In the end, it gave me the following results, which I found very interesting:

The results show that for small transactions, postgres_fdw performs much better. But when the transactions get large, Bitronix catches up and takes the lead. The graph shows a curve that may be part of an interesting trend, but it didn't seem worthwhile to test larger numbers of iterations per single transaction, because the larger transactions in the test are already very large compared to typical real-life workloads. It's difficult to see exactly what's going on in the center of the graph; here's a log rescaling of the data to make it clear what the numbers are up to.

All in all, it doesn't surprise me that postgres_fdw would be faster than Bitronix for small and medium-sized transactions. Being more tightly coupled to PostgreSQL, it has a faster path to get done what it wants to do, and in particular, isn't restricted to using two-phase commit, which is generally considered slow. I was surprised, however, to see that Bitronix managed to catch up for very large transactions.

End Point Partners with A-Zero to Expand Liquid Galaxy Services in South Korea

End Point Corporation continues its global leadership for Liquid Galaxy development and professional services, and has signed a partnership agreement with A-Zero of South Korea to further expand those services for the display platform and its associated professional services.

This partnership promises to be beneficial for the South Korean market. Already, A-Zero has lined up a number of engagements where the Liquid Galaxy could be deployed, bringing the incredible display platform together with the wealth of data resources in one of the most online-savvy countries in the world.

“We look forward to great business opportunities with our new friends at End Point,” said Cho Hyungwan, Director of Business Development for A-Zero. “We can see many uses for the platform here in our market.” A-Zero is a systems integrator and software development company based in Seoul with skills in GIS data manipulation, complex system deployments, and close relations with Google Enterprise partners in the region.

To kick-off this partnership, End Point brought a Liquid Galaxy Express to Seoul to show the platform at a nation-wide GIS conference together with A-Zero. The trade show was a great success, leading to several leads and near constant crowds at the booth.

As the lead agency for development and installation, End Point has brought Liquid Galaxies to over 50 locations around the world, including corporate offices, trade shows, and museum exhibits. End Point has installed numerous Liquid Galaxies in the United States and around the world, including permanent displays in Berlin, Brussels, Hamburg, Jakarta, London, Paris, Mexico City, Monaco, Moscow, Singapore, and Tokyo, and has set up and supported Liquid Galaxies at events in Amsterdam, Berlin, London, Jeju Island, Milan, Munich, Singapore, Sochi, Stockholm, Madrid, Munich and Paris.

Originally developed by Google, the Liquid Galaxy is a multi-screen display for 3D-rendered environments such as Google Earth, Google Street View, panoramic photos, videos, and GIS-data visualizations. End Point developers continue to extend that functionality with new data types, system integrations, visual interfaces for navigation, and content management for the display platform.

End Point is based in New York City, and has been providing technical development and solutions to complex problems for their clients since 1995. With over 35 developers and visualization specialists, End Point is the lead agency for providing turn-key installation, customization, and ongoing support for the Liquid Galaxy platform.

Use Ansible/Jinja2 templates to change file content based on target OS

In the End Point hosting team we really love automating repetitive tasks, especially when it involves remembering many little details which can over time be forgotten, like differences of coreutils location between some versions of Ubuntu (Debian), CentOS (Red Hat) and OpenBSD variants.

In our environment we bind the backup SSH user authorized_keys entry to a custom command in order to have it secured by being, among other aspects, tied to a specific rsync call.

So in our case the content of our CentOS authorized_keys would be something like:

command="/bin/nice -15 /usr/bin/rsync --server --daemon .",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAB3[...]Q== endpoint-backup

Sadly that's only true for CentOS systems so that if you want to automate the distribution of authorized_keys (as we'll show in another post) to different Linux distributions (like Ubuntu) you may need to tweak it to comply to the new standard "/usr/bin" location, which will be eventually adopted by all new Linux versions overtime.. RHEL 7.x onward included.

To do the OS version detection we decided to use an Ansible/Jinja2 template by placing the following line in the Ansible task:

- name: Deploy /root/.ssh/authorized_keys
  template: src=all/root/.ssh/authorized_keys.j2
            dest=/root/.ssh/authorized_keys
            owner=root
            group=root
            mode=0600

And inside the actual file place a slightly modified version of the line above:

command="{% if ansible_os_family != "RedHat" %}/usr{% endif %}/bin/nice -15 /usr/bin/rsync --server --daemon .",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAB3[...]Q== endpoint-backup"

So that if the target OS is not part of the "RedHat" family it will add the "/usr" in front of the "/bin/nice" absolute path.

Easy peasy, ain't it?
Now go out there and exploit this feature to all your needs.

Getting navigation bar to look good in iOS7

Apple has recently released iOS 7 — a major upgrade to its operating system for mobile devices. Whether users like it or not — developers have to treat it seriously. There is nothing worse in the world of technology than being viewed as passé.

From the point of view of users, the new look and feel resembles somewhat the overall movement in the user interface design. The flat UI style is the new hotness nowadays.

On the developers' side of the fence though, this means lots of hard work. Apple has introduced lots of changes so that many iOS apps had to be changed and even redesigned to look acceptable in iOS 7.

Some applications have already dropped support for older versions of iOS

One of them is... Evernote! Its team has decided that supporting older systems would be too costly and that they have decided to dump it. The only way to have the Evernote app is to have it installed before the release of its latest version.

The troublesome navigation bar

One issue I encountered while working on an iOS app lately was that the app started to display oddly. The top bar was overlapping with the contents of the views.

The reason is because now the top bar overlaps with the UI underneath. It applies a blur on whatever there is behind, making apps look a bit more integrated with the OS.

Solution hidden in the XIB designer

If you were using only the designer — you're very lucky. In the latest Xcode, there is an option to set deltas for UI element positions.

The UI delta is nothing more than a value that a particular measurement should be modified by if the app is being run on an iOS version lower than 7.

So keeping in mind that the top bar with the navigation view buttons area take 64 points of height — you have to provide -64 as y delta value. So that the UI in the designer looks great and it will also look nicely on a pre iOS 7 device.

What about views consisting purely of code?

In my case, I had to resort to some workarounds in the code. Most views in the application I was working on were created dynamically. There were no *.xib files to edit with the editor, hence — no way to set those deltas.

The easiest solution I found was to just edit view's frame values. Making the y of the frame at 64 points.

CGRect frame = tableViewController.view.frame;
frame.origin.y = 64;
tableViewController.view.frame = frame;

Supporting older versions in code

The last step was to simulate the behavior of the designer and allow the code to apply changes based on the iOS system version on which the app is currently being executed:

float topValue = 0;
if([[[UIDevice currentDevice] systemVersion] floatValue] >= 7.0f)
{
    topValue = 64;
}
CGRect frame = tableViewController.view.frame;
frame.origin.x = 0;
frame.origin.y = topValue;
tableViewController.view.frame = frame;

More to read

http://www.fastcolabs.com/3016423/open-company/developing-for-ios-7-the-good-the-bad-the-flat-and-the-ugly
https://developer.apple.com/library/ios/documentation/userexperience/conceptual/transitionguide/Bars.html

3 common misconceptions about Ruby on Rails

1) Rails is easy to learn.

Rails has a steep learning curve and that's never going away. From the beginning it was conceived of as an advanced tool for experienced web developers and having a low cost of entry for beginners is not part of the Rails way. Rails makes web development faster by depending on conventions that the developer needs to learn and utilize. There are a lot of these conventions and for a beginner the cost of learning them will be extremely low throughput initially. However, the investment pays off much later down the road when the throughput skyrockets due to familiarity with the framework. Another overlooked thing that makes Rails development hard to learn is that there is a lot more to learn than just Ruby and Rails, there's also things like source code management, databases, front end servers, testing and while these subjects are common to web development in general, you'll still need to learn the "Rails way" to approaching these satellite concerns that make up a typical production application. Brook Riggio created this fantastic chart that shows all the subjects that a Rails dev needs to be familiar with in the excellent article "Why Learning Rails is Hard", one look at the sheer number of items should illustrate why it takes a long time to learn Rails.

2) Rails is equally suited for all types of web apps.

To the extent that your web project veers away from being a dynamic website backed by a relational database with a page-centric design, or to put it another way to the extent that your app is different than Basecamp by 37signals is the extent to which you will be moving out of the sweet spot for developer productivity with Rails. What distinguishes Ruby on Rails from other web frameworks is that it was initially created by extracting it from a real world application rather than developing the framework first. The real world application it was extracted from is Basecamp by 37signals and even to this day the continued development of new Rails features is driven primarily by 37signals' need to "solve our own problems." While the web app landscape is evolving to include JavaScript layers that open up the possibilities for new client side architectures, it should be noted that Rails itself is optimized for building HTML pages and displaying them and any other type of architecture that works with Rails is working orthogonal to this goal.

3) Rails means you are working with Ruby and Ruby is slow.

Twitter is still sometimes cited as Ruby on Rails' biggest failure. Twitter was originally developed in Rails and shortly after its creation Twitter went through a phase where rapid growth of the site caused a major implosion of the site's ability to support the growth and the site was often down during a time that was known as the "Fail Whale" era. The solution to fixing Twitter's massive scaling problems was to strip out Rails as the engine of the site and introduce a massive messaging infrastructure made up of a multitude of different technologies. Rails was still a part of the infrastructure but only in a very superficial way serving up the web pages for the Twitter web client. This led some to the idea that Twitter was proof that Rails can't scale and anything "serious" should be done in a different technology. What the Twitter naysayers missed understanding is that the massive messaging behemoth that is Twitter was way outside of what Rails is designed to handle. More importantly however is that the ethos of the Ruby on Rails community is the "right tool for the right job", and that includes using non-Ruby and non-Rails technologies to support whatever solution you are after. If Java makes it go faster, then use Java. There's no reason why it has to be in Ruby, and while you might choose to use a Java search engine rather than a Ruby one chances are that Ruby still can act as an excellent "glue language" connecting your Java search engine to other technologies much like the Twitter messaging infrastructure was laid below Rails displaying pages in the web client. Rails devs should be comfortable using things outside of Ruby. A final comment on the Twitter controversy is that it's not likely Twitter would have ever come into existence without Rails because the low investment needed to try out an idea is precisely how Twitter came to be, and that in itself should be an example of one of Ruby on Rails' greatest successes.

New Kamelopard version

I recently pushed new Kamelopard version (v0.0.14), and thought I should briefly mention it here. This release includes a few bug fixes, including one that fatally affected several v0.0.13 installations, but its major improvement is a greatly expanded test suite. For quite some time many Kamelopard functions have had only placeholder tests, marked as "pending" in the code, or no test at all. In particular, this includes many of the more complex (or in other words, difficult to test) functions. Version 0.0.14 added 35 new tests, including for the frequently used orbit() function as well as for the relatively new multidimensional function logic.

The typical Kamelopard test creates a Kamelopard object, test that it responds to the right set of methods, renders it to KML, and finally inspects the result for correctness. This can quickly become complicated, as some KML objects can take many different forms. Here are a few selections from one of the new tests, as an example. This is for the ColorStyle object, which is an abstract class handling part of the options in other style objects.

This first section indicates that this test includes several other tests, defined elsewhere. Many objects in Kamelopard descend from Kamelopard::Object, for instance, this is far from the only test that refers to its behaviors.

shared_examples_for 'Kamelopard::ColorStyle' do
    it_should_behave_like 'Kamelopard::Object'
    it_should_behave_like 'KML_includes_id'
    it_should_behave_like 'KML_producer'

The KML spec defines a limited set of "color modes" allowed in a valid ColorStyle object, so we'll test the code that validates these modes, here.

it 'should accept only valid color modes' do
        @o.colorMode = :normal
        @o.colorMode = :random
        begin
            @o.colorMode = :something_wrong
        rescue RuntimeError => f
            q = f.to_s
        end
        q.should =~ /colorMode must be either/
    end

KML asks for its color constants in a different order than I'm used to. HTML asks for three byte color constants, with one byte each for red, green, and blue values, in that order. OpenGL's glColor function variants expect their arguments red first, then green, and then blue, with an optional alpha value at the end. So I sometimes get confused when KML wants alpha values first, then blue, then green, and finally red. Fortunately Kamelopard's ColorStyle object lets you set color and alpha values independently, and can sort out the proper order for you. This test verifies that behavior.

it 'should get settings in the right order' do
        @o.alpha = 'de'
        @o.blue = 'ad'
        @o.green = 'be'
        @o.red = 'ef'
        @o.color.should == 'deadbeef'
    end

Finally, this last segment renders the ColorStyle to KML and tests its validity. This particular test uses a helper function called get_obj_child_content(), defined elsewhere, because its particular XML parsing requirements are very common, but many of these tests which require more complex parsing make heavy use of XPath expressions to test the XML documents Kamelopard produces.

it 'should do its KML right' do
        color = 'abcdefab'
        colorMode = :random
        @o.color = color
        @o.colorMode = colorMode
        get_obj_child_content(@o, 'color').should == color
        get_obj_child_content(@o, 'colorMode').should == colorMode.to_s
    end

This new Kamelopard version also includes the beginnings of what could be a very useful feature. The idea is that Kamelopard objects should be able to create themselves from their KML representation. So, for instance, you could provide some Kamelopard function with a KML file, and it could create a Kamelopard representation which you can then process further. We already support each_placemark(), which iterates through each placemark in a KML document and returns the data therein, but this would expand that ability. Right now we're far from having all Kamelopard objects support parsing themselves from KML, but when it's completed it will open up interesting possibilities. For instance, it was originally conceived as a way to take a pre-existing tour and make a multicam version automatically. This, too, is still some way off.

Python decorator basics

Python decorators have been around since 2005, when they were included in the release of Python 2.4.1. A decorator is nothing more than syntax for passing a function to another function, or wrapping functions. Best put, a decorator is a function that takes a function as an argument and returns either the same function or some new callable. For example,

@foo
@bar
@baz
@qux
def f():
    pass

is shorthand for:

def f():
    pass
f = foo(bar(baz(qux(f))))

Say we have some functions we are debugging by printing out debug comments:

def mul(x, y):
    print __name__
    return x*y

def div(x, y):
    print __name__
    return x/y

The printing in the functions can be extracted out into a decorator like so:

def debug(f):            # debug decorator takes function f as parameter
    msg = f.__name__     # debug message to print later
    def wrapper(*args):  # wrapper function takes function f's parameters
        print msg        # print debug message
        return f(*args)  # call to original function
    return wrapper       # return the wrapper function, without calling it

Our functions get decorated with:

@debug
def mul(x, y):
    return x*y

@debug
def div(x, y):
    return x/y

Which again is just shorthand for:

def mul(x, y):
    return x*y
mul = debug(mul)

def div(x, y):
    return x/y
div = debug(div)

Looking at the definition of the debug function we see that debug(mul) returns wrapper, which becomes the new mul. When we now call mul(5, 2) we are really calling wrapper(5, 2). But how do subsequent calls to wrapper have access to the initial f parameter passed to debug and to the msg variable defined in debug? Closures. Taken from aaronasterling's response to this stackoverflow question, "A closure occurs when a function has access to a local variable from an enclosing scope that has finished its execution." You can read more about closures here, here, and here. So, at the moment that mul is decorated, debug(mul) is executed returning wrapper, which has access to the original mul function and to the msg variable, which is then set as the new mul.

By decorating, we remove code duplication and if the need to ever change the debug logic arises, we only need to do so in one place. Now, decorators with (non-optional) arguments get a bit trickier, but only because the syntax is a bit hard to grasp at first sight. Say that we want to pass the debug message as a parameter to the decorator like so:

@debug("Let's multiply!")
def mul(x, y)
    return x*y

Then the debug decorator would be:

def debug(msg):
    def actual_decorator(f):    # from here to
        def wrapper(*args):     # ...
            print msg           # ...
            return f(*args)     # ...
        return wrapper          # here, looks just like our debug decorator from above!
    return actual_decorator

A decorator with arguments should return a function that takes a function as an argument and returns either the same function or some new callable (what a mouthful, eh?). In other words, a decorator with arguments returns a decorator without arguments.

Looking at what the decorator syntax is shorthand for we can follow along as debug gets executed:

mul = debug("Let's multiply")(mul)

The debug function returns actual_decorator, to which we pass the mul function as the parameter, which then returns wrapper. So again, mul becomes wrapper which has access to msg and f because of closure.

What about decorators with optional arguments? That I'll leave for a future blog post :)

Managing Multiple Hosts and SSH Identities with OpenSSH

When I started working at End Point I was faced with the prospect of having multiple SSH identities for the first time. I had historically used an RSA SSH key with the default length of 2048 bits, but for my work at End Point I needed to generate a new key that was 4096 bits long.

Although I could have used ssh-copy-id to copy my new SSH public key to all of my old servers, I liked the idea of maintaining separate "personal" and "work" identities and decided to look for a way to automatically use the right key based on the server I was trying to connect to.

For the first few days I was specifying my new identity on the command line using:
ssh -i .ssh/endpoint_rsa patrick@server.example.com

That worked, but I often forgot to specify my new SSH identity when connecting to a server, only realizing my mistake when I was prompted for a password instead of being authenticated automatically.

Host Definitions


I had previously learned the value of creating an ssh_config file when I replaced a series of command-line aliases with equivalent entries in the SSH config file.

Instead of creating aliases in my shell:

alias server1='ssh -p 2222 -L 3389:192.168.1.99:3389 patrick@server1.example.com'

I learned that I could add an equivalent entry to my ~/.ssh/config file:

Host server1
  HostName server1.example.com
  Port 2222
  User patrick
  LocalForward 3389 192.168.1.99:3389

Then, to connect to that server, all I needed to do was run ssh server1 and all of the configuration details would be pulled in from the SSH config file. Replacing my series of shell aliases with Host definitions had the added benefit of automatically carrying over to other tools like git and mosh which read the same configuration.

Switching Identities Automatically


There's an easy solution to managing multiple SSH identities if you only use one identity per server; use ssh-add to store all of your keys in the SSH authentication agent. For example, I used ssh-add ~/.ssh/endpoint_rsa to add my new key, and ssh-add -l to verify that it was showing up in the list of known keys. After adding all of your keys to the agent, it will automatically try them in order for SSH connections until it finds one that authenticates successfully.

Manually Defining Identities


If you need more control over which identity an SSH session is using, the IdentityFile option in ssh_config lets you specify which key will be used to authenticate. Here's an example:

Host server2
  HostName server2.example.com
  User patrick
  IdentityFile ~/.ssh/endpoint_rsa

This usage is particularly helpful when you have a server that accepts more than one of your identities and you need to control which one should be used.

An update to the email_verifier gem has been released

I have just released a newer version of the email_verifier gem. It now supports Rails localization out of the box.
If you are one of gem's users — you can safely update your Bundles!

With this release, the project has also gained its first contributors. Some time ago, one of gem's users — Franscesco Gnarra has asked me about a possibility of having validation messages localized. While I was planning to do it myself — Francesco took an action and contributed to the project himself.

The project also received a pull request from another Rails developer — Maciej Walusiak. His commit provided a way to handle Dnsruby::NXDomain exceptions.

It's great to see that our work is helpful for others. If you'd like to contribute yourself — I invite you to do so.

Email verifier at GitHub: https://github.com/kamilc/email_verifier

A review of The Rails 4 Way

With a brand new project on the horizon (and a good candidate for Ruby on Rails 4 at that), I thought it would be a productive use of my time to bone up on some of the major differences introduced with this new version. My goal was to trade a little research for as many precluded that’s-not-how-we-do-it-anymore-500-errors as I could when fleshing out my first scaffold.

With this goal in mind, I purchased a copy of The Rails 4 Way by Obie Fernandez, Kevin Faustino, Vitaly Kushner, and Ari Lemer. Considering the free-nature of everything else Rails related, I will admit to a slight aversion to paying money for a Rails book. For those of you out there with similar proclivities, I felt compelled to share my experience.

The Rails 4 Way presents itself not as a tutorial of Ruby on Rails, but as “a day-to-day reference for the full-time Rails developer.” The tone of the book and the depth of information presented hints that the authors don’t want to teach you how to write Ruby on Rails applications. They want to share insight that will help you write better Ruby on Rails applications. Much of this is accomplished by fleshing out—utilizing a plentiful source of examples and snippets—the methods and features the framework has to offer.

Early on the authors concede the point, “There are definitely sections of the text that experienced Rails developer will gloss over. However, I believe that there is new knowledge and inspiration in every chapter, for all skill levels.” I would classify myself as an experienced Rails developer and I found this statement to be completely true. When settling down to read the book for the first time, I scrolled through the pages until a specific topic caught my eye. As a result of this, I started my read part-way through chapter 4. Upon completion of the last section, I wrapped back around to the beginning and dutifully read everything including the foreword. I did this specifically because each of the chapters, if not sections, had contained at least one little nugget of information that I found valuable. I felt compelled to go back and look for the extra tidbits that I’d passed over with the initial, casual flicks of my mouse.

I mentioned that the authors focus more on helping you produce better code, and to this end, they don’t dance around any issues. My favorite quote from the book illustrates this, “Frankly, there’s not much of a reason why that should be a problem unless you’ve made some pretty bad data-modeling decisions.” By all means, call it like it is.

The book is still a work-in-progress. Somewhere around 10% of the sections contain “TBD” rather than a body. I gather that the book is based on its Rails 3 counterpart and some of the sections have yet to be enhanced with the Rails-4-specific content. A majority share of the Rails 3 content is still applicable, so this shortcoming did not mar my experience at all. Especially since I will be given access to the new content once it is added. At the time of my download, the last change was authored 12 days ago, so for that reason also, I am unconcerned.

In conclusion, I took enough new information away, and enjoyed the process enough that I feel confident in recommending The Rails 4 Way to friends and coworkers. The bulk of this book reads like an API document, and in those sections, there’s nothing technical that you can’t find on the Rails API page itself. What the book does offer is back-story into what these methods were designed to do and what they pair well with. I would compare the Rails API and this book to a map in your hand and a local who knows the area. Most of the time you can figure out how to get where you need to go by carefully reading your unfolded map. But sometimes it’s faster to ask a local who can tell you, “See that red sign off in the distance? Head over to it and take a right.”

How to Enable MySQL Event Scheduler

You may think that you already know what's the opposite of "DISABLED", but with MySQL Event Scheduler you'll be wrong.

In fact MySQL Event Scheduler may have three different states[1][2]:

DISABLED -  The Event Scheduler thread does not run [1]. In addition, the Event Scheduler state cannot be changed at runtime.
OFF (default) - The Event Scheduler thread does not run [1]. When the Event Scheduler is OFF it can be started by setting the value of event_scheduler to ON.
ON - The Event Scheduler is started; the event scheduler thread runs and executes all scheduled events.

So if you're going to find it in the DISABLED state and instinctively set it to ENABLED you'll end up with a non-starting MySQL daemon.
Be warned and stay safe out there!


[1]: http://dev.mysql.com/doc/refman/5.5/en/events-configuration.html
[2]: When the Event Scheduler is not running does not appear in the output of SHOW PROCESSLIST

Testing Your Imagination

The usual blog post follows a particular format:

"I learned something new, as part of a task that I succeeded at. Here's what I did, here's why it worked so well, thank you for reading."

This one's a little different. I made a mistake, it seemed like a pretty simple thing, and then it got me thinking about why I (and in general, we software types), fall into that mistake, and how hard it is to correct.

Here's the background: I was working on a very small bit of code that was supposed to check a ZIP code and do one of two things. The test was, in Perl,

$zip =~ /^94|95/
Screetch!

Perhaps you have already spotted the bug. Don't feel so smug just yet. The particulars here are vital to understanding my point, but the bug could have been something more complex, or even simpler, and I am willing to bet a cubic yard of virtual cash that you have made equally embarrassing errors. I'm just more willing to talk about mine, that's all.

Back to our tale. I wrote that mistaken bit of code (and worse, it's not the first time I've made that mistake in code), and then I proceeded to test it.

  • Set $zip to '12345', doesn't match, no false positive. Check!
  • Set $zip to '94001', does match. Check!
  • Sest $zip to '95001', does match. Check!

(If you got this far and still haven't figured out the punch line: the regular expression matches "94" at the beginning of a string, which is good, but it matches "95" anywhere in the string, which is bad. So $zip containing "89501" would match, which is ... not good.)

It doesn't matter if you tested this in an external script, or went through the motions of operating the larger system (an e-commerce checkout) with the appropriate values of $zip, or wrote a full test-driven development exercise – the problem isn't the testing methodology, it's the imagination of the person desiging the test. I "knew" (nope) what the regular expression was written to do, so I tested to make sure it did that.

The only ways to catch this particular bug would be (a) exhaustively testing all values from 00000–99999, or (b) imagining ways that the regular expression might be broken. And that's the challenge here, pertaining to my title. How do you rig your imagination to construct test cases that are "outside the box"?

Darned good question. If my brain continues to write

$zip =~ /^94|95/

instead of

$zip =~ /^(?:94|95)/

then that same brain will continue to construct test cases that are "close but not quite". And if your assumptions about your code can be flawed in a simple case, how much more so when you are dealing with 100 lines? 1,000? An arbitrarily large system, or one that had "special" challenges?

I don't have an answer here, and I suspect no one does. Certainly it helps if you have a fellow engineer get involved and review your code (and your testing!), but not every project has a budget for that. Certainly it helps if your client gets involved, and can test with their (hopefully!) better understanding of business rules and conditions. (And when I say "test", I mean both "actually operate the system" and "actively contribute to test cases" such as "have you tried 89501? Have you tried A1A 1A1?")

I just know that we have to find a better way to construct test cases than relying on the imagination from the same brain that made the bug in the first place, before we start putting software in charge of important things like this or this Thank you for raeding. Er, reading.

Using JavaScript in PostgreSQL

This time I will describe two things: installing a new extension using pgxn and using JavaScript for writing a PostgreSQL stored procedure.

The last time I was describing a couple of nice features of the incoming PostgreSQL 9.3, I wrote about merging JSONs in Postgres using a stored procedure written in Python. In one of the comments there was a suggestion that I should try using JavaScript for that, as JSON is much more native there.

So let's try JavaScript with PostgreSQL.

Installing PL/V8

PL/V8 is a PostgreSQL procedural language powered by V8 JavaScript Engine. This way we can have JavaScript backed, something funny which could be used to create something like NoSQL database, with JavaScript procedures and storing JSON.

To have this procedural language, you need to install it as a separate extension. This can be done with system packages, if your system provides them. For Ubuntu, which I use, there are packages ready, however I use PostgreSQL compiled from source, and I keep it in my local directory, so I had to install it in a little bit different way.

I keep my PostgreSQL in ~/postgres directory. The ~/postgres/bin directory is added to environmnent $PATH variable. It is important, as the further steps will use pg_config program, which prints lots of information about the PostgreSQL installation.

The code for PL/V8 can be found in the project's PGXN page. You can of course download source, and install it. However there is much simpler way to do it. PGXN provides a tool for managing extensions stored there.

To get the client for pgxn, it's enough to write:

$ pip install pgxnclient

To install PL/V8 you also need to have the developer library for the V8 engine:

$ sudo apt-get install libv8-dev
$ pgxn install plv8

Now you can install the extension with a simple command:

$ pgxn install plv8

This should download, compile and copy all the files into a proper directory described by the pg_config program.

Use PL/V8

For each database where you want to use this extension, you need to create it:

plv8=# CREATE EXTENSION plv8;
CREATE EXTENSION

Write Procedure in JavaScript

This is an example I used in the previous post:

WITH j AS (
  SELECT
    '{"a":42, "b":"test"}'::JSON a,
    '{"a":1764, "x":"test x"}'::JSON b
)
SELECT a, b
FROM j;
          a           |            b             
----------------------+--------------------------
 {"a":42, "b":"test"} | {"a":1764, "x":"test x"}

The implementation of the merging function in JavaScript can look like this one:

CREATE OR REPLACE FUNCTION merge_json_v8(left JSON, right JSON)
RETURNS JSON AS $$
  for (var key in right) { left[key] = right[key]; }
  return left;
$$ LANGUAGE plv8;

You can use it exactly like the previous Python version:

WITH j AS (
  SELECT
    '{"a":42, "b":"test"}'::JSON a,
    '{"a":1764, "x":"test x"}'::JSON b
)
SELECT
  a,
  b,
  merge_json_v8(a, b)
FROM j;

Zero Downtime Deploys with Unicorn

I was recently deploying a new Ruby on Rails application that used NGINX and Unicorn for production. During the deploy, my common practice was to stop the Unicorn processes and then restart them. I would do this by finding the PID (process id) of the running process, stop it using the kill command and then run the unicorn_rails command from my application root directory. This worked well enough that I put together a simple unicorn_init shell script to handle running the commands for me.

After a couple of deploys using this init script, I found that there was a significant inturruption caused to the site. This was due to the approximately 20 seconds it took for the Unicorn workers to launch. This was unaccepatble and I started a search for how to perform a zero downtime deploy for Unicorn.

My search lead me to the Unicorn Signal Handling documentation. Unicorn makes use of POSIX Signals for inter-process communication. You can send a signal to a process using the unfortunately named kill system command. Reading through the different signals and what message they send to the Unicorn master and workers, I found a better approach to restarting my Unicorn processes that would result in no delay or inturruption to the website.

On the Signal Handling page (linked above) is a section called Procedure to replace a running unicorn executable. The key to my problem lied in this explanation:

You may replace a running instance of Unicorn with a new one without losing any incoming connections. Doing so will reload all of your application code, Unicorn config, Ruby executable, and all libraries.

This was exactly what I needed. I did a quick search online to see if anyone had put together an init script that used this method for restarting Unicorn and found one on a gist on github. With a few modifications, I assembled my new init script:

#!/bin/sh
# based on https://gist.github.com/jaygooby/504875
set -e

sig () {
  test -s "$PID" && kill -$1 `cat "$PID"`
}

oldsig () {
  test -s "$OLD_PID" && kill -$1 `cat "$OLD_PID"`
}

cmd () {
  case $1 in
    start)
      sig 0 && echo >&2 "Already running" && exit 0
      echo "Starting in environment $RAILS_ENV for config $CONFIG"
      $CMD
      ;;
    stop)
      sig QUIT && echo "Stopping" && exit 0
      echo >&2 "Not running"
      ;;
    force-stop)
      sig TERM && echo "Forcing a stop" && exit 0
      echo >&2 "Not running"
      ;;
    restart|reload)
      echo "Restarting with wait 30"
      sig USR2 && sleep 30 && oldsig QUIT && echo "Killing old master" `cat $OLD_PID` && exit 0
      echo >&2 "Couldn't reload, starting '$CMD' instead"
      $CMD
      ;;
    upgrade)
      sig USR2 && echo Upgraded && exit 0
      echo >&2 "Couldn't upgrade, starting '$CMD' instead"
      $CMD
      ;;
    rotate)
      sig USR1 && echo "rotated logs OK" && exit 0
      echo >&2 "Couldn't rotate logs" && exit 1
      ;;
    *)
      echo >&2 "Usage: $0 <start|stop|restart|upgrade|rotate|force-stop>"
      exit 1
      ;;
    esac
}

setup () {
  cd /home/mfarmer/camp2/rails # put your own path here
  export PID=/home/mfarmer/camp2/var/run/unicorn.pid # put your own path to the pid file here
  export OLD_PID="$PID.oldbin"
  export CONFIG=/home/mfarmer/camp2/unicorn/unicorn.conf # put your own path to the unicorn config file here
  export RAILS_ENV=development # Change for use on production or staging

  CMD="bundle exec unicorn_rails -c $CONFIG -E $RAILS_ENV -D"
}

start_stop () {
  . $CONFIG
  setup
  cmd $1
}

ARGS="$1"
start_stop $ARGS

The script is pretty self explanitory and it supports the major term signals outlined in the Unicorn documentation. With this new init script in hand, I now have a better way to restart Unicorn without negatively impacting the user experience.

There are a couple of caveats to this approach that you should be aware of before just slapping it into your system. First, in order to perform the restart, your application needs to essentially run twice until the old master is killed off. This means your hardware should be able to support running two instances of your application in both CPU and RAM at least temporarily. Second, you'll notice that the actual command being run looks something like bundle exec unicorn_rails -C /path/to/config/file -E development -D when interpolated by the script. This means that when you perform a rolling restart, those same parameters are used for the new application. So if anything changes in your config file or if you want to switch environments, you will need to completely stop the Unicorn processes and start them again for those changes to take effect.

Another thing you should be aware of is that your old application will be running for the 30 seconds it takes for the new application to load so if you perform any database migrations that could break the old version of your application, you may be better served by stopping the Unicorn process, running the migration, and then starting a new process. I'm sure there are ways to mitigate this but I just wanted to mention it here to help you be aware of the issue with this script in particular.

Piggybak Dependency & Demo Updates

Things have been quiet on the Piggybak front lately, but we recently upgraded the demo to Ruby 2.0.0 via rbenv, Rails 3.2.15, and Postgres 9.3. The Piggybak demo runs on Debian 7 with nginx and Unicorn. The upgrade went fairly smoothly, with the exception of jQuery related issues, described below.

As of jQuery 1.7, the live() method is deprecated, replaced with the on() method. As of jQuery 1.10.*, the live() method no longer exists. The previous version of Rails that was used on the demo, Rails 3.2.12, required the jquery-rails gem version which included an older version of jQuery. Upon upgrading to Rails 3.2.15, the attached jquery-rails gem now includes jQuery 1.10.*, resulting in the live() method no longer existing. As a result, several of the dependencies needed to be updated to accomodate this change (Rails_admin, the Piggybak Coupon gem, and the Piggybak Gift Cert gem, jQuery Nivo Slider).

What's next for Piggybak? Our future plans include an upgrade to support Rails 4.0. Additional features described on our last Roadmap Update include advanced taxonomy, reviews & ratings, saved cart, wishlist functionality, and saved address support. Piggybak continues to be a great mountable ecommerce solution for Ruby on Rails, but End Point has a great deal of experience with other popular Ruby on Rails ecommerce platforms as well.

Mooving to the Mobile Web

With the rise of the myriad of mobile phones, tablets and other devices that are connected to the internet, the potential users for a given website have both increased in number and morphed in their needs in terms of a user experience. As anyone who has attempted to use a website not designed for a mobile phone browser with frantic pinch-zooms to find a tiny page control, right next to four other controls that do something totally different in a menu, can attest this is really not ideal.

And meanwhile from the other perspective, for web developers, the notion of a fragmented user base over everything from a desktop PC with a modern browser to an embedded PC built into my new LCD TV needing to view your pages gracefully can be a scary prospect. The thought of maintaining independent versions of your web infrastructure that fit each of these major use cases would likely scare everyone in your company, especially the finance people cutting all the checks.

So what is a company facing this new reality of the modern web to do? One particular solution can help to alleviate one of the more troublesome issues with new devices browsing the Internet, mobile phone display. While the phones themselves are starting to come with resolutions comparable to a modern desktop or laptop PC, the screen size relative to your finger that is selecting inputs, leaves a far less precise pointer than a mouse. This is precisely the problem described earlier, as those page control designs with multiple options in a menu, comes from the design era when a precise mouse click as the input method was the norm. Additionally, with the smaller screen size, it is necessary to more prominently highlight certain aspects of the page design, like images of featured products, to keep the same level of emphasis of those images on the user who will view them.

One firm that is implementing a version of this solution that I recently helped implement for a client is Moovweb. For web developers, this technology will allow them to accomplish those goals of making page controls more usable on mobile phones, make featured elements of page layout stand out more effectively on a mobile phone browser, without actually maintaining a separate version of their site, optimized for mobile users. Moovweb will make a request to your site, optimize the content for mobile (behind the scenes) and the send that optimized response back to the user's device. In this manner, the page contents will always be updated automatically for the mobile sites, based on what is present on your current live site. Which page elements are selected to be displayed on the mobile page, and how they are displayed are all configurable options within Moovweb's controls, and you are also able to use a number of pre-built templates based on web software packages that are common.

Technical Details

How does Moovweb accomplish this sleight of hand -- tailoring the response based on the device requesting the information? The secret lies both in JavaScript and DNS. Firstly, in order to setup your domain for Moovweb, you need to create a sub-domain that requests from mobile devices would be forwarded to, which would actually point to Moovweb with a CNAME record. Here is an example from the setup documentation:

If your domain was example.com, and the mobile sub-domain you had selected was m.example.com you would create a CNAME record for:

m.example.com.     IN     CNAME     m.example.com.moovdns.net.

This would point any request to m.example.com over to the Moovweb servers, which will then carry out the forwarding of the requests back to the mobile browser of the user once the template had been applied to the page design and crafted the mobile version of the site.

For the JavaScript setup, a <script> tag must be added to the design of each page's <head> tag in order to perform redirection of requests for mobile browsers. This script that is added is created for each customer by Moovweb, and is used to match the User-Agent setting of each request against a list of known mobile browsers. Conceivably, with this in place on every page, whether the user is attempting to load the main page, or perhaps a deeper link to something like a product page or category page, every request from the mobile browsers should be automatically redirected to the mobile domain that we setup the CNAME record for.

How it looks

When working on deploying Moovweb for a client, tigerdistrict.com, I was introduced to the technology for the first time, and I was impressed with the mobile site experience. The page controls that were modified to make them easier to tap with your finger, and also making the page layout more portrait which fits the mobile phone form factor. Here are some examples of the mobile and non mobile site:
Desktop Version

Mobile Version
One of my favorite features was how Moovweb could handle page navigation menus, like the one you see on the left margin of the page in the desktop version. On a mobile device, attempting to get a precise enough point to select only the correct one of those options, and not mistakenly clicking others, would be painful to say the least. However after the site has been converted to the mobile version, two new page elements are added to the bar at the top of the page. There is a cart icon representing the all important shopping cart, and one of the legendary "Hamburger button" controls that opens up the page navigation menu. Here is what it looks like on the mobile browser:
No Dialing Wand Needed
As you can see, it replicates the same menu tree, with the same ability to expand into any of the available categories, or use the text search, all within an interface that is easy to use within a mobile browser.

The Future of the Web - Devices, Devices, Devices

One thing is clear from the rise of mobile devices and tablets on the web over the past few years, is that these devices are here to stay, and if anything will continue to grow and dominate the marketplace. For developers seeking to harness, or even to just stay ahead of this trend, will need to address the problems of mobile browsers and the limitations of the devices themselves. 

Creating websites that provide a desirable user experience to these many flavors of devices can be a daunting challenge, but in a way this shows us that the true issue is not the fragmentation itself. The real benefits from this type of mobile web development take advantage of your existing infrastructure, and gives you a method to tailor it as best you can to fit each of these new device's abilities. Duplicating effort is wasteful and maintaining multiple versions the the same content with slightly different presentation is an example of this wasted effort. Reusing the current web infrastructure already available and already being invested in the maintenance of, allows the presentation of these multiple user experiences in a cost effective way. 

Copying Rows Between PostgreSQL Databases

A recurring question is: ‘how can I copy a couple of rows from one database to another’? People try to set up some replication, or dump entire database, however the solution is pretty simple.

Example

For this blog post I will create two similar tables, I will be copying data from one to another. They are in the same database, but in fact that doesn’t matter, you can use this example to copy to another database as well. Even on another server, that’s enough to change arguments for the psql commands.

The tables are:

test=# CREATE TABLE original_table (i INTEGER, t TEXT);
CREATE TABLE
test=# CREATE TABLE copy_table (i INTEGER, t TEXT);
CREATE TABLE

Now I will insert two rows, which I will copy later to the “copy_table”.

test=# INSERT INTO original_table(i, t) VALUES
  (1, 'Lorem ipsum dolor sit amet'),
  (2, 'consectetur adipiscing elit');
INSERT 0 2

test=# SELECT * FROM original_table ;
 i |              t              
---+-----------------------------
 1 | Lorem ipsum dolor sit amet
 2 | consectetur adipiscing elit
(2 rows)

test=# SELECT * FROM copy_table;
 i | t 
---+---
(0 rows)

The Solution

Of course I can set up replication, which is too much effort for ad hoc copying two rows. Of course I could dump entire database, but imagine a database with millions of rows, and you just want to copy those two rows.

Fortunately there is “copy” command. However simple copy saves file on the same server as PostgreSQL is running, and usually you don’t have access to the file system there. There is another command, that’s internal command for psql, it is named “\copy”. It behaves exactly like copy, but it writes files on the machine you run psql at.

Save To File

The first and simplest solution we could save those two rows into a file, and load it later on another database.

First, let’s find out how “\copy” works:

$ psql test -c \
"\copy (SELECT i, t FROM original_table ORDER BY i) TO STDOUT"

1 Lorem ipsum dolor sit amet
2 consectetur adipiscing elit

As you can see, the main part of this command is the select query which allows to choose rows we want to export. You can provide there any “where” clause you want.

So now we can save it to a file:

$ psql test -c \
"\copy (SELECT i, t FROM original_table ORDER BY i) TO STDOUT" > /tmp/f.tsv

Loading now is also pretty easy with the same “\copy” command.

psql test -c "\copy copy_table (i, t) FROM STDIN"

Don’t Save to File

Saving to a file has one drawback: if the data amount is huge, then the file will be huge as well, it will waste disk space, and can be slower than using a pipe to load data. You can use a pipe to join the output of one psql command with input of another one. This is as simple as:

psql test -c \
"\copy (SELECT i, t FROM original_table ORDER BY i) TO STDOUT" | \
psql test -c "\copy copy_table (i, t) FROM STDIN"

test=# SELECT * FROM copy_table ;
 i |              t              
---+-----------------------------
 1 | Lorem ipsum dolor sit amet
 2 | consectetur adipiscing elit
(2 rows)

As you can see, that’s much simpler than setting up replication or dumping whole database.

SELinux fix for sudo PAM audit_log_acct_message() failed

I was just reading my co-worker Lele's blog post about making SELinux dontaudit AVC denial messages visible and realized it was likely the solution to a mystery I ran into a few days ago.

As Lele explains, the SELinux dontaudit flag suppresses certain very common SELinux AVC denials to keep the audit logs from bloating beyond belief and being too hard to use. But sometimes a commonly harmless denial can be the cause of further errors. You can tell this is the case if temporarily disabling SELinux enforcing (setenforce 0) makes the problem go away, but /var/log/audit/audit.log still doesn't show any AVC denial actions being allowed through.

In my somewhat unusual case there is an Apache CGI shell script that calls sudo to invoke another program as a different user without using setuid or suEXEC. Everything works fine with SELinux enforcing, but there are some strange errors in the logs. In /var/log/secure:

sudo: PAM audit_log_acct_message() failed: Permission denied

And in the Apache error_log is the apparently strangely unbuffered output:

[error] sudo
[error] : 
[error] unable to send audit message
[error] : 
[error] Permission denied
[error]

To show the dontaudit AVC denials, I ran semodule -DB as Lele explained, and then I saw in /var/log/audit/audit.log:

type=AVC msg=audit(1384959223.974:4192): avc:  denied  { write } for  pid=14836 comm="sudo" scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:system_r:httpd_sys_script_t:s0 tclass=netlink_audit_socket
type=AVC msg=audit(1384959223.975:4194): avc:  denied  { read } for  pid=14836 comm="sudo" scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:system_r:httpd_sys_script_t:s0 tclass=netlink_audit_socket
type=AVC msg=audit(1384959223.999:4196): avc:  denied  { write } for  pid=14836 comm="sudo" scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:system_r:httpd_sys_script_t:s0 tclass=key

Somewhat as expected from the error messages we saw before, sudo is being denied permission to send a message into the kernel. Now that we have the AVC errors from the audit log it's easy to make a local SELinux policy module to allow this.

The spurious error messages go away, and we can run semodule -B to re-suppress the dontaudit messages.

Thanks, Lele. :)

Asynchronous Page Switches with Django

Now that the newly rebuilt http://www.endpoint.com/ website is up and running, you may have noticed it does something fancy: internal links within the site are fetched in the background, and the page is replaced dynamically with a script. That eliminates the 'flicker' of normal website navigation, and removes the need for the browser to re-parse CSS and JavaScript, making it feel more responsive.

Recently I did some work on a Django project that uses jQuery for some AJAX calls to send information back to the database. It was a fairly simple $.post() call, but it got me thinking about Django's template inheritance and how it could be used to render parts of templates and update those client-side without having to render the whole thing. The idea being, if your base template is complex and has a number of built-in queries or calculations, for example if you have a dynamic navigation menu, why put extra load on Postgres, on the web server, or have the browser reload the CSS, JS, images, or other resources, to load in what could be otherwise static data into a content column?

The idea's a little half-baked, just the result of a little after-hours tinkering over a couple evenings. Certainly hasn't been fleshed out in a production environment, but seems to work okay in a test. I probably won't develop it much more, but maybe the concepts will help someone looking for something similar.

There's a few options out there, apparently, between django-ajax-blocks (which seems to do something similar to what I'm about to describe) and Tastypie, which lets you easily build with REST-based frameworks. Django's usually pretty good about that, having projects available that build functionality on top of it. But not having researched those at the time, I put together this basic technique for doing the same:

  1. Create a replacement template render function that detects AJAX-y requests.
  2. Update your page templates to use a variable in {% extends %}.*
  3. Create a simple XML base template with the blocks you use in your normal base template.
  4. Add a little JavaScript to perform the content switch.
* Yes, this works, which was a bit of a surprise to me. It's also the part I'm least happy with. More on that in a bit.

The details...

Template Render Function

This takes after the handy django.shortcuts.render function. In fact, it leans on it fairly heavily.

def my_render(request, template, context={}, html_base='main.html', xml_base='main.xml'):
    if request.is_ajax():
        context['base_template'] = xml_base
        return render(request, template, context, content_type='application/xml')
    else:
        context['base_template'] = html_base
        return render(request, template, context)

Giving html_base and xml_base as parameters lets views override those. This then injects a new variable, base_template, into the context passed to it with the appropriate base template.

Update Page Templates

Your page templates, assuming they now at the top say {% extends 'main.html' %}, replace with {% extends base_template %}. You shouldn't have to make any other changes to it.

But again, this is the bit I'm least happy about. It takes the selection out of the page template, and puts it in code. That takes away some of the decoupling an MVC environment like this is supposed to provide. Haven't come up with a way around it, though.

Create XML Base Template

In templates/main.xml (or whatever you want to call it above) create XML nodes for the blocks in your main.html file. Or at least the blocks your pages will replace:

<?xml version="1.0" encoding="UTF-8"?>
<content>
 <title><![CDATA[{% block title %}Django Site!{% endblock %}]]></title>
 <main_content><![CDATA[{% block main_content %}{% endblock %}]]></main_content>
</content>

Like your main.html, you can have defaults for the blocks here, such as a default title.

Why XML? I'd originally envisioned using JSON, but it has escaping rules, of course. Django, so far as I'm aware, will always drop the contents of a block into place verbatim, without an opportunity to escape it into a JSON string. That's where XML's CDATA construct came in handy, allowing a segment of HTML to be put right in place. Assuming, of course, "]]>" doesn't appear in your HTML.

JavaScript Page Switches

That takes care of the back-end Django bit. The last bit involves the front end JavaScript that takes care of the page switch/content replacement. This example leans on jQuery fairly heavily. In essence we'll: A) take over the link's click event, B) send off an AJAX-type request for the same href, C) set up a callback to do the actual content switch. Or, to put it another way:

$('a.ajax').click(function (e) {
  // Suppress the default navigate event
  e.preventDefault();
  // Instead, do the GET in the background
  $.get(this.href).done(function (response) {
    // The XML is automatically parsed and can be traversed in script
    var contentNodes = response.firstChild.childNodes;
    for (var i = 0; i < contentNodes.length; i++) {
      // Ignore any textNodes or other non-elements
      if (contentNodes[i].nodeType != 1) continue;

      // Handle each XML element appropriately:
      if (contentNodes[i].nodeName == 'title')
        document.title = contentNodes[i].firstChild.nodeValue;
      if (contentNodes[i].nodeName == 'main_content')
        $('#main_content').html(contentNodes[i].firstChild.nodeValue);
    }
  });
});

JavaScript, I'll admit, isn't a language I work in all that often. There's probably a better way of parsing and handling that, but that seemed to work okay in testing. And, of course, it's fairly bare bones as far as functionality. But it shouldn't be difficult to add in a spinner, error handling, etc.

Anyway, it was fun. Even the slightly frustrating update-JS/refresh/try-again cycle. Again it's still fairly rough, and quite untested at any scale. But maybe the idea will help someone out there.

Pagination days are over? Infinite scrolling technique

Love it or hate it, but the infinite scrolling technique became a big part of UX. Google Images use it, Flickr uses it, Beyonce's official website uses it. Twitter, Tumblr and the Facebook feed have it as well. The technique allows users to seamlessly scroll through content. When the user reaches the end of the page new content will automatically load at the bottom.

In my opinion, it allows for a much more natural and immersive experience while viewing images or articles, much better than pagination or once popular image slideshow galleries. In the real life you don't click on pagination links to get through your day, right?

To create the infinite scrolling page with images we will use the jQuery Infinite Scroll plugin and Masonry to lay out the images. Here is the demo of what we are going to accomplish. The code is below.

First step is to include the necessary scripts:




Add the container element. The navigation element is a very important part that triggers the loading of the subsequent page. After the second page the page number will automatically increment to fetch the subsequent pages:

<% @images.each do |i| %>
<% end %>

Ready to init the infinite scrolling script:

$(function(){
    var container = $('#container');
    container.masonry({
       itemSelector: '.item'
     });

    container.infinitescroll({
      navSelector  : '#page-nav', 
      nextSelector : '#page-nav a',
      itemSelector : '.item',
      loading: {
          finishedMsg: 'No more pages to load.',
          img: 'http://i.imgur.com/6RMhx.gif'
        }
      },

      function( newElements ) {
        var newElems = $( newElements ).css({ opacity: 0 });
          newElems.animate({ opacity: 1 });
          container.masonry( 'appended', $newElems, true );
      }
    );
  });

Once the user scrolls down to the bottom of the first page, the script will fetch the second page and filter out the new elements by the item selector. The controller action will look like this:

def list
  @images = Image.paginate(:page => params[:p] || 1, :per_page => 25)
end

Finally, we will add the styles to create a three-column layout and some nice animations to render the newly loaded items:

#container .item {
  width: 33%;
}
.transitions-enabled.masonry .masonry-brick {
  transition-property: left, right, top;
}
.transitions-enabled.masonry, .transitions-enabled.masonry .masonry-brick {
  transition-duration: 0.7s;
}

Post Login Action in Interchange

A while back, I sent a request to a few coworkers looking for a post login hook in Interchange, meaning that I'd like to execute some code after a user logs in that would not require modifying the core Interchange code. This was prompted by the need to transfer and create database records of uploaded images (uploaded while not logged in) to be tied to a specific user after they log in. Mark found a simple and elegant solution and I've described it below.

postlogin_action

The first step to adding a post login hook or method is to add the following to catalog.cfg:

UserDB  default  postlogin_action  transfer_user_images

The above code results in a call to the catalog or global sub transfer_user_images after a user logs in.

Defining the Global Sub

Next, the sub needs to be defined. In our code, this looks like:

# custom/GlobalSub/transfer_user_images.sub
GlobalSub transfer_user_images IC::GlobalSubs::transfer_user_images
# custom/lib/IC/GlobalSubs.pm
sub transfer_user_images {
  #code here
}

In the above example a transfer_user_images sub points to a Perl module that contains all of our custom global subroutines. The GlobalSubs Perl module contains the code executed upon login.

Add code!

After the simple steps above, code can be added inside the GlobalSub transfer_user_images subroutine. For this particular method, the simplified pseudocode looks something like:

sub transfer_user_images {
  # Create database connection
  # Foreach image stored in the session
  #   Move image to user specific location
  #   Record image to database, tied to user
  # Delete session images variable
}

Internal Tidbits: Links, Resources, Tools

Here at End Point, we have a broad range and depth of knowledge in many areas of web development, both server side and client side. This shows itself in form of many internal emails. Whenever I get an internal email with some tidbit of information I'd like to read later on, I file it in my Internal folder to read about later. That folder is overflowing now, and I wanted to take some time to clean it out and share the contents in blog form.

My Internal folder is now clean and I'm ready to hear about more great tips & tools from my coworkers.