Welcome to End Point’s blog

Ongoing observations by End Point people

Postgres Open 2012

Ruby on Rails software developer needed

We're looking for another talented Ruby on Rails developer to consult with our clients and develop their Ruby on Rails web applications. If you like to focus on solving business problems and can take responsibility for getting a job done well without intensive oversight, please read on!

End Point is a 17-year-old web consulting company based in New York city, with 28 full-time employees working mostly remotely from home offices. Our team is made up of strong ecommerce, database, and system administration talent, working together using ssh, Screen and tmux, IRC, Google+ Hangouts, Skype, and good old phones.

We serve over 200 clients ranging from small family businesses to large corporations, using a variety of open source technologies including Ruby, Python, Perl, Git, PostgreSQL, MySQL, RHEL/CentOS, Debian, and Ubuntu.

What is in it for you?

  • Work from your home office
  • Flexible full-time work hours
  • Annual bonus opportunity
  • Health insurance benefit
  • 401(k) retirement savings plan
  • Ability to move without being tied to your job location

What you will be doing:

  • Consult with clients to determine their web application needs
  • Build, test, release, and maintain web applications for our clients
  • Work with open source tools and contribute back as opportunity arises
  • Use your desktop platform of choice: Linux, Mac OS X, Windows

What you will need:

  • Professional experience building reliable Ruby on Rails apps
  • Good front-end web skills with HTML, CSS, and JavaScript
  • Experience with PostgreSQL, MySQL, or another relational database
  • A customer-centered focus
  • A passion for building flexible and, where needed, scalable web applications
  • Strong verbal and written communication skills
  • Experience directing your own work, and working from home
  • Ability to learn new technologies

Bonus points for experience:

  • Building and supporting ecommerce systems, including with Spree
  • Working with other languages and web app frameworks
  • Contributing to gems or other open source projects
  • Handling system administration and deployment

Please email us an introduction to to apply. Include a resume, your GitHub or LinkedIn URLs, or whatever else that would help us get to know you. We look forward to hearing from you!

Are you sure you want a MacBook Pro?

The "UltraBooks" trend has been been made popular by Apple's efforts at making the computer an appliance. There are clear benefits to this, but for enthusiasts, it means less control to upgrade to hardware of their choosing. Let's contrast this experience with my own Dell Vostro 3500. It was purchased two years ago and yet through upgrades, it has many features the recently refreshed MacBook Pro with Retina display lacks, while offering the same processing power.

MacBook Pro Advantages

Let's be clear, the MacBook Pro (MBP) has a number advantages over my Dell. The Retina display of course trounces my 1376 x 768 resolution on a 15" display. For designers, photographers, and video editors, Retina is a great asset. For someone who spends their day inside VIM and SSH, not so much. Similarly, the MBP's seven hour battery life is impressive, although I'm sure with my processor intensive work load, it would be lower. But for me, I'm not terribly mobile. I need to be mobile from room to room perhaps, but I don't find myself without power and needing to complete mission critical work less frequently than my replaceable 90 watt-hour battery can provide over 3 to 4 hours. Similarly, weight is not my number one concern; I simply stash the laptop in my backpack and go.

Incremental Upgrades

With my Dell, I purchased the baseline Vostro 3500 for $1,000 which included next business day service and accidental damage protection for three years. This means for parts which I cannot easily replace myself, like the display and motherboard, I'm covered. Over the years, I've upgraded the:

  • processor to an i5-580M, with Turbo Boost up 3.3GHz. Great for single threaded Ruby ($225).
  • wireless card to an Intel Wifi Link 5300 for better signal strength and 802.11n support ($42).
  • 65whr battery at it's end of life with a 90whr battery ($160 ouch!).
  • original HDD with a 160GB Intel 320 series SSD ($220).
  • optical drive with a 2.5" drive caddy ($52) and a 500GB Seagate Momentus XT Hybrid drive ($150).
  • RAM to 8GB ($99).

For $950 of upgrades over two years, I've got a machine with specs that are powerful in the places I care about, at a price I can afford and pay for incrementally. To get the same processor and warranty on the MacBook Pro, I need to shell out $3,150 and give up my eSATA, HDMI, VGA, PCI-Express, Gigabit Ethernet ports and 500GB of storage (upgradable to 1TB for $100). In short, I'm expected to pay a premium price for features which aren't of my choosing, while giving up ports.

While this might not be possible for all people, developers should look long and hard at what it is they value in their workstation and not blindly accept whatever Apple hypes. You might find you don't care quite as much about the "features" Apple wants you to.

Interchange "on-the-fly" items

Interchange has a handy feature (which, in my almost-seven-years of involvement, I'd not seen or suspected) allowing you to create an item "on-the-fly", without requiring any updates to your products table. Here's a recipe for making this work.

First, you need to tell Interchange that you're going to make use of this feature (in catalog.cfg).

OnFly onfly

Simple, no? The "OnFly" directive names a subroutine that is called to pre-process the custom item before it's added to the cart. The default "onfly" routine can be found in the system tag "onfly": code/SystemTag/onfly.coretag in the standard Interchange installation. (If you need more that what it provides, that's beyond the scope of my post, so good luck, bon voyage, and please write back to let us know what you learned!)

Then, you need to submit some special form parameters to set up the cart:

  • mv_order_item: the item number identifying this line
  • mv_order_fly: a structured string with | (vertical bar) delimiters. Each sub-field specifies something about the custom item, thus:
    description=My custom item|price=12.34

Now, in my particular case, I was encapsulating an XML feed of products from another site (a parts supplier) so that the client (a retail seller) could offer replacement parts, but not have to incorporate thousands of additional lines in the "products" table. So after drilling down to the appropriate model and showing the available parts, each item got the following bit of JavaScript (AJAX) code associated with its add-to-cart button:

var $row = $(this).parents('tr');
        url: '/cgi-bin/mycat/process',
        data: {
            mv_todo: 'refresh',
            mv_order_quantity: 1,
            mv_order_item: $row.find('td.item_number').html(),
            mv_order_fly: 'description='
                + $row.find('td.description').html().replace('|','')
                + '|'
                + 'price='
                + $row.find('td.price').html().replace('$','').replace(',','')
        method: 'POST',
        success: function(data, status) {
            $('#msg_div').html('Added to cart.')

And that's all it took. With Interchange, you don't even need a special "landing page" for your AJAX submission; Interchange handles all the cart-updating out of sight.

I still need to add some post-processing to handle errors, and update the current page so I can see the new cart line count, but the basics are done.

cPanel Exim false positive failure & restart fix

I'm not a big fan of add-on graphical control panels for Linux such as cPanel, Webmin, Ensim, etc. They deviate from the distributor's standard packages and locations for files, often simultaneously tightening security in various ways and weakening security practically by making several more remotely accessible administration logins.

On one of the few servers we maintain that has cPanel on it, today we did a routine Red Hat Network update and reboot to load the latest RHEL 5 kernel, and all seemed to go well.

However, within a few minutes we started getting emailed reports from the cPanel service monitor saying that the Exim mail server had failed and been restarted. These emails began coming in at roughly 5-minute intervals:

Date: Tue, 24 Jul 2012 14:21:05 -0400
From: cPanel ChkServd Service Monitor <cpanel@[SNIP]>
To: [SNIP]
Subject: exim on [SNIP] status: failed

exim failed @ Tue Jul 24 14:21:04 2012. A restart was attempted automagically.

Service Check Method:  [socket connect] 

Reason: TCP Transaction Log: 
<< 220-[SNIP] ESMTP Exim 4.77 #2 Tue, 24 Jul 2012 14:21:04 -0400 
>> EHLO localhost
<< 250-[SNIP] Hello localhost.localdomain []
<< 535 Incorrect authentication data
exim: ** [535 Incorrect authentication data != 2]
: Died at /usr/local/cpanel/Cpanel/TailWatch/ line 689, <$socket_scc> line 10.

Number of Restart Attempts: 1

Startup Log: Starting exim: [  OK  ]

And the relevant entry in /var/log/exim_mainlog was:

2012-07-24 14:08:05 fixed_plain authenticator failed for localhost.localdomain (localhost) []:48454: 535 Incorrect authentication data (set_id=__cpane

I wasn't able to find a way to fix this in any reasonable amount of time, so I opened a trouble ticket with cPanel support and they had asked for server access, logged in, and fixed the problem within a little over an hour. It was about as painless as tech support ever gets, so kudos to cPanel for that!

The solution was to run this as root:

/scripts/upcp --force

Which resyncs cPanel so that chkservd reports Exim as up and the unwanted service restarts no longer happen.

Here's to responsive tech support.

Automated VM cloning with PowerCLI

Most small businesses cannot afford the high performance storage area networks (SANs) that make traditional redundancy options such high availability and fault tolerance possible. Despite this, the APIs available to administrators of virtualized infrastructure using direct attached storage (DAS) make it possible to recreate many of the benefits of high availability.

High Availability on SAN vs DAS

A single server failure in a virtualized environment can mean many applications and services can become unavailable simultaneously; for small organizations, this can be particularly damaging. High availability with SANs minimize the downtime of applications and services when a host fails by keeping virtual machine (VM) storage off the host and on the SAN. VMs on a failed host can then be automatically restarted on hosts with excess capacity. This of course requires SAN infrastructure to be highly redundant, adding to the already expensive and complex nature of SANs.

Alternatively, direct attached storage (DAS) is very cost effective, performant, and well understood. By using software to automate the snapshot and cloning of VMs via traditional gigabit Ethernet from host to host, we can create a "poor man's" high availability system.

It's important for administrators to understand that there is a very real window of data loss that can range from hours to days depending on the number of systems backed up and hardware in use. However, for many small businesses who may not have trustworthy backups, automated cloning is an excellent step forward.

Automated cloning with VMWare's PowerCLI

Although End Point is primarily an open source shop, my introduction virtualization was with VMWare. For automation and scripting, PowerCLI, the PowerShell based command line interface for vSphere, is the platform on which we will build. The process is as follows:

  • A scheduled task executes the backup script.
  • Delete all old backups to free space.
  • Read CSV of VMs to be backed up and the target host and datastore.
  • For each VM, snapshot and clone to destination.
  • Collect data on cloning failures and email report.

I have created a public GitHub repository for the code and called it powercli_cloner.

Currently, it's fairly customized around the needs of the particular client it was implemented for, so there is much room for generalization and improvement. One area of improvement is immediately obvious: only delete a backup after successfully replacing it. Also, the script must be run as a Windows user with administrator vSphere privileges, as the scripts assumes pass-through authentication is in place. This is probably best for keeping credentials out of plain text. The script should be run during non-peak hours, especially if you have I/O intensive workloads.

Hopefully this tool can provide opportunities to develop backup and disaster recovery procedures that are flexible, cost-effective, and simple. I'd welcome pull requests and other suggestions for improvement.

GoRuCo 2012 Recap

A few weeks ago, End Point graciously agreed to send me to GoRuCo, "the premier Ruby conference in New York City". I was excited to try and apply the lessons from our own Greg Sabino Mullane, who gave a talk about attending conferences during End Point's 2012 Company Meeting. He emphasized a focus on interacting with the speakers and attendees instead of the presentation content.

Pre Party at Pivotal Labs

The pre-party was located just blocks away from our offices, making it a convenient after work stop. On my walk to the party, I tried to think about the type of connections I wanted to make and the topics I wanted to discuss. I was intrigued by Pivotal Lab's policy on pair programming, but realized I could continue to read up about that extensively online. What was a question I could ask that would be interesting for both asker and askee, providing me something a Google search couldn't?

After the usual introductory chit chat, I found myself asking, "so, what are you struggling with in your work right now?" It was broad and open, giving the speaker a chance to perhaps go somewhere outside of the normal conversation. I asked Haris Amin this question and got what would become a fairly typical answer. He summarized his struggle as one of keeping up with the firehose of his interests. So many great technologies are springing from all corners of creation, there simply aren't enough hours in the day to investigate them all. This resonated against our own Jon Jensen's comments during the company meeting about it being a golden age for technology.

I also got to meet and chat with Coby Randquist, the creator of Confreaks. It was great learning about his transition from implementer to manager. As a manager, he found he loved that he could solve much bigger problems than as a developer by bringing together talent and keeping "everyone else out of their way".

A Sampling of the Main Event

Of course there were so many great presentations, which you can watch at your leisure, but because there are always more presentations than time, I'll focus on one that I found particularly interesting. Dr. Nic gave a talk called The Future of Deployment (slides) which he laid out an argument for being able to able to version control and deploy not only our applications, but the infrastructure which supports them, all with a unified tool.

He lays out some interesting demands of this tool, including:

  • An explicit description of all infrastructure dependencies
  • The ability to manage all parts of this infrastructure, from the size of our AWS instances, to the particular version of Nginx we're running.
  • Full version control/history of these descriptions
  • A central API for all activities
As if this weren't challenging enough, he mentions it'd be nice to have:
  • Independent install paths; if we can use .../releases/TIMESTAMP for Capistrano deployments, why not the same for Ruby and Nginx?
  • Portable enterprise deployments; we should be able to offer our applications as a self building system *behind* the firewall

So what's the magical tool which will finally solve all our deployment problems? He offers BOSH, "an open source tool chain for release engineering, deployment and lifecycle management of large scale distributed services". It was created to manage *really* big deployments, so Dr. Nic argues perhaps it can also meet our smaller needs. BOSH is kind of a bear to get your head around, so Dr. Nic helpfully created a more palatable Getting Started series that can help bootstrap your experience.

If you're interested in hearing more about why we should care about BOSH, I'd say watch the presentation, but if you don't need to be sold on the idea and know you need it, the presentation is *not* about BOSH really at all, outside of it offering to address some of Dr. Nic's requirements above.

Liquid Galaxy at the INFO Summit

End Point was tasked with setting up and manning a Liquid Galaxy at the two day INFO Summit at the Four Seasons Hotel in Westlake Village, CA. Josh Ausborne and Alejandro Ramon were in attendance representing End Point, and performed the setup and the running of the system.

INFO is the short name for Illicit Networks: Forces in Opposition. The INFO Summit is an event designed to get people to understand how drug smugglers, arms dealers, and human traffickers operate, and how people can get involved to develop technology in an effort to disrupt and thwart the networks. You can read an overview about the summit and its purpose here.

Jared Cohen, Director of Google Ideas, was interviewed by Fast Company to discus the breaking down of illicit networks. According to Cohen, Google "focused on disrupting violent and coercive illicit networks, including drug cartels, the mafia, human trafficking rings, organ harvesters, illicit arms dealers, and forced labor networks." You can read the article here where he explains some of the ways that Google is stepping up the fight.

End Point's involvement in the summit is as the installer and presenter of a Liquid Galaxy tour that shows the various illicit network hotspots around the world. These locations include various places such as Africa, the Middle East, Azerbaijan, Russia, Mexico, and even the United States. The tour is a simple KML tour with text popups to give information on a given hotspot before flying to the next location. It provides general Illicit Network information that the people are looking for, but not so much that they are unable to finish reading it before the tour moves on to the next location. We could have included some more fancy design in the tour bubbles, but in this case we didn't want to distract from the content itself.

We have had numerous people approach us about how the Liquid Galaxy could be utilized in various situations, and we have presented ideas for each of them. One of them was a person involved with a US military school, and she is interested in how the LG could be used in a military education setting.

Another is head of security and emergency response for a very large international facility, and was wondering how the LG could be used to show watershed concerns, construction, and high security areas. We explained the idea of screen overlays to identify areas of high value, as well as weather updates and tracking.

During the conference, we have had a chance to sit and discuss the use of KML with Sean Askay, Developer Support Engineer of the Google Earth Outreach group. We talked about the current and future use of KML within Google Earth, how we expect it to grown in the future, as well as how End Point can continue to grow in its Liquid Galaxy / Google Earth role.

Overall, the tour has been a success as far as the Liquid Galaxy is concerned. We have shown a good tour on a beautiful system, have made contact with many interested people, and have been able to show how the LG can be utilized to meet the various needs of the conference attendees.

RHEL 6 glibc IPv6 DNS resolution bug

We ran into an unpleasant bug in a Red Hat Enterprise Linux 6 glibc update a couple of weeks ago. It since has made its way into CentOS 6 as well.

The problem manifested itself in our particular case with this error from the Postfix mailer:

Jun 29 01:55:23 sl37 kernel: smtp[7093]: segfault at 1 ip 00007ffc0e455596 sp 00007fff99948f60 error 6 in[7ffc0e449000+16000]

But it affects all DNS resolution on the host, not just for mail.

If you have any IPv6 resolvers at all listed in /etc/resolv.conf, all your DNS resolution is likely to be broken with this version of glibc:


To work around the problem, you can either:

  • Use only IPv4 DNS resolvers (comment out the IPv6 resolvers for now)
  • or downgrade to the previous version of glibc using yum downgrade

Red Hat is aware of the bug and you can track progress toward a resolution in Bugzilla bug #835090.

If you're using IPv6, watch out for this! If not, you're fine.

Changing Passenger Nginx Timeouts

It may frighten you to know that there are applications which take longer than Passenger's default timeout of 10 minutes. Well, it's true. And yes, those application owners know they have bigger fish to fry. But when a customer needs that report run *today* being able to lengthen a timeout is a welcomed stopgap.

Tracing the timeout

There are many different layers at which a timeout can occur, although these may not be immediately obvious to your users. Typically they receive a 504 and an ugly "Gateway Time-out" message from Nginx. Review the Nginx error logs both at the reverse proxy and application server, you might see a message like this:

upstream timed out (110: Connection timed out) while reading response header from upstream

If you're seeing this message on the reverse proxy, the solution is fairly straight forward. Update the proxy_read_timeout setting in your nginx.conf and restart. However, it's more likely you've already tried that and found it ineffective. If you expand your reading of the Nginx error you might notice another clue.

upstream timed out (110: Connection timed out) while reading response header from upstream, 
upstream: "passenger://unix:/tmp/passenger.3940/master/helper_server.sock:"

This is the kind of error message you'd see on the Nginx application server when a Passenger process takes longer than the default timeout of 10 minutes. If you're seeing this message, it'd be wise to review the Rails logs to get a sense for how long this process actually takes to complete so you can make a sane adjustment to the timeout. Additionally, it's good to see what task is actually taking so long so you can offload the job into the background eventually.

Changing nginx-passenger module's timeout

If you're unable to address the slow Rails process problem and must extend the length of the time out, you'll need to modify the Passenger gem's Nginx configuration. Start by locating the Passenger gem's Nginx config with locate nginx/Configuration.c and edit the following lines:

                              prev->upstream.read_timeout, 60000);
Replace the 60000 value with your desired timeout in milliseconds. Then run sudo passenger-install-nginx-module to recompile nginx and restart.

Improving Error Pages

Another lesson worth addressing here is that Nginx error pages are ugly and unhelpful. Even if you have a Rails plugin like exception_notification installed, these kind of Nginx errors will be missed, unless you use the error_page directive. In other applications I've setup explicit routes to test exception_notification properly sends an email by creating a controller action that simple raises an error. Using Nginx's error_page directive, you can call an exception controller action and pass useful information along to yourself as well as present the user with a consistent error experience.

Company Presentation: Ember, Backbone and Friends

We had a "virtual" company meeting today using Big Blue Button with Free Conference Call HD for audio. I gave an overview on the upcoming crop of JavaScript MVC Frameworks and how they attempt to mitigate some of the issues faced by JavaScript-heavy web applications. Later on, I spoke more specifically about Backbone.js and Ember and demonstrated an example Todo List application from Addy Osmani's TodoMVC project. If you are interested in evaluating and learning more about client-side frameworks, TodoMVC is definitely worth a look.

Slides from the talk

I used Google Presentations for this as I have for several other talks lately and have found it really great to use and collaborate on presentations with others. Check out the slide deck from my talk below:


During the Q&A time afterward we discussed LocalStorage a little bit as it's used for the storage layer for each of the TodoMVC todo list applications. We covered some of the options which allow client-side MVC frameworks to sync data between the client and RESTful server-side web services.

We also covered how much the Chrome Developer Tools have improved recently and I encouraged my colleagues to try them out if they haven't already done so. Many End Pointers including myself use Firebug and we talked a little about where it excels compared to the Chrome Dev Tools. Logging and inspecting XHR requests was a great feature Steph pointed out.

It's great to have some many helpful tools and software like this to work with in the open source community.

Independence Belarus

This month of July I am working from Belarus, my home country.

On July, 3 we celebrated Independence Day of Belarus here, quite in sync with the Independence Day in United States.

Belarus was occupied by German troops for 3 years since 1941 during World War II, and on July, 3, 1944 the Soviet army managed to release the capital of Belarus - Minsk.

Independence Day celebration starts with the military parade including tanks, jets, helicopters and other bulky equipment.
The rehearsal of the parade is usually carried out a day before the parade in my neighborhood. Sometimes it is pretty unnerving to hear all the noise tanks and jets make on the streets and in the air.

The celebration continues with outdoor concerts and traditional Belorussian folk songs performances in many beautiful parks of the city.

It finishes with the fireworks display near "Stella" - a monument built to memorialize the fortieth anniversary of the victory in the Great Patriotic War. Great Patriotic War is how we call a considerable part of World War II that happened on the territory of former Soviet Union.

Though not as famous as Macy's fireworks, "Stella" fireworks display was very cool this year. I captured it on video along with some traditional folk Belorussian songs from the concert earlier that day. Please, enjoy!

Code School: Journey into Mobile Review

Yesterday, I took the Journey into Mobile course over at Code School, an online training program with a handful of web technology courses. I just started a large mobile project for Paper Source, so this was good timing. Because a paid membership to Code School is required to participating in the training, I won't share too many of the course details here. But I'll share a few tidbits that will hopefully interest you in taking the course or learning more about mobile development.

The course was divided into 5 high level lessons (or levels):

  • Relative Font Size
  • Fluid Layouts
  • Adaptive Design
  • Responsiveness Adventures
  • Responsive Media

The first two topics covered working with proportional font sizes and content regions, and how to convert your existing layout to proportions (percentage or ems) to create a fluid layout which included proportional font sizes. In the Adaptive Design lesson, the CSS3 supported @media query was introduced. I've used the media query on the responsive design for The Best Game Apps and will be using it for Paper Source. Some examples of @media queries include:

@media (min-width: 655px) and (max-width: 1006px) {
  # styles specific to browser width 655-1006 pixels
@media only screen and (device-width: 768px) {
  # styles specific to browser width 768 pixels
@media (min-device-width: 320px) and (max-device-width: 655px) {
  # styles specific to browser width 320-655 pixels
@media (min-device-width: 450px) and (orientation:landscape) {
  # styles specific to browser width 450 pixels and landscape orientation

For each of the above @media queries, specific "break points" are determined to adjust styles as certain elements break as the browser width changes. For example, if elements begin to overlap as the screen narrows, the browser width at which this begins to happen is one break point, and new styles are defined for that width.

The last two levels of the training course covered Responsiveness Adventures and Responsiveness Media. Responsive design also leverages the @media query to design responsively for changing browser widths. One interesting topic covered in the Responsive Media lesson was how Retina Images are addressed on devices where the pixel density is 1.5-2 times regular pixel density. This was a topic I hadn't come across in mobile development. The lesson presented a couple of options for dealing with Retina images, including use of the @media query and picture HTML tag.

Overall, it was a decent course with a good overview. I would recommend it to anyone planning to get involved in mobile development.