End Point

News

Welcome to End Point's blog

Ongoing observations by End Point people.

Streaming Live with Red5 Media Server




Are you excited about Google+ Hangouts? Would you be even more excited to implement live streaming yourself?
Today we will take a look under the hood of broadcasting and even start implementing our own mini hangout.



A bit of theory

Live streaming is based on RTMP protocol. It is able to transfer video, audio and generally any data in real-time over Internet between server and client.
The most popular combination for video/audio streaming is Adobe Flash Player as a client software and a proprietary Adobe Flash Media Server as a server software. Another option is Wowza Streaming solutions.
Luckily for us there is an open-source Red5 Media Server – the most popular if not the only one stable of all open-source media streaming servers. We will be leveraging Red5 to dive into RTMP world and the cutting edge streaming technologies.


Exploring Red5 Media Server

Download and install Red5 Media Server for your environment from here. Now it is time to implement a sweeping live stream between our two laptops.
Red5 comes with the set of demo applications that are very handy to base the development on. Demo applications can be installed via the Installer. For the purpose of this article we need to install the "oflaDemo" application.
After installation is complete, we can launch the demo for Simple Broadcaster application. In Simple Broadcaster we click "Connect" button, grant Flash Player permission to use camera and microphone, and  – voila! – we should now be broadcasting live via Simple Broadcaster application to oflaDemo application.
If we use another browser  to open Simple Subscriber application, or even another laptop with the IP  of the broadcasting computer, we will be able to stream the video live. It is quite noticeable that quality drops in Simple Subscriber.



In this example we also learned that rtmp://localhost/oflaDemo is the server communication point to where we publish and stream videos.


Customizing SimpleBroadcaster and SimpleSubscriber

Next comes the most interesting part - embedding SimpleBroadcaster and SimpleSubscriber into the web page.
We can't really use their default versions "as is". We want to remove input box, "Connect" button and Red5 connection indicator. We also want the streaming to start automatically.
It's more convenient to implement "Start" button in HTML and trigger the loading of SWF on "click", because then we can easily change it, while changing SWF requires a lot of effort.  You will be able to understand how exactly much effort it takes in a minute.
My plan is to use Flash CS5 to modify SimpleBroadcaster.fla and SimpleSubscriber.fla. Flash Builder 4.6 to modify ActionScript classes for SimpleBroadcaster and SimpleSubscriber. Finally I will use Flash compiler to generate SimpleBroadcaster.swf and SimpleSubscriber.swf from .as and .fla files.
Finally, some coding versus talking!
Red5 sources are conveniently available as a Google Code project. It comes in server-side and the most interesting to us flash-side

 > svn checkout http://red5.googlecode.com/svn/flash/trunk/ red5

Step 1.
Let's open simpleBroadcaster.fla located in the very root of the download in Flash CS5.




and take a pick at the Library panel.




Let's double-click on the Embedded Video 1, open Window -> Properties panel and modify the size of the video to occupy the full documents area. Alternatively, we may change the document size to fit the video: right-click on the grey area, pick Document settings from the context menu and adjust dimensions. Either way, now we should drag the video rectangle to fit the document. We will save the document and repeat the same sequence of actions for simpleSubscriber.fla, because they are really no different.

Step 2.
Let's now open the "red5" directory that we checked out in FlashBuilder.




We immediately find the main class for simpleBroadcaster.fla org.red5.samples.livestream.broadcaster.Main in "classes" directory.
The key function here is "configUI".

private function configUI():Void   
{  
    // setup the tooltip defaults  
    Tooltip.options = {size:10, font:"_sans", corner:0};  
    // setup cam  
    cam = Camera.get();  
    cam.setMode(480, 320, 15);  
    cam.setQuality(0,80);  
    // setup mic
    mic = Microphone.get();  
    // get notified of connection changes  
    connector.addEventListener("connectionChange", this);  
    // set the uri  
    Connector.red5URI = "rtmp://localhost/oflaDemo";  
    // initialize the connector  
    connector.configUI();   
} 

Let's look into Connector class org.red5.utils.Connector.


 // UI Elements:
 // ** AUTO-UI ELEMENTS **
 private var alert:SimpleDialog;
 private var connect:IconButton;
 private var disconnect:IconButton;
 private var light:ConnectionLight;
 private var uri:TextInput;
 // ** END AUTO-UI ELEMENTS **
 
 public function configUI():Void 
 {
  // instantiate the con  qnection
  connection  = new Connection();

  // register the connection with the light so it can add a listener
  light.registerNC(connection);

  // hide disconnect button
  disconnect._visible = false;

  // set the URI
  uri.text = red5URI;

  // setup the buttons
  connect.addEventListener("click", Delegate.create(this, makeConnection));
  disconnect.addEventListener("click", Delegate.create(this, closeConnection));
  connect.tooltip = "Connect to Red5";
  disconnect.tooltip = "Disconnect from Red5";

  // add listener for connection changes
  connection.addEventListener("success", Delegate.create(this, manageButtons));
  connection.addEventListener("close", Delegate.create(this, manageButtons));
 }

 public function makeConnection(evtObj:Object):Void
 {
  if(uri.length > 0) 
  {
    var goodURI = connection.connect(uri.text, getTimer());
    if(!goodURI) alert.show("Please check connection URI String and try again.");
  }
 }

We will now make quick and dirty changes to get the proof of concept working. We will hide all the unnecessary UI elements, then convert the listener to the regular function, and call it from the configUI function.


public function configUI():Void 
 {
  disconnect._visible = false;
  connect._visible = false;
  uri._visible = false;
  light._visible = false;

  connection  = new Connection();
  light.registerNC(connection);
  if (makeConnection(Connector.red5URI)) {
    dispatchEvent({type:"connectionChange", connected: true, connection:connection});
  }
 }

 public function makeConnection(uri:String):Boolean
 {
  var result:Boolean = false;
  if(uri.length > 0) 
  {
    result = connection.connect(uri, getTimer());
    if(!result) {
      alert.show("Failed to connect.");
    }
  }
  return result;
 }

Done! After making these changes we are good with SimpleSubscriber as well, because both classes use the same Connector class.


Step3
Unfortunately Red5 is written in ActionScript 2. We can't really use any of the newer mxml compilers installed on our systems, because they compile ActionScript 3. Flex SDK is also only compatible with ActionScript 3.  I spent quite a lot of time searching for a simple and working AS2 compiler on the Internet, and luckily, found one called MTASC.
The compiler needs to be supplied with the correct MX Component libraries to compile Red5 classes. Flash CS6 has those libraries, but in case you don't have CS6 handy I put them on GitHub for you.

> export PATH="/path/to/mtasc-mx/bin:/path/to/mtasc-mx/std:$PATH"
> cd "/path/to/red5/classes" 
> mtasc -cp "." -swf ../simpleBroadcaster.swf -mx org/red5/samples/livestream/broadcaster/Main.as -v
> mtasc -cp "." -swf ../simpleSubscriber.swf -mx org/red5/samples/livestream/broadcaster/Main.as -v
The compiled swf's can be found in the root of the current directory.
At this point it is possible to create a broadcast and stream it just by launching the swf's, first broadcaster, then subscriber. However, we would like to be able to broadcast via network.


Are you ready to broadcast?

In order to enable broadcasting in our web application we will need to drop the compiled swf files into its public directory.
We will then download swfobject.js JavaScript library and drop it into the public directory of our web application as well.
We will create broadcast.html file.

<html>
 <head>
 <title>Red5 Demo - Simple Broadcaster</title>
 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
 <script type="text/javascript" src="swfobject.js"></script>
 <script type="text/javascript">
 swfobject.embedSWF("simpleBroadcaster.swf","myContent","360","240","8.0.0","jwplayer/expressInstall.swf");
 </script>
<style media="screen" type="text/css">#myContent {visibility:hidden}</style>
 </head>
 <body>
 <div id="myContent">
 <h1>You need the Adobe Flash Player for this demo, download it by clicking the image below.
</h1>
 <p><a href="http://www.adobe.com/go/getflashplayer">
<img src="http://www.adobe.com/images/shared/download_buttons/get_flash_player.gif" alt="Get Adobe Flash player"/>
</a></p>
 </div>
 </body>
</html>

Then we will create watch.html file.

<html>
<head>
  <title>Red5 Demo - Simple Subscriber</title>
 <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
 <script type="text/javascript" src="swfobject.js"></script>
 <script type="text/javascript">
 swfobject.embedSWF("simpleSubscriber.swf","myContent","360","240","8.0.0","assets/expressInstall.swf");
 </script>
<style media="screen" type="text/css">#myContent {visibility:hidden}</style>
 </head>
 <body>
 <div id="myContent">
 <h1>You need the Adobe Flash Player for this demo, download it by clicking the image below.
</h1>
 <p><a href="http://www.adobe.com/go/getflashplayer">
<img src="http://www.adobe.com/images/shared/download_buttons/get_flash_player.gif" alt="Get Adobe Flash player"/>
</a>
</p>
 </div>
</body>
</html>

At this point we are able to start a broadcast at http://localhost/broadcast.html and watch it at http://localhost/watch.html! In this case I am broadcasting from Macbook and streaming to Windows laptop.




Sweet! Next step is to implement the actual two-way hangout with Red5 Media Server!

Read on here: Streaming Live with Red5 Media Server: Two-Way

RailsConf 2012: What's Next in Rails?

I tend to walk away from every RailsConf feeling that there was a trend for each conference. In 2009, I thought the trend was Passenger, in 2010, NoSQL and Rails 3 was all the rage, and in 2011, many of the talks were related to CoffeeScript and SASS. While there was some of these technical specifics at this conference, I felt that the overarching trend this year is that several folks that are extremely involved in Rails (DHH, Yehuda Katz, Aaron Patterson, etc.) are very passionately involved in the course correction needed to keep Rails relevent.

As I mentioned in my blog article on Day 1, DHH talked about progress. DHH focused on the limitations of progress (loss aversion) and how and why people move from curious to suspicious when working with technology. He didn't delve into any of the Rails 4 details, but simply ended by sharing that Rails 4 will break and challenge things.

On Day 2, Aaron Patterson covered a discussion about what it means to incur technical debt, and how developers have different thresholds of how much technical debt is tolerated.

He talked about the types of refactoring needed to reduce technical debt. He ended his talk with a slide that said, "be prepared", which I understood to mean that there's a lot of movement, deep consideration, and no doubt passion. Aaron also participated in a Rails core panel on Day 2. While it was thoroughly entertaining (coloring book contest included!), there was some discussion over where Rails is going and the various perspectives. Aaron talked about how he is critical of Rails because he cares a lot, and the critical analysis of Rails is happening to keep Rails relevent in the future.

On Day 3, Yehuda Katz presented his talk on Rails: The Next Five Years. Katz noted that his perspective is not representative of the core team nor of his employer, that he comes from the perspective of developing very rich client-side web applications (that may also translate to mobile applications), and that part of the justification of his talk was to evangelize some of the technical features that he believes should be included in the core to keep Rails relevant.

For him, every time his company starts a new project and reevaluates whether Rails meets their needs means that something about Rails isn't working right now. Yehuda covered his examination of why Rails was great in the first place. And the underlying answer to that is convention over configuration. Rails eliminates trivial choices like code organization, assets, naming, routing, etc. and there is a shared benefit in this solution with convention. Gaining deep knowledge about hard problems is hard (e.g. CSRF prevention) and Rails makes these decisions for us with convention. Yehuda's perspective is [now] that any convention is better than no convention, because adding any convention upfront will allow something to be built out and improved incrementally over time rather than introducing divergent paths for trivial decisions.

<sidebar>

Yehuda took a short time in his talk to discuss justification for getting things in the core, even if they need follow-up iterations:

  1. Getting new features into core gets a lot of eyeballs on it.
  2. More importantly, getting new features into core gets a lot of eyeballs attached to people who have a lot of domain expertise on the problem that the feature is solving.
  3. Getting new features into core helps reveal conceptual blindspots and additional required functionality.

</sidebar>

After talking about Rails and his desire to get new relevent features into Rails from at a high level, Yehuda dug into a few specific wants:

  • serialization: This took up a large proportion of the talk. I can't give justice to everything covered here, so check out the slides and I'll post a link to the video later.
  • bulk editing
  • better convention for application structure (e.g. assets)
  • API between JS framework and Rails, as the popularity of JS frameworks grows


Coding with convention (middle) allows developers* to avoid making diverging trivial decisions (left). Conventions provide the ability to share resources and domain expertise and add new features quickly, even if course correction (right) is needed on those conventions.
* not to be confused with sperm :)

What Now?

I think what Yehuda said during QnA resonated with me most: Decoupling the data from the representation is really important because the data then becomes represented in a variety of ways. Because of the extremely quick growth in technology and tools of various media consumption devices (browser, mobile web, native client apps), data is represented in an increasing number of ways and with varied degrees of interactivity, so what I took away from the conference is the view-level is where there may be some serious surgery in Rails. Obviously, there are a lot of core members looking at a variety of things in the stack, but I perceive that Rails needs to solve the problem of offering a strong convention for data representation to keep it relevant to new and growing technologies, while allowing Rails to continue to flourish.

RailsConf 2012: Day Three

It's Day 3 of RailsConf 2012 here in Austin Texas. Check out my posts from Day 1 and Day 2.

RailsConf 5k

To kick off the day, I ran the low-key RailsConf 5k along the lake here in Austin. I'm glad that this event was organized — I think it's good to take a mental break from all the conference activities.

RubyRogues

This morning kicked off with a RubyRogues live session. The topic for today's podcast was a discussion about what Rails developers care about and/or should care about. I thought that the answers apply to developers in general. Below are some of the topics covered in the talk.

  • James Edward Gray II
    • James focused on the point that experimentation is important and to perceive all code as experimental.
  • Avdi Grimm
    • Avdi's main argument was a suggestion to practice deliberate and mindful coding. While he believes that there is a place for the "shut up and code" mentality in some projects depending on risk, it's not acceptable to have this perspective when it comes to community sharing of knowledge.
    • Avdi recommends that exercising introspection by reviewing your code and explaining it to someone else (how and why) will make you a better programmer and communicator.
  • David Brady
    • David shared that Ruby is not a block oriented lexically scoped language.
    • He suggests that you should a) learn Ruby as if it's unlike the other languages you know b) learn CoffeeScript because it writes better JavaScript than you will c) join the community and d) don't be a douchebag (translation: be humble, not arrogant) because d-bags make a bad stereotype for Rails programmers
  • Josh Susser
    • Josh suggests to exercise prudence, or acting with or showing care and thought for the future by building a strong foundation.
    • He also discussed how while he agrees with comments of previous keynotes (loss aversion is pillar of conservatism, experienced developers have a lower tolerance for technical debt), that he also has the perspective that he has a limited budget for risk in projects. From his experience, bundler was a huge win in minimizing risk, but asset pipeline (in its current state) spent all of his risk budget.
  • Charles Max Wood
    • Charles focused on how the Rails community should try to encourage and bring other developers to the community by evangelizing it. We should reach out to other communities and groups to encourage them to try Rails.

Lightning Talks

The last session I attended at RailsConf was the Lightning Talk session. I know I've abused lists in my RailsConf posts a bit, but Lightning Talks are meant to be presented in list form! Here's a quick rundown of some of the topics presented in the Lightning Talks, which were a mix of 1-minute and 5-minute talks. I'll update with more links as they become available.

  • NSRails is a gem that bridges objective-C and rails, and will give Objective-C a bunch of methods similar to Rails methods (.all,.find(x), CRUD behavior)
  • WindTunnel is a JavaScript testing framework without hassle.
  • config.threadsafe is a setting in configuration that allows threading. Tony Arcieri provided a lot more details — I'll update this post to include a link to the slides later.
  • Worker is a threading JavaScript tool.
  • iwanttolearnruby.com is a resource for Ruby/Rails learning resources
  • @marksim talked about how private teams should work like open source by a) doing more code reviews and b) using distributed communication tools like HipChat, Campfire, IRC, message boards, mailing lists, internal blogs, Google+ Hangout, and Skype to have success like open source does.
  • sidekiq is a distributed message queue (like resque)
  • erd is a gem that was written during RailsConf to allow visualization and changes of the database. I'll post a link to it when it becomes available.
  • Rowe is a development technique that means trading results for money (as opposed to time).
  • The job_interview gem is a silly (and extremely entertaining) gem created during RailsConf for helping you answer those tough interview questions.
  • In Rails Engines, routes should a) always be drawn on the engine, and b) the engine should call isolate_namespace.
  • Tokaido is a kickstarter project that Yehuda's been working on.
  • railcar is an isolated, compiler-optional Rails environment (arcturo/Railcar).
  • ninjascript is a JavaScript tool for unobtrusively adding event listeners, and more.
  • tddium is a continuous testing integration tool.
  • wicked is a gem for rendering wizards.
  • flash_s3_rails is a gem for embedding a multi-file upload widget into your application.
  • javascript-and-friends is a Google Group for discussing and acting on embedding JavaScript in Ruby.
  • graphite is a scalable real-time monitoring tool.

A Little Bit More...

I also attended Yehuda Katz's talk on Rails: The Next Five Years. I'm saving a write-up on this for another post, because I'd like to wrap it in with some of the other keynotes discussions.

RailsConf 2012: Day Two

I'm here in Austin for Day 2 of RailsConf 2012. Here's a quick run-down of a few talks I attended today. Read about a couple of sessions I attended on Day 1 here.

Let's Make the Web Faster

One of the talks I attended on Day 2 was Let's Make the Web Faster — tips from the trenches @ Google by Ilya Grigorik. Ilya works on the MTWF (make the web faster) team at Google; one of their goals is not only to make Google's web applications faster, but also to make a bigger impact on web performance by developing and contributing to tools. I found a few main takeaways from Ilya's talk:

  • Navigation Timing Spec: The navigation timing spec defining an interface for web applications to access timing information related to navigation and elements. You can look at this timing information by checking out the performance object in a console:

    Example output of navigation timing spec data.
  • Site Speed Tools in Google Analytics: Google Analytics now has elements of the page speed broken down that allows you to compare page load times and examine what the bottlenecks in the system are. To get there, you go to Content => Site Speed => Page Timings in Google Analytics. You can also measure and examine this data using Segments and Advanced segments.
    Example metrics shown on Google Site Speed breakdown.
  • WebPageTest.org is a tool that we've blogged about before a few times. Now it also provides a video of how the page loads to examine how the page load is perceived.
  • Speed Index: Speed index is a new metric for measuring "visual completeness" of a page loading. This is an interesting/valuable metric to look at when you want to try to understand and improve user perceived latency. For example, can you force loading some images before peripheral scripts to improve the user perceived latency?
  • A few Google tools that Ilya reviewed were PageSpeed Online and its API, PageSpeed SDK, and PageSpeed Service, which is in beta mode.
  • Ilya's recap was that you should a) measure performance b) optimize performance (with specific attention to improve user perceived latency) and c) use tools to automate performance.

Check out the full presentation here.

Presenters and Decorators: A Code Tour

Another talk I attended was Presenters and Decorators: A Code Tour by Mike Moore. He started with a brief explanation of the justification (more later) for presenters — as a "pain driven developer", Mike tries to reduce the pain of advanced view logic with presenters. Mike then went through a complex example of rendering which mapped through a view to a helper to an app/component file to a class in the library back to the app to a haml view; this is painful to dissect, maintain, and test, so presenters are a good option here to avoid this spaghetti code.

Mike then went through coding examples and thought processes that lead to using a presenter-like solution. He presented an example where a view contained view-specific logic and instantiation and use of variables. The natural first step for refactoring the view here would be to pull the logic into the model, but the argument against this is that because it's view specific (as opposed to domain specific), it should not live in the model. Another progression step in pulling the logic out of the view would be to create a class that's initialized in the view, and the various instance methods on that class are called in the view. Here Mike introduced the ActiveDecorator plugin. This plugin automagicallly includes class decorators for instances in the view, which eliminates the need for instantiation of the presenter object. Mike also reviewed a standard serialization example, where a JSON-ified view can have logic that should probably live elsewhere. One option here is to create a class serializer option which has methods to render data to json. Another option here is to use ActiveModel::Serializers which looks for the rendering of JSON to remove logic from the JSON rendering view. These examples and another thorough code example are best viewed in the slides.

Mike explained that it's difficult to define Presenters and that he perceives presenters on a spectrum between the model and view which is an object that represents state and behavior of the view. The justification for presenters is to write easier to read, easier to maintain, easier to hand off to a designer, and easier to test code. Mike also noted that this isn't something that a Rails noob should get into and also that if it's a problem that you don't see in your code base, it's not a problem you necessarily need to solve.

Ten 42 Things You Didn't Know Rails Could Do

Another fun talk I attended on Day 2 was Ten Things You Didn't Know Rails Could Do by James Edward Gray II. In list-presentation style, James reviewed 42 things you can do with Rails that you might not know about [which work in Rails 3.2.3 and may or may not work in previous versions of Rails]. This talk reminded me of some of the protips I picked up from taking the Rails Best Practices course over at Code School because there were so many usable tips. I summarized the presentation into my own list of takeaways, but note that I left out some of the advanced tips that required extensive code examples. View the slides here.

  • The command rake notes will output all your FIXME, TODO, and OPTIMIZE notes in your Rails application. You can also pass in custom annotations to output here as well.
  • In the console, you can call helper methods (e.g. helper.number_to_currency(100)
  • You can automagically add indexes via migration shorthand (e.g. rails g resource user name:index). And you can automagically add associations (e.g. rails g resource article user:references or rails g resource article:belongs_to) via migration shorthand.
  • rake db:migrate:status will tell you where your migration is at
  • You can use the pluck command instead of the map command. (e.g. User.pluck(:email), User.uniq.pluck(:email))
  • You can override association methods to "hook" in custom behavior.
  • You can use limitless strings in PostgreSQL. This requires a bit of code shared in the presentation. I know, my fellow PostgreSQL expert coworkers are going to balk that this is on the list, but unfortunately Rails doesn't allow this without some code.
  • You can utilize PostgreSQL's full text search with a bit of code.
  • You can use a different database for each user. This one could be great for multi-domain ecommerce, which I recently blogged about.
  • You can merge nested hashes via the deep_merge method.
  • You can remove a specific key from a hash via the except method.
  • You can add defaults to a hash without overriding via the reverse_merge method.
  • You can override form helpers, and a lot of other view stuff, but this was a little hard to encapsulate in a blog post, so it'll be best viewed in the slides.
  • You can route exceptions (e.g. match "/404", to "errors#not_found")
  • You can route to Sinatra. The example James gave here was to allow display of the Resque web interface inside a Rails application, but we have another client who would benefit from this technique as well.

Ta Da

Hopefully this article was more Rails-ey than yesterday's RailsConf 2012: Day One post which focused on Backbone.js and CoffeeScript. I'm looking forward to a couple of tomorrow's sessions — and hopefully they will be blogworthy. Stay tuned!

Using CarrierWave without an ORM in Spree

I was recently adding some settings to Spree (an open source Ruby on Rails ecommerce framework) using Spree's built-in preferences system and found myself needing to upload an image as part of the settings. Our application was already using CarrierWave to handle some other uploads and wanted to use it to handle the uploading for my current image as well. Since the only setting I was setting was the image and there would only ever be one image, I didn't want to tie the setting to ActiveRecord. So I did some reading and found that you can tie a CarrierWave to a class without an ORM. The documentation on this is very sparse so I resorted to reading through the code to figure out how to get it working. After hours of playing with it I finally got it working. With the current movement in the Rails community to move more toward an Object Oriented approach to building applications, I've decided to document this here so that others will be able to quickly tie CarrierWave uploaders to their non-persisted classes.

The uploader I'll be working on is to add a banner to the checkout process in Spree (0.60.x). The idea is to store the image url in Spree::Config (the general preferences mechanism within Spree). For those who unfamiliar, Spree::Config acts very similar to a Hash object that is persisted to the preferences table in the database.

The RSpec

Here are the tests that I will be using for the class. Hopefully these are fairly self explanatory.

# spec/checkout_view_setting_spec.rb
require 'spec_helper'
require File.join(Rails.root, 'lib', 'checkout_view_setting.rb')

describe CheckoutViewSetting do
  include ActionDispatch::TestProcess
  before do 
    @config = mock("Config")
    @cv = CheckoutViewSetting.new(@config)
    @uploaded_file = Rack::Test::UploadedFile.new(Rails.root.join('spec/support/test_image.png'), 'image/png')
  end

  it "should store the image in the config" do
    @config.should_receive(:[])
      .with(CheckoutViewSetting::IMAGE_ID_CONFIG_KEY)
      .and_return nil
    @config.should_receive(:set)
      .with({CheckoutViewSetting::IMAGE_ID_CONFIG_KEY => @uploaded_file.original_filename})

    @cv.image = @uploaded_file
    @cv.image_cache.include?(@uploaded_file.original_filename).should be_true
    @cv.write_image_identifier
    @cv.store_image!
  end

  it "should be able to set the image as form params" do
    @config.should_receive(:[])
      .with(CheckoutViewSetting::IMAGE_ID_CONFIG_KEY)
      .and_return nil

    @config.should_receive(:set)
      .with({CheckoutViewSetting::IMAGE_ID_CONFIG_KEY => @uploaded_file.original_filename})
      .and_return(@uploaded_file.original_filename)

    @cv.update_config(:image => @uploaded_file)
  end

  it "should be able to remove the image as form params" do
    @config.should_receive(:[])
      .with(CheckoutViewSetting::IMAGE_ID_CONFIG_KEY)
      .and_return nil
    @config.should_receive(:set)
      .with({CheckoutViewSetting::IMAGE_ID_CONFIG_KEY => nil})
      .and_return(nil)

    @cv.update_config(:remove_image => "1")
  end
end

The first thing I do here is mock the Spree config so that I can watch when the configuration gets written to make sure that works ok. I'm going to trust CarrierWave to do its thing as far as storing and removing the image. But what I'm really interested in for my tests is that the correct identifiers get read and set in the Spree::Config object. I'm also using the exact uploaded file object that will be passed to a Rails controller during the form submission so I don't have to worry about whether the image is the proper format.

In the next test, I want to make sure that I can use the standard conventions for setting the image straight from the class. The next two test cases I want to test the ability to pass a set of params as they would come from a form submission to save and remove an image. This matches the behavior you get when attaching CarrierWave to an ActiveRecord object.

Building the Class

One of the really nice things about CarrierWave is the design of the library. The record that is going to hold the information about the uploaded file can be any class so long as it implements a few methods for handling the reading and the writing of a serialized identifier. In ActiveRecord, this is a string field. For our example here, we are going to use the Spree::Config object. But it could be any method you choose to implement.

In its simplest form, our class is going to look like this:

# lib/checkout_view_setting.rb
require 'carrierwave/mount'

class CheckoutViewSetting
  extend CarrierWave::Mount
  mount_uploader :image do
    def default_url
      ""
    end
  end
  
  IMAGE_ID_CONFIG_KEY = :checkout_view_setting_image_id
  
  attr_reader :config
  
  def initialize(config)
    @config = config
  end

  def configured?
    ! @config[IMAGE_ID_CONFIG_KEY].nil?
  end

  # called to read the serialized identifier
  def read_uploader(column)
    @config[IMAGE_ID_CONFIG_KEY]
  end

  # called to write the serialized uploader
  # In Spree 0.60.x you cannot set attributes directly through []= like you can in the newer versions. 
  # So I use the set method to assign the value
  def write_uploader(column, identifier)
    @config.set IMAGE_ID_CONFIG_KEY => identifier
  end

  def write_remove_image_identifier
    write_uploader(:image, nil)
  end

  def store_remove_image!
    remove_image!
  end
end

The code above will allow our first test to pass. There's a bunch of stuff going on here, so I'll walk through the important parts:

  • require 'carrierwave/mount' just loads the CarrierWave mount library.
  • extend CarrierWave::Mount will add the mount_uploader method to our class so that we can designate which uploader we are going to use. Each file that is uploaded requires an uploader, so if our class were going to upload a banner and footer image, we would need an uploader for both of those.
  • IMAGE_ID_CONFIG_KEY = :checkout_view_setting_image_id is a unique key in the Spree::Config preferences system that is going to contain the image identifier.

The two most important methods here are the write_uploader and read_uploader. These methods will be called by CarrierWave to get and set your identifier. Here, I'm just setting those in the config. The configured? method is just for convenience to see if the value is set. The other two methods are used when removing the image.

When mount_uploader is called, a bunch of methods are automatically defined in your class and they will all have the name of your uploader in the method name. It also takes a configuration block where you can configure your uploader. Another option is to create your uploader class like you would in a typical CarrierWave setup, but I chose to use an anonymous uploader here for brevity. Oh, and all those methods that were defined? Here's a list of methods that were defined in my class:

  • image
  • image=
  • image?
  • image_url
  • image_cache
  • image_cache=
  • remote_image_url
  • remote_image_url=
  • remove_image
  • remove_image!
  • remove_image=
  • remove_image?
  • store_image!
  • image_integrity_error
  • image_processing_error
  • write_image_identifier
  • image_identifier
  • store_previous_model_for_image
  • find_previous_model_for_image
  • remove_previously_stored_image

Pretty nice! You can see how this is done by checking out the source code on GitHub. Now that these methods are in place, my class will act just like an ActiveRecord object with CarrierWave!

Well, almost. CarrierWave lets you do some pretty cool stuff with ActiveRecord's update_attributes method. For example, setting a value of :remove_image => 1 in the params with a checkbox will call the remove_image= method on your class. Since my class doesn't have an update_attributes method, I'm going to implement something similar with a method called update_config that will accept params from a form and act in a similar fashion to ActiveRecord. Here's the code I came up with (borrowed heavily from ActiveRecord).

# stores the setting similar update_attributes in ActiveRecord
# if a remove_image param is present, the image is removed instead of added.
def update_config(params)
  return @config if params.empty?
  begin
    safe_params = params.reject {|k,v| ! %w(image remove_image).include?(k.to_s) }
    safe_params.each do |k,v| 
      send("#{k}=", v) 
      send("write_#{k}_identifier")
      send("store_#{k}!")
    end
  rescue => e
    Rails.logger.error "Error uploading Checkout View Setting: #{e.message}"
    return false
  end
  return @config
end

First, I do some parameter checking so that only params that I want to processed are actually sent. Since I'm using metaprogramming here, I don't want to introduce a case where someone could run various methods by passing bad params (much like mass-assignment vulnerabilities). Then I loop through the params and run the commands needed to save the image.

The Implementation

To use my class in my controller I just setup the following update method:

class Admin::CheckoutViewSettingsController < Admin::BaseController
  before_filter :get_checkout_view_settings
  def update
    config = @cv.update_config(params[:checkout_view_settings])
    if config
      flash[:notice] = t("admin.success")
    else
      flash[:error] = t("admin.fail")
    end
    redirect_to admin_checkout_view_settings_path
  end

  private
  def get_checkout_view_settings
    @cv = CheckoutViewSetting.new(Spree::Config)
  end

end

And then in my form I have the following (haml):

= form_tag admin_checkout_view_settings_path, :method => :put, :multipart => true do
  #formTop
    %fieldset
      .field
        = label_tag :image, t(".image.label")
        %br/
        = file_field_tag 'checkout_view_settings[image]', :accept => 'image/png,image/gif,image/jpeg'
    %fieldset
      .field
        = check_box_tag "checkout_view_settings[remove_image]"
        = label_tag "checkout_view_settings[remove_image]", t(".remove_image.label"), :style => "display: inline"
  #formBottom
    = image_tag(@cv.image.url) if @cv.configured?
    %br/
    = submit_tag

CarrierWave is an awesome library and a very suitable replacement for the more popular Paperclip library. I couldn't find anywhere online for showing a complete solution for implementing CarrierWave in an non-ActiveRecord class so I hope this helps anyone else that wants to go this route.

Read more about End Point's Spree development and consulting services here.

RailsConf 2012: Day One

Today kicks off the official first day of RailsConf 2012, in Austin, Texas (yesterday RailsConf unofficially kicked off with Ignite Rails). This is my fourth RailsConf, and I've found that I get increasingly more out of each one as my time working with Rails has accumulated.

The morning started with a keynote by DHH. DHH talked about progress and keeping an interest in something because it continues to make progress. Initially, the Rails community had a negative reaction to progress like Ruby 1.9, Rails 3, the Asset Pipeline and CoffeeScript, but after the initial learning curve, the new tools and progress was appreciated by the community. DHH talked about how there have been many studies on what prevents people from progressing, and one of those causes is loss aversion, meaning that people hate losing or failing enough that it prevents them from advancing. To me, the point of his talk was to prepare everyone for Rails 4 and how it will challenge and break things. Rails 4 is admittedly not coming out anytime soon, so he didn't touch on any specifics.

Using Backbone.js with Rails

One session talk I attended was Using Backbone.js with Rails by Sarah Mei of Pivotal Labs. Backbone is not an MVC framework in JavaScript, but it is lumped into traditional JavaScript MVC frameworks. It serves as a lightweight infrastructure for organizing code like templates, interactive elements, and events. Because Backbone is a relatively immature framework, like other JavaScript frameworks, there isn't a standard way of writing code, so the implementation of Backbone varies widely.

Sarah talked about the love story between JS and Rails:

She touched on the basics of Backbone from a Rails perspective:

  • Models: Backbone and Rails models are very similar. An application will likely have Backbone models that mirror Rails models.
  • Templates: Backbone templates are akin to Rails views or a mustache template.
  • Views: Views are similar to Rails controllers. They set a template, class name, and define events.

Next, Sarah talked about two methods of implementation using Backbone.js and went through some code examples:

  • Greenfield App
    • This approach tends to be more API driven and makes sense for an API-driven application that also requires buildout of native mobile apps.
    • In this approach, the server side returns [JSON, etc.] data and doesn't render HTML.
    • Sarah noted that the amount of code required here can be substantial because the framework is lightweight, so she would recommended considering alternatives (such as ember.js).
  • "Backbone as Frosting" Implementation
    • In this implementation, there is a mixture of server-side and client-side rendering.
    • Sarah noted here that the lightweightedness of Backbone works nicely.

I haven't used Backbone.js in any client projects, but I've looked into it recently out of interest. I like the general idea of improving JavaScript code organization via the frosting approach described by Sarah. A few code examples were shared during the presentation — check out the slides here.

CoffeeScript for the Rubyist

Another talk I attended today was CoffeeScript for the Rubyist by Mark Bates. CoffeeScript is a JS meta language that compiles into JS and can easily integrate with your current JavaScript. It's easy to read, write, and maintain. I took the CoffeeScript lesson over at Code School recently (highly recommended!). Mark went through a brief history of JavaScript and explained that like the evolution to early assembly languages to C to Ruby (with a few missing steps), CoffeeScript's aim is to be an improved tool for writing readable and maintainable code. The syntax of CoffeeScript is a hybrid of Ruby and Python which is simple, uses no semicolons, typically no curly braces, no "function" keyword, relaxed parenthesis, and significant whitespace.

Mark then went into a review of Ruby-like CoffeeScript behavior. Below are some of the important topics in CoffeeScript that he covered, as well as links to the documentation:

In addition to the talks on Backbone.js and CoffeeScript, I'll post the links to slides and video for the other talks I attended when they become available.

What's Next?

It's funny (but not surprising) how many of the talks I attended on Day One weren't actually about Rails or the specifics of Rails. Perhaps I'll try to pick a few more Rails-centric talks in the next couple of days of the conference and report on those. Stay tuned!

Byte-swap an entire file using perl

I recently needed to byte-swap an input file, and came up with an easy way with perl:

$ perl -0777ne 'print pack(q{V*},unpack(q{N*},$_))' inputfile > outputfile

This will byte-swap 4-byte sequences.  If you need to byte-swap 2-byte sequences, you can just adjust the formats for pack/unpack to the lower-case version like so:

$ perl -0777ne 'print pack(q{v*},unpack(q{n*},$_))' inputfile > outputfile

(Of course there are more efficient ways to handle this, but for a quick and dirty job this may just be what you need.)

We use the -0777 option to ensure perl reads the input file in its entirety and doesn't affect newlines, etc.

Deconstructing an OO Blog Designs in Ruby 1.9

I've become interested in Avdi Grimm's new book Object on Rails, however I found the code to be terse. Avdi is an expert Rubyist and he makes extensive use of Ruby 1.9 with minimal explanation. In all fairness, he lobbies you to buy Peter Cooper's Ruby 1.9 Walkthrough. Instead of purchasing the videos, I wanted to try and deconstruct them myself.

In his first chapter featuring code, Mr. Grimm creates a Blog and Post class. For those of you who remember the original Rails blog demo, the two couldn't look more different.

Blog#post_source

In an effort to encourage Rails developers to think about relationships between classes beyond ActiveRecord::Relation, he creates his own interface for defining how a Blog should interact with a "post source".

# from http://objectsonrails.com/#sec-5-2
class Blog
  # ...
  attr_writer :post_source
  
  private
  def post_source
    @post_source ||= Post.public_method(:new)
  end
end

The code above defines the Blog class and makes available post_source= via the attr_writer method. Additionally, it defines the attribute reader as a private method. The idea being that a private method can be changed without breaking the class's API. If we decide we want a new default Post source, we can do it safely.

The magic of this code is in defining the post source as a class's method, in this case, Post.public_method(:new). The #public_method method is defined by Ruby's Object class and is similar to #method method. In short, it gives us a way of not directly calling Post.new, but instead, referring to the method that's responsible for creating new posts. This is logical if you remember that the name of this method is #post_source.

Now let's look how he puts post_source into action.

class Blog
  # ...
  def new_post
    post_source.call.tap do |p|
      p.blog = self
    end
  end
  # ...
end

During my first reading, it wasn't clear at all what was going on here, but if we remember that post_source is responsible for returning the method need to "call", we know that post_source.call is equivalent to Post.new. For the sake of clarity-while-learning for those not familiar with post_source.call, let's substitute it with something more readable so we can understand how tap is being employed.

class Blog
  # ...
  def new_post
    Post.new.tap do |p|
      p.blog = self
    end
  end
end

The tap method is available to all Ruby objects and serves as a way to have a block "act on" the method's caller and return the object called. Per the docs, "the primary purpose of this method is to 'tap into' a method chain, in order to perform operations on intermediate results within the chain". For some examples on using tap see MenTaLguY's post on Eavesdropping on Expressions. As he says in his post, "you can insert your [code] just about anywhere without disturbing the flow of data". Neat.

In this case, it's being used to tap into the process of creating a new blog post and define the blog to which that post belongs. Because tap returns the object it modifies, #new_post returns the post now assigned to the blog.

Brining it All Together

Avdi's approach may seem cumbersome at first, and it is compared to "the Rails way." But in general, that's the whole point of Object on Rails; to challenge you to see beyond a generic solution to a problem (in this case defining relationships between classes) so you can build more flexible solutions. Let's see some interesting things we might be able to do with this more flexible Blog class. We can imagine this same Blog class being able to handle posts from all sorts of different sources. Let's see if we can get creative.

class EmailPost < ActionMailer::Base
  def receive(message)
    @blog = Blog.find_by_owner_email(message.from)
    @blog.post_source = EmailPost.public_method(:new)
    @email_post = @blog.new_post(params[:email_post])
    @email_post.publish
  end
end

With this little snippet, we're able to use the Blog class to process a different sort of post. We simply let the blog know the method to call when we want a new post and pass along the arguments we'd expect. Let's see if we can think of something else that's creative.

require 'feedzirra'
# execute regularly with cronjob call like curl -d "blog_id=1&url=http://somefeed.com" http://myblog.com/feed_poster"

class FeedPostersController
  def create
    @feed = Feedzirra::Feed.fetch_and_parse(params[:url])
    @blog = Blog.find(params[:blog_id])
    @post.post_source = FeedPost.public_method(:new)
    @feed.entries.each do |entry|
      @blog.new_post(entry)
    end
  end
end

We could imagine the FeedPost.new method being the equivalent of a retweet for your blog using an RSS feed! Try having the blog class doing this with an ActiveRecord association! Seems to me the Blog class might need to get a bit more complex to support all these Post sources which makes post_source.call.tap look pretty good!

UTOSC, here I come


Recently the Utah Open Source Foundation announced their schedule for this year's UTOSC, scheduled for May 3-5 at Utah Valley University. I'm not sure I've ever before felt sufficiently ambitious to submit two talk proposals for a conference, but I did this time, and both were accepted, so I'll give one talk on database constraints, from simple to complex, and another on geospatial data visualization such as we've been doing a whole lot of lately. Jon Jensen will also be there with two talks of his own, one on website performance and the other a "screen vs. tmux faceoff". We use screen pretty heavily company-wide, but I've wanted to learn tmux for quite a while, so this one is on my list to see.


DATABASE CONSTRAINTS

Database constraints are something I've always strongly encouraged, and my commitment to clearly constrained data has become more complete recently after a few experiences with various clients and inconsistent data. The idea of a database constraint is to ensure all stored data meets certain criteria, such as that a matching record exists in another table, or the "begin" date is prior to the "end date", or even simply that a particular field is not empty. Applications and frameworks may claim to validate data for you, such that the database doesn't need to worry about it, but in practice those claims often prove only marginally true. In recent years, PostgreSQL especially has become much more flexible and powerful in this regard, with the addition of exclusion constraints, but developers still frequently disregard database constraints, at their peril.


GEOSPATIAL VISUALIZATION

Geospatial data visualization — that is, displaying data that has a location component — is a huge field, and well-explored, but Google Earth and Liquid Galaxy are an impressive recent leap forward. End Point has done a great deal of work to make the Liquid Galaxy platform easily configurable and manageable, and has created powerful tools for development of these visualizations. My visualization presentation will cover the basics of Google Earth and Liquid Galaxy, and show how to begin turning data into visualizations.


UTOSC started a few years ago to cater to the large open source movement in Salt Lake City, Provo, and surrounding areas, and the conference has grown to something truly remarkable. Expect several hundred attendees for the three days full of talks and tutorials. The community is a vibrant one, and the attendees, as well as the sponsors, come from many different domains and with many different areas of expertise. End Point will be one of those sponsors this year: we're going to man a booth with a scaled-down Liquid Galaxy, so come see what we've been talking about. We demonstrate several visualizations of the sort we've done recently, to include panoramic video, automatic visualization builders, and innovative systems for real-time control. Registration is now open, so sign up and come say hi.

Integrating Propel ORM

Propel ORM Integration

Most of us have worked in environments that are organized in an less than desirable manner. A common recurring problem is the organization and communication between the business logic and the database model. One helpful methodology to help with a problem like this is implementing an ORM (Object-Relational Mapping) system. There are many to choose from, but I would like to discuss use and integration of Propel ORM.

Propel is currently only used in PHP, but supports many different databases. Currently Propel supports the following databases: MySQL, Postgres, Sqlite, MsSQL, and Oracle.

Installation and Setup

The main point of this post is to show how easily you can start integrating an ORM into your working environment. The explanation and examples below assume that you have installed the correct packages and configured Propel to work with your environment properly.

The Propel website offers great documentation on how to do that:

Installation
Configuration

Integration

After you have set everything up, in particular the build.properties file, you can now generate your schema.xml file. This generated file describes your database in XML, everything form datatypes to the relationships between tables. Run the following command to generate this XML file:

  propel-gen reverse

Now we want to generate the PHP classes that you will use to interact with your data. Do this by running the following command.

  propel-gen .

If everything went well you should now have a directory called 'build/' where you ran the propel_gen commands. Look in 'build/classes/' you will see the name of the project you named earlier in build.properties. Within this directory you will see a list of files, one for each of your tables. Copy these somewhere you can easily include them from a PHP file.

From now on each time you make a change to your database you will need to either modify your schema.xml to reflect that change, or simply run the propel-gen reverse to have it generated for you. You will also need to copy the generated PHP classes to your project when changes are made. There are many different ways you can use Propel for data model management, this is just one way to get you up and running.

Usage

Propel offers very elaborate functionality for interfacing with your data-model. I have included two quick examples to give you an idea of how you can easily begin to use it. I have included two examples showing some simple retrieval functionality. For this example let's say we have a table called 'Customers' with a few common columns like name and address.

Simple Select Statement

$query = new CustomerQuery();
$customers = CustomerQuery::create()->orderByName()->limit(2)->find();
$first_customer = $query->findPK(1); // This would give us the first in the list

Custom SQL

$con = Propel::getConnection(CustomerPeer::DATABASE_NAME);
$sql = "SELECT * FROM customer WHERE name ilike ('%:name%');
$stmt = $con->prepare($sql);
$stmt->execute(array(':name' => 'Terry'));

These are just two 'very' simple examples but Propel offers many ways to access and update your data. Please visit the Propel website if you would like to read more about this ORM.

Liquid Galaxy Website Launch

We have just launched a new site to promote our Liquid Galaxy Turn-Key systems, our suite of Liquid Galaxy Services as well as presenting the current range of the Liquid Galaxy's Capabilities.

Check the Liquid Galaxy Website here

End Point has been developing the Google Liquid Galaxy project into a commercially available platform over the last two years. In case you are unfamiliar with the Liquid Galaxy project here is a brief rundown:

  • Originally developed by engineers at Google on their 20% time
  • Provides an immersive viewing environment for Google Earth by running multiple instances of the software synced across any number of displays
  • The core software is available as open source

So far the majority of the people using this system (only a small number to date) are advanced hackers and hobbyists who have set up mini versions using computer display monitors. Some of these talented developers have completed projects like porting open source video games or experimenting with different display configurations.

Meanwhile, End Point has been hard at work developing a standardized, portable, and robust turn-key version of the Liquid Galaxy system. Through trial and error and the experience of configuring and installing over 30 Liquid Galaxies worldwide, we've come up with the best and most reliable approach to building and maintaining a Liquid Galaxy system for real world use.

We've added remote monitoring and support, optimized the code and performance of the system, created custom stands and mounts, and sourced high quality components. This allows us to easily configure, install, and support a Liquid Galaxy system for your organization. No technical knowledge is needed from your company to take advantage of this exciting new display technology.

If you are interested in learning more about purchasing or renting a turn-key Liquid Galaxy system head on over to our new Liquid Galaxy website and use the contact form or dial our number and talk to us directly. We love talking to anyone about the Liquid Galaxy, whether you're interested in purchasing a system, or just curious to learn more. You can even schedule a visit to our New York office to see it up close in person.

Monitoring cronjob exit codes with Nagios

If you're like me, you've got cronjobs that make email noise if there is an error. While email based alerts are better than nothing, it'd be best to integrate this kind of monitoring into Nagios. This article will break down how to monitor the exit codes from cronjobs with Nagios.

Tweaking our cronjob

The monitoring plugin depends on being able to read some sort of log output file which includes an exit code. The plugin also assumes that the log will be truncated with every run. Here's an example of a cronjob entry which meets those requirements:

rsync source dest 2>&1 > /var/log/important_rsync_job.log; echo "Exit code: $?" >> /var/log/important_rsync_job.log

So let's break down a couple of the more interesting points in this command:

  • 2>&1 sends the stderr output to stdout so it can be captured in our log file
  • Notice the single > which will truncate the log every time it is run
  • $? returns the exit code of the last run command
  • Notice the double >> which will append to the log file our exit code

Setting up the Nagios plugin

The check_exit_code plugin is available on GitHub, and couldn't be easier to setup. Simply specify the log file to monitor and the frequency with which it should be updated. Here's the usage statement:

Check log output has been modified with t minutes and contains "Exit code: 0".
        If "Exit code" is not found, assume command is running.
        Check assumes the log is truncated after each command is run.
        
        --help      shows this message
        --version   shows version information

        -f          path to log file
        -t          Time in minutes which a log file can be unmodified before raising CRITICAL alert

The check makes sure the log file has been updated within t minutes because we want to check that our cronjob is not only running successfully, but running regularly. Perhaps this should be an optional parameter, or this should be called check_cronjob_exit_code, but for right now, it's getting the job done and cutting back on email noise.

An Introduction to Google Website Optimizer

On End Point’s website, Jon and I recently discussed testing out use of Google Website Optimizer to run a few A/B tests on content and various website updates. I've worked with a couple of clients who use Google Website Optimizer, but I've never installed it from start to finish. Here are a few basic notes that I made during the process.

What’s the Point?

Before I get into the technical details of the implementation, I’ll give a quick summary of why you would want to A/B test something. A basic A/B test will test user experiences of content A versus content B. The goal is to decide which of the two (content A or content B) leads to higher conversion (or higher user interactivity that indirectly leads to conversion). After testing, one would continue to use the higher converting content. An example of this in ecommerce may be product titles or descriptions.

A/B tests in Google Website Optimizer

I jumped right into the Google Website Optimizer sign-up and wanted to set up a simple A/B test to test variations on our home page content. Unfortunately, I found right away that basic A/B tests in Google Website optimizer require two different URLs to test. In test A, the user would be see index.html, and in test B, the user would see index_alt.html. This is unfortunate because for SEO and technical reasons, I didn’t want to create an alternative index page.

Test A: Keep existing text

Test B: Remove some paragraph text in first section

Multivariate Testing

Rather than implement a basic A/B tests in Google Website optimizer, I decided to implement a multivariate test with just the two options (A and B). The basic setup required this:

  • Copy provided JavaScript into my test page just above </head>
  • Wrap <script>utmx_section("stephs_test")</script>...</noscript> around the section of text that will be modified by the test.
  • Copy provided JavaScript into my converting test page just above </head>

Google Website Optimizer will verify the code and provide a user interface to enter the variations of the test text.

I used multivariate testing to test the homepage changes described above. After a couple of weeks of testing, my test results were inconclusive:


Example multivariate test result in Google Website Optimizer

A Limitation with Multivariate Testing

One thing we wanted to test was a site-wide CSS change. Unfortunately, the multivariate testing in place is designed to test on page content only rather than global CSS changes. You could potentially come up with a “creative” hack and set a cookie inside the variation to specify which layout option you would use. And then the page would always look at that cookie while rendering to apply the special CSS behavior. However, this requires a bit of customization and development.

Easy Creating Ramdisk on Ubuntu

Hard drives are extremely slow compared to RAM. Sometimes it is useful to use a small amount of RAM as a drive.

However, there are some drawbacks to this solution. All the files will be gone when you reboot your computer, so in fact it is suitable only for storing some temporary files - those which are generated during some process and are not useful later.

I will mount the ramdisk in my local directory. I use Ubuntu 11.10, my user name is 'szymon', and my home directory is '/home/szymon'.

I create the directory for mounting the ramdisk in my home dir:

mkdir /home/szymon/ramdisk

When creating the ramdisk, I have a couple of possibilities:

  • ramdisk - there are sixteen standard block devices at /dev/ram* (from /dev/ram0 to /dev/ram15) which can be used for storing ram data. I can format it with any of the filesystems I want, but usually this is too much complication
  • ramfs - a virtual filesystem stored in ram. It can grow dynamically, and in fact it can use all available ram, which could be dangerous.
  • tmpfs - another virtual filesystem stored in ram, but because it has a fixed size, it cannot grow like ramfs.

I want to have a ramdisk that won't be able to use all of my ram, and I want to keep it as simple as possible; therefore, I will use tmpfs.

The following command will mount a simple ramdisk in my new local directory.

$ sudo mount -t tmpfs -o size=512M,mode=777 tmpfs
/home/szymon/ramdisk

I can even unmount it with:

$ sudo umount /home/szymon/ramdisk

Let's check if the ramdisk is really there. I can do so in a couple of ways.

I can use df -h to check the size of the mounted device:

$ df -h | grep szymon
tmpfs                 512M     0  512M   0% /home/szymon/ramdisk

I can also use mount to report on the mounted devices:

$ mount | grep ramdisk
tmpfs on /home/szymon/ramdisk type tmpfs (rw,size=512M,mode=777)

There is one more thing to do - make the ramdisk load automatically at machine start. This can be done by adding the following line into /etc/fstab:

tmpfs    /home/szymon/ramdisk    tmpfs    rw,size=512M,mode=777 0    0

Two things to be aware of:

  • Data stored in ramdisk will be removed at the machine restart. Creating a script for saving the files to hard drive at machine shutdown won't persist the data during a machine crash or reset.
  • The computer won't be able to boot normally if the entry in the fstab file has any errors in it. If you do make such an error, you can always boot the computer in recovery mode, so you have the root console and can fix the fstab file.

XOR ROX

Recently a co-worker posed an interesting issue:

Given a non-zero integer $delta,

an array of structures with two key/value pairs,
{ flag => boolean, quantity => non-zero value }

Sort the array so that the first structures are those where either

flag is true and (sign of $delta and sign of $quantity are different)
or
flag is false and (sign of $delta and sign of $quantity are the same)

Secondarily, sort on the absolute value of $quantity.

A solution fairly leaped out at me, but I’m not claiming incredible programming skill: in fact, the solution suggested an XOR operation, which was the second time in about as many weeks that I’d gotten to use XOR in Perl code. (It’s one of those things that you can literally write tens of thousands of lines of code without ever needing, so a second opportunity within the same decade was pretty pleasing in a code-geek kind of way.)

The key to recognizing XOR in your problem solution is a pattern like:

A AND (B != C) or ~A AND (B == C)

or more simply:

(A AND ~B) or (~A AND B)

which is nothing more complex than the expanded equivalent of (A XOR B), from your college symbolic-logic class. The daunting sort problem becomes:

@sorted = sort {(
        ($a->{flag} xor ($a->{quantity} > 0 xor $delta > 0))
        <=>
        ($b->{flag} xor ($b->{quantity} > 0 xor $delta > 0))
) || abs($a->{quantity} <=> abs($b->{quantity})} @items;

That’s more “xor” operations in one statement than I’ve used in the last year.

Tips for job applicants

Consider this a public service announcement. This is a list of assorted suggestions to job applicants I’ve collected over the years based on my own observations and conversations I’ve had with others who were hiring.

General

  • When you apply for a job, make sure you actually have time to pursue it. It’s rude to say you’re too busy to interview except late in the evening or for 15 minutes on your cell phone in the parking lot outside work. Finding a job takes some investment and you shouldn’t waste potential employers’ time if you’re not ready to put some of your vacation time on the line.
  • This is old advice, but important as always: Don’t claim to have abilities or experience that you don’t! Be honest about your strengths and if directly asked, admit what you lack. We will find out anyway, and it’s pretty embarrassing to all of us to have to see you’re dishonest and clueless about something you claimed as a skill.
  • If you’re applying for a programming job, be prepared to show recent code samples. Saying that all your work is under NDA with employers may be true, but doesn’t help me see what your code is like. If you can’t show any code from work, you’ll need to have code from side projects or open source projects to show, most conveniently on GitHub. How else is anyone supposed to get familiar with your work?

Do some research

  • Take a few minutes to become at least minimally familiar with our company. If conversation reveals that you have not even reviewed the major sections of their website, it shows you’re not really interested and probably just spammed your application as widely as possible.
  • Read the complete job posting and pay attention to the details. You should already know whether it’s a contract or salaried position, full- or part-time, in office or telecommuting, etc. if the job description made that clear.
  • In correspondence and conversation with a potential employer, pay attention to details. Spell and pronounce names of people, companies, and software correctly. Use good capitalization and punctuation, and proofread your writing. It only takes an extra few seconds, but shows you care. You probably don’t want to work for an employer who is completely unforgiving on such matters, but an employer probably doesn’t want to hire you if you seem oblivious of all this. (For example, here’s an industry-specific tip: The short form of “PostgreSQL” is “Postgres” ending with a hard unvoiced “s”. Never “Postgre”!)

Watch your email

  • Sending me an empty email with only a resume attached is really lame.
  • When responding about a job by email, write a cover letter directly in the email body, specific to our company and the job.
  • Why would you use anything other than your first & last name in your email From: header when writing about a job? Cute nicknames look silly in this context and your first or last name only will not be easily searchable. The From text is often the first thing someone sees in an inbox, so make it presentable.
  • Send your resume in standard PDF format. Other formats are far more likely to have rendering oddities on computers other than your own. A link to your online resume is ok, but if that’s all you provide, make sure your website stays up! Yes, I’ve really had candidates provide only a link to an online resume and the website was down or displaying errors when I went to look at it.
  • While you’re looking for a job, check your email spam folder regularly to look for false positives from potential employers. It’s your loss if you overlook email that wrongly goes there.

Interviewing

  • When it’s time to schedule an interview, don’t share irrelevant details about how busy you are, and what you’re doing when that makes scheduling difficult. Just make time for the interview at one of the times when the interviewer can meet.
  • Be on time for scheduled interviews, whether on the phone or in person. That means paying attention to travel time, and time zone differences. And tracking your appointments on some kind of calendar.
  • During a phone interview, make sure you’re in a fairly quiet place and have a reliable phone connection. Cell phones often have poor audio quality, but if you must use one, use it somewhere that you know has good reception and stay put.
  • Give an interview your undivided attention. You shouldn’t be answering your phone, reading email or text messages, or spacing out and saying “sorry, can you repeat that?”
  • Be prepared to have a video chat on a Google+ Hangout, Skype, or similar.

This wasn’t meant to be an exhaustive list, but will help the attentive job-seeker. Much of it applies to all of us in normal business communication with our co-workers and others as well.

On a distantly related note, I recently came across this list of tips called How to Get the Best Customer Service (Anywhere). It appeared on SoftLayer’s blog and has a similar overall theme of approaching others respectfully, and communicating effectively.

Make your code search-friendly

Here's something about coding style that you may not have considered: is your code "search-friendly"? That is, does the format of your code help or hinder someone who might be searching it for context while debugging, extending, or just learning how it works?

Seriously Contrived Example (from Perl):

my $string = q{Your transaction could not be} .
   q{ processed due to a charge} .
   q{ card error.};
return $string;

Now someone's going to experience this error and wonder where it occurs. So armed with grep, or ack, or git-grep, they set off into the wilderness:

$ git grep 'could not be processed'
$ git grep 'charge card error'
$ git grep -e 'transaction.*charge.*error'
$ alsdkjgalkghkf

(The last simulates pounding the keyboard with both fists.) I would suggest humbly that "strings you emit as a line should appear as a line in your code", if for no other reason than that it makes it so much easier for you or others to find them. Thus:

my $string = <<'MSG';
Your transaction could not be processed due to a charge card error.
MSG
return $string;

Three Things: Photography, Facebook on WordPress, and the watch command

1. Photography News

There’s been some recent news in the photography space. Adobe announced Photoshop CS6, and Lightroom 4, and Adobe Photoshop Touch recently. The Lytro [Light Field] camera has picked up recognition lately. The new Nikon D4 recently became available, as well as the Canon 5D MK III, both high end DSLRs.

2. Facebook Comments for WordPress

Last week, I was working on WordPress development for The Best Game Apps, and I came across a strange Facebook integration error about compatibility mode, show in the screen below:

The site uses the WordPress plugin Facebook Comments for WordPress. After some research, I decided to dig into the Facebook documentation and the code to make the following change myself to specify the post URL as the href attribute in the facebook markup:

219c219
-    echo "\t<fb:comments xid='$xid' url='$postUrl' $siteisdark ",
+    echo "\t<fb:comments xid='$xid' href='$postUrl' url='$postUrl' $siteisdark ",

In the context of:

219                     echo "\t<fb:comments xid='$xid' href='$postUrl' url='$postUrl' $siteisdark ",
220                         "numposts='{$fbc_options['numPosts']}' ",
221                         "width='{$fbc_options['width']}' ",
222                         "publish_feed='$publishToWall' ",
223                         "migrated='1'></fb:comments>";

With this change, the Facebook error message disappeared and all appeared to be working as expected. I’m surprised that this hasn’t been fixed in the plugin itself, and the documentation wasn't extremely helpful. Sometimes the best option is to go right to the code. And of course, I'm recording this so I can revisit it if it breaks again with a WordPress plugin upgrade.

3. The watch Command

I was talking to Jon recently about monitoring disk usage and he mentioned the watch command. Specifically, the watch command might be a good tool to monitor disk usage if you expect to see the disk fill up quickly while you are actively developing on it. I'm always learning random tips from End Point's sysadmin experts, and this one may come in handy at some point.

Guide to Ubuntu 11.10 on a Samsung Netbook

12.04 UPDATE: Unsurprisingly, after installing 12.04 which includes the 3.2.x kernel, brightness controls work perfectly out of the box. Sadly, it's still necessary to tweak modprobe to get wireless working after a suspend/resume. Also, don't forget to reboot after making the changes!


After reading a few too many reviews about netbook hardware and its compatibility with Linux, I settled on a Samsung NF310 netbook. However, like all things worth doing, it wasn't nearly as easy as I'd hoped. This post highlights some of the lessons learned from my first days with Linux on my Samsung netbook.

Pre-Installation

I had been eagerly awaiting the arrival of my new hardware and had already gotten a fresh copy of Fedora 16 ready. The crippled Windows 7 Starter Edition was going to be off that netbook as a first order of business, as a matter of principle. If I had it to do over again, I would have installed the Windows-only BIOS update for the Netbook first. I'm working through using BartPE, but the installation onto the USB drive hasn't gone well. Best to do these kinds of Windows-only activities while Windows is still easily available.

Additionally, I would have used YUMI multiboot USB creator to put on a copy of Ultimate Boot CD as well as a handful of live Linux distros for testing. When I later had issues installing, I found myself wondering, is it the hardware? I ended up going back and running the tests (which passed), but had I done this first, I would not have been distracted from the real problem.

Installation

Aside from choosing a distro, I found getting the installation to complete successfully to be a chore. It seems the disk's partition table was GPT and so both Ubuntu and Fedora's installation processes failed.

ext4 Filesystem check failure on/dev/mapper/vgsamsungnetbook-lvroot:

Operational error.

Errors like this usually mean there is a problem with the filesystem that will require user interaction to repair. Before restarting installation reboot to rescue mode or another system that allows you to repair the filesystem interactively. Restart installation after you have corrected the problems on the filesystem.

You can see how an error like this might prompt you to run hardware checks. To work around this, simply fire up one of the Live CD installations on your YUMI stick and offer Linux the following incantation:

parted
mklabel msdos
quit

Test Driving Fedora 16

After getting past installation issues, I loaded Fedora 16's XCFE spin. Despite the dual core Atom processors and the 2GB RAM, I was looking for a lightweight distro; snappy UI and battery life were my first concerns and I knew less demands from the OS would leave more resources for the work I wanted to do.

The sparse XCFE desktop experience offered few bells and whistles but exactly what (I thought) I wanted. It was responsive and stayed out of the way. However, I quickly discovered that after resuming from suspend, my wireless connection would not work. Was this Fedora? Was this the kernel? Was this hardware failure? Only trial and error would help me figure it out...

Troubleshooting wireless connectivity after resume

The Samsung came with a Broadcom chip, which was recognized by Fedora, but I have a low opinion of Broadcom hardware and know that its driver support is relatively new in the kernel mainline. I noticed kernel errors related to the driver and decided to swap out my other laptop's Intel Ultimate N Wifi Link 5300 card and see what happened. To Linux's credit, it detected the new hardware without a hitch and connected right away after booting. However, upon another round of suspend/resume, wireless connectivity couldn't be established.

Next, I decided to see if this could be a distro issue. I tried Ubuntu 11.10 to find this was not an issue with a particular distro. Of course with both attempts I fully patched all the software, so I didn't think it a kernel issue; Fedora was running the 3.3 kernel!

Sadly, it's 2012 and Linux still can't handle simple power management issues out the box. Fortunately for you, lucky reader, I have the magic incantations you need to have wireless network connectivity after suspend/resume. I knew I couldn't be the only one experiencing the issue but was simply blown away this wasn't better documented. As always, log files tend to be the best source for searching Google; results seem more accurate and from more authoritative sources. Here's what I was seeing from /var/log/kern.log after resuming:

kernel: [  326.394858] wlan0: direct probe to 00:13:10:9e:88:5f (try 1/3)
kernel: [  326.592242] wlan0: direct probe to 00:13:10:9e:88:5f (try 2/3)
kernel: [  326.792214] wlan0: direct probe to 00:13:10:9e:88:5f (try 3/3)
kernel: [  326.992198] wlan0: direct probe to 00:13:10:9e:88:5f timed out

Fixing Wireless Connectivity after suspend/resume

After a long and winding hunt through Launchpad bug reports and unhelpful forum threads, the first step is to identify which kernel module is providing your wifi driver.

lspci -k | grep -i -E "network|wireless|wifi|wlan|802.11" -A3

#my output...
05:00.0 Network controller: Intel Corporation Ultimate N WiFi Link 5300
 Subsystem: Intel Corporation Device 1101
 Kernel driver in use: iwlwifi
 Kernel modules: iwlwifi

On my machine, it was the iwlwifi module responsible for wireless. Armed with the module name, we now need just one simple line, in one simple file:

# /etc/modprobe.d/iwl.conf
options iwlwifi bt_coex_active=0

With that simple modification AND A REBOOT - Linux sanity is restored. The wireless works perfectly (and quite quickly) after resume.

For those who don't happen to be using the iwlwifi module, it's not obvious what the modprobe.d config file naming conventions are, but in general, Googling for your kernel module name plus bt_coex_active=0 should yield something useful.

Samsung screen brightness control, out of control

After having made the switch to Ubuntu to help troubleshoot my wireless issues, I found that the screen brightness would occasionally start randomly cycling through various levels repeatedly and rapidly. To stop this behavior temporarily, I found I could jump to a tty session by pressing ALT+F2 and return to X with ALT+F7.

For a more permanent fix, a kernel upgrade was required. While, Ubuntu 11.10's stock 3.0.x kernel did not include the fixed "samsung_laptop" modules, the 3.2.x kernel did. Fortunately, with the upcoming 12.04 release, the 3.2.x kernel is actively being packaged for Ubuntu and can be downloaded from kernel.ubuntu.com. More specifically, you can browse the available kernel packages and choose your release of choice. Because I knew 3.2.x was going to be supported for the 12.04 release and I was looking for stability and didn't see any must-have features in the 3.3.x kernel, I went with these downloads and installations:

# 64 bit installation
su
cd ~
mkdir src
wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.14-precise/linux-headers-3.2.14-030214_3.2.14-030214.201204021356_all.deb http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.14-precise/linux-headers-3.2.14-030214-generic_3.2.14-030214.201204021356_amd64.deb http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.14-precise/linux-image-3.2.14-030214-generic_3.2.14-030214.201204021356_amd64.deb
dpkg -i linux-headers-3.2.14-030214_3.2.14-030214.201204021356_all.deb linux-headers-3.2.14-030214-generic_3.2.14-030214.201204021356_amd64.deb linux-image-3.2.14-030214-generic_3.2.14-030214.201204021356_amd64.deb

reboot

After updating the kernel the screen brightness worked flawlessly and updating the kernel this way seemed more reasonable than compiling. However, I did notice that Update Manager wanted to reinstall the stock kernel. Running these lines as root will prevent that:

echo linux-generic hold | dpkg --set-selections
echo linux-headers-generic hold | dpkg --set-selections
echo linux-image-generic hold | dpkg --set-selections

Choosing a Distro and Desktop Shell

While I had originally planned to use Fedora 16 XCFE spin, I find myself wanting to stay with Ubuntu and Unity. The polish in the interface may be worth a bit of lag, and it seems it's easy enough to switch shells at login with the little gear icon next to your username. You can easily add additional shells and experiment with what works for you.

Unity2D works reasonably well, despite a bit of delay. Moreover, adding lenses to Dash seems like it will make the interface even more usable. I guess I did want some of the bells and whistles, but not always, especially when I need to save battery life. But isn't this kind of flexibility the point of Linux and Free software in general?

Closing thoughts: MacBook Air alternative?

Several times over the last few days of banging my head against the wall, I found myself thinking about Apple's MacBook Air. It's hard to argue the hardware is world class and OS X can be made to do most things a Linux distro can, while providing rock-solid power and device management with a slick UI. However, I just couldn't bring myself to shell out $999 upfront, with the obligatory because-you're-on-your-own-without AppleCare ($249), plus all the little Apple lock-in-gotchas that I knew were down the road.

Instead, I paid $315 for hardware that I can actually modify with additional batteries ($75/each/4hrs life) and a SSD upgrade ($85 for 40 GB). I guess the bottom line is, if you're a developer, or a system administrator, it may seem like the Apple product is a way to avoid the pain of running Linux on something other than a server. However, by deciding to fight the good fight, I've saved $773 (after planned upgrades!) but spent 20 hours of time, which means I paid myself about $40/hr to learn a few things. That's a great rate of return and now that the pain is over, it all seems worth it.