End Point

News

Welcome to End Point's blog

Ongoing observations by End Point people.

Working with Annotator: Part 2

A while back, I wrote about my work on Ruby on Rails based H2O with Annotator, an open source JavaScript library that provides annotation functionality. In that article, I discussed the history of my work with annotations, specifically touching on several iterations with native JavaScript functionality developed over several years to handle colored highlight overlapping on selected text.

Finally, last July we completed the transition to Annotator from custom JavaScript development, with the caveat that the application had quite a bit of customization hooked into Annotator's easily extensible library. But, just a few months after that, I revisited the customization, described in this post.

Separation of UI Concerns

In our initial work with Annotator, we had extended it to offer a few additional features:

  • Add multiple tags with colored highlights, where content can be tagged on the fly with a color chosen from a set of predefined colors assigned to the tag. Text would then be highlighted with opacity, and colors combined (using xColor) on overlapping highlights.
  • Interactive functionality to hide and show un-annotated text, as well as hide and show annotated text with specific tags.
  • Ability to link annotated text to other pieces of content.

But you know the paradox of choice? The extended Annotator, with so many additional features, was offering too many choices in a cluttered user interface, where the choices were likely not used in combination. See:

Too many annotation options: Should it be hidden? Should it be tagged?
Should it be tagged with more than one tag? Can text be tagged and hidden?

So, I separated concerns here (a common term in software) to intentionally separate annotation features. Once a user selects text, a popup is shown to move forward with adding a comment, adding highlights (with or without a tag name), adding a link, or hiding that text:


The new interface, where a user chooses the type of annotation they are saving, and only relevant fields are then visible.


After the user clicks on the highlight option, only highlight related fields are shown.

Annotator API

The functionality required to intercept and override Annotator's default behavior and required core overrides, but the API has a few nice hooks that were leveraged to accommodate this functionality in the H2O plugin:

  • annotationsLoaded: called after annotation data loaded
  • annotationsEditorSubmit: called after user saves annotation, before data sent to server
  • annotationCreated: after annotation created
  • annotationUpdated: after annotation updated
  • annotationDeleted: after annotation deleted

While these hooks don't mean much to someone who hasn't worked with Annotator, the point is that there are several ways to extend Annotator throughout the CRUD (create, read, update, destroy) actions on an annotation.

Custom Data

On the backend, the four types of annotation are contained in a single table, as was the data model prior to this work. There are several additional data fields to indicate the type of annotation:

  • hidden (boolean): If true, text is not visible.
  • link (text): Annotation can link to any other URL.
  • highlight (color only, no tag): Annotation can be assigned a single colored highlight.
  • highlight + tag (separate table, using various Rails plugins (acts_as_taggable_on)): Annotation can be assigned a single tag with a corresponding colored highlight.

Conclusion

The changes here resulted in a less cluttered, more clear interface. To a user, each annotation has a single concern, while utilizing Annotator and saving to a single table.

SCaLE 13x

SCaLE Penguin

I recently went to the Southern California Linux Expo (SCaLE). It takes place in Los Angeles at the Hilton, and is four days of talks, classes, and more, all focusing around Linux. SCaLE is the largest volunteer run open source conference. The volunteers put a lot of work into the conference, from the nearly flawless wireless network to the AV team making it as easy as plugging in a computer to start a presentation.


One large focus of the conference was the growing DevOps community in the Linux world. The more DevOps related talks drew the biggest crowds, and there was even a DevOps focused room on Friday. There are a wide range of DevOps related topics but the two that seemed to draw the largest crowds were configuration management and containerization. I decided to attend a full day talk on Chef (a configuration management solution) and Docker (the new rage in containerization).


The Thursday Chef talk was so full that they decided to do an extra session on Sunday. The talk was more of an interactive tutorial than a lecture, so everyone was provided with an AWS instance to use as their Chef playground. The talk started with the basics of creating a file, installing a package, and running a service. It was all very interactive; there would be a couple of slides explaining a feature and then there was time provided to try it out. During the talk there was a comment from someone about a possible bug in Chef, concerning the suid bit being reset after a change of owner or group to a file. The presenter, who works for the company that creates Chef, wasn't sure what would happen and said, "Try it out." I did try it out, and there was a bug in Chef. The presenter suggested I file an issue on github, so I did and I even wrote a patch and made a pull request later on that weekend.


Containers were the other hot topic that weekend, with the half day class on Friday, and a few other talks throughout the weekend. The Docker Talk was also set up in a learn by doing style. We learned the basics of downloading and running Docker images from the Docker Hub through the command line. We added our own tweaks to the tops of those images and created new images of our own. The speaker, Jerome Petazzoni, usually gives a two or three day class on the subject, so he picked the parts he thought most interesting to share with us. I really enjoyed making a Docker File which describes the creation of a new machine from a base image. I also thought one of the use cases described for Docker to be very interesting, creating a development environment for employees at a company. There is usually some time wasted moving things from machine to machine, whether upgrading a personal machine or transferring a project from one employee to another, especially when they are using different operating systems. Docker can help to create a unified state for all development machines in a company to the point where setting a new employee up with a workspace can be accomplished in a matter of minutes. This also helps to bring the development environment closer to the production environment.


One sentiment I heard reiterated in multiple DevOps talks was the treatment of servers as Pets vs. Cattle. Previously servers were treated as pets. We gave servers names, we knew what they liked and didn't like, when they got sick we'd nurse them back to health. This kind of treatment for servers is time consuming and not manageable at the scale that many companies face. The new trend is to treat servers like cattle. Each server is given a number, they do their job, and if they get sick they are "put down". Tools like Docker and Chef make this possible, servers can be set up so quickly that there's no reason to nurse them back to health anymore. This is great for large companies that need to manage thousands of servers, but it can save time for smaller companies as well.

Temporary tables in SQL query optimization

SQL queries can get complex in a big hurry. If you are querying multiple tables, and in particular if your query involves operations like UNION and INTERSECT, then you can find yourself in a big, messy pile of SQL. Worse, that big pile will often run slowly, sometimes to the point where a web application times out!

I won't inflict a real-world example on you; that would be cruel. So let's look at a "toy" problem, keeping in mind that this won't illustrate any time-savings, just the technique involved.

Here's the original SQL:

SELECT p.* FROM products p
JOIN (SELECT * FROM inventory WHERE /* complex clause here */) i USING (sku)
UNION ALL
SELECT p.* FROM clearance_products p
JOIN (SELECT * FROM inventory WHERE /* complex clause here */) i USING (sku)

Bonus hint: using "UNION ALL" instead of just "UNION" will allow the query processor to skip an unnecessary step here. "UNION ALL" says you know the rows on either side of the clause are unique. "UNION" means the results will be post-processed to remove duplicates. This might save you more than a smidgen of time, depending on how large the two sub-queries get.

Now, many times the query optimizer will just do the right thing here. But sometimes (cough, cough-MySQL), your database isn't quite up to the task. So you have to shoulder the burden and help out. That's where we can apply a temporary table.

Temporary tables are created for the length of the database session; that's different than a transaction. For a web application, that's usually (not always) the length of the request (i.e., from the time your web application opens a database connection, until it explicitly closes it, or until it returns control to the web server, usually by passing it a completed page). For a script, it's a similar duration, e.g. until the script exits.

CREATE TEMPORARY TABLE cross_inventory AS
SELECT * FROM inventory WHERE /* complex clause here */;

CREATE INDEX cross_inv_sku ON cross_inventory(sku);
There's no significant difference for our purposes between a "permanent" and a "temporary" table. However, you do have to keep in mind that these tables are created without indexes, so if your goal is to improve the speed of queries involving the data here, adding an index after creating the table is usually desirable.

With all this in place, now we can:

SELECT p.* FROM products p
JOIN cross_inventory i USING (sku)
UNION
SELECT p.* FROM clearance_products p
JOIN cross_inventory i USING (sku)

Sometimes your temporary table will be built up not by a straightforward "CREATE ... AS SELECT ...", but by your application:

CREATE TEMPORARY TABLE tmp_inventory AS SELECT * FROM inventory WHERE false;
CREATE INDEX tmp_inv_sku ON tmp_inventory(sku);

And then within the application:

# Pseudocode
while (more_data) {
  row = build_inv_record(more_data);
  sql_do('INSERT INTO tmp_inventory VALUES (?,?,...)', row);
}

Here, we are creating an empty "inventory" table template as a temporary table ("SELECT * FROM inventory WHERE false"), then adding rows to it from the application, and finally running our query. Note that in a practical application of this, it's not likely to be a lot faster, because the individual INSERT statements will take time. But this approach may have some utility where the existing "inventory" table doesn't have the data we want to JOIN against, or has the data, but not in a way we can easily filter.

I've used temporary tables (in a MySQL/Interchange/Perl environment) to speed up a query by a factor of two or more. It's usually in those cases where you have a complex JOIN that appears in two or more parts of the query (again, usually a UNION). I've even had big-win situations where the same temporary table was used in two different queries during the same session.

A similar approach is the Common Table Expression (CTE) found in PostgreSQL starting with version 8.4. This allows you to identify the rows you would be pouring into your temporary table as a named result-set, then reference it in your query. Our "toy" example would become:

WITH cross_inventory AS
(SELECT * FROM inventory WHERE /* complex clause here */)
SELECT p.* FROM products p
JOIN cross_inventory i USING (sku)
UNION
SELECT p.* FROM clearance_products p
JOIN cross_inventory i USING (sku)

I've not had an opportunity to use CTEs yet, and of course they aren't available in MySQL, so the temporary-table technique will still have a lot of value for me in the foreseeable future.

Postgres ON_ERROR_ROLLBACK explained

Way back in 2005 I added the ON_ERROR_ROLLBACK feature to psql, the Postgres command line client. When enabled, any errors cause an immediate rollback to just before the previous command. What this means is that you can stay inside your transaction, even if you make a typo (the main error-causing problem and the reason I wrote it!). Since I sometimes see people wanting to emulate this feature in their application or driver, I thought I would explain exactly how it works in psql.

First, it must be understood that this is not a Postgres feature, and there is no way you can instruct Postgres itself to ignore errors inside of a transaction. The work must be done by a client (such as psql) that can do some voodoo behind the scenes. The ON_ERROR_ROLLBACK feature is available since psql version 8.1.

Normally, any error you make will throw an exception and cause your current transaction to be marked as aborted. This is sane and expected behavior, but it can be very, very annoying if it happens when you are in the middle of a large transaction and mistype something! At that point, the only thing you can do is rollback the transaction and lose all of your work. For example:

greg=# CREATE TABLE somi(fav_song TEXT, passphrase TEXT, avatar TEXT);
CREATE TABLE
greg=# begin;
BEGIN
greg=# INSERT INTO somi VALUES ('The Perfect Partner', 'ZrgRQaa9ZsUHa', 'Andrastea');
INSERT 0 1
greg=# INSERT INTO somi VALUES ('Holding Out For a Hero', 'dx8yGUbsfaely', 'Janus');
INSERT 0 1
greg=# INSERT INTO somi BALUES ('Three Little Birds', '2pX9V8AKJRzy', 'Charon');
ERROR:  syntax error at or near "BALUES"
LINE 1: INSERT INTO somi BALUES ('Three Little Birds', '2pX9V8AKJRzy'...
greg=# INSERT INTO somi VALUES ('Three Little Birds', '2pX9V8AKJRzy', 'Charon');
ERROR:  current transaction is aborted, commands ignored until end of transaction block
greg=# rollback;
ROLLBACK
greg=# select count(*) from somi;
 count
-------
     0

When ON_ERROR_ROLLBACK is enabled, psql will issue a SAVEPOINT before every command you send to Postgres. If an error is detected, it will then issue a ROLLBACK TO the previous savepoint, which basically rewinds history to the point in time just before you issued the command. Which then gives you a chance to re-enter the command without the mistake. If an error was not detected, psql does a RELEASE savepoint behind the scenes, as there is no longer any reason to keep the savepoint around. So our example above becomes:

greg=# \set ON_ERROR_ROLLBACK interactive
greg=# begin;
BEGIN
greg=# INSERT INTO somi VALUES ('Volcano', 'jA0EBAMCV4e+-^', 'Phobos');
INSERT 0 1
greg=# INSERT INTO somi VALUES ('Son of a Son of a Sailor', 'H0qHJ3kMoVR7e', 'Proteus');
INSERT 0 1
greg=# INSERT INTO somi BALUES ('Xanadu', 'KaK/uxtgyT1ni', 'Metis');
ERROR:  syntax error at or near "BALUES"
LINE 1: INSERT INTO somi BALUES ('Xanadu', 'KaK/uxtgyT1ni'...
greg=# INSERT INTO somi VALUES ('Xanadu', 'KaK/uxtgyT1ni', 'Metis');
INSERT 0 1
greg=# commit;
COMMIT
greg=# select count(*) from somi;
 count
-------
     3

What about if you create a savepoint yourself? Or even a savepoint with the same name as the one that psql uses internally? Not a problem - Postgres allows multiple savepoints with the same name, and will rollback or release the latest one created, which allows ON_ERROR_ROLLBACK to work seamlessly with user-provided savepoints.

Note that the example above sets ON_ERROR_ROLLBACK (yes it is case sensitive!) to 'interactive', not just 'on'. This is a good idea, as you generally want it to catch human errors, and not just plow through a SQL script.

So, if you want to add this to your own application, you will need to wrap each command in a hidden savepoint, and then rollback or release it. The end-user should not see the SAVEPOINT, ROLLBACK TO, or RELEASE commands. Thus, the SQL sent to the backend will change from this:

BEGIN; ## entered by the user
INSERT INTO somi VALUES ('Mr. Roboto', 'H0qHJ3kMoVR7e', 'Triton');
INSERT INTO somi VALUES ('A Mountain We Will Climb', 'O2DMZfqnfj8Tle', 'Tethys');
INSERT INTO somi BALUES ('Samba de Janeiro', 'W2rQpGU0MfIrm', 'Dione');

to this:

BEGIN; ## entered by the user
SAVEPOINT myapp_temporary_savepoint ## entered by the application
INSERT INTO somi VALUES ('Mr. Roboto', 'H0qHJ3kMoVR7e', 'Triton');
RELEASE myapp_temporary_savepoint

SAVEPOINT myapp_temporary_savepoint
INSERT INTO somi VALUES ('A Mountain We Will Climb', 'O2DMZfqnfj8Tle', 'Tethys');
RELEASE myapp_temporary_savepoint

SAVEPOINT myapp_temporary_savepoint
INSERT INTO somi BALUES ('Samba de Janeiro', 'W2rQpGU0MfIrm', 'Dione');
ROLLBACK TO myapp_temporary_savepoint

Here is some pseudo-code illustrating the sequence of events. To see the actual implementation in psql, take a look at bin/psql/common.c


run("SAVEPOINT myapp_temporary_savepoint");
run($usercommand);
if (txn_status == ERROR) {
  run("ROLLBACK TO myapp_temporary_savepoint");
}
if (command was "savepoint" or "release" or "rollback") {
  ## do nothing
}
elsif (txn_status == IN_TRANSACTION) {
  run("RELEASE myapp_temporary_savepoint");
}

While there is there some overhead in constantly creating and tearing down so many savepoints, it is quite small, especially if you are using it in an interactive session. This ability to automatically roll things back is especially powerful when you remember that Postgres can roll everything back, including DDL (e.g. CREATE TABLE). Certain other expensive database systems do not play well when mixing DDL and transactions.

Testing your chef repo pull requests with chef-zero, Vagrant and Jenkins

All Liquid Galaxy setups deployed by End Point are managed by Chef. Typical deployment consists of approx 3 to 9 Linux boxes from which only 1 is managed and the rest is an ISO booted from this machine via network with copy-on-write root filesystem. Because of this, typical deployment involves more steps than just updating your code and restarting application. Deployment + rollback may be even 10 times longer compared with typical web application. Due to this fact, we need to test our infrastructure extensively.

What are we to do in order to make sure that our infrastructure is tested well before it hits production?


Scary? - It's not.


Workflow broken down by pieces



  • lg_chef.git repo - where we keep cookbooks, environments and node definitions
  • GitHub pull request - artifact of infrastructure source code tested by Jenkins
  • Vagrant - virtual environment in which chef is run in order to test the artifact. There's always 1 master node and few Vagrant boxes that boot an ISO from master via tftp protocol
  • chef-zero - Chef flavor used to converge and test the infrastructure on the basis of GitHub pull request
  • chef-server/chef-client - Chef flavor used to converge and test production and pre-production environment
  • Jenkins - Continuous Integration environment that runs the converge process and part of the tests
  • Tests - two frameworks used - BATS (for the integration tests on the top) and minitest (for after-converge tests)
  • lg-live-build - our fork of Debian live build used to build the ISO that is booted by Vagrant slaves

Workflow broken down by the order of actions

  1. User submits GitHub pull request to lg_chef.git repo
  2. GitHub pull request gets picked up by Jenkins
  3. Jenkins creates 1 master Vagrant node and several slave nodes to act as slaves
  4. chef-zero converges master Vagrant box and runs minitest
  5. BATS tests run on the freshly converged Vagrant master box. Few steps are performed here: ISO is built, it's distributed to the slaves, slaves boot the ISO and final integration tests are run to see whether slaves have all the goodness.
  6. If points 1 to 5 are green, developer merges the changes, uploads the updated cookbooks, node definitions, roles and environments and runs the final tests.

What didn't work for us and why

  • kitchen-vagrant  - because it didn't play well with Jenkins (or JVM itself) and didn't know how to use advanced Vagrant features for specifying multiple networking options, interfaces and drivers. However it supports using your own Vagrantfile.erb
  • We've had some doubts about keeping all the cookbooks, environments and node definitions in one repo because chef-server/chef-client tests can only test your stuff if it's uploaded to the Chef server, but chef-zero came in handy

The code

As previously mentioned, we needed our own vagrant template file.

Vagrant.configure("2") do |config|
  <% if @data[:chef_type] == "chef_zero" %>
  config.chef_zero.enabled = true
  config.chef_zero.chef_server_url = "<%= @data[:chef_server_url] %>"
  config.chef_zero.roles = "../../roles/"
  config.chef_zero.cookbooks = "../../cookbooks/"
  config.chef_zero.environments = "../../environments/"
  config.chef_zero.data_bags = "../integration/default/data_bags/"
  <% else %>
  config.chef_zero.enabled = false
  <% end %>
  config.omnibus.chef_version = "<%= @data[:chef_version] %>"
  config.vm.define "<%= @data[:headnode][:slug] %>" do |h|
  h.vm.box = "<%= @data[:headnode][:box] %>"
    h.vm.box_url = "<%= @data[:headnode][:box_url] %>"
    h.vm.hostname = "<%= @data[:headnode][:hostname] %>"
    h.vm.network(:private_network, {:ip => '10.42.41.1'})
    h.vm.synced_folder ".", "/vagrant", disabled: true
    h.vm.provider :virtualbox do |p|
      <% @data[:headnode][:customizations].each do |key, value| %>
        p.customize ["modifyvm", :id, "<%= key %>", "<%= value %>"]
      <% end %>
    end
    h.vm.provision :chef_client do |chef|
      <% if @data[:chef_type] == "chef_zero" %>
      chef.environment = "<%= @data[:headnode][:provision][:environment] %>"
      chef.run_list = <%= @data[:run_list] %>
      chef.json = <%= @data[:node_definition] %>
      chef.chef_server_url = "<%= @data[:chef_server_url] %>"
      <% else %>
      chef.chef_server_url = "<%= @data[:headnode][:provision][:chef_server_url] %>"
      <% end %>
      chef.validation_key_path = "<%= @data[:headnode][:provision][:validation_key_path] %>"
      chef.encrypted_data_bag_secret_key_path = "<%= @data[:headnode][:provision][:encrypted_data_bag_secret_key_path] %>"
      chef.verbose_logging = <%= @data[:headnode][:provision][:verbose_logging] %>
      chef.log_level = "<%= @data[:headnode][:provision][:log_level] %>"
  <% end %>
  config.omnibus.chef_version = "<%= @data[:chef_version] %>"
  config.vm.define "<%= @data[:headnode][:slug] %>" do |h|
  h.vm.box = "<%= @data[:headnode][:box] %>"
    h.vm.box_url = "<%= @data[:headnode][:box_url] %>"
    h.vm.hostname = "<%= @data[:headnode][:hostname] %>"
    h.vm.network(:private_network, {:ip => '10.42.41.1'})
    h.vm.synced_folder ".", "/vagrant", disabled: true
    h.vm.provider :virtualbox do |p|
      <% @data[:headnode][:customizations].each do |key, value| %>
        p.customize ["modifyvm", :id, "<%= key %>", "<%= value %>"]
      <% end %>
    end
    h.vm.provision :chef_client do |chef|
      <% if @data[:chef_type] == "chef_zero" %>
      chef.environment = "<%= @data[:headnode][:provision][:environment] %>"
      chef.run_list = <%= @data[:run_list] %>
      chef.json = <%= @data[:node_definition] %>
      chef.chef_server_url = "<%= @data[:chef_server_url] %>"
      <% else %>
      chef.chef_server_url = "<%= @data[:headnode][:provision][:chef_server_url] %>"
      <% end %>
      chef.validation_key_path = "<%= @data[:headnode][:provision][:validation_key_path] %>"
      chef.encrypted_data_bag_secret_key_path = "<%= @data[:headnode][:provision][:encrypted_data_bag_secret_key_path] %>"
      chef.verbose_logging = <%= @data[:headnode][:provision][:verbose_logging] %>
      chef.log_level = "<%= @data[:headnode][:provision][:log_level] %>"
      chef.node_name = "<%= @data[:headnode][:provision][:node_name] %>"
    end
  end

  #display nodes
  <% @data[:display_nodes][:nodes].each do |dn| %>
  config.vm.define "<%= dn[:slug] %>" do |dn_config|
    dn_config.vm.box_url = "<%= @data[:display_nodes][:global][:box_url] %>"
    dn_config.vm.hostname = "<%= dn[:hostname] %>"
    dn_config.vm.box = "<%= @data[:display_nodes][:global][:box] %>"
    dn_config.vm.synced_folder ".", "/vagrant", disabled: true
    dn_config.vm.boot_timeout = 1
    dn_config.vm.provider :virtualbox do |p|
    <% @data[:display_nodes][:global][:customizations].each do |key, value| %>
      p.customize ["modifyvm", :id, "<%= key %>", "<%= value %>"]
    <% end %>
      p.customize ["modifyvm", :id, "--macaddress1", "<%= dn[:mac] %>"]
      p.customize ["createhd", "--filename", "../files/<%= dn[:slug] %>.vmdk", "--size", 80*1024]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "none"]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "../files/<%= dn[:slug] %>.vmdk"]
      p.customize ["storagectl", :id, "--name", "SATA Controller", "--add", "sata",  "--controller", "IntelAHCI", "--hostiocache", "on"]
      p.customize ["storageattach", :id, "--storagectl", "SATA Controller", "--port", 1, "--device", 0, "--type", "hdd", "--medium", "../files/ipxe_<%= dn[:slug] %>.vmdk"] 
    end
  end
  <% end %>
end

It renders a Vagrant File out of following data:

{
  "description" : "This file is used to generate Vagrantfile and run_test.sh and also run tests. It should contain _all_ data needed to render the templates and run teh tests.",
  "chef_version" : "11.12.4",
  "chef_type" : "chef_zero",
  "vagrant_template_file" : "vagrantfile.erb",
  "run_tests_template_file" : "run_tests.sh.erb",
  "chef_server_url" : "http://192.168.1.2:4000",
  "headnode" :
    {
    "slug" : "projectX-pull-requests",
    "box" : "opscode-ubuntu-14.04",
    "box_url" : "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box",
    "hostname" : "lg-head",
    "bats_tests_dir" : "projectX-pr",
    "customizations" : {
      "--memory" : "2048",
      "--cpus": "2",
      "--nic1" : "nat",
      "--nic2": "intnet",
      "--nic3": "none",
      "--nic4" : "none",
      "--nictype1": "Am79C970A",
      "--nictype2": "Am79C970A",
      "--intnet2": "projectX-pull-requests"
    },
    "provision" : {
      "chef_server_url" : "https://chefserver.ourdomain.com:40443",
      "validation_key_path" : "~/.chef/validation.pem",
      "encrypted_data_bag_secret_key_path" : "~/.chef/encrypted_data_bag_secret",
      "node_name" : "lg-head-projectXtest.liquid.glx",
      "environment" : "pull_requests",
      "verbose_logging" : true,
      "log_level" : "info"
    }
  },
  "display_nodes" : {
    "global" : {
      "box" : "opscode-ubuntu-14.04",
      "box_url" : "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box",
      "customizations" : {
        "--memory" : "2048",
        "--cpus" : "1",
        "--boot1" : "floppy",
        "--boot2" : "net",
        "--boot3" : "none",
        "--boot4" : "none"
    },
    "provision" : {
      "chef_server_url" : "https://chefserver.ourdomain.com:40443",
      "validation_key_path" : "~/.chef/validation.pem",
      "encrypted_data_bag_secret_key_path" : "~/.chef/encrypted_data_bag_secret",
      "node_name" : "lg-head-projectXtest.liquid.glx",
      "environment" : "pull_requests",
      "verbose_logging" : true,
      "log_level" : "info"
    }
  },
  "display_nodes" : {
    "global" : {
      "box" : "opscode-ubuntu-14.04",
      "box_url" : "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box",
      "customizations" : {
        "--memory" : "2048",
        "--cpus" : "1",
        "--boot1" : "floppy",
        "--boot2" : "net",
        "--boot3" : "none",
        "--boot4" : "none",
        "--intnet1" : "projectX-pull-requests",
        "--nicpromisc1": "allow-all",
        "--nic1" : "intnet",
        "--nic2": "none",
        "--nic3": "none",
        "--nic4": "none",
        "--nictype1": "Am79C970A",
        "--ioapic": "on"
      }
    },
    "nodes" : [
      {
      "slug" : "projectX-pull-requests-kiosk",
      "hostname" : "kiosk",
      "mac" : "5ca1ab1e0001"
    },
    {
      "slug" : "projectX-pull-requests-display",
      "hostname" : "display",
      "mac" : "5ca1ab1e0002"
    }
    ]
  }
}

As a result we get an on-the-fly Vagrantfile that's used during the testing:

Vagrant.configure("2") do |config|

  config.chef_zero.enabled = true
  config.chef_zero.chef_server_url = "http://192.168.1.2:4000"
  config.chef_zero.roles = "../../roles/"
  config.chef_zero.cookbooks = "../../cookbooks/"
  config.chef_zero.environments = "../../environments/"
  config.chef_zero.data_bags = "../integration/default/data_bags/"

  config.omnibus.chef_version = "11.12.4"
  config.vm.define "projectX-pull-requests" do |h|
  h.vm.box = "opscode-ubuntu-14.04"
    h.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box"
    h.vm.hostname = "lg-head"
    h.vm.network(:private_network, {:ip => '10.42.41.1'})
    h.vm.synced_folder ".", "/vagrant", disabled: true
    h.vm.provider :virtualbox do |p|
        p.customize ["modifyvm", :id, "--memory", "2048"]
        p.customize ["modifyvm", :id, "--cpus", "2"]
        p.customize ["modifyvm", :id, "--nic1", "nat"]
        p.customize ["modifyvm", :id, "--nic2", "intnet"]
        p.customize ["modifyvm", :id, "--nic3", "none"]
        p.customize ["modifyvm", :id, "--nic4", "none"]
        p.customize ["modifyvm", :id, "--nictype1", "Am79C970A"]
        p.customize ["modifyvm", :id, "--nictype2", "Am79C970A"]
        p.customize ["modifyvm", :id, "--intnet2", "projectX-pull-requests"]
    end
    h.vm.provision :chef_client do |chef|

      chef.environment = "pull_requests"
      chef.run_list = ["role[lg-head-nocms]", "recipe[lg_live_build]", "recipe[lg_tftproot]", "recipe[lg_projectX]", "recipe[lg_test]", "recipe[test_mode::bats]", "recipe[hostsfile::projectX]"]
      chef.json = {:sysctl=>{:params=>{:vm=>{:swappiness=>20}, :net=>{:ipv4=>{:ip_forward=>1}}}}, :test_mode=>true, :call_ep=>{:keyname=>"ProjectX CI Production", :fwdport=>33299}, :tftproot=>{:managed=>true}, :tags=>[], :lg_cms=>{:remote=>"github"}, :monitor=>true, :lg_grub=>{:cmdline=>"nomodeset biosdevname=0"}, :projectX=>{:repo_branch=>"development", :display_host=>"42-a", :kiosk_host=>"42-b", :sensors_host=>"42-b", :maps_url=>"https://www.google.com/maps/@8.135687,-75.0973243,17856994a,40.4y,1.23h/data=!3m1!1e3?esrch=Tactile::TactileAcme,Tactile::ImmersiveModeEnabled"}, :liquid_galaxy=>{:touchscreen_link=>"/dev/input/lg_active_touch", :screenshotd=>{:screen_rows=>"1", :screen_columns=>"1"}, :screenshot_service=>true, :display_nodes=>[{:hostname=>"42-c"}, {:allowed_pages=>["Google Maps", "Pacman Doodle", "jellyfish", "Doodle Selection", "ProjectX Video Player", "Composer kiosk"], :hostname=>"42-b", :mac=>"5c:a1:ab:1e:00:02", :features=>"mandatory_windows, plain_gray, starry_skies", :bad_windows_names=>"Google Earth - Login Status", :mandatory_windows_names=>"awesome", :screens=>[{:display=>":0", :crtc=>"default", :grid_order=>"0"}], :screen_rotation=>"normal", :audio_device=>"{type hw; card DGX; device 0}", :onboard_enable=>true, :keyboard_enable=>true, :mouse_enable=>true, :cursor_enable=>true, :background_extension=>"jpg", :background_mode=>"zoom-fill", :projectX=>{:extensions=>{:kiosk=>"ProjectX Kiosk", :google_properties_menu=>"Google Properties Menu", :onboard=>"Onboard", :no_right_click=>"Right Click Killer", :render_statistics=>"Render Statistics"}, :browser_slug=>"lgS0", :urls=>"https://www.google.com/maps", :ros_nodes=>[{:name=>"rfreceiver_reset", :pkg=>"rfreceiver", :type=>"kill_browser.py"}, {:name=>"proximity", :pkg=>"maxbotix", :type=>"sender.py"}, {:name=>"spacenav", :pkg=>"spacenav_node", :type=>"spacenav_node"}, {:name=>"leap", :pkg=>"leap_motion", :type=>"sender.py"}, {:name=>"projectX_nav", :pkg=>"projectX_nav", :type=>"projectX_nav"}, {:name=>"onboard", :pkg=>"onboard", :type=>"listener.py"}, {:name=>"rosbridge", :pkg=>"rosbridge_server", :type=>"rosbridge_websocket", :params=>[{:name=>"certfile", :value=>"/home/lg/etc/ros.crt"}, {:name=>"keyfile", :value=>"/home/lg/etc/ros.key"}]}]}, :browser_infinite_url=>"http://lg-head/projectX-loader.html"}, {:hostname=>"42-a", :mac=>"5c:a1:ab:1e:00:01", :features=>"mandatory_windows, plain_gray, starry_skies, erroneous_text", :bad_windows_names=>"Google Earth - Login Status", :mandatory_windows_names=>"awesome", :screens=>[{:display=>":0", :crtc=>"default", :grid_order=>"1"}], :keyboard_enable=>true, :mouse_enable=>true, :cursor_enable=>true, :background_extension=>"jpg", :background_mode=>"zoom-fill", :nvidia_mosaic=>true, :manual_layout=>{:default=>"1024x768+0+0"}, :projectX=>{:extensions=>{:display=>"ProjectX Large Display", :pacman=>"pacman", :render_statistics=>"Render Statistics"}, :browser_slug=>"lgS0", :urls=>"https://www.google.com/maps", :ros_nodes=>[{:name=>"geodata", :pkg=>"geodata", :type=>"geodata_server.py"}]}, :browser_infinite_url=>"http://lg-head/projectX-loader.html", :default_browser_bin=>"google-chrome", :allowed_pages=>["Google Maps", "Pacman Doodle", "jellyfish", "ProjectX Video Player", "Composer wall"]}], :has_cec=>false, :google_office=>false, :viewsync_master=>"42-b", :has_touchscreen=>false, :has_spacenav=>true, :support_name=>"projectX-ci", :podium_interface=>"http://lg-head", :podium_display=>"42-b:default"}}
      chef.chef_server_url = "http://192.168.1.2:4000"

      chef.validation_key_path = "~/.chef/validation.pem"
      chef.encrypted_data_bag_secret_key_path = "~/.chef/encrypted_data_bag_secret"
      chef.verbose_logging = true
      chef.log_level = "info"
      chef.node_name = "lg-head-projectXtest.liquid.glx"
    end
  end

  #display nodes

  config.vm.define "projectX-pull-requests-kiosk" do |dn_config|
    dn_config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box"
    dn_config.vm.hostname = "kiosk"
    dn_config.vm.box = "opscode-ubuntu-14.04"
    dn_config.vm.synced_folder ".", "/vagrant", disabled: true
    dn_config.vm.boot_timeout = 1
    dn_config.vm.provider :virtualbox do |p|
      p.customize ["modifyvm", :id, "--memory", "2048"]
      p.customize ["modifyvm", :id, "--cpus", "1"]
      p.customize ["modifyvm", :id, "--boot1", "floppy"]
      p.customize ["modifyvm", :id, "--boot2", "net"]
      p.customize ["modifyvm", :id, "--boot3", "none"]
      p.customize ["modifyvm", :id, "--boot4", "none"]
      p.customize ["modifyvm", :id, "--intnet1", "projectX-pull-requests"]
      p.customize ["modifyvm", :id, "--nicpromisc1", "allow-all"]
      p.customize ["modifyvm", :id, "--nic1", "intnet"]
      p.customize ["modifyvm", :id, "--nic2", "none"]
      p.customize ["modifyvm", :id, "--nic3", "none"]
      p.customize ["modifyvm", :id, "--nic4", "none"]
      p.customize ["modifyvm", :id, "--nictype1", "Am79C970A"]
      p.customize ["modifyvm", :id, "--ioapic", "on"]
      p.customize ["modifyvm", :id, "--macaddress1", "5ca1ab1e0001"]
      p.customize ["createhd", "--filename", "../files/projectX-pull-requests-kiosk.vmdk", "--size", 80*1024]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "none"]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "../files/projectX-pull-requests-kiosk.vmdk"]
      p.customize ["storagectl", :id, "--name", "SATA Controller", "--add", "sata",  "--controller", "IntelAHCI", "--hostiocache", "on"]
      p.customize ["storageattach", :id, "--storagectl", "SATA Controller", "--port", 1, "--device", 0, "--type", "hdd", "--medium", "../files/ipxe_projectX-pull-requests-kiosk.vmdk"]
    end
  end

  config.vm.define "projectX-pull-requests-display" do |dn_config|
    dn_config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box"
    dn_config.vm.hostname = "display"
    dn_config.vm.box = "opscode-ubuntu-14.04"
    dn_config.vm.synced_folder ".", "/vagrant", disabled: true
    dn_config.vm.boot_timeout = 1
    dn_config.vm.provider :virtualbox do |p|
      p.customize ["modifyvm", :id, "--memory", "2048"]
      p.customize ["modifyvm", :id, "--cpus", "1"]
      p.customize ["modifyvm", :id, "--boot1", "floppy"]
      p.customize ["modifyvm", :id, "--boot2", "net"]
      p.customize ["modifyvm", :id, "--boot3", "none"]
      p.customize ["modifyvm", :id, "--boot4", "none"]
      p.customize ["modifyvm", :id, "--intnet1", "projectX-pull-requests"]
      p.customize ["modifyvm", :id, "--nicpromisc1", "allow-all"]
      p.customize ["modifyvm", :id, "--nic1", "intnet"]
      p.customize ["modifyvm", :id, "--nic2", "none"]
      p.customize ["modifyvm", :id, "--nic3", "none"]
      p.customize ["modifyvm", :id, "--nic4", "none"]
      p.customize ["modifyvm", :id, "--nictype1", "Am79C970A"]
      p.customize ["modifyvm", :id, "--ioapic", "on"]
      p.customize ["modifyvm", :id, "--macaddress1", "5ca1ab1e0002"]
      p.customize ["createhd", "--filename", "../files/projectX-pull-requests-display.vmdk", "--size", 80*1024]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "none"]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "../files/projectX-pull-requests-display.vmdk"]
      p.customize ["storagectl", :id, "--name", "SATA Controller", "--add", "sata",  "--controller", "IntelAHCI", "--hostiocache", "on"]
      p.customize ["storageattach", :id, "--storagectl", "SATA Controller", "--port", 1, "--device", 0, "--type", "hdd", "--medium", "../files/ipxe_projectX-pull-requests-display.vmdk"]
    end
  end

end

Finally we have the environment stored in one Vagrantfile. The missing part is to how to run tests on it.
The testing script does the following:

#/bin/bash
set -e
# FUNCTIONS

function halt_vm () {
  vms=`vboxmanage list vms | grep "vagrant_$1_" | awk {'print $1'} | sed s/'"'//g`
  echo "Stopping VM $vms"
  stop_result=$(for vm in $vms ; do vboxmanage controlvm $vm poweroff; echo $?; done)
  echo "Output of stopping VM $1 : $stop_result"
}

function boot_vm () {
  vms=`vboxmanage list vms | grep "vagrant_$1_" | awk {'print $1'} | sed s/'"'//g`
  echo "Booting VM $vms"
  start_result=$(for vm in $vms ; do vboxmanage startvm $vm --type headless; echo $?; done)
  echo "Output of booting VM $1 : $start_result"
  echo "Sleeping additional 15 secs after peacefull boot"
  sleep 15
}

function add_keys () {
  for i in `find /var/lib/jenkins/.ssh/id_rsa* | grep -v '.pub'` ; do ssh-add $i ; done
}

#vars

knifeclient_name=lg-head-projectXtest.liquid.glx
headnode_name=projectX-pull-requests

# TEST SCENARIO

cd test/vagrant

# teardown of previous sessions
vagrant destroy projectX-pull-requests-kiosk -f
vagrant destroy projectX-pull-requests-display -f
vagrant destroy $headnode_name -f

echo "Not managing knife client because => chef_zero "
echo "All ssh keys presented below"
ssh-add -l

# headnode
vagrant up ${headnode_name}

# displaynodes

result=$(vboxmanage convertfromraw ../files/ipxe.usb ../files/ipxe_projectX-pull-requests-kiosk.vmdk --format=VMDK ; vagrant up projectX-pull-requests-kiosk; echo $?)
echo "projectX-pull-requests-kiosk : $result"

result=$(vboxmanage convertfromraw ../files/ipxe.usb ../files/ipxe_projectX-pull-requests-display.vmdk --format=VMDK ; vagrant up projectX-pull-requests-display; echo $?)
echo "projectX-pull-requests-display : $result"



# test phase
OPTIONS=`vagrant ssh-config  ${headnode_name} | grep -v ${headnode_name} | awk -v ORS=' ' '{print "-o " $1 "=" $2}'`
scp ${OPTIONS} ../integration/projectX-pr/bats/*.bats vagrant@${headnode_name}:/tmp/bats_tests

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/pre_build_checks.bats'

halt_vm projectX-pull-requests-kiosk
halt_vm projectX-pull-requests-display

echo "Building teh ISO (it may take a long time)"
ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/build_iso.bats'

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/set_grub_to_make_partitions.bats'

echo "Booting nodes"


boot_vm projectX-pull-requests-kiosk
boot_vm projectX-pull-requests-display


echo "Sleeping 30 secs for the DNS to boot and setting the grub to boot the ISO"
sleep 30

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/set_grub_to_boot_the_iso.bats'
echo "Sleeping for 4 mins for the displaynodes to boot fresh ISO"
sleep 240

echo "Running the tests inside the headnode:"

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/post_checks.bats'
So finally we get the following pipeline:

  1. Clone Chef pull request from GitHub
  2. Create Vagrantfile on the basis of Vagrantfile template
  3. Create run_tests.sh script for running the tests
  4. Destroy all previously created Vagrant boxes
  5. Create one Chef Vagrant box
  6. Create ISO Vagrant boxes with ipxe bootloader
  7. Converge the Vagrant box with Chef
  8. Copy BATS tests onto the headnode
  9. Run initial BATS tests that build an ISO
  10. Boot display nodes with the newly created ISO
  11. Run final integration tests on the stack
Elapsed time - between 40 and 50 minutes.

Postgres pg_dump implicit cast problem patched!

One of the many reasons I love Postgres is the responsiveness of the developers. Last week I posted an article about the dangers of reinstating some implicit data type casts. Foremost among the dangers was the fact that pg_dump will not dump user-created casts in the pg_catalog schema. Tom Lane (eximious Postgres hacker) read this and fixed it up - the very same day! So in git head (which will become Postgres version 9.5 someday) we no longer need to worry about custom casts disappearing with a pg_dump and reload. These same-day fixes are not an unusual thing for the Postgres project.

For due diligence, let's make sure that the casts now survive a pg_dump and reload into a new database via psql:


psql -qc 'drop database if exists casting_test'
psql -qc 'create database casting_test'
psql casting_test -xtc 'select 123::text = 123::int'
ERROR:  operator does not exist: text = integer
LINE 1: select 123::text = 123::int
                                 ^
HINT:  No operator matches the given name and argument type(s). You might need to add explicit type casts.
psql casting_test -c 'create function pg_catalog.text(int) returns text immutable language sql as $$select textin(i\
nt4out($1))$$'
CREATE FUNCTION

psql casting_test -c 'create cast (int as text) with function pg_catalog.text(int) as implicit'
CREATE CAST

psql casting_test -xtc 'select 123::text = 123::int'
 ?column? | t

psql -qc 'drop database if exists casting_test2'
psql -qc 'create database casting_test2'
pg_dump casting_test | psql -q casting_test2
psql casting_test2 -xtc 'select 123::text = 123::int'
 ?column? | t

Yay, it works! Thanks, Tom, for commit 9feefedf9e92066fa6609d1e1e17b4892d81716f). The fix even got back-patches, which means it will appear in Postgres version 9.5, but also versions 9.4.2, 9.3.7, 9.2.11, 9.1.16, and 9.0.20. However, does this mean that pg_dump is logically complete, or are there similar dangers lurking like eels below the water in the source code for pg_dump? You will be happy to learn that I could find no other exceptions inside of src/bin/pg_dump/pg_dump.c. While there are still many place in the code where an object can be excluded, it's all done for valid and expected reasons, such as not dumping a table if the schema it is in is not being dumped as well.

Updated End Point Blog Stats and Our Services

Today, I sat down to read through a few recent End Point blog articles and was impressed at the depth of topics in recent posts (PostgreSQL, Interchange, SysAdmin, Text Editors (Vim), Dancer, AngularJS) from my coworkers. The list continues if I look further back covering technologies in both front and back end web development. And, this list doesn't even cover the topics I typically write about such as Ruby on Rails & JavaScript.

While 5 years ago, we may have said we predominately worked with ecommerce clients, our portfolio has evolved to include Liquid Galaxy clients and many non-ecommerce sites as well. With the inspiration from reading through these recent posts, I decided to share some updated stats.

Do you remember my post on Wordle from early 2011? Wordle is a free online word cloud generator. I grabbed updated text from 2013 and on from our blog, using the code included in my original post, and generated a new word cloud from End Point blog content:


End Point blog Word cloud from 2013 to present

I removed common words from the word cloud not removed from the original post, including "one", "like", etc. Compared to the original post, it looks like database related topics (e.g. PostgreSQL) still have strong representation on the blog in terms of word count, and many other common developer words. Liquid Galaxy now shows up in the word cloud (not surprising), but many of the other technology specific terms are still present (Spree, Rails, Bucardo).

I also took a look at the top 10 blog posts by page views, as compared to this post:

The page views are not normalized over time, which means older blog posts would not only have more page views, but also have more time to build up traffic from search. Again, this list demonstrates qualitatively the broad range of topics for which our blog is popular, including both very technology specific posts as well as general development topics. I also suspect our traffic continues to attract long-tail keywords, described more in this post.

Finally, back in October, I visited End Point's Tennessee office and got into a discussion with Jon about how we define our services and/or how our business breaks down into topics. Here's a rough chart of what we came up with at the time:


How do End Point services break down?

Trying to explain the broad range and depth of our services can be challenging. Here are a few additional notes related to the pie chart:

  • Our Liquid Galaxy work spans across the topics of Hardware & Hosting, Cloud Systems, and Databases.
  • Our Ecommerce services typically includes work in the topics of Backend & Client Side Development, as well as Databases.
  • Our development in mobile applications spans Backend & Client Side Development.

All in all, I'm impressed that we've continued to maintain expertise in long-standing topics such as PostgreSQL and Interchange, but also haven't shied away from learning new technologies such as GIS as related to Liquid Galaxy and JavaScript frameworks.

PS

P.S. If you are interested in generating word statistics via command line, the following will get you the top 20 words given a text file:

tr -c '[:alnum:]' '[\n*]' < some_text_file.txt | sort | uniq -c | sort -nr | head  -20