News

Welcome to End Point’s blog

Ongoing observations by End Point people

RSpec's Anything Argument Matcher Trickery

If you're like me, and I know I am, then you've used RSpec's "anything" argument matcher to test that a method is getting called with the expected arguments. If you haven't, then I strongly recommend you check it out. Here's a basic example:

expect(object).to receive(:message).with(:foo, anything)

That test will pass if object.message() is called with :foo as its first argument, and any non-nil value as its second argument. The "anything" matcher can be used anywhere in the argument list, as well as multiple times. For example:

expect(object).to receive(:message).with(anything, :bar, anything)

That test will pass if object.message() is called with a non-nil value for its first argument, :bar for its second argument, and any non-nil value for its third argument.

I recently made one of those discoveries where the happiness of making the discovery quickly turned into the sneaking suspicion that I was actually the last person on the planet to make this discovery. So, I told a co-worker about my discovery. She hadn't heard of it before, which meant at worst I was the second to last person on the planet. "There could be others," I thought. "I must set alight our grail shaped beacon!" And here we are.

I discovered that you can use the "anything" matcher as a kind of wildcard inside a data structure to test for the presence of a non-nil value there. Here's the simplified version of what I came up with:

expect(subject.targets).to eq [
  {:id => anything, :name => 'Franchise 1'},
  {:id => anything, :name => 'Franchise 2'},
  {:id => anything, :name => 'Franchise 3'},
  {:id => anything, :name => 'X-Men'},
]

My particular test doesn't care about the values that the :id hash keys reference. They're going to be standard ActiveRecord integer IDs. It really only cares about the overall structure (an array of hashes in this specific order,) that each hash has an :id tuple with a non-nil value, and that the :name tuples have the expected values.

So, that's my discovery. It passes when it should pass. It fails when it should fail. However, it still looks a little suspect to me. It feels like I may be violating the spirit of the anything method, if not the syntax or functionality. I've trying consulting Uncle Google on the topic, but the "anything" search term makes it a tough to get relevant results.

To be honest, I'm half-hoping someone out there will shoot me down and describe why this is a terrible idea, how I'm a terrible person, and how they would test this differently. What say you, dear Internet?

Using jQuery Migrate plugin

We all know these tricky situations, like introducing a new feature on top of old code, when it seems we’re about to step into a tedious swamp of workarounds and dirty hacks. Fortunately, jQuery Migrate plugin is here to make these situations easier, at least in JavaScript. So for any of you who wondered about a real-life example of using jQuery Migrate plugin I have one!

My task was to add an editable combomonster, oh sorry, combobox (even though editable comboboxes remind me of UX Frankenstein's Monster, they are still requested a lot) to a rather old website built on jQuery v1.4.2.

I downloaded the most recent jQuery UI (at that time it was v1.10.4) and a very neat editable combobox component to go with it. It was expected that it wouldn’t work out of box with the rather outdated jQuery we had. It didn't work and the page produced the following JavaScript error:

TypeError: t.cssHooks is undefined
...t(" ");f(i,function(e,i){t.cssHooks[i]={set:function(e,n){var a,o,r="";if("trans...

No problem, I grab the newer compatible jQuery v1.10.2 from the website and yield it into head in that particular page.

<% content_for :head do %>
  javascript_include_tag "jquery-1.10.2.min.js"
  javascript_include_tag "jquery-ui-1.10.4.min.js"
  javascript_include_tag "jquery.editableselect.js"
<% end %>
There’s good news and bad news. Good - the old error is gone. Bad - there's a new one:
TypeError: jQuery.browser is undefined
if (jQuery.browser.safari && document.readyState != "complete”)
function stretchbar(){
  /* the if block is for safari 4, it was disrupting the display on refresh. */
  if (jQuery.browser.safari && document.readyState != "complete")
    {
     setTimeout( arguments.callee, 100 );
     return;
   }
We have a lot of old components on that page and they fail to work with the new jQuery. Why? All versions of jQuery after v.1.9 are stripped of certain components that are not 'core' ones. jQuery offers a migration technique to restore deprecated and removed functionality using jQuery Migrate plugin so the older code could work. The plugin can be included with versions of jQuery as old as 1.6.4. However, the plugin is only required for version 1.9.0 or higher.

The section "Changes of Note in jQuery 1.9" explains the particular error I got. Finally, I downloaded jQuery Migrate plugin v1.2.1 (the most recent at the time) and put it after the script for jQuery:

<% content_for :head do %>
  javascript_include_tag "jquery-1.10.2.min.js"
  javascript_include_tag "jquery-ui-1.10.4.min.js"
  javascript_include_tag "jquery.editableselect.js"
  javascript_include_tag "jquery-migrate-1.2.1.min.js"
<% end %>

Voila! The website gets an instant design boost with this brand new, sleek grey editable combobox:


Solving pg_xlog out of disk space problem on Postgres


pg_xlog with a dummy file
(image by Andrew Malone)

Running out of disk space in the pg_xlog directory is a fairly common Postgres problem. This important directory holds the WAL (Write Ahead Log) files. (WAL files contain a record of all changes made to the database - see the link for more details). Because of the near write‑only nature of this directory, it is often put on a separate disk. Fixing the out of space error is fairly easy: I will discuss a few remedies below.

When the pg_xlog directory fills up and new files cannot be written to it, Postgres will stop running, try to automatically restart, fail to do so, and give up. The pg_xlog directory is so important that Postgres cannot function until there is enough space cleared out to start writing files again. When this problem occurs, the Postgres logs will give you a pretty clear indication of the problem. They will look similar to this:


PANIC:  could not write to file "pg_xlog/xlogtemp.559": No space left on device
STATEMENT:  insert into abc(a) select 123 from generate_series(1,12345)
LOG:  server process (PID 559) was terminated by signal 6: Aborted
DETAIL:  Failed process was running: insert into abc(a) select 123 from generate_series(1,12345)
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally an
d possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at 2014-09-16 10:36:47 EDT
LOG:  database system was not properly shut down; automatic recovery in progress
FATAL:  the database system is in recovery mode
LOG:  redo starts at 0/162FE44
LOG:  redo done at 0/1FFFF78
LOG:  last completed transaction was at log time 2014-09-16 10:38:50.010177-04
PANIC:  could not write to file "pg_xlog/xlogtemp.563": No space left on device
LOG:  startup process (PID 563) was terminated by signal 6: Aborted
LOG:  aborting startup due to startup process failure

The "PANIC" seen above is the most severe log_level Postgres has, and it basically causes a "full stop right now!". You will note in the above snippet that a normal SQL command caused the problem, which then caused all other Postgres processes to terminate. Postgres then tried to restart itself, but immediately ran into the same problem (no disk space) and thus refused to start back up. (The "FATAL" line above was another client trying to connect while all of this was going on.)

Before we can look at how to fix things, a little background will help. When Postgres is running normally, there is a finite number of WAL files (roughly twice the value of checkpoint_segments) that exist in the pg_xlog directory. Postgres deletes older WAL files, so the total number of files never climbs too high. When something prevents Postgres from removing the older files, the number of WAL files can grow quite dramatically, culminating in the out of space condition seen above. Our solution is therefore two-fold: fix whatever is preventing the old files from being deleted, and clear out enough disk space to allow Postgres to start up again.

The first step is to determine why the WAL files are not being removed. The most common case is a failing archive_command. If this is the case, you will see archive-specific errors in your Postgres log. The usual causes are a failed network, downed remote server, or incorrect copying permissions. You might see some errors like this:

2013-05-06 23:51:35 EDT [19421]: [206-1] user=,db=,remote= LOG:  archive command failed with exit code 14
2013-05-06 23:51:35 EDT [19421]: [207-1] user=,db=,remote= DETAIL:  The failed archive command was: rsync --whole-file --ignore-existing --delete-after -a pg_xlog/000000010000006B00000016 backup:/archive/000000010000006B00000016
rsync: Failed to exec ssh: Permission denied (13)
# the above was from an actual bug report; the problem was SELinux

There are some other reasons why WAL would not be removed, such as failure to complete a checkpoint, but they are very rare so we will focus on archive_command. The quickest solution is to fix the underlying problem by bringing the remote server back up, fixing the permissions, etc. (To debug, try emulating the archive_command you are using with a small text file, as the postgres user. It is generally safe to ship non-WAL files to the same remote directory). If you cannot easily or quickly get your archive_command working, change it to a dummy command that always returns true:

# On Nix boxes:
archive_command = '/bin/true'
# On BSD boxes:
archive_command = '/usr/bin/true'
# On Windows boxes:
archive_command = 'REM'

This will allow the archive_command to complete successfully, and thus lets Postgres start removing older, unused WAL files. Note that changing the archive_command means you will need to change the archive_command back later and create fresh base backups, so do that as a last resort. Even after changing the archive_command, you cannot start the server yet, because the lack of disk space is still a problem. Here is what the logs would look like if you tried to start it up again:

LOG:  database system shutdown was interrupted; last known up at 2014-09-16 10:38:54 EDT
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at 0/162FE44
LOG:  redo done at 0/1FFFF78
LOG:  last completed transaction was at log time 2014-09-16 10:38:50.010177-04
PANIC:  could not write to file "pg_xlog/xlogtemp.602": No space left on device
LOG:  startup process (PID 602) was terminated by signal 6: Aborted
LOG:  aborting startup due to startup process failure

At this point, you must provide Postgres a little bit of room in the partition/disk that the pg_xlog directory is in. There are four approaches to doing so: removing non-WAL files to clear space, moving the pg_xlog directory, resizing the partition it is on, and removing some of the WAL files yourself.

The easiest solution is to clear up space by removing any non-WAL files that are on the same partition. If you do not have pg_xlog on its own partition, just remove a few files (or move them to another partition) and then start Postgres. You don't need much space - a few hundred megabytes should be more than enough.

This problem occurs often enough that I have a best practice: create a dummy file on your pg_xlog partition whose sole purpose is to get deleted after this problem occurs, and thus free up enough space to allow Postgres to start! Disk space is cheap these days, so just create a 300MB file and put it in place like so (on Linux):

dd if=/bin/zero of=/pgdata/pg_xlog/DO_NOT_MOVE_THIS_FILE bs=1MB count=300

This is a nice trick, because you don't have to worry about finding a file to remove, or determine which WALs to delete - simply move or delete the file and you are done. Once things are back to normal, don't forget to put it back in place.

The best way to get more room is to simply move your pg_xlog directory to another partition that has more space. Simply create a directory for it on the other partition, copy over all the files, then make pg_xlog a symlink to this new directory. (thanks to Bruce in the comments below)

Another way to get more space in your pg_xlog partition is to resize it. Obviously this is only an option if your OS/filesystem has been setup to allow resizing, but if it is, this is a quick and easy way to give Postgres enough space to startup again. No example code on this one, as the way to resize disks varies so much.

The final way is to remove some older WAL files. This should be done as a last resort! It is far better to create space, as removing important WAL files can render your database unusable! If you go this route, first determine which files are safest to remove. One way to determine this is to use the pg_controldata program. Just run it with the location of your data directory as the only argument, and you should be rewarded with a screenful of arcane information. The important lines will look like this:

Latest checkpoint's REDO location:    0/4000020
Latest checkpoint's REDO WAL file:    000000010000000000000005

This second line represents the last WAL file processed, and it should be safe to remove any files older than that one. (Unfortunately, older versions of PostgreSQL will not show that line, and only the REDO location. While the canonical way to translate the location to a filename is with the pg_xlogfile_name() function, it is of little use in this situation, as it requires a live database! Thus, you may need another solution.)

Once you know which WAL file to keep by looking at the pg_controldata output, you can simply delete all WAL files older than that one. (As Craig points out in the comments below, you can use the pg_archivecleanup program in standalone mode, which will actually work all the way back to version 8.0). As with all mass deletion actions, I recommend a three-part approach. First, back everything up. This could be as simple as copying all the files in the pg_xlog directory somewhere else. Second, do a trial run. This means seeing what the deletion would do without actually deleting the files. For some commands, this means using a --dry-run or similar option, but in our example below, we can simply leave out the "-delete" argument. Third, carefully perform the actual deletion. In our example above, we could clear the old WAL files by doing:

$ cp -r /pgdata/pg_xlog/* /home/greg/backups/
$ find -L /pgdata/pg_xlog -not -newer /pgdata/pg_xlog/000000010000000000000005 -not -samefile /pgdata/pg_xlog/000000010000000000000005| sort | less
$ find -L /pgdata/pg_xlog -not -newer /pgdata/pg_xlog/000000010000000000000005 -not -samefile /pgdata/pg_xlog/000000010000000000000005 -delete

It's worth a mention that to find files older than the specific file it's not sufficient to just do find -not -newer, because this would actually include the file being compared against, so deleting would be disastrous for your database cluster. Be sure to include the -not -samefile in the find command. Additionally, if you have a very busy system, it's possible that the modification timestamps on the WAL files will have the same timestamp, and so might get removed if you just blindly -delete everything. This is why it is very important to always review the output before actually deleting things.

Once you have straightened out the archive_command and cleared out some disk space, you are ready to start Postgres up. You may want to adjust your pg_hba.conf to keep everyone else out until you verify all is working. When you start Postgres, the logs will look like this:

LOG:  database system was shut down at 2014-09-16 10:28:12 EDT
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

After a few minutes, check on the pg_xlog directory, and you should see that Postgres has deleted all the extra WAL files, and the number left should be roughly twice the checkpoint_segments setting. If you adjusted pg_hba.conf, adjust it again to let clients back in. If you changed your archive_command to always return truth, remember to change it back as well as generate a new base backup

Now that the problem is fixed, how do you prevent it from happening again? First, you should use the 'tail_n_mail' program to monitor your Postgres log files, so that the moment the archive_command starts failing, you will receive an email and can deal with it right away. Making sure your pg_xlog partition has plenty of space is a good strategy as well, as the longer it takes to fill up, the more time you have to correct the problem before you run out of disk space.

Another way to stay on top of the problem is to get alerted when the pg_xlog directory starts filling up. Regardless of whether it is on its own partition or not, you should be using a standard tool like Nagios to alert you when the disk space starts to run low. You can also use the check_postgres program to alert you when the number of WAL files in the pg_xlog directory goes above a specified number.

In summary, things you should do now to prevent, detect, and/or mitigate the problem of running out of disk space in pg_xlog:

  1. Move pg_xlog to its own partition. This not only increases performance, but keeps things simple and makes things like disk resizing easier.
  2. Create a dummy file in the pg_xlog directory as described above. This is a placeholder file that will prevent the partition from being completely filled with WAL files when 100% disk space is reached.
  3. Use tail_n_mail to instantly detect archive_command failures and deal with them before they lead to a disk space error (not to mention the stale standby server problem!)
  4. Monitor the disk space and/or number of WAL files (via check_postgres) so that you are notified that the WALs are growing out of control. Otherwise your first notification may be when the database PANICs and shuts down!

In summary, don't panic if you run out of space. Do the steps above, and rest assured that no data corruption or data loss has occurred. It's not fun, but there are far worse Postgres problems to run into! :)

Some metaprogramming examples from RSpec

I'm quite the curious cat and one thing that has interested me for a while has been how RSpec and its describe and it methods work. In digging through the rspec-core gem source code (v3.1.4), specifically the example_group.rb file, I came across some syntax that I had not been exposed to:

# https://github.com/rspec/rspec-core/blob/v3.1.3/lib/rspec/core/example_group.rb

module RSpec
  module Core
    class ExampleGroup
      # ...

      def self.define_singleton_method(*a, &b)
        (class << self; self; end).__send__(:define_method, *a, &b)
      end

      # ...

      def self.define_example_method(name, extra_options={})
        define_singleton_method(name) do |*all_args, &block|
          
          # ...
          
        end
      end

      # ...

      define_example_method :it

      # ...

    end
  end
end

"What's with all this passing around of blocks? And what's :define_method doing?" I asked. The documentation for the define_method is straightforward enough, yet I still wondered what was being accomplished in the code above. In pursuit of answers, here's what I found out.

Metaprogramming

Metaprogramming is the writing of code that acts on other code instead of data, also commonly described as "code that writes code". As an example, let's reopen a class at runtime and add a method:

class Dog
  # empty class
end

dog = Dog.new
=> #<Dog:0x00000006add440>
dog.methods.include? :speak
=> false
Dog.__send__(:define_method, :speak, Proc.new { "Woof!" })
=> :speak
dog.speak
=> "Woof!"

The block passed to the define_method method is used as the body of the method being defined, which in our case is the speak method. Note the use of __send__ over send. Because some classes define their own send method, it's safer to use __send__ . As another example, let's define a method that we can use to create more class methods:

class Cat
  def create_method(method_name, &method_body)
    self.class.__send__(:define_method, method_name, &method_body)
  end
end

cat = Cat.new
 => #<Cat:0x00000005973c68> 
cat.methods.include? :speak
 => false
cat.create_method(:speak) { "Meow!" }
=> :speak
cat.speak
=> "Meow!"
cat2 = Cat.new
=> #<Cat:0x00000005962da0>
cat.2.speak
=> "Meow!"

Metaclasses

A metaclass is the class of an object that holds singleton methods, and a singleton method is a method which belongs to just one object. If we have an instance, dog, of a Dog class we can define a singleton method as follows:

class Dog
end

dog = Dog.new
=> #<Dog:0x00000006990e70>
def dog.sit
  "I'm sitting, now gimme a treat!"
end
=> :sit
dog.sit
=> "I'm sitting, now gimme a treat!"
dog2 = Dog.new
=> #<Dog:0x00000006977998>
dog2.methods.include? :sit
=> false

When Ruby looks for a method, it first looks in the object's metaclass. If it doesn't find it there, then it looks in the object's class and upwards through the inheritance chain. To access an object's metaclass we can use the following syntax:

class Dog
end

dog = Dog.new
=> #<Dog:0x00000006990e70>
def dog.sit
  "I'm sitting, now gimme a treat!"
end
=> :sit
metaclass = class << dog; self; end
=> #<Class:#<Dog:0x00000006990e70>>
metaclass.instance_methods.include? :sit
=> true

OK, with all this in mind let's define a class method that can be used to create singleton methods for that class. "Wait, wait, singleton methods for a class? But aren't singleton methods for objects?" I hear you. In Ruby, classes are objects. If they are objects, then they can have metaclasses. Let's see it in action:

class Cat
  def self.define_singleton_method(method_name, &method_body)
    (class << self; self; end).__send__(:define_method, method_name, &method_body)
  end

  define_singleton_method(:speak) {  "Meow!" }
  define_singleton_method(:purr) { "Purr!" }
  define_singleton_method(:hiss) { "Hiss!" }
end

Cat.speak
=> "Meow!"
Cat.methods.grep(/speak|purr|hiss/)
=> [:speak, :purr, :hiss]
cat = Cat.new
=> #<Cat:0x000000067e8d20>
cat.methods.grep(/speak|purr|hiss/)
[]

Hmm, the methods speak, purr, and hiss look like class methods, don't they? Aha! I've learned that class methods are actually singleton methods of the class object, or, instance methods of the class object's metaclass!

class Person
  def self.greet
    "Hello there!"
  end
end

metaclass = class << Person; self; end
=> #<Class: Person>
metaclass.instance_methods.include? :greet
=> true

If we go back and look at the code block where we defined class method define_singleton_method for the Cat class, we can see now that this method, by opening up the class' metaclass and sending it define_method, is basically just creating more class methods. And this is exactly what's going on in the example_group.rb file; the class methods :describe, :it, etc., are created via metaprogramming. Neat!

There's still lots more for me to learn when it comes to metaprogramming. Here's one blog article by Yehuda Katz that I found really helpful in understanding metaprogramming, especially the role of self.

Spree Authorization Failure for Customized Role

Hello again all. Recently I was working on another Spree site running Spree 2.1.1. The client wanted to create some custom roles. For example, the client wanted there to be a Sales Manager role. A Sales Manager could log in and have read and write access to all the orders. However, a sales manager should not have read/write access to products, configuration, promotions, users, etc. This was easily accomplished by following the steps in the Spree documentation. As I will describe, this documentation assumes that the custom role will have access to Orders#index.

The client wanted to create a second custom role that had create, read, update and delete access to the Training model and nothing more. The training model belongs to a taxon and has a unique event date and taxon id. An example would be a training instance with an event date of September 9th, 2014 that belongs to a taxon with the name "Fire Safety 101" and a description "Teaching fire safety in accordance with OSHA standards. 10 hours and lunch is provided". So, I planned to create a training personnel role that should be able to log in and only have read/write access to Trainings. However, the Spree documentation did not provide an explanation on how to create a custom role that does not have read or write access to orders.

 

Following the pattern described in the Spree documentation for creating custom roles and their respective authorization, I created an ability_decorator.rb with the contents:

class AbilityDecorator
     include CanCan::Ability
     def initialize(user)
       if user.respond_to?(:has_spree_role?) && user.has_spree_role?('sales_manager')
         can [:admin, :index, :show], Spree::Order
       end
       if user.respond_to?(:has_spree_role?) && user.has_spree_role?('training')
         can [:admin, :manage], Spree::Training
       end
     end
   end
 
   Spree::Ability.register_ability(AbilityDecorator)

However, after creating a training user and attempting to log in, I got an unauthorized error. So, I checked the logs:

The log output above shows that while I was logged in as a user with the training role, the application was checking for authorization on Spree::Admin::OrdersController#index (the orders list page), because the base admin URL ("/admin") points to this controller action. I reviewed the Devise documentation to modify where a user with the training role is redirected to upon login (via Spree Auth Devise's after_sign_in method), as shown in the code shown below.

def after_sign_in_path_for(resource)
    stored_location_for(resource) ||
      if resource.is_a?(Spree::User) && resource.has_spree_role?('training')
        admin_trainings_path
      else
        super
      end
  end

After making this change, I tried once again and was able to successfully log in as a training user and only have the desired access to Trainings.

 

To summarize, if you'd like to have a custom role and not give them access to Orders, you will need to make some adjustments outside the steps listed in Spree's documentation for custom role authorization.

Piggybak: Recent Updates and Upgrades

Piggybak, an open source Ruby on Rails ecommerce gem, implemented as a mountable solution, has continued to be upgraded and maintained over the last several months to keep up to date with Rails security releases and Ruby releases. Here are some quick notes on recent work:

  • Piggybak (version 0.7.5) is now compatible with Rails 4.1.6, which is the most up to date release of Rails. See the Rails release notes for more details on this recent release. The Piggybak Demo is now running on Rails 4.1.6.
  • Piggybak is compatible with Ruby 2.1.2, and the demo is running on Ruby 2.1.2
  • Recent updates in Piggybak include migration fixes to handle table namespace issues, and updates to remove methods that are no longer present in Rails (that were previously deprecated).
  • Recent updates to the demo include updates to the integration testing suite to allow testing to be compatible with Rails 4.1.6, as well as modifications to how the demo handles exceptions.

Make sure to check out Piggybak on github repository for more details on these recent updates.

Aliasin' and Redmine plugins

Recently I was tasked with creating a plugin to customize End Point's Redmine instance. In working through this I was exposed for the first time to alias_method_chain. What follows is my journey down the rabbit hole as I wrap my head around new (to me) Ruby/Rails magic.

The Rails core method alias_method_chain encapsulates a common pattern of using alias_method twice: first to rename an original method to a method "without" a feature, and second to rename a new method "with" a feature to the original method. Whaaaa? Let's start by taking a look at Ruby core methods alias and alias_method before further discussing alias_method_chain.

alias and alias_method

At first glance, they achieve the same goal with slightly different syntax:

class Person
  def hello
    "Hello"
  end

  alias say_hello hello
end

Person.new.hello
=> "Hello"
Person.new.say_hello
=> "Hello"
class Person
  def hello
    "Hello"
  end

  alias_method :say_hello, :hello
end

Person.new.hello
=> "Hello"
Person.new.say_hello
=> "Hello"

Let's see what happens when we have a class inherit from Person in each of the cases above.

class Person
  def hello
    "Hello"
  end

  # Wrapped in a class function to examine scope
  def self.apply_alias
    alias say_hello hello
  end
  apply_alias
end

class FunnyPerson < Person
  def hello
    "Hello, I'm funny!"
  end
  apply_alias
end

FunnyPerson.new.hello
=> "Hello, I'm funny!"
FunnyPerson.new.say_hello
=> "Hello"
class Person
  def hello
    "Hello"
  end

  # Wrapped in a class function to examine scope
  def self.apply_alias
    alias_method :say_hello, :hello
  end
  apply_alias
end

class FunnyPerson < Person
  def hello
    "Hello, I'm funny!"
  end
  apply_alias
end

FunnyPerson. new.hello
=> "Hello, I'm funny!"
FunnyPerson.new.say_hello
=> "Hello, I'm funny!"

Because alias is a Ruby keyword it is executed when the source code gets parsed which in our case is in the scope of the Person class. Hence, say_hello will always be aliased to the hello method defined in Person. Since alias_method is a method, it is executed at runtime which in our case is in the scope of the FunnyPerson class.

alias_method_chain

Suppose we want a child class to extend the hello method. We could do so with a couple of alias_method calls:

class Person
  def hello
    "Hello"
  end
end

class PolitePerson < Person
  def hello_with_majesty
    "#{hello_without_majesty}, your majesty!"
  end

  alias_method :hello_without_majesty, :hello
  alias_method :hello, :hello_with_majesty
end

PolitePerson.new.hello
=> "Hello, your majesty!"
PolitePerson.new.hello_with_majesty
=> "Hello, your majesty!"
PolitePerson.new.hello_without_majesty
=> "Hello"

What we did above in PolitePerson can be simplified by replacing the two alias_method calls with just one call to alias_method_chain:

class Person
  def hello
    "Hello"
  end
end

class PolitePerson < Person
  def hello_with_majesty
    "#{hello_without_majesty}, your majesty!"
  end

  alias_method_chain :hello, :majesty
end

class OverlyPolitePerson < Person
  def hello_with_honor
    "#{hello_without_humbling} I am honored by your presence!"
  end

  alias_method_chain :hello, :honor
end

PolitePerson.new.hello
=> "Hello, your majesty!"
OverlyPolitePerson.new.hello
=> "Hello, your majesty! I am honored by your presence!"

Neat! How does this play into Redmine plugins, you ask? Before we get into that there is one more thing to go over: a module's included method.

The included callback

When a module is included into another class or module, Ruby invokes the included method if defined. You can think of it as a sort of module initializer:

module Polite
  def self.included(base)
    puts "Polite has been included in class #{base}"
  end
end

class Person
  include Polite

  def hello
    "Hello"
  end
end
Polite has been included in class Person
=> Person

Now, what if you can't modify the Person class directly with the include line? No biggie. Let's just send Person a message to include our module:

class Person
  def hello
    "Hello"
  end
end

module Polite
  def self.include(base)
    puts "Polite has been included in class #{base}"
  end

  def polite_hello
    "Hello, your majesty!"
  end
end

Person.send(:include, Polite)
Polite has been included in class Person
=> Person

What if we now want to extend Person's hello method? Easy peasy:

class Person
  def hello
    "Hello"
  end
end

module Polite
  def self.included(base)
    base.send :include, InstanceMethods

    base.class_eval do
      alias_method_chain :hello, :politeness
    end
  end

  module InstanceMethods
    def hello_with_politeness
      "#{hello_without_politeness}, your majesty!"
    end
  end
end

Person.new.hello
=> "Hello"
Person.send :include, Polite
=> Person
Person.new.hello
=> "Hello, your majesty!"

How polite! Let's talk about what's going on in the Polite module. We defined our hello_with_politeness method inside an InstanceMethods module in order to not convolute the self.include method. In self.include we send an include call to the base class so that InstanceMethods is included. This will allow our base class instances access to any method defined in InstanceMethods. Next, class_eval is used on the base class so that the alias_method_chain method is called within the context of the class.

How this applies to Redmine

If you take a look at the Redmine plugin documentation, specifically Extending the Redmine Core, you'll see the above pattern as the recommended method to overwrite/extend Redmine core functionality. I'll include the RateUsersHelperPatch example from the documentation here so that you can see it compared with the above code blocks:

module RateUsersHelperPatch
  def self.included(base) # :nodoc:
    base.send(:include, InstanceMethods)

    base.class_eval do
      unloadable # Send unloadable so it will not be unloaded in development

      alias_method_chain :user_settings_tabs, :rate_tab
    end
  end

  module InstanceMethods
    # Adds a rates tab to the user administration page
    def user_settings_tabs_with_rate_tab
      tabs = user_settings_tabs_without_rate_tab
      tabs << { :name => 'rates', :partial => 'users/rates', :label => :rate_label_rate_history}
      return tabs
    end
  end
end

Sending an include to RateUsersHelper can be done in the plugin's init.rb file:

Rails.configuration.to_prepare do
  require 'rate_users_helper_patch'
  RateUsersHelper.send :include, RateUsersHelperPatch
end

So, the tabs variable is set using user_settings_tabs_without_rate_tab, which is aliased to the Redmine core user_settings_tabs method:

# https://github.com/redmine/redmine/blob/2.5.2/app/helpers/users_helper.rb#L45-L53
def user_settings_tabs
  tabs = [{:name => 'general', :partial => 'users/general', :label => :label_general},
          {:name => 'memberships', :partial => 'users/memberships', :label => :label_project_plural}
          ]
  if Group.all.any?
    tabs.insert 1, {:name => 'groups', :partial => 'users/groups', :label => :label_group_plural}
  end
  tabs
end

Then, a new hash is added to tabs. Because method user_settings_tabs is now aliased to user_settings_tabs_with_rate_tab, the users/groups partial will be included when the call to render user_settings_tabs is executed:

#https://github.com/redmine/redmine/blob/2.5.2/app/views/users/edit.html.erb#L9
<%= link_to l(:label_profile), user_path(@user), :class => 'icon icon-user' %> <%= change_status_link(@user) %> <%= delete_link user_path(@user) if User.current != @user %>
<%= title [l(:label_user_plural), users_path], @user.login %> <%= render_tabs user_settings_tabs %>

Although alias_method_chain is a pretty cool and very useful method, it's not without its shortcomings. There's a great, recent blog article about that here in which Ruby 2's Module#prepend as a better alternative to alias_method_chain is discussed as well.

Analyzer Reports with Geo Map Option in Pentaho 5 BI Server

The "Geo Map" option in Analyzer Reports provides a feature to visualize data with geographic locations. We will learn how to design a Mondrian schema and configure Pentaho to make use of the "Geo Map" feature in the Analyzer Reports. This article will show us how to set this feature up step by step.

Enable Geo Map feature on Geographic fields in Mondrian Schema


The Mondrian schema has two main categories called Dimensions and Measures. The Dimensions are defined as levels in the Mondrian schema. The Geographic fields should have two additional annotations to use Geo Map. The two annotations are:

1. Data.Role - defines the type of level generally; for this type of node, this must be set to  'Geography'.
2. Geo.Role - defines the geographical classification in a hierarchy. These can be either predefined roles ('country', 'state', 'city', 'postalcode') or custom roles.

Sample Level with Annotation:

        <Level name="Country Name" visible="true" column="country" type="String" uniqueMembers="false" levelType="Regular" hideMemberIf="Never">
          <Annotations>
            <Annotation name="Data.Role"><![CDATA[Geography]]></Annotation>
            <Annotation name="Geo.Role"><![CDATA[country]]></Annotation>
          </Annotations>
        </Level>


Geographic fields and datasets in database 


I have created a sample table with the fields containing geographic locations for dimensions and aggregated value for measures. The sample population table contains Pentaho-defined geographic locations 'country', 'state', 'city' and aggregated population count for those geographic fields.

'Population' table design and datasets: 


Here we create a sample population table with geographic fields and the population count in a PostgreSQL database.
CREATE TABLE population (
   id INT PRIMARY KEY   NOT NULL,
   country      TEXT    NOT NULL,
   state        TEXT    NOT NULL,
   city         TEXT   NOT NULL,
   count        INT    NOT NULL
);

Next we load population data into the table for 4 cities of 2 states in USA. (Population data for more geographic locations in USA are available at USA Population.)
# SELECT * FROM population;
 id | country |   state    |     city      |  count 
----+---------+------------+---------------+---------
  1 | USA     | California | Los Angeles   | 3857800
  2 | USA     | California | San Francisco |  825863
  3 | USA     | New York   | Hilton        |    5974
  4 | USA     | New York   | Johnsburg     |    2390

Download the sql dump file with table schema and datasets.


Design a Mondrian Schema with Geographic Support


Pentaho provides a tool called "Schema Work Bench" to design a Mondrian schema for a specific table's data. We can create a new Mondrian schema for the table by selecting File -> New -> Schema. The picture below depicts the hierarchy level of the Mondrian schema elements.



Publish the Mondrian schema to Pentaho 


The publish process requires the JDBC datasource to have access to database. Create a JDBC datasource in the manage datasources wizard with necessary input values.



Once the JDBC datasource has been created in Pentaho server, the Mondrian schema can be published from the Schema Work Bench.



Download the Mondrian schema xml to view the schema, cube, table, dimension, hierarchy, level, annotations, measures elements and corresponding attribute values.

The Mondrian schema xml can be imported directly into Pentaho server to create an analysis datasource.



Create a Analyzer Report with Geo Map


Add the necessary geographic fields under "Rows" and population count under "Measure" to create a basic analyzer report.



Change the report type to "Geo Map" through the right top corner options to view the visualized data. Congratulations, you're done!

Today's Internet Slowdown

Today End Point is participating in an Internet-wide campaign to raise awareness about net neutrality, the FCC's role in overseeing the Internet in the United States, and the possible effects of lobbying by large consumer Internet providers.

Many companies and individuals are in favor of specific "net neutrality" regulation by the FCC, and make good arguments for it, such as these by Battle for the Net, Etsy and ThoughtWorks and Reddit.

There are also plenty speaking out against certain specific regulatory proposals out there: TechFreedom, Google, Facebook, Microsoft, Yahoo!, Todd Wasserman, and with a jaunty propagandistic style, NCTA, the cable company's lobby.

I think we are all sympathetic to free-market arguments and support non-governmental solutions that allow companies and individuals to create things without getting permission, and to arrange services and peering as they see fit. It seems that most people and companies understand the need to pay for more bandwidth, and more data transfer. (Marketers are the ones promising unlimited everything, then hiding limits in the fine print!) Many of us are worried about further entrenching government in private networks, whether ostensibly for national security, "intellectual property" policing, or enforcing net neutrality.

But the market competition is hobbled when there are few competitors in a given geographic area. Many Internet users have few options if their ISP begins to filter or slow traffic by service type. I think we would all be better off with less false advertising of "unlimited downloads" and more realistic discussion of real costs. ISP backroom arm-twisting deals with companies just using the network as customers request can invisibly entrench existing players to the exclusion of new entrants.

Every Internet provider builds on lots of infrastructure that was funded by the public, platform popularity built by other companies and individuals, rights of way granted by local municipalities and others, research done by government-funded institutions, and finally, their own semi-monopoly positions that are sometimes enforced by government at various levels.

In any case there is not really a simple argument on either side either entirely for or against regulation. Some regulation is already there. The question is what form it will take, how it affects different groups now, and how it shapes the possibilities in the future.

End Point does not endorse any specific position or proposal on the table at the FCC, but we want to raise awareness about this Internet regulation discussion and hope that you will do some research and comment to the FCC about how you see things. It's worth letting your Internet provider, mobile phone carrier, and businesses you interact with online know how you feel too! Those outside the United States may find similar debates underway in their countries, perhaps not getting the broad attention they deserve.

Enhancing the labelsontime.com Spree application

Labels on Time is an online retailer that delivers top-quality thermal roll and direct thermal labels - and all on time, of course. They came to us last year to upgrade their Spree site, resolve bugs, and develop cutting-edge features, utilizing our expertise with the ecommerce platform. Spree Commerce is an open-source ecommerce solution built on Ruby on Rails, and manages all aspects of the fulfillment process, from checkout to shipping to discounts, and much more.

UPGRADING THE SPREE PLATFORM

There were quite a few challenges associated with the upgrade, since Labels on Time was still running on Spree's version 2.0, which was not yet stable. To keep some stability, we initially worked off a fork of Spree, and selectively brought in changes from 2.0 when we were sure they were stable and reliable enough.

USING SPREE GEMS

To date, some of the Spree gems we have used on the site include:

Active Shipping: This is a Spree plugin that can interface with USPS, UPS and FedEx. Label on Time’s active_shipping gem interacts with the UPS API, which is a big task to tackle since it requires a lot of configuration, especially every time Spree is updated.

Another important gem we use for Labels on Time is Volume Pricing. Volume Pricing is an extension to Spree that uses predefined ranges of quantities to determine the price for a particular product variant. When we first added this gem on the labelsontime.com checkout page, we kept finding that if a user increased the number of items in their cart sufficiently to activate the volume pricing and receive a discount per item, the standard Spree view did not show the new (discounted) price that was currently in effect (although it was correctly calculating the totals). To resolve this, our developer Matt Galvin created some custom JavaScript and Ruby code. Thanks to Matt’s ingenuity, the application can now return every price for every possible size and sort it accordingly.

WHAT WE’RE WORKING ON NEXT

Matt recently upgraded the application to 2.0.10, which was needed for security reasons. You can read more about the security fix here. We are also working on implementing a neat SEO gem called Canonical Rails, which helps search engines understand that any duplicate content URLs it can access all refer to the canonical URL.

Next up, we’re going to implement inventory management, where, according to a customer’s location, we can suggest the available inventory in the closest warehouse to that location.

Adventures in Downgrading my Linode Plan

I recently went through the process of downgrading and downsizing my Linode plan and I wanted to share a few of the [small] hoops that I had to jump through to get there, with the help of the Linode Support team.

Background

I've had a small personal WordPress site running for more than a few years now. I also use this server for personal Ruby on Rails development. When I began work on that site, I tried out a few shared hosting providers such as Bluehost and GoDaddy because of the low cost (Bluehost was ~$6/mo) at the time. However, I quickly encountered common limitations of shared server hosting:

  • Shared hosting providers typically make it very difficult to run Ruby on Rails, especially edge versions of Ruby on Rails. It's possible this has improved over the last few years, but when you are a developer and want to experiment (not locally), shared hosting providers are not going to give you the freedom to do so.
  • Shared hosting providers do not give you control of specific performance settings (e.g. use of mod_gzip, expires headers), so I was suffering from lack of control for my little WordPress site as well as my Rails sites. While this is another limitation that may have improved over the last few years, ultimately you are limited by non-root access as a shared server user.

Enter Linode

I looked to virtual server providers such as Linode, Slicehost, and Rackspace after experiencing these common limitations. At the time of my transition, Linode and Slicehost were comparatively priced, but because End Point had successful experiences with Linode for several clients up to that point, I decided to make the jump to Linode. I can't remember what my initial Linode size was (I think 512MB), but I chose the smallest available option at $20/mo, plus $5/mo for backups. I am not a sysadmin expert like my fellow coworkers (Richard, Jon, the list goes on ...), but I managed to get PHP and Ruby on Rails running on Apache with MySQL, and several of the best practices for speeding up your web site in place.

Fast forward about 2 years, and I've been very happy with Linode. I can only remember one specific instance where my server has gone down, and the support team has always been very responsive. They also release occasionally free upgrades at the $20/mo price point, and the current offering at that price point is the Linode 2GB (see more here). But lately, I've been hearing that Digital Ocean has been gaining momentum with a few cheaper options, and I considered making the jump. But I missed the recent announcement that Linode introduced a new $10/mo. plan back in June (hooray!), so I'm happy to stay with Linode at this lower price point that is suitable for my small, but optimized WordPress site and small Ruby on Rails experiments.

How to Downsize

In a perfect world, it would seem that to quickly downsize your Linode instance, you would first click on the "Resize" tab upon logging in to the Linode dashboard, click on the lower plan that you want, and then click "Resize this Linode now!", as shown in the screenshot below:

The Linode resize options.

Things went a little differently for me. First, I received this message when I tried to resize:
"Pending free upgrades must be performed before you are able to resize. Please visit the dashboard to upgrade."

So I headed to my dashboard and clicked on the free upgrade link on the bottom right in the dashboard. I then encountered this message:
"Linodes with configuration profiles referencing a 32 bit kernel are currently not eligible for this upgrade. For more information please see our switching kernels guide, or redeploy this Linode using a 64 bit distro."

My main Linode Configuration Profile was 64 bit, but my Restore Configuration Profile was running the 32 bit kernel. So, I first had to update that by clicking on the "Edit" link, selecting the right kernel, and saving those changes. That took a few minutes to take effect.


My two configuration profiles needed to be on 64 bit kernel to allow for the Linode upgrade.

Then, I was ready for the free upgrade, which took another few minutes after the server booted down, migrated, and booted back up. Next, I headed back to the "Resize" tab on the dashboard and tried to proceed on the downgrade. I immediately received an error message notifying me that my disk images exceeded the resized option I wanted to switch to (24GB). Upon examining my dashboard, my disk images showed ~28GB allocated to the various disk images:


My disk image space exceeded the 24GB allotted for the Linode 1024 plan.

I was directed by the Linode support team to edit the disk image to get under that 24GB allowed amount. They also explained that I must verify my current app(s) didn't exceed what I was going to downsize to, using "df -h" while logged into my server. I had already verified previously where disk space was going on my server and cleaned out some cached files and old log files, so I knew the space used was well under 24GB. The only additional step here was that I had to shut down my server first from the dashboard before reducing the disk image space. So I went through all that, and the disk image adjustment took another few minutes. After the disk image size was adjusted, I booted up the server again and verified it was still running.


Editing my Disk Image

Finally, after all that, I went to the "Resize" tab again and selected the Linode 1024 plan and proceeded. The new plan was implemented within a few minutes, automagically booting down my server and restarting it after completion. My billing information was also updated almost immediately, showing that I will now pay $12.50/mo for the Linode 1024 plan with backups.

Conclusion

In list form, here are the steps I went through to reach my final destination:

  • Updated kernels to 64 bit for all configuration profiles.
  • Applied pending, free upgrades.
  • Manually shut down server.
  • Applied change to reduce disk image space.
  • Rebooted server (not necessary, but I verified at this point things were still running).
  • Resized to Linode 1024 plan.

While this process wasn't as trivial as I had hoped, the support folks were super responsive, often responding within a minute or two when I had questions. I'm happy to stay with Linode at this offering and it allows them to remain competitive with both virtual private hosting providers and as an appealing alternative to shared hosting providers. The Linode 1024 plan is also a great starting point for a proof-of-concept or staging server that may be scaled up later as applications move to production and increase in traffic. Linode has plans ranging from the $10/mo plan I have (1GB of RAM, 24GB SSD storage, etc.) all the way up to a 96GB RAM, 1920 GB SSD storage plan at $960/mo.

Rsyslog property based filtering features

Do you need something more powerful than the usual, clunky selectors based Rsyslog filtering rules but still you don't see the benefit of going full throttle and use RainerScript?

Perhaps you weren’t aware, but there is an additional filtering rule you may not have used, which is a great alternative to the classic selector-based one, called property-based filtering.

This kind of filtering lets you create rules like:

:msg, contains, "firewall: IN=" -/var/log/firewall

There's a few more properties that you can use like hostname,fromhost,fromip and the number (and variety) is growing over time.

Instead of just verifying that a specific string is contained in the highlighted property, you could also be interested in operators like isempty, isequal or the powerful regex and ereregex which could be used to compare the string content against regexes, that we all love so much.

:fromhost, regex, ".*app-fe\d{2}" -/data/myapp/frontend_servers.log
:fromhost, regex, ".*app-db\d{2}" -/data/myapp/DB_servers.log

Also remember that you can always use the ! to negate the condition and the discard operator to block Rsyslog from further rules parsing for that specific content:

:msg, !contains, "firewall: IN=" -/data/myapp/all_logs_but_firewall_related.log
:fromhost, regex, ".*appfe\d{2}" ~ -/data/myapp/frontend_servers.log
:fromhost, regex, ".*appdb\d{2}" ~ -/data/myapp/DB_servers.log
*.* /data/myapp/all_logs_but_firewall_related_and_not_from_appfe_and_appdb_servers.log

In case you don't know what the - (dash) sign stands for, that's used to put the log writing process in async mode, so that Rsyslog can proceed with other filtering and won't wait for disk I/O to confirm a successful write before proceeding to something else.

Now go back to your logging system and let us know what nice set up you came up with!

Link-ography: