End Point

News

Welcome to End Point's blog

Ongoing observations by End Point people.

OpenSSL CSR with Alternative Names one-line

I find it hard to remember a period in my whole life in which I issued, reissued, renewed and revoked so many certificates.

And while that's usually fun and interesting, there's one thing I often needed and never figured out, till a few days ago, which is how to generate CSRs (Certificate Signing Requests) with AlternativeNames (eg: including www and non-www domain in the same cert) with a one-liner command.

This need is due to the fact that some certificate providers (like GeoTrust) don't cover the parent domain when requesting a new certificate (eg: CSR for www.endpoint.com won't cover endpoint.com), unless you specifically request so.

Luckily that's not the case with other Certificate products (like RapidSSL) which already offer this feature built-in.

This scenario is starting to be problematic more often since we're seeing a growing number of customers supporting sites with HTTPs connections covering both www and "non-www" subdomains for their site.

Luckily the solution is pretty simple and straight-forward and the only requirement is that you should type the CSR subject on the command line directly, basically without the use of the interactive question mechanism.

If you managed to understand how an SSL certificate works this shouldn't be a huge problem, anyway just as a recap here's the list of the meaning for the common Subject entries you'll need:

  • C => Country
  • ST => State
  • L => City
  • O => Organization
  • OU => Organization Unit
  • CN => Common Name (eg: the main domain the certificate should cover)
  • emailAddress => main administrative point of contact for the certificate

So by using the common syntax for OpenSSL subject written via command line you need to specify all of the above (the OU is optional) and add another section called subjectAltName=.

By adding DNS.n (where n is a sequential number) entries under the "subjectAltName" field you'll be able to add as many additional "alternate names" as you want, even not related to the main domain.

Obviously the first-level parent domain will be covered by most SSL products, unless specified differently.

So here's an example to generate a certificate which will cover www.endpoint.com and endpoint.com:

openssl req -new -key endpoint.com.key -sha256 -nodes
  -subj '/C=US/ST=New York/L=New York/O=End Point/OU=Hosting Team/CN=www.endpoint.com/
         emailAddress=administrative-not-existent-address@our-awesome-domain.com/
         subjectAltName=DNS.1=endpoint.com' > www.endpoint.com.csr

So here's another example with multiple DNS.n entries:

openssl req -new -key endpoint.com.key -sha256 -nodes
  -subj '/C=US/ST=New York/L=New York/O=End Point/OU=Hosting Team/CN=www.endpoint.com/
         emailAddress=administrative-not-existent-address@our-awesome-domain.com/
         subjectAltName=DNS.1=endpoint.com,
         DNS.2=usually-not-convered-domain.endpoint.com,
         DNS.3=multiple-domains-crt.endpoint.com' > www.endpoint.com.csr

warning: we had to split the command into multiple lines to make it readable, but you should keep it all on one line, otherwise you may lose some Subject details.

Now with that I'm able to generate proper multi-domain CSRs effectively.

Please note the use of the -sha256 to use the SHA256 algorithm to sign the CSR that, while not required, is appreciated considered the last round of concerning "attentions" to SHA1.

Prevent MediaWiki showing PHP version with new extension: ControlSpecialVersion


Sok Kwu Wan

I recently created a new MediaWiki extension named ControlSpecialVersion whose purpose is to allow some control over what is shown on MediaWiki's "special" page Special:Version. The latest version of this extension can be downloaded from Mediawiki.org. You can see it in action on the Special:Version page for bucardo.org. The primary purpose of the module is to prevent showing the PHP and database versions to the public.

As with most MediaWiki extensions, installation is easy: download the tarball, unzip it into your extensions directory, and add this line to your LocalSettings.php file:


require_once( "$IP/extensions/ControlSpecialVersion/ControlSpecialVersion.php" );

By default, the extension removes the PHP version information from the page. It also changes the PostgreSQL reported version from its revision to simply the major version, and changes the name from the terrible-but-official "PostgreSQL" to the widely-accepted "Postgres". Here is what the Software section of bucardo.org looks like before and after the extension is used:


Note that we are also eliding the git revision information (sha and date). You can also do things such as hide the revision information from the extension list, remove the versions entirely, or even remove an extension from showing up at all. All the configuration parameters can be found on the extension's page on mediawiki.org.

It should be noted that there are typically two other places in which your PHP version may be exposed, both in the HTTP headers. If you are running Apache, it may show the version as part of the Server heading. To turn this off, edit you httpd.conf file and change the ServerTokens directive to ProductOnly. The other header is known as X-Powered-By and is added by PHP to any pages it serves (e.g. MediaWiki pages). To disable this header, edit your php.ini file and make sure expose_php is set to Off.

While these methods may or may not make your server safer, there really is no reason to expose certain information to the world. With this extension, you at least have the choice now.

Another Round of Tidbits: Browser Tools, Performance, UI

It's been a while since I shared a blog article where I share End Point tidbits, or bits of information passed around the End Point team that don't necessarily merit a single blog post, but are worth mentioning and archiving. Here are some notes shared since that last post that I've been collecting:

  • Skeuocard and creditcard.js are intuitive user interface (JS, CSS) plugins for credit card form inputs (card number, security code, billing name).

    Skeuocard Screenshot
  • StackExchange UX is a Stack Overflow resource for user interface Q&A.
  • wpgrep is an available tool for grepping through WordPress databases.
  • Here is a nifty little tool that analyzes GitHub commits to report on language convention, e.g. space vs. tab indentation & spacing in argument definitions.

    Example comparison of single vs. double quote convention in JavaScript.
  • Ag (The Silver Searcher) is a document searching tool similar to ack, with improved speed. There's also a Ag plugin for vim.
  • GitHub released Atom earlier this year. Atom is a desktop application text editor; features include Node.js support, modular design, and a full feature list to compete with existing text editors.
  • SpeedCurve is a web performance tool built on WebPagetest data. It focuses on providing a beautiful user interface and minimizing data storage.

    Example screenshot from SpeedCurve
  • Here is an interesting article by Smashing Magazine discussing mobile strategy for web design. It covers a wide range of challenges that come up in mobile web development.
  • Reveal.js, deck.js, Impress.js, Shower, and showoff are a few open source tools available for in-browser presentation support.
  • Have you seen Firefox's 3D view? It's a 3D representation of the DOM hierarchy. I'm a little skeptical of its value, but the documentation outlines a few use cases such as identifying broken HTML and finding stray elements.

    Example screenshot of Firefox 3D view
  • Here is an interesting article discussing how to approach sales by presenting a specific solution and alternative solutions to clients, rather than the generic "Let me know how I can help." approach.
  • A coworker inquired looking for web based SMS providers to send text messages to customer cellphones. Responses included services recommended such as txtwire, twilio, The Callr, and Clickatell.

Updating Firefox and the Black Screen

If you are updating your Firefox installation for Windows and you get a puzzling black screen of doom, here's a handy tip: disable graphics acceleration.

The symptoms here are that after you upgrade Firefox to version 33, the browser will launch into a black screen, possibly with a black dialog box (it's asking if you want to choose Firefox to be your default browser). Close this as you won't be able to do much with it.

Launch Firefox by holding down the SHIFT key and clicking on the Firefox icon. It will ask if you want to reset Firefox (Nope!) or launch in Safe mode (Yes).

Once you get to that point, click the "Open menu" icon (three horizonal bars, probably at the far right of your toolbar). Choose "Preferences", "Advanced", and uncheck "Use hardware acceleration when available".

Close Firefox, relaunch as normal, and you should be AOK. You can try re-enabling graphics acceleration if and when your graphics driver is updated.

Reference: here.

Postgres copy schema with pg_dump


Manny Calavera (animated by Lua!)
Image by Kitt Walker

Someone on the #postgresql IRC channel was asking how to make a copy of a schema; presented here are a few solutions and some wrinkles I found along the way. The goal is to create a new schema based on an existing one, in which everything is an exact copy. For all of the examples, 'alpha' is the existing, data-filled schema, and 'beta' is the newly created one. It should be noted that creating a copy of an entire database (with all of its schemas) is very easy: CREATE DATABASE betadb TEMPLATE alphadb;

The first approach for copying a schema is the "clone_schema" plpgsql function written by Emanuel Calvo. Go check it out, it's short. Basically, it gets a list of tables from the information_schema and then runs CREATE TABLE statements of the format CREATE TABLE beta.foo (LIKE alpha.foo INCLUDING CONSTRAINTS INCLUDING INDEXES INCLUDING DEFAULTS). This is a pretty good approach, but it does leave out many types of objects, such as functions, domains, FDWs, etc. as well as having a minor sequence problem. It's also slow to copy the data, as it creates all of the indexes before populating the table via INSERT.

My preferred approach for things like this is to use the venerable pg_dump program, as it is in the PostgreSQL 'core' and its purpose in life is to smartly interrogate the system catalogs to produce DDL commands. Yes, parsing the output of pg_dump can get a little hairy, but that's always preferred to trying to create DDL yourself by parsing system catalogs. My quick solution follows.

pg_dump -n alpha | sed '1,/with_oids/ {s/ alpha/ beta/}' | psql

Sure, it's a bit of a hack in that it expects a specific string ("with_oids") to exist at the top of the dump file, but it is quick to write and fast to run; pg_dump creates the tables, copies the data over, and then adds in indexes, triggers, and constraints. (For an explanation of the sed portion, visit this post). So this solution works very well. Or does it? When playing with this, I found that there is one place in which this breaks down: assignment of ownership to certain database objects, especially functions. It turns out pg_dump will *always* schema-qualify the ownership commands for functions, even though the function definition right above it has no schema, but sensibly relies on the search_path. So you see this weirdness in pg_dump output:

--
-- Name: myfunc(); Type: FUNCTION; Schema: alpha; Owner: greg
--
CREATE FUNCTION myfunc() RETURNS text
    LANGUAGE plpgsql
    AS $$ begin return 'quick test'; end$$;

ALTER FUNCTION alpha.myfunc() OWNER TO greg;

Note the fully qualified "alpha.myfunc". This is a problem, and the sed trick above will not replace this "alpha" with "beta", nor is there a simple way to do so, without descending into a dangerous web of regular expressions and slippery assumptions about the file contents. Compare this with the ownership assignments for almost every other object, such as tables:

--
-- Name: mytab; Type: TABLE; Schema: alpha; Owner: greg
--
CREATE TABLE mytab (
    id integer
);

ALTER TABLE mytab OWNER TO greg;

No mention of the "alpha" schema at all, except inside the comment! Before going into why pg_dump is acting like that, I'll present my current favorite solution for making a copy of a schema: using pg_dump and some creative renaming:

$ pg_dump -n alpha -f alpha.schema
$ psql -c 'ALTER SCHEMA alpha RENAME TO alpha_old'
$ psql -f alpha.schema
$ psql -c 'ALTER SCHEMA alpha RENAME TO beta'
$ psql -c 'ALTER SCHEMA alpha_old TO alpha'

This works very well, with the obvious caveat that for a period of time, you don't have your schema available to your applications. Still, a small price to pay for what is most likely a relatively rare event. The sed trick above is also an excellent solution if you don't have to worry about setting ownerships.

Getting back to pg_dump, why is it schema-qualifying some ownerships, despite a search_path being used? The answer seems to lie in src/bin/pg_dump/pg_backup_archiver.c:

  /*                                                                                                                                                      
     * These object types require additional decoration.  Fortunately, the                                                                                  
     * information needed is exactly what's in the DROP command.                                                                                            
     */
    if (strcmp(type, "AGGREGATE") == 0 ||
        strcmp(type, "FUNCTION") == 0 ||
        strcmp(type, "OPERATOR") == 0 ||
        strcmp(type, "OPERATOR CLASS") == 0 ||
        strcmp(type, "OPERATOR FAMILY") == 0)
    {
        /* Chop "DROP " off the front and make a modifiable copy */
        char       *first = pg_strdup(te->dropStmt + 5);

Well, that's an ugly elegant hack and explains why the schema name keeps popping up for functions, aggregates, and operators: because their names can be tricky, pg_dump hacks apart the already existing DROP statement built for the object, which unfortunately is schema-qualified. Thus, we get the redundant (and sed-busting) schema qualification!

Even with all of that, it is still always recommended to use pg_dump when trying to create DDL. Someday Postgres will have a DDL API to allow such things, and/or commands like MySQL's SHOW CREATE TABLE, but until then, use pg_dump, even if it means a few other contortions.

Liquid Galaxy at the Ryder Cup 2014



End Point was proud to present the Liquid Galaxy for the French Golf Federation at this year’s Ryder Cup in Gleneagles, Scotland. The French Golf Federation will be hosting the cup in 2018 at Le Golf National, which is just outside of Paris and is also the current venue of the French Open.

Throughout the event, thousands of people came in and tried out the Liquid Galaxy. The platform displayed one of its many hidden talents and allowed golf fans from around the world to find and show off their home courses. One of the most interesting things to witness was watching golf course designers accurately guess the date of the satellite imagery based on which course changes were present.


This deployment presented special challenges: a remote location (the bustling tented village adjacent to the course) with a combination of available hardware from our European partners and a shipment from our Tennessee office. Despite these challenges, we assembled the system, negotiated the required network connectivity, deployed the custom interface, and delivered a great display for our sponsoring partners. The event was a great success and all enjoyed the unseasonably mild Scottish weather.




Rails Recursive Sorting for Multilevel Nested Array of Objects

Whenever you display data as a list of records, sorting them in a particular order is recommended. Most of the time, Rails treats data as an array, an array of objects, or as a nested array of objects (tree). We would like to use a general sorting mechanism to display the records in ascending or descending order, to provide a decent view to end users. Luckily, Rails comes with a sorting method called 'sort_by' which helps to sort the array of objects by specific object values.

Simple Array:

Trivially, an array can be sorted just by sorting using the “sort” method:
my_array = [ 'Bob', 'Charlie', 'Alice']

my_array = my_array.sort;  # (or just my_array.sort!)
This is the most basic way to sort elements in an array and is part of Ruby’s built-in API.

Array of Objects:

Usually, the result set of the Rails will have an array of objects and should be sorted based on specific attributes of the object in the array. Here is a sample array of objects which need to be sorted by date and fullname.
s_array =
[  
    {
        "date"=> "2014-05-07",
        "children"=> [],
        "fullname"=> "Steve Yoman"
    },
    {
        "date"=> "2014-05-06",
        "children"=> [],
        "fullname"=> "Josh Tolley"
    }
]

Solution:

1) Simple sorting

We can use the Rails sort_by method to sort the array of objects by date and fullname in order:
s_array = s_array.sort_by{|item| [ item['date'], item['fullname'] ]}
sort_by is passed an anonymous function which operates on each item, returning a value to be used as a sort key (returned as an anonymous array in this case). Because Ruby’s array have the Enumerable property, they will automatically be able to be used as sort keys as long as the elements containing them are as well. Because we are returning string properties, we get this for free. We can make use of Rails sort_by method to sort the array of objects by date and fullname in order.

2) Handling case on strings

Sometimes sorting directly on the object attribute will produce undesirable results, for instance if there is inconsistent case in the data. We can further normalize the case of the string used to get records to sort in the expected order:
s_array = s_array.sort_by{|item| [ item['date'], item['fullname'].downcase ]}
Here again we are returning an array to be used as a sort key, but we are using a normalized version of the input data to return.

Multilevel Nested Array of Objects

Sometimes objects in an array will contain the array as element and it continues multilevel. Sorting this kind of array requires recursive sorting to sort the all the levels of array of objects based on specific attributes in object. The following array has nested array and objects alternatively:

m_array =
[
    {
        "name"=> "Company",
        "children"=> [
            {
                "name"=> "Sales",
                "children"=> [
                    {
                        "date"=> "2014-05-07",
                        "children"=> [],
                        "fullname"=> "Steve Yoman"
                    },
                    {
                        "date"=> "2014-05-06",
                        "children"=> [],
                        "fullname"=> "Josh Tolley"
                    }
                ]
            },
            {
                "name"=> "Change Requests",
                "children"=> [
                    {
                        "name"=> "Upgrade Software",
                        "children"=> [
                            {
                                "date"=> "2014-05-01",
                                "children"=> [],
                                "fullname"=> "Selvakumar Arumugam"
                            },
                            {
                                "date"=> "2014-05-02",
                                "children"=> [],
                                "fullname"=> "Marina Lohova"
                            }
                        ]
                    },
                    {
                        "name"=> "Install Software",
                        "children"=> [
                            {
                                "date"=> "2014-05-01",
                                "children"=> [],
                                "fullname"=> "Selvakumar Arumugam"
                            },
                            {
                                "date"=> "2014-05-01",
                                "children"=> [],
                                "fullname"=> "Josh Williams"
                            }
                        ]
                    }
                ]
            }
        ]
    }
]

Solution:

In order to tackle this issue, we will want to sort all of the sub-levels of the nested objects in the same way. We will define a recursive function in order to handle this. We will also want to add additional error-handling.

In this specific example, we know each level of the data contains a “children” attribute, which contains an array of associated objects. We write our sort_multi_array function to recursively sort any such arrays it finds, which will in turn sort all children by name, date and case-insensitive fullname:
def sort_multi_array(items)
  items = items.sort_by{|item| [ item['name'], item['date'], item['fullname'].to_s.downcase ]}
  items.each{ |item| item['children'] = sort_multi_array(item['children']) if (item['children'].nil? ? [] : item['children']).size > 0 }
  items
end

m_array = sort_multi_array(m_array);

You can see that we first sort the passed-in array according to the object-specific attributes, then we check to see if there is an attribute ‘children’ which exists, and then we sort this array using the same function. This will support any number of levels of recursion on this data structure.

Notes about this implementation:

1. Case-insensitive sorting

The best practice when sorting the strings is to convert to one unique case (i.e upper or lower) on sorting. This ensures that records show up in the order that the user would expect, not the computer:
item['fullname'].downcase

2. Handling null values in case conversion

The nil values on the attributes need to be handled on the string manipulation process to avoid the unexpected errors. Here we converting to string before applying the case conversion:
item['fullname'].to_s.downcase

3. Handling null values in array size check

The nil values on the array attributes need to be handled on the sorting process to avoid the unexpected errors. Here we guard against the possibility of item[‘children’] being nil, and if it is, then we return an empty array instead:
(item['children'].nil? ? [] : item['children']).size