News

Welcome to End Point’s blog

Ongoing observations by End Point people

The New Earth



As many of you may have seen, earlier this week Google released a major upgrade to the Google Earth app. Overall, it's much improved, sharper, and a deeper experience for viewers. We will be upgrading/incorporating our managed fleet of Liquid Galaxies over the next two months after we've had a chance to fully test its functionality and polish the integration points, but here are some observations for how we see this updated app impacting the overall Liquid Galaxy experience.

  • Hooray! The new Earth is here! The New Earth is here! Certainly, this is exciting for us. The Google Earth app plays a key central role in the Liquid Galaxy viewing experience, so a major upgrade like this is a most welcome development. So far, the reception has been positive. We anticipate it will continue to get even better as people really sink their hands into the capabilities and mashup opportunities this browser-based Earth presents.

  • We tested some pre-release versions of this application, and successfully integrated them with the Liquid Galaxy and are very happy with how we are able to view-synchronize unique instances of the new Google Earth across displays with appropriate geometrically configured offsets.

  • What to look for in this new application:
    • Stability: The new Google Earth runs as a NaCl application in a Chrome browser. This is an enormous advance for Google Earth. As an application in Chrome it is instantly accessible to billions of new users with their established expectations. Because the new Google Earth uses Chrome the Google Earth developers will no longer need to engage in the minutiae of having to support multiple desktop operating systems, but now can instead concentrate on the core-functionality of Google Earth and leverage the enormous amount of work that the Chrome browser developers do to make Chrome a cross-platform application.
    • Smoother 3D: The (older) Google Earth sometimes has a sort of "melted ice cream" look to the 3D buildings in many situations. Often, buildings fail to fully load from certain viewpoints. From what we're seeing so far, the 3D renderings in the New Earth appear to be a lot sharper and cleaner.
    • Browser-based possibilities: As focus turns more and more to browser-based apps, and as JavaScript libraries continue to mature, the opportunities and possibilities for how to display various data types, data visualizations, and interactions really start to multiply. We can already see this with the sort of deeper stories and knowledge cards that Google is including in the Google Earth interface. We hope to take the ball and run with it, as the Liquid Galaxy can already handle a host of different media types. We might exploit layers, smart use controls, realtime content integration from other available databases, and... okay, I'm getting ahead of myself.

  • The New Google Earth makes a major point of featuring stories and deeper contextual information, rather than just ogling at the terrain: as pretty as the Grand Canyon is to look at, knowing a little about the explorers, trails, and history makes it such a nicer experience to view. We've gone through the same evolution with the Liquid Galaxy: it used to be just a big Google Earth viewer, but we quickly realized the need for more context and usable information for a richer interaction with the viewers by combining Earth with street view, panoramic video, 3D objects, etc. It's why we built a content management system to create presentations with scenes. We anticipate that the knowledge cards and deeper information that Google is integrating here will only strengthen that interaction.
We are looking to roll out the new Google Earth to the fleet in the next couple of months. We need to do a lot of testing and then update the Liquid Galaxies with minimal (or no) disturbance to our clients, many of whom rely on the platform as a daily sales and showcasing tool for their businesses. As always, if you have any questions, please reach us directly via email or call.

Job opening: Web developer

We are looking for another talented software developer to consult with our clients and develop web applications for them in Ruby on Rails, Django, AngularJS, Java, .NET, Node.js, and other technologies. If you like to solve business problems and can take responsibility for getting a job done well without intensive oversight, please read on!

End Point is a 20-year-old web consulting company based in New York City, with 45 full-time employees working mostly remotely from home offices. We are experts in web development, databases, and DevOps, collaborating using SSH, Screen/tmux, chat, Hangouts, Skype, and good old phones.

We serve over 200 clients ranging from small family businesses to large corporations. We use open source frameworks in a variety of languages including JavaScript, Ruby, Java, Scala, Kotlin, C#, Python, Perl, and PHP, tracked by Git, running mostly on Linux and sometimes on Windows.

What is in it for you?

  • Flexible full-time work hours
  • Paid holidays and vacation
  • For U.S. employees: health insurance subsidy and 401(k) retirement savings plan
  • Annual bonus opportunity
  • Ability to move without being tied to your job location

What you will be doing:

  • Work from your home office, or from our offices in New York City and the Tennessee Tri-Cities area
  • Consult with clients to determine their web application needs
  • Build, test, release, and maintain web applications for our clients
  • Work with open source tools and contribute back as opportunity arises
  • Use your desktop platform of choice: Linux, macOS, Windows
  • Learn and put to use new technologies
  • Direct much of your own work

What you will need:

  • Professional experience building reliable server-side apps
  • Good front-end web skills with responsive design using HTML, CSS, and JavaScript, including jQuery, Angular, Backbone.js, Ember.js, etc.
  • Experience with databases such as PostgreSQL, MySQL, SQL Server, MongoDB, CouchDB, Redis, Elasticsearch, etc.
  • A focus on needs of our clients and their users
  • Strong verbal and written communication skills

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of gender, race, religion, color, national origin, sexual orientation, age, marital status, veteran status, or disability status.

Please email us an introduction to jobs@endpoint.com to apply. Include a resume, your GitHub or LinkedIn URLs, or whatever else that would help us get to know you. We look forward to hearing from you! Full-time employment seekers only, please -- this role is not for agencies or subcontractors.

The mystery of the disappearing SSH key

SSH (Secure Shell) is one of the programs I use every single day at work, primarily to connect to our client's servers. Usually it is a rock-solid program that simply works as expected, but recently I discovered it behaving quite strangely - a server I had visited many times before was now refusing my attempts to login. The underlying problem turned out to be a misguided decision by the developers of OpenSSH to deprecate DSA keys. How I discovered this problem is described below (as well as two solutions).

The use of the ssh program is not simply limited to logging in and connecting to remote servers. It also supports many powerful features, one of the most important being the ability to chain multiple connections with the ProxyCommand option. By using this, you can "login" to servers that you cannot reach directly, by linking together two or more servers behind the scenes.

As as example, let's consider a client named "Acme Anvils" that strictly controls access to its production servers. They make all SSH traffic come in through a single server, named dmz.acme-anvils.com, and only on port 2222. They also only allow certain public IPs to connect to this server, via whitelisting. On our side, End Point has a server, named portal.endpoint.com, that I can use as a jumping off point, which has a fixed IP that we can give to our clients to whitelist. Rather than logging in to "portal", getting a prompt, and then logging in to "dmz", I can simply add an entry in my ~/.ssh/config file to automatically create a tunnel between the servers - at which point I can reach the client's server by typing "ssh acmedmz":

##
## Client: ACME ANVILS
##

## Acme Anvil's DMZ server (dmz.acme-anvils.com)
Host acmedmz
User endpoint
HostName 555.123.45.67
Port 2222
ProxyCommand ssh -q greg@portal.endpoint.com nc -w 180s %h %p

Notice that the "Host" name may be set to anything you want. The connection to the client's server uses a non-standard port, and the username changes from "greg" to "endpoint", but all of that is hidden away from me as now the login is simply:

[greg@localhost]$ ssh acmedmz
[endpoint@dmz]$

It's unusual that I'll actually need to do any work on the dmz server, of course, so the tunnel gets extended another hop to the db1.acme-anvils.com server:

##
## Client: ACME ANVILS
##

## Acme Anvil's DMZ server (dmz.acme-anvils.com)
Host acmedmz
User endpoint
HostName 555.123.45.67
Port 2222
ProxyCommand ssh -q greg@portal.endpoint.com nc -w 180s %h %p

## Acme Anvil's main database (db1.acme-anvils.com)
Host acmedb1
User postgres
HostName db1
ProxyCommand ssh -q acmedmz nc -w 180s %h %p

Notice how the second ProxyCommand references the "Host" of the section above it. Neat stuff. When I type "ssh acemdb1", I'm actually connecting to the portal.endpoint.com server, then immediately running the netcat (nc) command in the background, then going through netcat to dmz.acme-anvils.com and running a second netcat command on *that* server, and finally going through both netcats to login to the db1.acme-anvils.com server. It sounds a little complicated, but quickly becomes part of your standard tool set once you wrap your head around it. After you update your .ssh/config file, you soon forget about all the tunneling and feel as though you are connecting directly to all your servers. That is, until something breaks, as it did recently for me.

The actual client this happened with was not "Acme Anvils", of course, and it was a connection that went through four servers and three ProxyCommands, but for demonstration purposes let's pretend it happened on a simple connection to the dmz.acme-anvils.com server. I had not connected to the server in question for a long time, but I needed to make some adjustments to a tail_n_mail configuration file. The first login attempt failed completely:

[greg@localhost]$ ssh acmedmz
endpoint@dmz.acme-anvils.com's password: 

Although the connection to portal.endpoint.com worked fine, the connection to the client server failed. This is not an unusual problem: it usually signifies that either ssh-agent is not running, or that I forgot to feed it the correct key via the ssh-add program. However, I quickly discovered that ssh-agent was working and contained all my usual keys. Moreover, I was able to connect to other sites with no problem! On a hunch, I tried breaking down the connections into manual steps. First, I tried logging in to the "portal" server. It logged me in with no problem. Then I tried to login from there to dmz.acme-anvils.com - which also logged me in with no problem! But trying to get there via ProxyCommand still failed. What was going on?

When in doubt, crank up the debugging. For the ssh program, using the -v option turns on some minimal debugging. Running the original command from my computer with this option enabled quickly revealed the problem:

[greg@localhost]$ ssh -v acmedmz
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /home/greg/.ssh/config
debug1: /home/greg/.ssh/config line 1227: Applying options for acmedmz
debug1: Reading configuration data /etc/ssh/ssh_config
...
debug1: Executing proxy command: exec ssh -q greg@portal.endpoint.com nc -w 180s 555.123.45.67 2222
...
debug1: Authenticating to dmz.acme-anvils.com:2222 as 'endpoint'
...
debug1: Host 'dmz.acme-anvils.com' is known and matches the ECDSA host key.
...
debug1: Skipping ssh-dss key /home/greg/.ssh/greg2048dsa.key - not in PubkeyAcceptedKeyTypes
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/greg/.ssh/greg4096rsa.key
debug1: Next authentication method: password
endpoint@dmz.acme-anvils.com's password: 

As highlighted above, the problem is that my DSA key (the "ssh-dss key") was rejected by my ssh program. As we will see below, DSA keys are rejected by default in recent versions of the OpenSSH program. But why was I still able to login when not hopping through the middle server? The solution lays in the fact that when I use the ProxyCommand, *my* ssh program is negotiating with the final server, and is refusing to use my DSA key. However, when I ssh to the portal.endpoint.com server, and then on to the next one, the second server has no problem using my (forwarded) DSA key! Using the -v option on the connection from portal.endpoint.com to dmz.acme-anvils.com reveals another clue:

[greg@portal]$ ssh -v endpoint@dmz.acme-anvils.com:2222
...
debug1: Connecting to dmz [1234:5678:90ab:cd::e] port 2222.
...
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/greg/.ssh/endpoint2.ssh
debug1: Authentications that can continue: publickey,password
debug1: Offering DSA public key: /home/greg/.ssh/endpoint.ssh
debug1: Server accepts key: pkalg ssh-dss blen 819
debug1: Authentication succeeded (publickey).
Authenticated to dmz ([1234:5678:90ab:cd::e]:2222).
...
debug1: Entering interactive session.
[endpoint@dmz]$

If you look closely at the above, you will see that we first offered an RSA key, which was rejected, and then we successfully offered a DSA key. This means that the endpoint@dm account has a DSA, but not a RSA, public key inside of its ~/.ssh/authorized_keys file. Since I was able to connect to portal.endpoint.com, its ~/.ssh/authorized_keys file must have my RSA key.

For the failing connection, ssh was able to use my RSA key to connect to portal.endpoint.com, run the netcat command, and then continue on to the dmz.acme-anvils.com server. However, this connection failed as the only key my local ssh program would provide was the RSA one, which the dmz server did not have.

For the working connection, ssh was able to connect to portal.endpoint.com as before, and then into an interactive prompt. However, when I then connected via ssh to dmz.acme-anvils.com, it was the ssh program on portal, not my local computer, which negotiated with the dmz server. It had no problem using a DSA key, so I was able to login. Note that both keys were happily forwarded to portal.endpoint.com, even though my ssh program refused to use them!

The quick solution to the problem, of course, was to upload my RSA key to the dmz.acme-anvils.com server. Once this was done, my local ssh program was more than happy to login by sending the RSA key along the tunnel.

Another solution to this problem is to instruct your SSH programs to recognize DSA keys again. To do this, add this line to your local SSH config file ($HOME/.ssh/config), or to the global SSH config file (/etc/ssh/config):

PubkeyAcceptedKeyTypes +ssh-dss

As mentioned earlier, this whole mess was caused by the OpenSSH program deciding to deprecate DSA keys. Their rationale for targeting all DSA keys seems a little weak at best: certainly I don't feel that my 2048-bit DSA key is in any way a weak link. But the writing is on the wall now for DSA, so you may as well replace your DSA keys with RSA ones (and an ed25519 key as well, in anticipation of when ssh-agent is able to support them!). More information about the decision to force out DSA keys can be found in this great analysis of the OpenSSH source code.

SELINUX=disabled? Read this and think twice!

Not long ago, one of our customers had their website compromised because of a badly maintained, not-updated WordPress. At End Point we love WordPress, but it really needs to be configured and hardened the right way, otherwise it's easy to end up in a real nightmare.

This situation is worsened even more if there's no additional security enforcement system to protect the environment on which the compromised site lives. One of the basic ways to protect your Linux server, especially RHEL/Centos based ones, is using SELinux.

Sadly, most of the interaction people has with SELinux happens while disabling it, first on the running system:

setenforce disabled
# or
setenforce 0

and then permanently by manually editing the file /etc/sysconfig/selinux to change the variable SELINUX=enforcing to SELINUX=disabled.

Is that actually a good idea though? While SELinux can be a bit of a headache to tune appropriately and can easily be misconfigured, here's something that could really convince you to think twice before disabling SELinux once and forever.

Back to our customer's compromised site. While going through the customer's system for some post-crysis cleaning, I found this hilarious piece of bash_history:

ls
cp /tmp/wacky.php .
ls -lFa
vim wacky.php
set
ls -lFa
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
fg
ls -lFa
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
php wacky.php > THE-EVIL 2>&1
vim THE-EVIL
ls -lFA
less wacky.php
ls
less THE-EVIL
less wacky.php
cat /selinux/enforce
ls
less THE-EVIL
exit

As you can see, what happened was that the attacker was able to manage having a shell connection as the customer user, and started using a PHP files injected in /tmp as a possible further vector of attack.

Sadly, for the attacker at least, what happened was that SELinux was setup in enforcing mode with some strict rules and prevented all kind of execution on that specific script so after a few frantic attempts the attacker surrendered.

Looking into the /var/log/audit/auditd.log file I found all the AVC="denied" errors that SELinux was shouting while forbidding the attacker to pursue his nefarious plan.

Hilarious and good props to SELinux for saving the day.

less THE-EVIL, more SELinux! 🙂

mysqldump issues after Percona 5.7 update

During a recent CentOS 7 update, among other packages, we updated our Percona 5.7 installation to version 5.7.17-13.

Quickly after that, we discovered that mysqldump stopped working, thus breaking our local mysql backup script (that complained loudly).

What happened?


The error we received was:

mysqldump: Couldn't execute 'SELECT COUNT(*) FROM INFORMATION_SCHEMA.SESSION_VARIABLES WHERE VARIABLE_NAME LIKE 'rocksdb\_skip\_fill\_cache'': The 'INFORMATION_SCHEMA.SESSION_VARIABLES' feature is disabled; see the documentation for 'show_compatibility_56' (3167)

After a bit of investigation, we discovered this was caused by this regression bug, apparently already fixed but not yet available on CentOS:

Everything revolves around INFORMATION_SCHEMA being deprecated in version 5.7.6, when Performance Schema tables has been added as a replacement.

Basically, a regression caused mysqldump to try and use deprecated INFORMATION_SCHEMA tables instead of the new Performance Schema.

How to fix it?


Immediate workaround is to add this line to /etc/my.cnf or (more likely) /etc/percona-server.conf.d/mysqld.cnf, depending on how your configuration files are organized:

show_compatibility_56=1

This flag was both introduced and deprecated in 5.7.6. It will be there for some time to help with the transition.

It seems safe and, probably, good to keep if you have anything still actively using INFORMATION_SCHEMA tables, that would obviously be broken if not updated to the new Performance Schema since 5.7.6.

With this flag, it is possible to preserve the old behavior and keep your old code in a working state, while you upgrade it. Also, according to the documentation, it should not impact or turn off the new behavior with Performance Schema.

More information on how to migrate to the new Performance Schema can be found here.

Linode IPv6 issues with NetworkManager on CentOS 7

In End Point, we use different hosting providers based on the specific task needs. One provider we use extensively with good results is Linode.

During a routine CentOS 7 system update, we noticed a very strange behavior where our IPv6 assigned server address was wrong after restarting the server.

IPv6 on Linode and SLAAC


Linode is offering IPv6 on all their VPS, and IPv6 dynamic addresses are assigned to servers using SLAAC.

In the provided CentOS 7 server image, this is managed by NetworkManager by default. After some troubleshooting, we noticed that during the update the NetworkManager package was upgraded from 1.0.6 to 1.4.0.

This was a major update, and it turned out that the problem was a change in the configuration defaults between the two version.

Privacy stable addressing


Since 1.2, NetworkManager added the Stable Privacy Addressing feature. This allows for some form of tracking prevention, with the IPv6 address to be stable on a network but changing when entering another network, and still remain unique.

This new interesting feature has apparently become the default after the update, with the ipv6.addr-gen-mode property set to "stable-privacy". Setting it to “eui64” maintains the old default behavior.

Privacy Extension


Another feature apparently also caused some problems on our VPS: the Privacy Extension. This is a simple mechanism that somewhat randomizes the network hardware’s (MAC) address, to add another layer of privacy. Alas, this is used in address generation, and that randomization seemed to be part of the problem we were seeing.

This too has become the default, with the ipv6.ip6-privacy property set to 1. Setting it to 0 turns off the feature.

To sum it up


In the end, after the update, we could restore the old behavior and resolve our issues by running, in a root shell:

nmcli connection modify "Wired connection 1" ipv6.ip6-privacy 0
nmcli connection modify "Wired connection 1" ipv6.addr-gen-mode eui64

After a reboot, the IPv6 address finally matched the one actually assigned by Linode, and everything was working ok again.

If you want to know more on Privacy Extensions and Privacy Stable Addressing, this great blog post by Lubomir Rintel helped us a lot understanding what was going on.

Half day GlusterFS training in Selangor, Malaysia

On January 21, 2017, I had an opportunity to join a community-organized training on storage focused on GlusterFS. GlusterFS is an open source cloud-based filesharing network. The training was not a strictly structured training as the topic approached knowledge sharing from various experts and introduced GlusterFS to the ones who were new to it. The first session was delivered by Mr Adzmely Mansor from NexoPrima. He shared a bit of his view on GlusterFS and technologies that are related to it.

Mr Haris, a freelance Linux expert, later led a GlusterFS technical class. Here we created two virtual machines (we used Virtualbox) to understand how GlusterFS works in a hands-on scenario. We used Ubuntu 16.04 as the guest OS during technical training. We used Digital Ocean's GlusterFS settings as a base of reference. The below commands detail roughly what we did during the training.

In GlusterFS the data section is called as "brick". Hence we could have a lot of "bricks" if we have it more than once :) . As Ubuntu already had the related packages in its repository, we could simply run apt-get for the package installation. Our class notes were loosely based from Digital Ocean's GlusterFS article here. (Note: the article was based on Ubuntu 12.04 so some of the steps could be omitted).

The GlusterFS packages could be installed as a superuser with the following command:

apt-get install glusterfs-server

Since we were using a bridged VM during the demo, we simply edited the /etc/hosts in the each VM so they could communicate between each other by using hostname instead of using typing the IP manually.

root@gluster2:~# grep gluster /etc/hosts
192.168.1.11 gluster1
127.0.0.1 gluster2

Here we will try to probe the remote host whether it is reachable:

root@gluster2:~# gluster peer probe gluster1
peer probe: success. Host gluster1 port 24007 already in peer list

The following commands create the storage volume. Later, whatever we put in the /data partition will be reachable on the other gluster node.

gluster volume create datastore1 replica 2 transport tcp gluster1:/data gluster2:/data
gluster volume create datastore1 replica 2 transport tcp gluster1:/data gluster2:/data force
gluster volume start datastore1

Most of the parts here could be retrieved from the link that I gave above. But let's see what will happen later on when the mounting part is done.

cd /datastore1/
root@gluster2:/datastore1# touch blog
root@gluster2:/datastore1# ls -lth
total 512
-rw-r--r-- 1 root root  0 Mar 14 21:33 blog
-rw-r--r-- 1 root root 28 Jan 21 12:15 ujian.txt

The same output could be retrieved from gluster1

root@gluster1:/datastore1# ls -lth
total 512
-rw-r--r-- 1 root root  0 Mar 14 21:33 blog
-rw-r--r-- 1 root root 28 Jan 21 12:15 ujian.txt


Mr. Adzmely gave explanation on the overall picture of GlusterFS



Mr. Haris explained on the technical implementation of GlusterFS

In terms of the application, the redundancy based storage is good for situations where you have a file being updated on several servers and you need to ensure the file is there for retrieval even if one of the servers is down. One audience member shared his experience deploying GlusterFS in his workplace (a university) for the purpose of new intake of student's registration. If anyone ever uses Samba filesystem or NFS, this kind of similar, but GlusterFS is much more advanced. I recommend additional reading here.

DBA Revenge: How To Get Back at Developers

In the spirit of April 1st, resurrecting this old classic post:


Maybe you work at one of those large corporations that has a dedicated DBA staff, separate from the development team. Or maybe you're lucky and just get to read about it on thedailywtf.com. But you've probably seen battles between database folk and the developers that "just want a table with "ID " VARCHAR(255), name VARCHAR(255), price VARCHAR(255), post_date VARCHAR(255). Is that so much to ask?!"

Well if you ever feel the need to get back at them, here's a few things you can try. Quoted identifiers let you name your objects anything you want, even if they don't look like a normal object name...

CREATE TABLE "; rollback; drop database postgres;--" ("'';
delete from table order_detail;commit;" INT PRIMARY KEY,
";commit;do $$`rm -rf *`$$ language plperlu;" TEXT NOT NULL);

COMMENT ON TABLE "; rollback; drop database postgres;--"
IS 'DON''T FORGET TO QUOTE THESE';

Good advice, that comment. Of course, assuming they learn, they'll be quoting everything you give them. So, drop a quote right in the middle of it:

CREATE TABLE "messages"";rollback;update products set price=0;commit;--"
("am i doing this right" text);

[local]:5432|production=# \dt *messages*
 List of relations
 Schema |                           Name                           | Type  |   Owner   
--------+----------------------------------------------------------+-------+-----------
 public | messages";rollback;update products set price=0;commit;-- | table | jwilliams
(1 row)
A copy & paste later...
[local]:5432|production=# SELECT "am i doing this right" FROM "messages";rollback;update products set price=0;commit;--";
ERROR:  relation "messages" does not exist
LINE 1: select "am i doing this right" from "messages";
                                            ^
NOTICE:  there is no transaction in progress
ROLLBACK
UPDATE 100
WARNING:  there is no transaction in progress
COMMIT

Then again, if this is your database, that'll eventually cause you a lot of headache. Restores aren't fun. But UTF-8 can be...

CREATE TABLE suoıʇɔɐsuɐɹʇ (ɯnu‾ɹǝpɹo SERIAL PRIMARY KEY,
ǝɯɐuɹǝsn text REFERENCES sɹǝsn, ןɐʇoʇ‾ɹǝpɹo NUMERIC(5,2));

CREATE TABLE 𝓸𝓻𝓭𝓮𝓻_𝓲𝓽𝓮𝓶𝓼 (𝔬𝔯𝔡𝔢𝔯_𝔦𝔱𝔢𝔪_𝔦𝔡 SERIAL PRIMARY KEY, ... );

Postgres WAL files: best compression methods


Turtle turtle by WO1 Larry Olson from US Army

The PostgreSQL database system uses the write-ahead logging method to ensure that a log of changes is saved before being applied to the actual data. The log files that are created are known as WAL (Write Ahead Log) files, and by default are 16 MB in size each. Although this is a small size, a busy system can generate hundreds or thousands of these files per hour, at which point disk space becomes an issue. Luckily, WAL files are extremely compressible. I examined different programs to find one that offered the best compression (as indicated by a smaller size) at the smallest cost (as indicated by wall clock time). All of the methods tested worked better than the venerable gzip program, which is suggested in the Postgres documentation for the archive_command option. The best overall solution was using the pxz program inside the archive_command setting, followed closely by use of the 7za program. Use of the built-in wal_compression option was an excellent solution as well, although not as space-saving as using external programs via archive_command.


A database system is a complex beast, involving many trade-offs. An important issue is speed: waiting for changes to get written to disk before letting the client proceed can be a very expensive solution. Postgres gets around this with the use of the Write Ahead Log, which generates WAL files indicating what changes were made. Creating these files is much faster than performing the actual updates on the underlying files. Thus, Postgres is able to tell the client that the work is "done" when the WAL file has been generated. Should the system crash before the actual changes are made, the WAL files are used to replay the changes. As these WAL files represent a continuous unbroken chain of all changes to the database, they can also be used for Point in Time Recovery - in other words, the WAL files can be used to rewind the database to any single point in time, capturing the state of the database at a specific moment.

Postgres WAL files are exactly 16 MB in size (although this size may be changed at compilation time, it is extremely unheard of to do this). These files primarily sit around taking up disk space and are only accessed when a problem occurs, so being able to compress them is a good one-time exchange of CPU effort for a lower file size. In theory, the time to decompress the files should also be considered, but testing revealed that all the programs decompressed so quickly that it should not be a factor.

WAL files can be compressed in one of two ways. As of Postgres 9.5, the wal_compression feature can be enabled, which instructs Postgres to compress parts of the WAL file in-place when possible, leading to the ability to store much more information per 16 MB WAL file, and thus reducing the total number generated. The second way is to compress with an external program via the free-form archive_command parameter. Here is the canonical example from the Postgres docs, showing use of the gzip program for archive_command:

archive_command = 'gzip < %p > /var/lib/pgsql/archive/%f'

It is widely known that gzip is no longer the best compression option for most tasks, so I endeavored to determine which program was the best at WAL file compression - in terms of final file size versus the overhead to create the file. I also wanted to examine how these fared versus the new wal_compression feature.


To compare the various compression methods, I examined all of the compression programs that are commonly available on a Linux system, are known to be stable, and which perform at least as good as the common utility gzip. The contenders were:

  • gzip - the canonical, default compression utility for many years
  • pigz - parallel version of gzip
  • bzip2 - second only to gzip in popularity, it offers better compression
  • lbzip2 - parallel version of bzip
  • xz - an excellent all-around compression alternative to gzip and bzip
  • pxz - parallel version of xz
  • 7za - excellent compression, but suffers from complex arguments
  • lrzip - compression program targeted at "large files"

For the tests, 100 random WAL files were copied from a busy production Postgres system. Each of those 100 files were compressed nine times by each of the programs above: from the "least compressed" option (e.g. -1) to the "best compressed" option (e.g. -9). The tests were performed on a 16-core system, with plenty of free RAM and nothing else running on the server. Results were gathered by wrapping each command with /usr/bin/time -verbose, which produces a nice breakdown of results. To gather the data, the "Elapsed (wall clock) time" was used, along with size of the compressed file. Here is some sample output of the time command:

  Command being timed: "bzip2 -4 ./0000000100008B91000000A5"
  User time (seconds): 1.65
  System time (seconds): 0.01
  Percent of CPU this job got: 99%
  Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.66
  Average shared text size (kbytes): 0
  Average unshared data size (kbytes): 0
  Average stack size (kbytes): 0
  Average total size (kbytes): 0
  Maximum resident set size (kbytes): 3612
  Average resident set size (kbytes): 0
  Major (requiring I/O) page faults: 0
  Minor (reclaiming a frame) page faults: 938
  Voluntary context switches: 1
  Involuntary context switches: 13
  Swaps: 0
  File system inputs: 0
  File system outputs: 6896
  Socket messages sent: 0
  Socket messages received: 0
  Signals delivered: 0
  Page size (bytes): 4096
  Exit status: 0

The wal_compression feature was tested by creating a new Postgres 9.6 cluster, then running the pgbench program twice to generate WAL files - once with wal_compression enabled, and once with it disabled. Then each of the resulting WAL files was compressed using each of the programs above.


Table 1.
Results of compressing 16 MB WAL files - average for 100 files
CommandWall clock time (s)File size (MB)
gzip -10.2714.927
gzip -20.2924.777
gzip -30.3664.667
gzip -40.3814.486
gzip -50.4924.318
gzip -60.7344.250
gzip -70.9914.235
gzip -82.0424.228
gzip -93.6264.227
CommandWall clock time (s)File size (MB)
bzip2 -11.5403.817
bzip2 -21.5313.688
bzip2 -31.5703.638
bzip2 -41.6253.592
bzip2 -51.6673.587
bzip2 -61.7073.566
bzip2 -71.7313.559
bzip2 -81.7523.557
bzip2 -91.7843.541
CommandWall clock time (s)File size (MB)
xz -10.9623.174
xz -21.1863.066
xz -35.9112.711
xz -46.2922.682
xz -56.6942.666
xz -68.9882.608
xz -79.1942.592
xz -89.1172.596
xz -99.1642.597
Table 2.
Results of compressing 16 MB WAL file - average for 100 files
CommandWall clock time (s)File size (MB)
lrzip
-l -L1
0.2965.712
lrzip
-l -L2
0.3195.686
lrzip
-l -L3
0.3415.645
lrzip
-l -L4
0.3705.639
lrzip
-l -L5
0.3895.627
lrzip
-l -L6
0.9015.501
lrzip
-l -L7
2.0905.462
lrzip
-l -L8
2.8295.471
lrzip
-l -L9
5.9835.463
CommandWall clock time (s)File size (MB)
lrzip
-z -L1
3.5823.353
lrzip
-z -L2
3.5773.342
lrzip
-z -L3
3.6013.326
lrzip
-z -L4
11.9712.799
lrzip
-z -L5
11.8902.741
lrzip
-z -L6
11.9712.751
lrzip
-z -L7
12.8612.750
lrzip
-z -L8
30.0802.483
lrzip
-z -L9
33.1712.482
CommandWall clock time (s)File size (MB)
7za -bd -mx=1
a test.7za
0.1283.182
7za -bd -mx=2
a test.7za
0.1393.116
7za -bd -mx=3
a test.7za
0.3013.059
7za -bd -mx=4
a test.7za
1.2513.001
7za -bd -mx=5
a test.7za
3.8212.620
7za -bd -mx=6
a test.7za
3.8412.629
7za -bd -mx=7
a test.7za
4.6312.591
7za -bd -mx=8
a test.7za
4.6712.590
7za -bd -mx=9
a test.7za
4.6632.599
Table 3.
Results of compressing 16 MB WAL file - average for 100 files
CommandWall clock time (s)File size (MB)
pigz -10.0514.904
pigz -20.0514.755
pigz -30.0514.645
pigz -40.0514.472
pigz -50.0514.304
pigz -60.0604.255
pigz -70.0814.225
pigz -80.1404.212
pigz -90.2514.214
CommandWall clock time (s)File size (MB)
lbzip2 -10.1353.801
lbzip2 -20.1513.664
lbzip2 -30.1513.615
lbzip2 -40.1513.586
lbzip2 -50.1513.562
lbzip2 -60.1513.545
lbzip2 -70.1503.538
lbzip2 -80.1513.524
lbzip2 -90.1503.528
CommandWall clock time (s)File size (MB)
pxz -10.1353.266
pxz -20.1753.095
pxz -31.2442.746
pxz -42.5282.704
pxz -55.1152.679
pxz -69.1162.604
pxz -79.2552.599
pxz -89.2672.595
pxz -99.3552.591
Table 4.
Results of Postgres wal_compression option
ModificationsTotal size of WAL files (MB)
No modifications208.1
wal_compression enabled81.0
xz -28.6
wal_compression enabled PLUS xz -29.4

Table 1 shows some baseline compression values for the three popular programs gzip, bzip2, and xz. Both gzip and bzip2 show little change in the file sizes as the compression strength is raised. However, xz has a relatively large jump when going from -2 to -3, although the time cost increases to an unacceptable 5.9 seconds. As a starting point, something well under one second is desired.

Among those three, xz is the clear winner, shrinking the file to 3.07 MB with a compression argument of -2, and taking 1.2 seconds to do so. Both gzip and bzip2 never even reach this file size, even when using a -9 argument. For that matter, the best compression gzip can ever achieve is 4.23 MB, which the other programs can beat without breakng a sweat.

Table 2 demonstrates two ways of invoking the lrzip program: the -l option (lzo compression - described in the lrzip documentation as "ultra fast") and the -z option (zpaq compression - "extreme compression, extreme slow"). All of those superlatives are supported by the data. The -l option runs extremely fast: even at -L5 the total clock time is still only .39 seconds. Unfortunately, the file size hovers around an undesirable 5.5 MB, no matter what compression level is used. The -z option produces the smallest file of all the programs here (2.48 MB) at a whopping cost of over 30 seconds! Even the smallest compression level (-L1) takes 3.5 seconds to produce a 3.4 MB file. Thus, lrzip is clearly out of the competition.


Compression in action (photo by Eric Armstrong)

The most interesting program is without a doubt 7za. Unlike the others, it is organized around creating archives, and thus doesn't do in-place compression as the others do. Nevertheless, the results are quite good. At the lowest level, it takes a mere 0.13 seconds to produce a 3.18 MB file. As it takes xz 1.19 seconds to produce a nearly equivalent 3.10 MB file, 7za is the clear winner ... if we had only a single core available. :)

It is rare to find a modern server with a single processor, and a crop of compression programs have appeared to support this new paradigm. First up is the amazing pigz, which is a parallel version of gzip. As Table 3 shows, pigz is extraordinarily fast on our 16 core box, taking a mere 0.05 seconds to run at compression level -1, and only 0.25 seconds to run at compression level -9. Sadly, it still suffers from the fact that gzip is simply not good at compressing WAL files, as the smallest size was 4.23 MB. This rules out pigz from consideration.

The bzip2 program has been nipping at the heels of gzip for many years, so naturally it has its own parallel version, known as lbzip2. As Table 3 shows, it is also amazingly fast. Not as fast as pigz, but with a speed of under 0.2 seconds - even at the highest compression level! There is very little variation among the compression levels used, so it is fair to simply state that lbzip2 takes 0.15 seconds to shrink the WAL file to 3.5 MB. A decent entry.

Of course, the xz program has a parallel version as well, known as pxz. Table 3 shows that the times still vary quite a bit, and reach into the 9 second range at higher compression levels, but does very well at -2, taking a mere 0.17 seconds to produce a 3.09 MB file. This is comparable to the previous winner, 7za, which took 0.14 seconds to create a 3.12 MB file.

So the clear winners are 7za and pxz. I gave the edge to pxz, as (1) the file size was slightly smaller at comparable time costs, and (2) the odd syntax for 7za for both compressing and decompressing was annoying compared with the simplicity of "xz -2" and "xz -d".

Now, what about the built-in compression offered by the wal_compression option? As Table 4 shows, the compression for the WAL files I tested went from 208 MB to 81 MB. This is a significant gain, but only equates to compressing a single WAL file to 6.23 MB, which is a poor showing when compared to the compression programs above. It should be pointed out that the wal_compression option is sensitive to your workload, so you might see reports of greater and lesser compressions.

Of interest is that the WAL files generated by turning on wal_compression are capable of being further compressed by the archive_command option, and doing a pretty good job of it as well - going from 81 MB of WAL files to 9.4 MB of WAL files. However, using just xz in the archive_command without wal_compression on still yielded a smaller overall size, and means less CPU because the data is only compressed once.

It should be pointed out that wal_compression has other advantages, and that comparing it to archive_command is not a fair comparison, but this article was primarily about the best compression option for storing WAL files long-term.

Thus, the overall winner is "pxz -2", followed closely by 7za and its bulky arguments, with honorable mention given to wal_compression. Your particular requirements might guide you to another conclusion, but hopefully nobody shall simply default to using gzip anymore.


Thanks to my colleague Lele for encouraging me to try pxz, after I was already happy with xz. Thanks to the authors of xz, for providing an amazing program that has an incredible number of knobs for tweaking. And a final thanks to the authors of the wal_compression feature, which is a pretty nifty trick!

Wrocloverb part 2 - The Elixir Hype

One of the main reasons I attend Wrocloverb almost every year, is that it's a great forum for confronting ideas. It's almost a tradition to have at least 2 very enjoyful discussion panels during this conference. One of them was devoted to Elixir and why Ruby [1] community is so hyping about it.


Why Elixir is “sold” to us as “new better Ruby” while its underlying principles are totally different? Won’t it result in Elixir programmers that do not understand Elixir (like Rails programmers that do not know Ruby)?


Panelists discussed briefly the history of Elixir:

Jose Valim (who created Elixir) was working on threading in Rails and he was searching for better approaches for threading in web frameworks. He felt like lots of things were lacking in Erlang and Elixir is his approach for better Exceptions, better developer experience.

Then they've jumped to Elixir's main goals which are:
  • Compatibility with Erlang (all datatypes)
  • Better tooling
  • Improving developer's experience

After that, they've started speculating about problems that Elixir solves and RoR doesn't:

Ruby on Rails addresses many problems in ways that may be somehow archaic to us in the ever-scaling world of 2017. There are many approaches to it - e.g. "actor model" which is implemented in Ruby by Celluloid, in Scala with Akka and also Elixir and Phoenix (Elixir's rails-like framework) has it's own actor model.

Phoenix ("Rails for Elixir") is just an Elixir app - unlike Rails, it is not separate from Elixir. Moreover Elixir is exactly the same language as Erlang so:

Erlang = Elixir = Phoenix

Great comment:


Then a discussion followed during which panelists speculated about the price of the jump from Rails to Elixir:

Java to Rails jump was caused by business/productivity. There's no such jump for Phoenix/Elixir. Elixir code is more verbose (less instance variables, all args are passed explicitly to all functions)

My conclusions

A reason why this discussion was somehow shallow and pointless was that those two world have different problems. Great comment:

Elixir solves a lot of technical problems with scaling thanks to Erlang's virtual machine. Such problems are definitely only a small part of what typical ruby problems solvers deal with on a daily basis. Hearing Elixir and Ruby on Rails developers talk past each other was probably the first sign of the fact that there's no hyping technology right now. Each problem can be addressed by many tech tools and frameworks.

[1] Wrocloverb describes itself as a "best Java conference in Ruby world". It's deceiving:

Wrocloverb 2017 part 1

Wrocloverb is a single-track 3-day conference that takes place in Wrocław (Poland) every year in March.

Here's a subjective list of most interesting talks from the first day

# Kafka / Karafka (by Maciej Mensfeld)

Karafka is another library that simplifies Apache Kafka usage in Ruby. It lets Ruby on Rails apps benefit from horizontally scalable message busses in a pub-sub (or publisher/consumer) type of network.

Why Kafka is (probably) better message/task broker for your app:
- broadcasting is a real power feature of kafka (http lacks that)
- author claims that its easier to support it  rather than ZeroMQ/RabbitMQ
- it's namespaced with topics (similar to Robot Operating System)
- great replacement for ruby-kafka and Poseidon



# Machine Learning to the Rescue (Mariusz Gil)


This talk was devoted to Machine Learning success (and failure) story of the author.

Author underlined that Machine Learning is a process and proposed following workflow:


  1. define a problem
  2. gather you data
  3. understand your data
  4. prepare and condition the data
  5. select & run your algorithms
  6. tune algorithms parameters
  7. select final model
  8. validate final model (test using production data)
Mariusz described few ML problems that he has dealt with in the past. One of them was a project designed to estimate cost of a code review. He outlined the process of tuning the input data. Here's a list of what comprised the input for a code review estimation cost:
  • number of lines changed
  • number of files changed
  • efferent coupling
  • afferent coupling
  • number of classes
  • number of interfaces
  • inheritance level
  • number of method calls
  • lloc metric
  • lcom metric (whether single responsibility pattern is followed or not)

# Spree lightning talk by sparksolutions.co

One of the lightning talks was devoted to Spree. Here's some interesting latest data from spree world:

  • number of contributors of spree - 700
  • it's very modular modular
  • it's api driven
  • it's one of the biggest repos on github
  • very large number of extensions
  • it drives thousands of stores worldwide
  • Spark Solutions is a maintainer
  • Popular companies that use spree: Go Daddy, Goop, Casper, Bonobos, Littlebits, Greetabl
  • it support rails 5, rails 4.2 and rails 3.x
Author also released newest 3.2.0 stable version during the talk:








Shell efficiency: mkdir and mv

Little tools can be a nice improvement. Not everything needs to be thought-leaderish.

For example, once upon a time in my Unix infancy I didn't know that mkdir has the -p option to make intervening directories automatically. So back then, in order to create the path a/b/c/ I would've run: mkdir a; mkdir b; mkdir c when I could instead have simply run: mkdir -p a/b/c.

In working at the shell, particularly on my own local machine, I often find myself wanting to move one or several files into a different location, to file them away. For example:

mv -i ~/Downloads/Some\ Long\ File\ Name.pdf ~/some-other-long-file-name.tar.xz ~/archive/new...

at which point I realize that the subdirectory of ~/archive that I want to move those files into does not yet exist.

I can't simply move to the beginning of the line and change mv to mkdir -p without removing my partially-typed ~/archive/new....

I can go ahead and remove that, and then after I run the command I have to change the mkdir back to mv and add back the ~/archive/new....

In one single day I found I was doing that so often that it became tedious, so I re-read the GNU coreutils manpage for mv to see if there was a relevant option I had missed or a new one that would help. And I searched the web to see if a prebuilt tool is out there, or if anyone had any nice solutions.

To my surprise I found nothing suitable, but I did find some discussion forums full of various suggestions and many brushoffs and ill-conceived suggestions that either didn't work for me or seemed much overengineered.

The solution I came up with was very simple. I've been using it for a few months and am happy enough with it to share it and see if it helps anyone else.

In zsh (my main local shell) add to ~/.zshrc:

mkmv() {
    mkdir -p -- "$argv[-1]"
    mv "$@"
}

And in bash (which I use on most of the many servers I access remotely) add to ~/.bashrc:

mkmv() {
    mkdir -p -- "${!#}"
    mv "$@"
}

To use: Once you realize you're about to try to move files or directories into a nonexistent directory, simply go to the beginning of the line (^A in standard emacs keybindings) and type mk in front of the mv that was already there:

mkmv -i ~/Downloads/Some\ Long\ File\ Name.pdf ~/some-other-long-file-name.tar.xz ~/archive/new...

It makes the directory (or directories) and then completes the move.

There are a few important considerations that I didn't foresee in my initial naive implementation:

  • Having the name be somethingmv meant less typing than something requiring me to remove the mv.
  • For me, it needs to support not just moving one thing to one destination, but rather a whole list of things. That meant accessing the last argument (the destination) for the mkdir.
  • I also needed to allow through arguments to mv such as -i, -v, and -n, which I often use.
  • The -- argument to mkdir ensures that we don't accidentally end up with any other options and that we can handle a destination with a leading - (however unlikely that is).
  • The mv command needs to have a double-quoted "$@" so that the original parameters are each expanded into double-quoted arguments, allowing for spaces and other shell metacharacters in the paths. (See the zsh and bash manpages for details on the important difference in behavior of "$@" compared to "$*" and either of them unquoted.)

This doesn't support GNU extensions to mv such as a --target-directory that precedes the source paths. I don't use that interactively, so I don't mind.

Because this is such a small thing, I avoided for years bothering to set it up. But now that I use it all the time I'm glad I have it!

360° Panoramic Video on Liquid Galaxy



End Point’s Liquid Galaxy recently had an exciting breakthrough! We can now show 360° panoramic video effectively and seamlessly on our Liquid Galaxy systems.

Since our integration of seamless 360° panovideo, our marketing and content teams have been loading up our system with the best 360° panoramic videos. The Liquid Galaxy system at our headquarter office has 360° panoramic videos from National Geographic, The New York Times, Google Spotlight, gaming videos, and more.


The technical challenge with panoramic video is to get multiple instances of the video player app to work in sync. On the Liquid Galaxy, you're actually seeing 7 different video player apps all running at the same time and playing the same video file, but with each app showing a slightly adjusted slice of the 360° panoramic video. This synchronization and angle-adjustment is at the heart of the Liquid Galaxy platform, and allows a high-resolution experience and surrounding immersion that doesn't happen on any other video wall system.

We anticipate that our flawless 360° panoramic video will resonate with many industries, one of whom is the gaming industry. Below we’ve included a video of 360° gaming videos, and how they appear on Liquid Galaxy. Mixed with all the other exciting capabilities on Liquid Galaxy, we anticipate the ability to view angle-adjusted and 3D-immersive 360° video on the system will be a huge crowd-pleaser.

If you are interested in learning more about 360° panoramic video on Liquid Galaxy, don’t hesitate to contact us or visit our Liquid Galaxy website for more information.

Become A File Spy With This One Easy Trick! Sys Admins Love This!

We had an interesting problem to track down. (Though I suppose I wouldn't be writing about it if it weren't, yes?) Over the years a client had built up quite the collection of scripts executed by cron to maintain some files on their site. Some of these were fairly complex, taking a long while to run, and overlapping with each other.

One day, the backup churn hit a tipping point and we took notice. Some process, we found, seemed to be touching an increasing number of image files: The contents were almost always the same, but the modification timestamps were updated. But digging through the myriad of code to figure out what was doing that was proving to be somewhat troublesome.

Enter auditd, already present on the RHEL host. This allows us to attach a watch on the directory in question, and track down exactly what was performing the events. -- Note, other flavors of Linux, such as Ubuntu, may not have it out of the box. But you can usually install it via the the auditd package.
(output from a test system for demonstration purposes)
# auditctl -w /root/output
# tail /var/log/audit/audit.log
type=SYSCALL msg=audit(1487974252.630:311): arch=c000003e syscall=2 success=yes exit=3 a0=b51cf0 a1=241 a2=1b6 a3=2 items=2 ppid=30272 pid=30316 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=1 comm="script.sh" exe="/usr/bin/bash" key=(null)
type=CWD msg=audit(1487974252.630:311):  cwd="/root"
type=PATH msg=audit(1487974252.630:311): item=0 name="output/files/" inode=519034 dev=fd:01 mode=040755 ouid=0 ogid=0 rdev=00:00 objtype=PARENT
type=PATH msg=audit(1487974252.630:311): item=1 name="output/files/1.txt" inode=519035 dev=fd:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 objtype=CREATE
The most helpful logged items include the executing process's name and path, the file's path, operation, pid and parent pid. But there's a good bit of data there per syscall.

Don't forget to auditctl -W /root/output to remove watch. auditctl -l will list what's currently out there:
# auditctl -l
-w /root/output -p rwxa
That's the short version. auditctl has a different set of parameters that are a little bit more verbose, but have more options. The equivalent of the above would be: auditctl -a always,exit -F dir=/root/output -F perm=rwxa ... with options for additional rule fields/filters on uid, gid, pid, whether or not the action was successful, and so on.

FOSDEM 2017: experience, community and good talks

In case you happen to be short on time: my final overall perspective about FOSDEM 2017 is that it was awesome... with very few downsides.

If you want the longer version, keep reading cause there's a lot to know and do at FOSDEM and never enough time, sadly.

This year I actually took a different approach than last time and decided to concentrate on one main track per day, instead of (literally) jumping from one to the other. While I think that overall this may be a good approach if most of the topics covered in a track are of your interest, that comes at the cost of missing one of the best aspects of FOSDEM which is "variety" in contents and presenters.

Day 1: Backup & Recovery

For the first day I chose the Backup & Recovery track which hosted talks revolving around three interesting and useful projects: namely REAR (Relax and Recovery), DRLM, a wrapper and backup management tool based on REAR and Bareos, which is a backup solution forked from Bacula in 2010 and steadily proceeding and improving since then. Both REAR and DLRM were explained and showcased by some of the respective projects main contributors and creators. As a long time system administrator, I particularly appreciated the pride in using Bash as the main "development platform" for both projects. As Johannes Meixner correctly mentioned, using bash facilitates introduces these tools into your normal workflow with knowledge that you'll most likely already have as a System Administrator or DevOps, thus allowing you to easily "mold" these scripts to your specific needs without spending weeks to learn how to interact with them.

During the Day 1 Backup & Recovery track there were also a few speeches from two Bareos developers (Jörg Steffens and Stephan Dühr) that presented many aspects of their great project, ranging from very introductory topics, to providing a common knowledge ground for the audience, up to more in depth topics like software capabilities extension through Python Plugins, or a handful of best practices and common usage scenarios. I also enjoyed the speech about automated testing in REAR, presented by Gratien D'haese, which showed how to leverage common testing paradigms and ideas to double-check a REAR setup for potential unexpected behaviors after updates or on new installations or simply as a fully automated monitoring tool to do sanity checks on the backup data flow. While this testing project was new, it's already functional and impressive to see at work.

Day 2: Cloud Microservices

On the second day I moved in a more "cloudy" section of the FOSDEM where most of the conferences revolved around Kubernetes, Docker and more in general the microservices landscape. CoreOS (the company behind the open source distribution) was a major contributor and I liked their Kubernetes presentation by Josh Wood and Luca Bruno which respectively explained the new Kubernetes Operators feature and how containers work under the hood in Kubernetes.

Around lunch time there was a "nice storm of lightning talks" which kept most of the audience firmly on their seats, especially since the Microservices track room didn't have a free seat for the entire day. I especially liked the talk from Spyros Trigazis about how CERN created and is maintaining a big OpenStack Magnum (the container integrated version of OpenStack) cloud installation for their internal use.

Then it was Chris Down's turn and, while he's a developer from Facebook, his talk gave the audience a good perspective on the future of CGROUPs in the Linux kernel and how they are already relatively safe and usable, even if not yet officially marked as production ready. While I already knew and used "sysdig" in past as a troubleshooting and investigation tool, it was nice to see one of the main developers, Jorge Salamero, using it and showing alternative approaches such as investigating timeout issues between Kubernetes Docker containers by just sysdig and its many modules and filters. It was really impressive seeing how easy it is to identify cross-containers issues and data flow.

Atmosphere

There were a lot of Open Source communities with "advertising desks" and I had a nice talk with a few interesting developers from the CoreOS team or from FSFE (Free Software Foundation Europe). Grabbing as many computer stickers as possible is also mandatory at FOSDEM, so I took my share and my new Thinkpad is way more colorful now. In fact, on a more trivial note, this year the FOSDEM staff decided to sell on sale all the laptops that were used during the video encoding phase for the streaming videos before the upload. These laptops were all IBM Thinkpad X220 and there were only a handful of them (~30) at a very appealing price. In fact, this article is being written from one of those very laptops now as I was one of the lucky few which managed to grab one before they were all gone within an hour or so. So if you're short of a laptop and happen to be at FOSDEM next year, keep your eyes open cause I think they'll do it again!

So what's not to like in such a wonderful scenario? While I admit that there was a lot to be seen and listened to, I sadly didn't see any "ground-shaking" innovation this year at FOSDEM. I did see many quality talks and I want to send a special huge "thank you" to all the speakers for the effort and high quality standards that they keep for their FOSDEM talks - but I didn't see anything extraordinarily new from what I can remember.

Bottom line is that I still have yet to find someone who was ever disappointed at FOSDEM, but the content quality varies from presenter to presenter and from year to year, so be sure to check the presentations you want to attend carefully before hand.

I think that the most fascinating part of FOSDEM is meeting interesting, smart, and like-minded people that would be difficult to reach otherwise.

In fact, while a good share of the merit should be attributed to the quality of the content presented, I firmly believe that the community feeling that you get at FOSDEM is hard to beat and easy to miss when skipped even for one year.




I'll see you all next year at FOSDEM then.

Full Cesium Mapping on the Liquid Galaxy

A few months ago, we shared a video and some early work we had done with bringing the Cesium open source mapping application to the Liquid Galaxy. We've now completed a full deployment for Smartrac, a retail tracking analytics provider, using Cesium in a production environment! This project presented a number of technical challenges beyond the early prototype work, but also brought great results for the client and garnered a fair amount of attention in the press, to everyone's benefit.

Cesium is an open source mapping application that separates out the tile sets, elevation, and markup language. This separation allows for flexibility at each major element:

  • We can use a specific terrain elevation data set while substituting any one of several map "skins" to drape on that elevation: a simple color coded map, a nighttime illumination map, even a water-colored "pirate map" look.
  • For the terrain, we can download as much or as little is needed: As the Cesium viewer zooms in on a given spot, Cesium uses a sort of fractal method to download finer and finer resolution terrains in just the surrounding area, eventually getting to the data limit of the set. This gradual approach balances download requirements with viewable accuracy. In our case, we downloaded an entire terrain set up to level 14 (Earth from high in space is level 1, then zooms in to levels 2, 3, 4, etc.) which gave us a pretty good resolution while conserving disk space. (The data up to level 14 totaled about 250 GB.)
  • Using some KML tools we have developed for past projects and adapting to CZML ("cesium language", get it?), we were able to take Smartrac's supply chain data and show a comprehensive overview of the product flow from factories in southeast Asia through a distribution center in Seattle and on to retail stores throughout the Western United States.
The debut for this project was the National Retail Federation convention at the Javitz Center in New York City. Smartrac (and we also) wanted to avoid any show-stoppers that might come from a sketchy internet connection. So, we downloaded the map tiles, a terrain set, built our visualizations, and saved the whole thing locally on the head node of the Liquid Galaxy server stack, which sat in the back of the booth behind the screens.

The show was a great success, with visitors running through the visualizations almost non-stop for 3 days. The client is now taking the Liquid Galaxy and the Cesium visualizations on to another convention in Europe next month. The NRF, IBM, and several other ecommerce bloggers wrote up the platform, which brings good press for Smartrac, Cesium, and the Liquid Galaxy.

Liquid Galaxy Success at U.S. Embassy’s Cultural Center

The U.S. Embassy to Jakarta features a high-tech cultural center called “@america”. @america’s mission is to provide a space for young Indonesians to learn more about the United States through discussions, cultural performances, debates, competitions and exhibitions.

Since Google generously donated it six years ago, @america has had a Liquid Galaxy deployed for use at the center. Not until recently, however, has @america taken advantage of our Content Management System. This past year, End Point developed and rolled out a revamped and powerful Content Management System for the fleet of Liquid Galaxies we support. With the updated Content Management System, End Point’s Content Team created a specialized Interactive Education Portal on @america's Liquid Galaxy. The Education Portal featured over 50 high quality, interactive university experiences. Thanks to the CMS, the Liquid Galaxy now shows campus videos, university statistics, and fly-tos and orbits around the schools. The campus videos included both recruitment videos, as well as student-created videos on topics like housing, campus sports, and religion. These university experiences allow young Indonesians the opportunity to learn more about U.S. Universities and culture.

@America and the US Embassy report that from December through the end of January, already more than 16,500 Indonesians have had the opportunity to engage with the Education Portal while visiting @america. We are thankful to have had the opportunity to help the US Embassy use their Liquid Galaxy for such a positive educational cause.


Liquid Galaxy systems are installed at educational institutions, from embassies to research libraries, around the world. If you’d like to learn more about Liquid Galaxy, please visit our Liquid Galaxy website or contact us here.

Smartrac's Liquid Galaxy at National Retail Federation

Last week, Smartrac exhibited at the retail industry’s BIG Show, NRF 2017, using a Liquid Galaxy with custom animations to showcase their technology.

Smartrac provides data analytics to retail chains by tracking physical goods with NFC and Bluetooth tags that combine to track goods all the way from the factory to the distribution center to retail stores. It's a complex but complete solution. How best to visualize all that data and show the incredible value that Smartrac brings? Seven screens with real time maps in Cesium, 3D store models in Unity, and browser-based dashboards, of course. End Point has been working with Smartrac for a number of years as a development resource on their SmartCosmos platform, helping them with UX and back-end database interfaces. This work included development of REST-based APIs for data handling, as well as a Virtual Reality project utilizing the Unity game engine to visualize data and marketing materials directly on several platforms including the Oculus Rift, the Samsung Gear 7 VR, and WebGL. Bringing that back-end work forward in a highly visible platform for the retail conference was a natural extension for them, and the Liquid Galaxy fulfilled that role perfectly. The large Liquid Galaxy display allowed Smartrac to showcase some of their tools on a much larger scale.

For this project, End Point deployed two new technologies for the Liquid Galaxy:
  • Cesium Maps - Smartrac had two major requirements for their data visualizations: show the complexity of the solution and global reach, while also making the map data offline wherever possible to avoid the risk of sketchy Internet connections at the convention center (a constant risk). For this, we deployed Cesium instead of Google Earth, as it allowed for a fully offline tileset that we could store locally on the server, as well as providing a rich data visualization set (we've shown other examples before).
  • Unity3D Models - Smartrac also wanted to show how their product tracking works in a typical retail store. Rather than trying to wire a whole solution during the short period for a convention, however, they made the call to visualize everything with Unity, a very popular 3D rendering engine. Given the multiple screens of the Liquid Galaxy, and our ability to adjust the view angle for each screen in the arch around the viewers, this Unity solution would be very immersive and able to tell their story quite well.
Smartrac showcased multiple scenes that incorporated 3D content with live data, labels superimposed on maps, and a multitude of supporting metrics. End Point developers worked on custom animation to show their tools in a engaging demo. During the convention, Smartrac had representatives leading attendees through the Liquid Galaxy presentations to show their data. Video of these presentations can be viewed below.



Smartrac’s Liquid Galaxy received positive feedback from everyone who saw it, exhibitors and attendees alike. Smartrac felt it was a great way to walk through their content, and attendees both enjoyed the content and were intrigued by the display on which they were seeing the content. Many attendees who had never seen Liquid Galaxy inquired about it.

If you’d like to learn more about Liquid Galaxy or new projects we are working on or having custom content developed, please visit our Liquid Galaxy website or contact us here.

TriSano Case Study


Overview

End Point has been working with state and local health agencies since 2008. We host disease outbreak surveillance and management systems and have expertise providing clients with the sophisticated case management tools they need to deliver in-house analysis, visualization, and reporting - combined with the flexibility to comply with changing state and federal requirements. End Point provides the hosting infrastructure, database, reporting systems, and customizations that agencies need in order to service to their populations.

Our work with health agencies is a great example of End Point’s ability to use our experience in open source technology, Ruby on Rails, manage and back up large secure datasets, and integrate reporting systems to build and support a full-stack application. We will discuss one such client in this case study.

Why End Point?

End Point is a good fit for this project because of our expertise in several areas including reporting and our hosting capabilities. End Point has had a long history of consultant experts in PostgreSQL and Ruby on Rails, which are the core software behind this application.

Also, End Point specializes in customizing open-source software, which can save not-for-profit and state agencies valuable budget dollars they can invest in other social programs.

Due to the secure nature of the medical data in these database, we and our clients must adhere to all HIPAA and CDC policies regarding hosting of data handling, server hosting, and staff authorization and access auditing.




Team

Steve Yoman

Steve serves as the project manager for both communication and internal development for End Point’s relationship with the client. Steve brings many years in project management to the table for this job and does a great job keeping track of every last detail, quote, and contract item.


Selvakumar Arumugam

Selva is one of those rare engineers who is gifted with both development and DevOps expertise. He is the main developer on daily tasks related to the disease tracking system. He also does a great job navigating a complex hosting environment and has helped the client make strides towards their future goals.


Josh Tolley

Josh is one of End Point’s most knowledgeable database and reporting experts. Josh’s knowledge of PostgreSQL is extremely helpful to make sure that the data is secure and stable. He built and maintains a standalone reporting application based on Pentaho.




Application

The disease tracking system consists of several applications including a web application, reporting application, two messaging areas, and SOAP services that relay data between internal and external systems.

TriSano: The disease tracking web app is an open source Ruby on Rails application based on the TriSano product, originally built at the Collaborative Software Initiative. This is a role-based web application where large amounts of epidemiological data can be entered manually or by data transfer.

Pentaho: Pentaho is a PostgreSQL reporting application that allows you to run a separate reporting service or embed reports into your website. Pentaho has a community version and an enterprise version, which is what is used on this particular project. This reporting application provides OLAP services, dashboarding, and generates ad hoc and static reports. Josh Tolley customized Pentaho so that the client can download or create custom reports depending on their needs.

Two Messaging Area applications: The TriSano system also serves as the central repository for messaging feeds used to collect data from local health care providers, laboratories throughout the state, and the CDC.

SOAP services run between the TriSano web app, the Pentaho reporting application, and the client’s data systems translate messages into the correct formats and relay the information to each application.

Into the Future

Based on the success over 9+ years working on this project, the client continues to work with their End Point team to manage their few non open-source software licenses, create long term security strategies, and plan and implement all of their needs related to the continuous improvement and changes in epidemiology tracking. We partner with the client so they can focus their efforts on reading results and planning for the future health of their citizens. This ongoing partnership is something End Point is very proud to be a part of and we hope to continue our work in this field well into the future.