News

Welcome to End Point’s blog

Ongoing observations by End Point people

The New Earth



As many of you may have seen, earlier this week Google released a major upgrade to the Google Earth app. Overall, it's much improved, sharper, and a deeper experience for viewers. We will be upgrading/incorporating our managed fleet of Liquid Galaxies over the next two months after we've had a chance to fully test its functionality and polish the integration points, but here are some observations for how we see this updated app impacting the overall Liquid Galaxy experience.

  • Hooray! The new Earth is here! The New Earth is here! Certainly, this is exciting for us. The Google Earth app plays a key central role in the Liquid Galaxy viewing experience, so a major upgrade like this is a most welcome development. So far, the reception has been positive. We anticipate it will continue to get even better as people really sink their hands into the capabilities and mashup opportunities this browser-based Earth presents.

  • We tested some pre-release versions of this application, and successfully integrated them with the Liquid Galaxy and are very happy with how we are able to view-synchronize unique instances of the new Google Earth across displays with appropriate geometrically configured offsets.

  • What to look for in this new application:
    • Stability: The new Google Earth runs as a NaCl application in a Chrome browser. This is an enormous advance for Google Earth. As an application in Chrome it is instantly accessible to billions of new users with their established expectations. Because the new Google Earth uses Chrome the Google Earth developers will no longer need to engage in the minutiae of having to support multiple desktop operating systems, but now can instead concentrate on the core-functionality of Google Earth and leverage the enormous amount of work that the Chrome browser developers do to make Chrome a cross-platform application.
    • Smoother 3D: The (older) Google Earth sometimes has a sort of "melted ice cream" look to the 3D buildings in many situations. Often, buildings fail to fully load from certain viewpoints. From what we're seeing so far, the 3D renderings in the New Earth appear to be a lot sharper and cleaner.
    • Browser-based possibilities: As focus turns more and more to browser-based apps, and as JavaScript libraries continue to mature, the opportunities and possibilities for how to display various data types, data visualizations, and interactions really start to multiply. We can already see this with the sort of deeper stories and knowledge cards that Google is including in the Google Earth interface. We hope to take the ball and run with it, as the Liquid Galaxy can already handle a host of different media types. We might exploit layers, smart use controls, realtime content integration from other available databases, and... okay, I'm getting ahead of myself.

  • The New Google Earth makes a major point of featuring stories and deeper contextual information, rather than just ogling at the terrain: as pretty as the Grand Canyon is to look at, knowing a little about the explorers, trails, and history makes it such a nicer experience to view. We've gone through the same evolution with the Liquid Galaxy: it used to be just a big Google Earth viewer, but we quickly realized the need for more context and usable information for a richer interaction with the viewers by combining Earth with street view, panoramic video, 3D objects, etc. It's why we built a content management system to create presentations with scenes. We anticipate that the knowledge cards and deeper information that Google is integrating here will only strengthen that interaction.
We are looking to roll out the new Google Earth to the fleet in the next couple of months. We need to do a lot of testing and then update the Liquid Galaxies with minimal (or no) disturbance to our clients, many of whom rely on the platform as a daily sales and showcasing tool for their businesses. As always, if you have any questions, please reach us directly via email or call.

Job opening: Web developer

We are looking for another talented software developer to consult with our clients and develop web applications for them in Ruby on Rails, Django, AngularJS, Java, .NET, Node.js, and other technologies. If you like to solve business problems and can take responsibility for getting a job done well without intensive oversight, please read on!

End Point is a 20-year-old web consulting company based in New York City, with 45 full-time employees working mostly remotely from home offices. We are experts in web development, databases, and DevOps, collaborating using SSH, Screen/tmux, chat, Hangouts, Skype, and good old phones.

We serve over 200 clients ranging from small family businesses to large corporations. We use open source frameworks in a variety of languages including JavaScript, Ruby, Java, Scala, Kotlin, C#, Python, Perl, and PHP, tracked by Git, running mostly on Linux and sometimes on Windows.

What is in it for you?

  • Flexible full-time work hours
  • Paid holidays and vacation
  • For U.S. employees: health insurance subsidy and 401(k) retirement savings plan
  • Annual bonus opportunity
  • Ability to move without being tied to your job location

What you will be doing:

  • Work from your home office, or from our offices in New York City and the Tennessee Tri-Cities area
  • Consult with clients to determine their web application needs
  • Build, test, release, and maintain web applications for our clients
  • Work with open source tools and contribute back as opportunity arises
  • Use your desktop platform of choice: Linux, macOS, Windows
  • Learn and put to use new technologies
  • Direct much of your own work

What you will need:

  • Professional experience building reliable server-side apps
  • Good front-end web skills with responsive design using HTML, CSS, and JavaScript, including jQuery, Angular, Backbone.js, Ember.js, etc.
  • Experience with databases such as PostgreSQL, MySQL, SQL Server, MongoDB, CouchDB, Redis, Elasticsearch, etc.
  • A focus on needs of our clients and their users
  • Strong verbal and written communication skills

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of gender, race, religion, color, national origin, sexual orientation, age, marital status, veteran status, or disability status.

Please email us an introduction to jobs@endpoint.com to apply. Include a resume, your GitHub or LinkedIn URLs, or whatever else that would help us get to know you. We look forward to hearing from you! Full-time employment seekers only, please -- this role is not for agencies or subcontractors.

The mystery of the disappearing SSH key

SSH (Secure Shell) is one of the programs I use every single day at work, primarily to connect to our client's servers. Usually it is a rock-solid program that simply works as expected, but recently I discovered it behaving quite strangely - a server I had visited many times before was now refusing my attempts to login. The underlying problem turned out to be a misguided decision by the developers of OpenSSH to deprecate DSA keys. How I discovered this problem is described below (as well as two solutions).

The use of the ssh program is not simply limited to logging in and connecting to remote servers. It also supports many powerful features, one of the most important being the ability to chain multiple connections with the ProxyCommand option. By using this, you can "login" to servers that you cannot reach directly, by linking together two or more servers behind the scenes.

As as example, let's consider a client named "Acme Anvils" that strictly controls access to its production servers. They make all SSH traffic come in through a single server, named dmz.acme-anvils.com, and only on port 2222. They also only allow certain public IPs to connect to this server, via whitelisting. On our side, End Point has a server, named portal.endpoint.com, that I can use as a jumping off point, which has a fixed IP that we can give to our clients to whitelist. Rather than logging in to "portal", getting a prompt, and then logging in to "dmz", I can simply add an entry in my ~/.ssh/config file to automatically create a tunnel between the servers - at which point I can reach the client's server by typing "ssh acmedmz":

##
## Client: ACME ANVILS
##

## Acme Anvil's DMZ server (dmz.acme-anvils.com)
Host acmedmz
User endpoint
HostName 555.123.45.67
Port 2222
ProxyCommand ssh -q greg@portal.endpoint.com nc -w 180s %h %p

Notice that the "Host" name may be set to anything you want. The connection to the client's server uses a non-standard port, and the username changes from "greg" to "endpoint", but all of that is hidden away from me as now the login is simply:

[greg@localhost]$ ssh acmedmz
[endpoint@dmz]$

It's unusual that I'll actually need to do any work on the dmz server, of course, so the tunnel gets extended another hop to the db1.acme-anvils.com server:

##
## Client: ACME ANVILS
##

## Acme Anvil's DMZ server (dmz.acme-anvils.com)
Host acmedmz
User endpoint
HostName 555.123.45.67
Port 2222
ProxyCommand ssh -q greg@portal.endpoint.com nc -w 180s %h %p

## Acme Anvil's main database (db1.acme-anvils.com)
Host acmedb1
User postgres
HostName db1
ProxyCommand ssh -q acmedmz nc -w 180s %h %p

Notice how the second ProxyCommand references the "Host" of the section above it. Neat stuff. When I type "ssh acemdb1", I'm actually connecting to the portal.endpoint.com server, then immediately running the netcat (nc) command in the background, then going through netcat to dmz.acme-anvils.com and running a second netcat command on *that* server, and finally going through both netcats to login to the db1.acme-anvils.com server. It sounds a little complicated, but quickly becomes part of your standard tool set once you wrap your head around it. After you update your .ssh/config file, you soon forget about all the tunneling and feel as though you are connecting directly to all your servers. That is, until something breaks, as it did recently for me.

The actual client this happened with was not "Acme Anvils", of course, and it was a connection that went through four servers and three ProxyCommands, but for demonstration purposes let's pretend it happened on a simple connection to the dmz.acme-anvils.com server. I had not connected to the server in question for a long time, but I needed to make some adjustments to a tail_n_mail configuration file. The first login attempt failed completely:

[greg@localhost]$ ssh acmedmz
endpoint@dmz.acme-anvils.com's password: 

Although the connection to portal.endpoint.com worked fine, the connection to the client server failed. This is not an unusual problem: it usually signifies that either ssh-agent is not running, or that I forgot to feed it the correct key via the ssh-add program. However, I quickly discovered that ssh-agent was working and contained all my usual keys. Moreover, I was able to connect to other sites with no problem! On a hunch, I tried breaking down the connections into manual steps. First, I tried logging in to the "portal" server. It logged me in with no problem. Then I tried to login from there to dmz.acme-anvils.com - which also logged me in with no problem! But trying to get there via ProxyCommand still failed. What was going on?

When in doubt, crank up the debugging. For the ssh program, using the -v option turns on some minimal debugging. Running the original command from my computer with this option enabled quickly revealed the problem:

[greg@localhost]$ ssh -v acmedmz
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /home/greg/.ssh/config
debug1: /home/greg/.ssh/config line 1227: Applying options for acmedmz
debug1: Reading configuration data /etc/ssh/ssh_config
...
debug1: Executing proxy command: exec ssh -q greg@portal.endpoint.com nc -w 180s 555.123.45.67 2222
...
debug1: Authenticating to dmz.acme-anvils.com:2222 as 'endpoint'
...
debug1: Host 'dmz.acme-anvils.com' is known and matches the ECDSA host key.
...
debug1: Skipping ssh-dss key /home/greg/.ssh/greg2048dsa.key - not in PubkeyAcceptedKeyTypes
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/greg/.ssh/greg4096rsa.key
debug1: Next authentication method: password
endpoint@dmz.acme-anvils.com's password: 

As highlighted above, the problem is that my DSA key (the "ssh-dss key") was rejected by my ssh program. As we will see below, DSA keys are rejected by default in recent versions of the OpenSSH program. But why was I still able to login when not hopping through the middle server? The solution lays in the fact that when I use the ProxyCommand, *my* ssh program is negotiating with the final server, and is refusing to use my DSA key. However, when I ssh to the portal.endpoint.com server, and then on to the next one, the second server has no problem using my (forwarded) DSA key! Using the -v option on the connection from portal.endpoint.com to dmz.acme-anvils.com reveals another clue:

[greg@portal]$ ssh -v endpoint@dmz.acme-anvils.com:2222
...
debug1: Connecting to dmz [1234:5678:90ab:cd::e] port 2222.
...
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/greg/.ssh/endpoint2.ssh
debug1: Authentications that can continue: publickey,password
debug1: Offering DSA public key: /home/greg/.ssh/endpoint.ssh
debug1: Server accepts key: pkalg ssh-dss blen 819
debug1: Authentication succeeded (publickey).
Authenticated to dmz ([1234:5678:90ab:cd::e]:2222).
...
debug1: Entering interactive session.
[endpoint@dmz]$

If you look closely at the above, you will see that we first offered an RSA key, which was rejected, and then we successfully offered a DSA key. This means that the endpoint@dm account has a DSA, but not a RSA, public key inside of its ~/.ssh/authorized_keys file. Since I was able to connect to portal.endpoint.com, its ~/.ssh/authorized_keys file must have my RSA key.

For the failing connection, ssh was able to use my RSA key to connect to portal.endpoint.com, run the netcat command, and then continue on to the dmz.acme-anvils.com server. However, this connection failed as the only key my local ssh program would provide was the RSA one, which the dmz server did not have.

For the working connection, ssh was able to connect to portal.endpoint.com as before, and then into an interactive prompt. However, when I then connected via ssh to dmz.acme-anvils.com, it was the ssh program on portal, not my local computer, which negotiated with the dmz server. It had no problem using a DSA key, so I was able to login. Note that both keys were happily forwarded to portal.endpoint.com, even though my ssh program refused to use them!

The quick solution to the problem, of course, was to upload my RSA key to the dmz.acme-anvils.com server. Once this was done, my local ssh program was more than happy to login by sending the RSA key along the tunnel.

Another solution to this problem is to instruct your SSH programs to recognize DSA keys again. To do this, add this line to your local SSH config file ($HOME/.ssh/config), or to the global SSH config file (/etc/ssh/config):

PubkeyAcceptedKeyTypes +ssh-dss

As mentioned earlier, this whole mess was caused by the OpenSSH program deciding to deprecate DSA keys. Their rationale for targeting all DSA keys seems a little weak at best: certainly I don't feel that my 2048-bit DSA key is in any way a weak link. But the writing is on the wall now for DSA, so you may as well replace your DSA keys with RSA ones (and an ed25519 key as well, in anticipation of when ssh-agent is able to support them!). More information about the decision to force out DSA keys can be found in this great analysis of the OpenSSH source code.

SELINUX=disabled? Read this and think twice!

Not long ago, one of our customers had their website compromised because of a badly maintained, not-updated WordPress. At End Point we love WordPress, but it really needs to be configured and hardened the right way, otherwise it's easy to end up in a real nightmare.

This situation is worsened even more if there's no additional security enforcement system to protect the environment on which the compromised site lives. One of the basic ways to protect your Linux server, especially RHEL/Centos based ones, is using SELinux.

Sadly, most of the interaction people has with SELinux happens while disabling it, first on the running system:

setenforce disabled
# or
setenforce 0

and then permanently by manually editing the file /etc/sysconfig/selinux to change the variable SELINUX=enforcing to SELINUX=disabled.

Is that actually a good idea though? While SELinux can be a bit of a headache to tune appropriately and can easily be misconfigured, here's something that could really convince you to think twice before disabling SELinux once and forever.

Back to our customer's compromised site. While going through the customer's system for some post-crysis cleaning, I found this hilarious piece of bash_history:

ls
cp /tmp/wacky.php .
ls -lFa
vim wacky.php
set
ls -lFa
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
fg
ls -lFa
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
vim wacky.php
php wacky.php 2>&1 | less
php wacky.php > THE-EVIL 2>&1
vim THE-EVIL
ls -lFA
less wacky.php
ls
less THE-EVIL
less wacky.php
cat /selinux/enforce
ls
less THE-EVIL
exit

As you can see, what happened was that the attacker was able to manage having a shell connection as the customer user, and started using a PHP files injected in /tmp as a possible further vector of attack.

Sadly, for the attacker at least, what happened was that SELinux was setup in enforcing mode with some strict rules and prevented all kind of execution on that specific script so after a few frantic attempts the attacker surrendered.

Looking into the /var/log/audit/auditd.log file I found all the AVC="denied" errors that SELinux was shouting while forbidding the attacker to pursue his nefarious plan.

Hilarious and good props to SELinux for saving the day.

less THE-EVIL, more SELinux! 🙂

mysqldump issues after Percona 5.7 update

During a recent CentOS 7 update, among other packages, we updated our Percona 5.7 installation to version 5.7.17-13.

Quickly after that, we discovered that mysqldump stopped working, thus breaking our local mysql backup script (that complained loudly).

What happened?


The error we received was:

mysqldump: Couldn't execute 'SELECT COUNT(*) FROM INFORMATION_SCHEMA.SESSION_VARIABLES WHERE VARIABLE_NAME LIKE 'rocksdb\_skip\_fill\_cache'': The 'INFORMATION_SCHEMA.SESSION_VARIABLES' feature is disabled; see the documentation for 'show_compatibility_56' (3167)

After a bit of investigation, we discovered this was caused by this regression bug, apparently already fixed but not yet available on CentOS:

Everything revolves around INFORMATION_SCHEMA being deprecated in version 5.7.6, when Performance Schema tables has been added as a replacement.

Basically, a regression caused mysqldump to try and use deprecated INFORMATION_SCHEMA tables instead of the new Performance Schema.

How to fix it?


Immediate workaround is to add this line to /etc/my.cnf or (more likely) /etc/percona-server.conf.d/mysqld.cnf, depending on how your configuration files are organized:

show_compatibility_56=1

This flag was both introduced and deprecated in 5.7.6. It will be there for some time to help with the transition.

It seems safe and, probably, good to keep if you have anything still actively using INFORMATION_SCHEMA tables, that would obviously be broken if not updated to the new Performance Schema since 5.7.6.

With this flag, it is possible to preserve the old behavior and keep your old code in a working state, while you upgrade it. Also, according to the documentation, it should not impact or turn off the new behavior with Performance Schema.

More information on how to migrate to the new Performance Schema can be found here.

Linode IPv6 issues with NetworkManager on CentOS 7

In End Point, we use different hosting providers based on the specific task needs. One provider we use extensively with good results is Linode.

During a routine CentOS 7 system update, we noticed a very strange behavior where our IPv6 assigned server address was wrong after restarting the server.

IPv6 on Linode and SLAAC


Linode is offering IPv6 on all their VPS, and IPv6 dynamic addresses are assigned to servers using SLAAC.

In the provided CentOS 7 server image, this is managed by NetworkManager by default. After some troubleshooting, we noticed that during the update the NetworkManager package was upgraded from 1.0.6 to 1.4.0.

This was a major update, and it turned out that the problem was a change in the configuration defaults between the two version.

Privacy stable addressing


Since 1.2, NetworkManager added the Stable Privacy Addressing feature. This allows for some form of tracking prevention, with the IPv6 address to be stable on a network but changing when entering another network, and still remain unique.

This new interesting feature has apparently become the default after the update, with the ipv6.addr-gen-mode property set to "stable-privacy". Setting it to “eui64” maintains the old default behavior.

Privacy Extension


Another feature apparently also caused some problems on our VPS: the Privacy Extension. This is a simple mechanism that somewhat randomizes the network hardware’s (MAC) address, to add another layer of privacy. Alas, this is used in address generation, and that randomization seemed to be part of the problem we were seeing.

This too has become the default, with the ipv6.ip6-privacy property set to 1. Setting it to 0 turns off the feature.

To sum it up


In the end, after the update, we could restore the old behavior and resolve our issues by running, in a root shell:

nmcli connection modify "Wired connection 1" ipv6.ip6-privacy 0
nmcli connection modify "Wired connection 1" ipv6.addr-gen-mode eui64

After a reboot, the IPv6 address finally matched the one actually assigned by Linode, and everything was working ok again.

If you want to know more on Privacy Extensions and Privacy Stable Addressing, this great blog post by Lubomir Rintel helped us a lot understanding what was going on.

Half day GlusterFS training in Selangor, Malaysia

On January 21, 2017, I had an opportunity to join a community-organized training on storage focused on GlusterFS. GlusterFS is an open source cloud-based filesharing network. The training was not a strictly structured training as the topic approached knowledge sharing from various experts and introduced GlusterFS to the ones who were new to it. The first session was delivered by Mr Adzmely Mansor from NexoPrima. He shared a bit of his view on GlusterFS and technologies that are related to it.

Mr Haris, a freelance Linux expert, later led a GlusterFS technical class. Here we created two virtual machines (we used Virtualbox) to understand how GlusterFS works in a hands-on scenario. We used Ubuntu 16.04 as the guest OS during technical training. We used Digital Ocean's GlusterFS settings as a base of reference. The below commands detail roughly what we did during the training.

In GlusterFS the data section is called as "brick". Hence we could have a lot of "bricks" if we have it more than once :) . As Ubuntu already had the related packages in its repository, we could simply run apt-get for the package installation. Our class notes were loosely based from Digital Ocean's GlusterFS article here. (Note: the article was based on Ubuntu 12.04 so some of the steps could be omitted).

The GlusterFS packages could be installed as a superuser with the following command:

apt-get install glusterfs-server

Since we were using a bridged VM during the demo, we simply edited the /etc/hosts in the each VM so they could communicate between each other by using hostname instead of using typing the IP manually.

root@gluster2:~# grep gluster /etc/hosts
192.168.1.11 gluster1
127.0.0.1 gluster2

Here we will try to probe the remote host whether it is reachable:

root@gluster2:~# gluster peer probe gluster1
peer probe: success. Host gluster1 port 24007 already in peer list

The following commands create the storage volume. Later, whatever we put in the /data partition will be reachable on the other gluster node.

gluster volume create datastore1 replica 2 transport tcp gluster1:/data gluster2:/data
gluster volume create datastore1 replica 2 transport tcp gluster1:/data gluster2:/data force
gluster volume start datastore1

Most of the parts here could be retrieved from the link that I gave above. But let's see what will happen later on when the mounting part is done.

cd /datastore1/
root@gluster2:/datastore1# touch blog
root@gluster2:/datastore1# ls -lth
total 512
-rw-r--r-- 1 root root  0 Mar 14 21:33 blog
-rw-r--r-- 1 root root 28 Jan 21 12:15 ujian.txt

The same output could be retrieved from gluster1

root@gluster1:/datastore1# ls -lth
total 512
-rw-r--r-- 1 root root  0 Mar 14 21:33 blog
-rw-r--r-- 1 root root 28 Jan 21 12:15 ujian.txt


Mr. Adzmely gave explanation on the overall picture of GlusterFS



Mr. Haris explained on the technical implementation of GlusterFS

In terms of the application, the redundancy based storage is good for situations where you have a file being updated on several servers and you need to ensure the file is there for retrieval even if one of the servers is down. One audience member shared his experience deploying GlusterFS in his workplace (a university) for the purpose of new intake of student's registration. If anyone ever uses Samba filesystem or NFS, this kind of similar, but GlusterFS is much more advanced. I recommend additional reading here.