Welcome to End Point’s blog

Ongoing observations by End Point people

Liquid Galaxy at ASTC 2017

End Point is pleased to be participating in ASTC 2017, alongside our partners BWC Visual Technology. ASTC, which stands for the Association of Science-Technology Centers, is holding their annual conference at The Tech Museum of Innovation, located in San Jose, CA. We were excited to hear that the conference takes place at The Tech Museum, as we have a Liquid Galaxy set up in the museum!

A 3-screen desktop Liquid Galaxy display will be set up by Liquid Galaxy Engineer Josh Ausborne at Booth 1103. This display will be showcasing content that includes Sketchfab and Unity 3D Models, Cesium content with interactive weather data, 360 panoramic video, Google Earth/Google Streetview content, and engaging presentations about National Parks and National Marine Sanctuaries.

We are very excited to be showcasing our technology with BWC Visual Technology. BWC is a distributor and licensed re-seller of state-of-the-art, interactive exhibit technology for museums and science centers. We have great respect for their team and technologies, and are excited to be showcasing Liquid Galaxy with them.

Liquid Galaxy is currently featured in many science and technology centers around the world. Please view the information featured below to learn more. This handout will be available to all attendees at ASTC, and can be picked up at booth 1103.

PKIX path validation failed - Debugging

I recently ran into a case working on an application with a PKIX path validation error on a site that had a valid certificate. I was able to solve the issue using OpenSSL to debug.

Typically, the PKIX path validation error arises due to SSL certificate expiry, but I ran into the same error even when the system was configured with a valid certificate. There are two web applications in our scenario, AppX and AppY. AppX uses AppY's authentication mechanism to allow the users to login with same user account. AppX sends a POST request using HttpClient with necessary arguments to SSL enabled AppY and allows the user to login based on the response.
HttpClient httpclient = new DefaultHttpClient();
// ...
HttpPost httppost = new HttpPost("");

try {
    resp = httpclient.execute(httppost);
catch (Exception e) {
    throw new Exception("Exception: ", e);


The AppX was isolated to new server and it started throwing PKIX path validation failed error while sending requests to AppY.
Exception: PKIX path validation failed: timestamp check failed
PKIX (Public-Key Infrastructure - X.509) is standard for key based encryption mechanism. The PKIX path related errors come up due to the failure establishing the connection with SSL applications.


It is good to identify the root cause of the problem since there are few possible reasons for the same error. Let's start debugging...

  • Check the Certificate status and expiration date in your browser
  • The browser reports that the certificate is valid and will expire at a future date for AppY's domain name. So now on to the detailed debugging using OpenSSL.
  • OpenSSL validation
  • The openssl tool is a handy utility to validate the SSL certificate for any domain. It reports error with return code 20 (unable to get local issuer certificate) when checking the status of certificate - which is in contrast with browser's report on the same certificate.
$ echo -n | openssl s_client -CApath /etc/ssl/certs/ -connect 
Start Time: 1482921042
Timeout   : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)

$ echo -n | openssl s_client -CApath /etc/ssl/certs/ -connect </dev/null | openssl x509 -noout -dates

verify error:num=20:unable to get local issuer certificate
notBefore=May  4 00:00:00 2013 GMT
notAfter=May 14 23:59:59 2015 GMT

Root Cause

The openssl tool reported certificate details another unused and expired domain. Since this is configured on the same server, it is causing the error in our case. The same scenario happened to the AppX when sending request to AppX. It may have tried to establish connection through the expired certificate. So, the lesson here is that it is necessary to clean up the expired certificates when the connection is established through HttpClient utilities. Also, a specific domain name can be validated by passing the -servername option (for SNI) to the openssl, which in this case reports has valid certificate.
$ echo -n | openssl s_client -CApath /etc/ssl/certs/ -connect -servername
Start Time: 1482920942
Timeout   : 300 (sec)
Verify return code: 0 (ok)

$ echo -n | openssl s_client -CApath /etc/ssl/certs/ -connect -servername </dev/null | openssl x509 -noout -dates
verify return:0
notBefore=Sep 26 11:52:51 2015 GMT
notAfter=Apr  1 12:35:52 2018 GMT


In most cases, the PKIX path validation error comes up when the SSL certificate is expired for the domain name, however, there may be different reasons such as certificate expiry, picking wrong certificate, etc. It is always helpful to debug with the openssl tool to identify the root cause. This specific issue was fixed by removing the unused expired certificate.

Disaster Recovery - Miami to Dallas in one hit

Hurricane Irma came a knocking but didn’t blow Miami away.

Hurricanes are never fun, and neither is knowing that your business servers may be in the direct path of a major outage due to a natural disaster.

Recently we helped a client prepare a disaster recovery environment, 4 days out pending Hurricane Irma approaching Miami — where it happens their entire server ecosystem sat. In recent months we had been discussing a long-term disaster recovery plan for this client’s infrastructure, but the final details hadn’t been worked out yet, and no work begun on it, when the news of the impending storm started to arrive.

Although the Miami datacenter they are using is highly regarded and well-rated, the client had the foresight to think about an emergency project to replicate their entire server ecosystem out of Miami to somewhere a little safer.

Fire suits on, battle helmets ready, GO! Two team members jumped in and did an initial review of the ecosystem: six Linux servers, one Microsoft SQL Server database with about 50 GB of data, and several little minions. All of this hosted on a KVM virtualization platform. OK, easy, right? Export the VM disks and stand up in a new datacentre on a similar KVM-based host.

But no downtime could be scheduled at that time, and cloning the database would take too much time to make a replica. If we can’t clone the VM disks without downtime we need another plan. Enter our save-the-day idea: use the database clone from the development machine.

So first things first: Build the servers in the new environment using the cloned copy, rsync the entire disk across, hoping that a restart would return the machine into a working state, take the development database and use the Microsoft tools at our disposal to replicate one current snapshot, then replicate row level details, in preparation of reducing the amount of lost data, should the datacenter lose connection.

Over the course of the next 16 hours, with 2 DevOps engineers, we replicated 6 servers, migrated 100+ GB of data twice and had a complete environment ready for a manual cutover.

Luckily the hurricane didn’t cause any damage to the datacenter, which never lost power or connectivity, so the sites never went down.

Even though it wasn’t used in this case, this emergency work did have a silver lining: Until this client’s longer-term disaster recovery environment is built, these system replicas can serve as cold standby, which is much better than having to rebuild from backups if their primary infrastructure goes out.

That basic phase is out of the way, so the client can breathe easier. Along the way, we produced some updated documentation, and gained deeper knowledge of the client’s software ecosystem. That’s a win!

The key takeaways here: Don’t panic, and do a little planning prior. Leaving things to the last minute rarely works well.
And preferable to all of the above: Plan and build your long-term disaster recovery environment now, well before any possible disaster is on the horizon!

To all the people affected by Hurricane Irma we wish a speedy rebuild.

AngularJS is not Angular

Why I'm doing this?

I often see that people use AngularJS and Angular names interchangeably. It's an obvious mistake for someone familiar with the front-end programming.

If you don't believe me (why would you?) just go to the Angular team repository or Wikipedia (Angular and AngularJS).

AngularJS (1.X)

AngularJS is a popular open-source JavaScript framework created by Miško Hevery released in 2010 that was subsequently taken over by Google. It's not the same thing as Angular. The latest stable version of the framework is 1.6.6.

Angular (2+)

Angular is an open-source TypeScript-based framework created by Google from scratch. It's been totally rewritten and not compatible with AngularJS. The latest stable version of the framework is 4.4.4. They started from version 2 to avoid even more confusion.


I feel I have made the world better. Go now and correct everyone (it's such fun)!