Welcome to End Point’s blog

Ongoing observations by End Point people

Become A File Spy With This One Easy Trick! Sys Admins Love This!

We had an interesting problem to track down. (Though I suppose I wouldn't be writing about it if it weren't, yes?) Over the years a client had built up quite the collection of scripts executed by cron to maintain some files on their site. Some of these were fairly complex, taking a long while to run, and overlapping with each other.

One day, the backup churn hit a tipping point and we took notice. Some process, we found, seemed to be touching an increasing number of image files: The contents were almost always the same, but the modification timestamps were updated. But digging through the myriad of code to figure out what was doing that was proving to be somewhat troublesome.

Enter auditd, already present on the RHEL host. This allows us to attach a watch on the directory in question, and track down exactly what was performing the events. -- Note, other flavors of Linux, such as Ubuntu, may not have it out of the box. But you can usually install it via the the auditd package.
(output from a test system for demonstration purposes)
# auditctl -w /root/output
# tail /var/log/audit/audit.log
type=SYSCALL msg=audit(1487974252.630:311): arch=c000003e syscall=2 success=yes exit=3 a0=b51cf0 a1=241 a2=1b6 a3=2 items=2 ppid=30272 pid=30316 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=1 comm="" exe="/usr/bin/bash" key=(null)
type=CWD msg=audit(1487974252.630:311):  cwd="/root"
type=PATH msg=audit(1487974252.630:311): item=0 name="output/files/" inode=519034 dev=fd:01 mode=040755 ouid=0 ogid=0 rdev=00:00 objtype=PARENT
type=PATH msg=audit(1487974252.630:311): item=1 name="output/files/1.txt" inode=519035 dev=fd:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 objtype=CREATE
The most helpful logged items include the executing process's name and path, the file's path, operation, pid and parent pid. But there's a good bit of data there per syscall.

Don't forget to auditctl -W /root/output to remove watch. auditctl -l will list what's currently out there:
# auditctl -l
-w /root/output -p rwxa
That's the short version. auditctl has a different set of parameters that are a little bit more verbose, but have more options. The equivalent of the above would be: auditctl -a always,exit -F dir=/root/output -F perm=rwxa ... with options for additional rule fields/filters on uid, gid, pid, whether or not the action was successful, and so on.

FOSDEM 2017: experience, community and good talks

In case you happen to be short on time: my final overall perspective about FOSDEM 2017 is that it was awesome... with very few downsides.

If you want the longer version, keep reading cause there's a lot to know and do at FOSDEM and never enough time, sadly.

This year I actually took a different approach than last time and decided to concentrate on one main track per day, instead of (literally) jumping from one to the other. While I think that overall this may be a good approach if most of the topics covered in a track are of your interest, that comes at the cost of missing one of the best aspects of FOSDEM which is "variety" in contents and presenters.

Day 1: Backup & Recovery

For the first day I chose the Backup & Recovery track which hosted talks revolving around three interesting and useful projects: namely REAR (Relax and Recovery), DRLM, a wrapper and backup management tool based on REAR and Bareos, which is a backup solution forked from Bacula in 2010 and steadily proceeding and improving since then. Both REAR and DLRM were explained and showcased by some of the respective projects main contributors and creators. As a long time system administrator, I particularly appreciated the pride in using Bash as the main "development platform" for both projects. As Johannes Meixner correctly mentioned, using bash facilitates introduces these tools into your normal workflow with knowledge that you'll most likely already have as a System Administrator or DevOps, thus allowing you to easily "mold" these scripts to your specific needs without spending weeks to learn how to interact with them.

During the Day 1 Backup & Recovery track there were also a few speeches from two Bareos developers (Jörg Steffens and Stephan Dühr) that presented many aspects of their great project, ranging from very introductory topics, to providing a common knowledge ground for the audience, up to more in depth topics like software capabilities extension through Python Plugins, or a handful of best practices and common usage scenarios. I also enjoyed the speech about automated testing in REAR, presented by Gratien D'haese, which showed how to leverage common testing paradigms and ideas to double-check a REAR setup for potential unexpected behaviors after updates or on new installations or simply as a fully automated monitoring tool to do sanity checks on the backup data flow. While this testing project was new, it's already functional and impressive to see at work.

Day 2: Cloud Microservices

On the second day I moved in a more "cloudy" section of the FOSDEM where most of the conferences revolved around Kubernetes, Docker and more in general the microservices landscape. CoreOS (the company behind the open source distribution) was a major contributor and I liked their Kubernetes presentation by Josh Wood and Luca Bruno which respectively explained the new Kubernetes Operators feature and how containers work under the hood in Kubernetes.

Around lunch time there was a "nice storm of lightning talks" which kept most of the audience firmly on their seats, especially since the Microservices track room didn't have a free seat for the entire day. I especially liked the talk from Spyros Trigazis about how CERN created and is maintaining a big OpenStack Magnum (the container integrated version of OpenStack) cloud installation for their internal use.

Then it was Chris Down's turn and, while he's a developer from Facebook, his talk gave the audience a good perspective on the future of CGROUPs in the Linux kernel and how they are already relatively safe and usable, even if not yet officially marked as production ready. While I already knew and used "sysdig" in past as a troubleshooting and investigation tool, it was nice to see one of the main developers, Jorge Salamero, using it and showing alternative approaches such as investigating timeout issues between Kubernetes Docker containers by just sysdig and its many modules and filters. It was really impressive seeing how easy it is to identify cross-containers issues and data flow.


There were a lot of Open Source communities with "advertising desks" and I had a nice talk with a few interesting developers from the CoreOS team or from FSFE (Free Software Foundation Europe). Grabbing as many computer stickers as possible is also mandatory at FOSDEM, so I took my share and my new Thinkpad is way more colorful now. In fact, on a more trivial note, this year the FOSDEM staff decided to sell on sale all the laptops that were used during the video encoding phase for the streaming videos before the upload. These laptops were all IBM Thinkpad X220 and there were only a handful of them (~30) at a very appealing price. In fact, this article is being written from one of those very laptops now as I was one of the lucky few which managed to grab one before they were all gone within an hour or so. So if you're short of a laptop and happen to be at FOSDEM next year, keep your eyes open cause I think they'll do it again!

So what's not to like in such a wonderful scenario? While I admit that there was a lot to be seen and listened to, I sadly didn't see any "ground-shaking" innovation this year at FOSDEM. I did see many quality talks and I want to send a special huge "thank you" to all the speakers for the effort and high quality standards that they keep for their FOSDEM talks - but I didn't see anything extraordinarily new from what I can remember.

Bottom line is that I still have yet to find someone who was ever disappointed at FOSDEM, but the content quality varies from presenter to presenter and from year to year, so be sure to check the presentations you want to attend carefully before hand.

I think that the most fascinating part of FOSDEM is meeting interesting, smart, and like-minded people that would be difficult to reach otherwise.

In fact, while a good share of the merit should be attributed to the quality of the content presented, I firmly believe that the community feeling that you get at FOSDEM is hard to beat and easy to miss when skipped even for one year.

I'll see you all next year at FOSDEM then.

Full Cesium Mapping on the Liquid Galaxy

A few months ago, we shared a video and some early work we had done with bringing the Cesium open source mapping application to the Liquid Galaxy. We've now completed a full deployment for Smartrac, a retail tracking analytics provider, using Cesium in a production environment! This project presented a number of technical challenges beyond the early prototype work, but also brought great results for the client and garnered a fair amount of attention in the press, to everyone's benefit.

Cesium is an open source mapping application that separates out the tile sets, elevation, and markup language. This separation allows for flexibility at each major element:

  • We can use a specific terrain elevation data set while substituting any one of several map "skins" to drape on that elevation: a simple color coded map, a nighttime illumination map, even a water-colored "pirate map" look.
  • For the terrain, we can download as much or as little is needed: As the Cesium viewer zooms in on a given spot, Cesium uses a sort of fractal method to download finer and finer resolution terrains in just the surrounding area, eventually getting to the data limit of the set. This gradual approach balances download requirements with viewable accuracy. In our case, we downloaded an entire terrain set up to level 14 (Earth from high in space is level 1, then zooms in to levels 2, 3, 4, etc.) which gave us a pretty good resolution while conserving disk space. (The data up to level 14 totaled about 250 GB.)
  • Using some KML tools we have developed for past projects and adapting to CZML ("cesium language", get it?), we were able to take Smartrac's supply chain data and show a comprehensive overview of the product flow from factories in southeast Asia through a distribution center in Seattle and on to retail stores throughout the Western United States.
The debut for this project was the National Retail Federation convention at the Javitz Center in New York City. Smartrac (and we also) wanted to avoid any show-stoppers that might come from a sketchy internet connection. So, we downloaded the map tiles, a terrain set, built our visualizations, and saved the whole thing locally on the head node of the Liquid Galaxy server stack, which sat in the back of the booth behind the screens.

The show was a great success, with visitors running through the visualizations almost non-stop for 3 days. The client is now taking the Liquid Galaxy and the Cesium visualizations on to another convention in Europe next month. The NRF, IBM, and several other ecommerce bloggers wrote up the platform, which brings good press for Smartrac, Cesium, and the Liquid Galaxy.

Liquid Galaxy Success at U.S. Embassy’s Cultural Center

The U.S. Embassy to Jakarta features a high-tech cultural center called “@america”. @america’s mission is to provide a space for young Indonesians to learn more about the United States through discussions, cultural performances, debates, competitions and exhibitions.

Since Google generously donated it six years ago, @america has had a Liquid Galaxy deployed for use at the center. Not until recently, however, has @america taken advantage of our Content Management System. This past year, End Point developed and rolled out a revamped and powerful Content Management System for the fleet of Liquid Galaxies we support. With the updated Content Management System, End Point’s Content Team created a specialized Interactive Education Portal on @america's Liquid Galaxy. The Education Portal featured over 50 high quality, interactive university experiences. Thanks to the CMS, the Liquid Galaxy now shows campus videos, university statistics, and fly-tos and orbits around the schools. The campus videos included both recruitment videos, as well as student-created videos on topics like housing, campus sports, and religion. These university experiences allow young Indonesians the opportunity to learn more about U.S. Universities and culture.

@America and the US Embassy report that from December through the end of January, already more than 16,500 Indonesians have had the opportunity to engage with the Education Portal while visiting @america. We are thankful to have had the opportunity to help the US Embassy use their Liquid Galaxy for such a positive educational cause.

Liquid Galaxy systems are installed at educational institutions, from embassies to research libraries, around the world. If you’d like to learn more about Liquid Galaxy, please visit our Liquid Galaxy website or contact us here.