Welcome to End Point’s blog

Ongoing observations by End Point people

Xen MAC mismatch VNC mouse escape HOWTO

This is a story that probably shouldn't need to be told if everything is in documentation somewhere. I'm not using any fancy virtualization management tools and didn't have an easy time piecing everything together, so I thought it'd be worth writing down the steps of the manual approach I took.

Dramatis personæ:

  • Server: Red Hat Enterprise Linux 5.4 with Xen kernel
  • Guest virtual server: CentOS 5.4 running paravirtualized under Xen
  • Workstation: Ubuntu 9.10

The situation: I updated the CentOS 5 Xen virtual guest via yum and rebooted to load the new Linux kernel and other libraries such as glibc. According to Xen as reported by xm list, the guest had started back up fine, but the host wasn't reachable over the network via ping, http, or ssh, including from the host network.

The guest wasn't using much CPU (as shown by xm top), so I figured it wasn't just a slow-running fsck during startup. And I was familiar with the iptables firewall rules on this guest, so I was fairly sure I wasn't being blocked there. I needed to get to the console to see what was wrong.

The way I've done this before is using VNC to access the virtual console remotely. The Xen host was configured to accept VNC connections on localhost, which I could see by looking in /etc/xen/xend-config.sxp:

(vnc-listen '')

There are 11 Xen guests, with consoles listening on TCP ports 5900-5910. Which one was the one? I don't know any simple way to get a list that maps ports to Xen guests, but I did it this way:

ps auxww | grep qemu-dm

I noted the PID of the process that was running for my guest as revealed in its command line. Then I looked for the listener running under that PID:

netstat -nlp

I looked for $pid/qemu-dm in the PID/Program Name column and could then see the TCP port in the Local Address column. In my case it was

So I set up an ssh tunnel to the server for my VNC traffic:

ssh -f -N -L 5903:localhost:5903 root@$remote_host &

Then I opened the default Ubuntu/GNOME VNC viewer, labeled "Remote Desktop Viewer" under the Internet menu. This program is actually called Vinagre, and is basic but works. I connected to localhost:5903, since I'd forwarded my own local TCP port 5903 to the remote port 5903.

The remote console came up, and I was presented with the login banner and prompt. If I hadn't had the root password, I would have needed to reboot the guest and start it in runlevel 0 to get a root shell without a password and change the password. But I did have the password, so that wasn't necessary.

Then when trying to change to another window on my desktop, I ran into the biggest snag of the whole exercise: Getting control of the mouse out of the VNC remote desktop window and back in my own desktop! I couldn't find anything accurate on this in any documentation, forums, etc. Finally I stumbled across the trick: Press F10, which pulls down the Machine menu in Vinagre and as a side-effect takes control of the mouse away from the remote desktop. It was nice not to have to ssh in from another machine to kill Vinagre. But it makes me wonder how I'd send an F10 through to the remote console ...

Armed with the root password I was able to log into the guest and discover that only the lo (loopback) interface started on boot. The eth0 and eth1 interfaces failed because there was no virtual NIC available with the MAC addresses specified in /etc/sysconfig/network-scripts/ifcfg-eth0 and eth1.

That was because the virtual machine image had been cloned from another one and hadn't been given new unique MAC addresses. The problem was easily fixed by updating the ifcfg-eth0 and ifcfg-eth1 files with the MAC addresses actually given to the interfaces, as seen by ifconfig, which were ultimately assigned by the Xen host in /etc/xen/$host in the vif parameter. (You can also specify no MAC addresses in the guest at all and it will use whatever it gets.)

Then after running service network restart the networking was back, and I rebooted to make sure it started correctly on its own.


thelsdj said...

I'm sure you have a reason, but I'm curious why you don't use serial console emulation so you can just run 'xm console ' to get a text console on the guest?

Jon Jensen said...

That's a good question! In this case, I didn't even think about it, and should have.

I long ago chose to avoid serial consoles where possible because of the lack of curses support (editor pain), usually missing scrollback (no clue about kernel panics that had already happened), etc.

The lack of curses support is still a problem with Xen's console, but the scrollback is actually there, which is great. So next time I'd do this first and save some hassle.

I just checked the docs and read that Xen only has a console to paravirtualized guests, which would work fine here, but not for the fully virtualized guests, so that's something to note.

Thanks for the comment!

Jon Jensen said...

Actually I just tried using xm console with vim, emacs, and top, and contrary to what I expected from the xm manpage, they worked just fine.

I had to manually set the COLUMNS and LINES environment variables for my console, since they defaulted to 80x24. And the TERM defaulted to a pretty simple type, but setting TERM to screen fixed that.

So xm console is really quite usable. Thanks again!

thelsdj said...

For fully virtualized guests, xm console may still be an option. You can actually make the guest think xm console is a physical serial device so OS's that support outputting to console (like OpenBSD) can support xm console when fully virtualized!