Welcome to End Point’s blog

Ongoing observations by End Point people

RHEL 5 SELinux initscripts problem

I ran into a strange problem updating Red Hat Enterprise Linux 5 a few months ago, and just ran into it again and this time better understood what went wrong.

The problem was serious: After a `yum upgrade` of a RHEL 5 x86_64 server with SELinux enforcing, it never came back after a reboot. Logging into the console I could see that it was stuck in single user mode because there were no init scripts! Investigation showed that indeed the initscripts package was completely missing.

I searched on looking for any reported problems and didn't find any. I reinstalled initscripts, rebooted, and the server was fine, but it was not happytimes to have that unexpected downtime.

Tonight I ran into the problem again, this time on a build server where downtime didn't matter so I could investigate more leisurely.

The error from yum looked like this (the same problem affected the screen package as affected initscripts):

Downloading Packages:
screen-4.0.3-4.el5.i386.rpm          | 559 kB      00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
groupadd: unable to open group file
error: %pre(screen-4.0.3-4.el5.i386) scriptlet failed, exit status 10
error:   install: %pre scriptlet failed (2), skipping screen-4.0.3-4.el5

  screen.i386 0:4.0.3-4.el5

# cat /selinux/enforce

The way I dealt with this initially was to temporarily disable SELinux enforcing, update the package, then reboot (to also load a kernel update):

# setenforce 0
# yum -y upgrade
# shutdown -r now

But following up on the specific error message showed:

# ls -lFaZ /etc/group
-rw-r--r--  root root system_u:object_r:file_t:s0      /etc/group

Aha! The SELinux context is wrong. Given that this has happened a couple of different machines, I'm guessing some past upgrade broke the context. What should it be? Let's check /etc/passwd for reference:

# ls -lFaZ /etc/passwd
-rw-r--r--  root root system_u:object_r:etc_t:s0       /etc/passwd

That's confirmed the correct context for /etc/group on another working server. To fix:

# chcon system_u:object_r:etc_t:s0 /etc/group

Then the updates proceed fine. It would be nice to find out what past action set the context wrong on /etc/group.


Dyoma said...

this is bad way.
better find in fedora new selinux rules or rewrite it.

Jon Jensen said...

Rewrite what specifically? Or new SELinux rules for what? The context was wrong on /etc/group, so I think initscripts and screen were fine.

The mystery is: What set the /etc/group context wrong in the first place?

Kiel said...

Try making minor changes using 'vigr'. Maybe it's atomic change methods aren't SELinux friendly.

Jon Jensen said...

That's a good thought, Kiel.

I tried it and vigr did the right thing, at least in this case.

However, I also found a handful of shared libraries in /lib had the wrong context, a generic system_u:object_r:file_t:s0 instead of the system_u:object_r:lib_t:s0 they need.

Included were libsemanage, libwrap, libtermcap, libattr, and others. Especially affected were symlinks from e.g. /usr/lib/ to /usr/lib/

It appears that some major update in the past messed up quite a lot of SELinux contexts!

Armor said...

I am admining a RHEL 5.x server and had the same issue with "Screen> could not get it update after repeated attempts.
I used chcon system_u:object_r:etc_t:s0 /etc/group and the issue was cured. I know I should also delve into the issue, but the fix on this operational system was welcome relief.
Thanks for the accurate post.

Jon Jensen said...

Tonight I updated 3 more RHEL 5 x86_64 machines with SELinux enabled, and 2 of the 3 had the same problem with broken contexts for /etc/group and other files.

On those two servers I knew an SELinux relabling would be safe, as there were no custom labels applied, so I did:

fixfiles relabel

As described in Red Hat's documentation:

And all was well. Still a mystery how /etc contexts got out of whack.