Feb 01

Problem Spotlight: Cannot Start Microsoft Exchange System Attendant and Event ID 9157 is Logged in the Event Viewer

Two weeks ago, I got an emergency call on a Friday afternoon. A local Scottsdale Arizona business was having issues with its email server. They stated that their Microsoft Exchange Server was down. When I arrived onsite, the business’ IT support personnel filled me in on the issue and what they had tried to fix the issue. The server apparently had gone down the previous night after it was rebooted to install OS updates. They had been trying to fix the issue for most of the day.

After inspecting the server, I realized that the Information Store was dismounted. They were running Microsoft Exchange Server 2003. After closer inspection, I found out that the Microsoft Exchange System Attendant service couldn’t start, and the event ID 9157 was logged in the Event Viewer. The following is a description of that event: “Microsoft Exchange Server computer system attendant does not have sufficient rights to read Exchange Server configuration objects in Active Directory. System attendant will try again in approximately one minute.” This was, obviously, a problem.

The Microsoft Exchange System Attendant is critical to Exchange Server’s stability and performance. Its components and sub components work together to facilitate Active directory communication from and to clients (Outlook) and Exchange Services. Also, many Exchange related Services will not function without it. If the Microsoft Exchange System Attendant is not running, many services will not run including the Information Store. This was the reason their Information Store was dismounted, and their Exchange Server was down.

Based on the error message, instinct would tell us to start checking and troubleshooting permissions. Fortunately, I had seen this problem before and was able to find a solution quickly without wasting much time troubleshooting. It turns out that when the Microsoft System Attendant service starts, it looks for certain groups ONLY in the default Users Active Directory Container. The groups the System Attendant service looks for are the following:

  • Exchange Services
  • Exchange Domain Servers
  • Exchange Enterprise Servers

If any of these groups have been moved to a different container, the System Attendant service will not find these groups and will not start. To fix this problem just move these groups back to the default Users container and restart the System Attendant service.

That was all I needed to do to fix my Customer’s issue. It turned out they had hired a “Network Engineer” to maintain their network a week before. This person started to “clean” Active Directory. One of the things he did to “clean” Active Directory was move all the contents of the default Users container to an Organizational Unit created by him. Then a week later the server was restarted and the System Attendant Service did not start. I was called in because he wasn’t able to fix the problem.

Well, there you have it. It was a simple fix. Until next time!

Mar 06

Blue Screen of Death after installing Microsoft update.

This past February, before I put this blog online, I had an issue with some of my customers’ computers. Some of my customers called me asking for help because their computers wouldn’t load. They said their computers kept rebooting. I know it’s been almost a month now, but I think this issue deserves a post in this blog.

After troubleshooting the issue I traced down the source of the problem to a Microsoft update. More specifically to update KB977165/MS10-015. As soon as I removed this update the computer was able to load windows perfectly fine.

This was odd though. Other computers where the same update was installed didn’t have this issue. This indicated that the problem was caused by something else – not the Microsoft Update. I scanned the computer for viruses and found nothing. I re-installed the update and the computer entered the boot loop once again. After some more troubleshooting I traced the cause of the stop error to the file %System32\drivers\atapi.sys. There was a problem with this file. It had to be infected. I uploaded the file to virustotal.com, and the results came back clean in all but one of the scans. It just said that it was probably infected with a root kit, but it wouldn’t give me more information.

This made sense. Rootkits are designed to hide themselves or other malware from antivirus applications. This is probably why the anti-virus scan I ran didn’t catch anything.

I decided to take a different approach, and I took the hard-drive out of the computer. I connected the hard-drive to another computer and scanned it using an up to date anti-virus (Kaspersky). The scan found several items and cleaned them successfully. One of the infected files was atapi.sys. After this, I installed the update and the computer didn’t reboot again.

So there you have it. The problem was caused by a an infection on the PC.

My suspicions were later confirmed by Microsoft. They apparently took some customers’ computers with them to check them and found the source of the problem. They state on their security response blog that the problem was caused by the Alureon root kit.

They state on their blog:

the presence of Alureon does not allow for a successful boot of the compromised system. The Windows Engineering team continued testing different configurations, as well as retesting several third party applications, leading to our firm conclusion that the blue screen issue is the result of the Alureon rootkit.

So there you have it. They later released a version of the update that does not install if it detects the system is in a state that will cause it to enter this reboot loop.

If you are a victim of this problem, make sure your computer is free from infection. Microsoft recommends to re-install your operating system if you cannot get rid of the infection.

I don’t think this is necessary. If you are in the Phoenix Arizona area and need assistance with this issue, you can contact us and we will gladly help you.