Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Panel
borderColorblack
borderStylesolid
titleXymon

Xymon is the web based monitoring tool of choice for Administrators and staff supporting ITSO Windows and Unix processes on the Purdue Campus. It performs a valuable duty, giving the trained eye a quick overview of the hardware and processes that may be out of sync for several key University areas.

In general, Operations staff will only need to respond to Critical (Red) alerts that affect production devices (Servers, Virtual Machines, etc.) All production Xymon alerts should also be reflected in Squared Up. Alarms for McAfee (mcshield.exe) should not be reported unless they last for several hours.

While Operations staff are primarily concerned with Critical (Red) alerts, they should also be familiar with the various other colors of Xymon alerts and their meanings:

Xymon icons.PNG

Note
titlemcshield.exe

If a system has a red CPU alert, check the processes to see if it is due to mcshield. It will be on top and a high percentage.

This is McAfee running an anti-virus sweep on this machine. It will take up a lot of CPU cycles, but can be ignored.

1. After a new Xymon critical alert pops up for a production machine, begin considering which group to contact, if any. If the alert is still present after 20 minutes IOC will need to call the system owner.

2. Is the alert for a clustered device? Some systems, like Mailhub, are clustered, and thus can have several alerts before one needs to take action. Generally clustered machines will NOT alarm in Squared Up for individual boxes, but they will in Xymon. This clue can help an operator determine the severity of the Xymon alert. There are some clustered systems which react in the opposite manner - They will alert in Squared Up but not Xymon until critical mass is reached. In these cases ensure enough machines are in alert before contacting the appropriate admins.

3. Locate the night's planned maintenance in the Footprints Change and Release Management Workspace Calendar to ensure the device is not scheduled to be down.

4. Xymon - Click the status icon (pictured below) along the row corresponding to the trouble server. 

5. Click on the server name to find further instructions on who to contact or if the alarm should be ignored. Follow any special instructions for the machine OR use the Configuration Management Database (CMDB) to locate the appropriate on-call. For further use on how to use CMDB: Configuration Management Database (CMDB) .

6. Call the on-call and inform them of the situation, affected device, and any other issues that may be cropping up due to the alert. Send a follow-up email. For CPU, memory, and disk alerts, paste the Xymon alert text into the email.

Warning

If there is no answer from the groups group's on call number, leave a voicemail. Call back again in 10 20 minutes. If there is no answer from the group by phone or email after this time, contact the group's manager. If you have questions, consult with your supervisor or on call CSC / IOC supervisor.

7. Log the contact, and appropriate follow-up activities. 

How to find the system owner (Admin to call):

      1. If the system name is clickable, there will be special instructions. Follow those instructions

System name.png

    • These instructions are many times instructions about when NOT to call.

instructions.PNG

         2. Find the system in the Footprints Change and Release management CMDB.

    • Search for the name of the system
    • Click on the related CI(s) link.
    • Click the bubble button CI’s to “Named server”
    • Click on the Managed by Relationship, and click “Go To”
    • The on call group information should be listed here.

          3. If there is no information from the previous steps

    • Search the communication log for the alarming system.
    • Review past correspondence and determine who to call.
    • If the server starts with a “W” it is most likely the windows on call
    • If the server starts with an “L” it is most likely the unix on call.(Linux)
    • Consult with your coworkers.
    • Consult your supervisor, or on call supervisor.
Info
iconfalse
titlePurple Alerts

Treat a production machine with numerous purple status alerts as if it were a red alert.


...