The IOC operator's duties are primarily ones of observation and proactive intervention. The job revolves around inputs from Xymon and Squared UpGrafana and other monitoring apps, as well as human feedback from sources such as the CSCPurdue It Service Desk, Classrooms, etc.
| Panel | |||||||
|---|---|---|---|---|---|---|---|
| |||||||
|
| Anchor | ||||
|---|---|---|---|---|
|
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Admin Follow-up Email TemplateAfter calling an admin to report an issue, a follow-up email should be sent to the admin's group (not the admin themselves) using the following template:
|
| Anchor | ||||
|---|---|---|---|---|
|
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
On your workstation:
On the monitoring computers:
|
| Anchor | ||||
|---|---|---|---|---|
|
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
The IOC Shift Log (found here) is how records are kept during the shift and between shifts. While you are monitoring, one of your duties is to maintain the log. This includes being aware of existing log entries and their implications, making new log entries when certain events occur, and updating log entries as their status changes or new information is reported. The following events should be noted in the log when they occur:
Further information on the process for updating and maintaining the shift log can be found here: IOC: Shift Log Entries. |
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Squared Up will notify IOC of two different types of alerts: Xymon alarms (covered in the next subsection) and Network Device Alerts (covered in the next subsection) . Xymon Alerts:Before calling on any alarm review the following:
After a new Squared Up alert pops up for a production machine, If the alert is still present after 20 minutes Operations will need to call the group responsible for the system. |
...
| borderColor | black |
|---|---|
| borderStyle | solid |
| title | Xymon |
Xymon is the web based monitoring tool of choice for Administrators and staff supporting ITSO Windows and Unix processes on the Purdue Campus. It performs a valuable duty, giving the trained eye a quick overview of the hardware and processes that may be out of sync for several key University areas.
In general, Operations staff will only need to respond to Critical (Red) alerts that affect production devices (Servers, Virtual Machines, etc.) All production Xymon alerts should also be reflected in Squared Up. Alarms for McAfee (mcshield.exe) should not be reported unless they last for several hours.
While Operations staff are primarily concerned with Critical (Red) alerts, they should also be familiar with the various other colors of Xymon alerts and their meanings:
1. After a new Xymon critical alert pops up for a production machine, begin considering which group to contact, if any. If the alert is still present after 20 minutes IOC will need to call the system owner.
2. Is the alert for a clustered device? Some systems, like Mailhub, are clustered, and thus can have several alerts before one needs to take action. Generally clustered machines will NOT alarm in Squared Up for individual boxes, but they will in Xymon. This clue can help an operator determine the severity of the Xymon alert. There are some clustered systems which react in the opposite manner - They will alert in Squared Up but not Xymon until critical mass is reached. In these cases ensure enough machines are in alert before contacting the appropriate admins.
3. Locate the night's planned maintenance in the Footprints Change and Release Management Workspace Calendar to ensure the device is not scheduled to be down.
4. Xymon - Click the status icon (pictured below) along the row corresponding to the trouble server.
5. Click on the server name to find further instructions on who to contact or if the alarm should be ignored. Follow any special instructions for the machine OR use the Configuration Management Database (CMDB) to locate the appropriate on-call. For further use on how to use CMDB: Configuration Management Database (CMDB) .
6. Call the on-call and inform them of the situation, affected device, and any other issues that may be cropping up due to the alert. Send a follow-up email. For CPU, memory, and disk alerts, paste the Xymon alert text into the email.
| Warning |
|---|
If there is no answer from the group's on call number, leave a voicemail. Call back again in 20 minutes. If there is no answer from the group by phone or email after this time, contact the group's manager. If you have questions, consult with CSC / IOC supervisor. |
7. Log the contact, and appropriate follow-up activities.
How to find the system owner (Admin to call):
1. If the system name is clickable, there will be special instructions. Follow those instructions
- These instructions are many times instructions about when NOT to call.
2. Find the system in the Footprints Change and Release management CMDB.
- Search for the name of the system
- Click on the related CI(s) link.
- Click the bubble button CI’s to “Named server”
- Click on the Managed by Relationship, and click “Go To”
- The on call group information should be listed here.
3. If there is no information from the previous steps
- Search the communication log for the alarming system.
- Review past correspondence and determine who to call.
- If the server starts with a “W” it is most likely the windows on call
- If the server starts with an “L” it is most likely the unix on call.(Linux)
- Consult with your coworkers.
- Consult your supervisor, or on call supervisor.
| Info | ||||
|---|---|---|---|---|
| ||||
Treat a production machine with numerous purple status alerts as if it were a red alert. |
...
| borderColor | black |
|---|---|
| borderStyle | solid |
| title | Network Device Alerts |
| Note |
|---|
|
8:00 AM to 5:00 PM, Monday through Friday:
...
- Wireless/PAL issues that are being experienced by multiple clients.
- Reports of multiple data PIC service issues in an area.
- Report of single HIGH PRIORITY data PIC or wireless PAL outage.
- Call from I-Light / GigaPOP
...
- Follow the normal on-call procedures
Outside of 8:00 AM to 5:00 PM, Monday through Friday:
...
- Wireless/PAL issues that are being experienced by multiple clients in different buildings.
...
- Send an email to itns-pdnhlog-ext@lists.purdue.edu to notify Data Networking. If this issue seems to be high priority or widespread it may be justified to escalate to our normal on-call procedures.
...
- Reports of multiple data PIC service issues in an area.
- Report of single HIGH PRIORITY data PIC or wireless PAL outage.
- Call from I-Light / GigaPOP
...
- Follow the normal on-call procedures.
...
Network Incident On-Call Process
For all SQUARED UP alarms and trouble calls follow the on-call triage process:
For alarms originating from StruxureWare, DCM (Data Center Management) personnel should be notified in addition to proceeding with the Data Networking (ITIS) notification steps outlined below.
...
To determine if a device is down on Squared Up due to power outage please start checking the middle screen that networking maintains.
• If it is on this list do not call.
• If a building near the building listed on the power outage list is on Squared Up do not call. (Please only call if it is still showing on Squared Up once the power outage time listed is over.)
(You can look on Purdue Campus Map https://www.purdue.edu/campus_map/ to tell if where buildings are located)
...
For alarms originating from StruxureWare, DCM personnel should be notified in addition to proceeding with the Data Networking notification steps outlined below.
...
Call Justin McIntyre at 630-675-7640. If no answer, call Richard Letts 206-790-5837. Wait 5 minutes before proceeding to Step 8.
...
Special Notes:
...
Cases where it is outside of business hours (as is also defined below), please do not call the data networks on-call staff for Squared Up alarms in any of the locations listed in the table down below. Essentially if the hostname value of the alarm contains a value from the “Squared Up Value” column of the list down below, it should be ignored. These values are typically fairly static but should any updates need to occur we’ll forward you an updated list.
- Business Hours – Monday through Friday – 7:00 AM to 5:00 PM
- (Official Purdue Holidays and Weekends Adjoining those Holiday Dates are Considered to be Non-Business Hours, Superseding the Monday through Friday Standard)
- Example: 5/25/19 through 5/27/19 (Memorial Day and the weekend adjoining it) would not be considered business hours
...
Building Short Name
...
Building Long Name
...
SquaredUp Value
...
-
...
Any APC Devices
...
"-apc******-"
...
-
...
Any TRP Devices
...
"-trp******-"
...
2550
...
State Farm
...
"2550-"
...
844S
...
844 South River Road
...
"844s-"
...
AC22
...
Field Research Facility (ACRE Farm)
...
"ac22-"
...
AC35
...
Pest Lab and Storage Facility (ACRE Farm)
...
"ac35-"
...
AC41
...
Grain Drying Complex - Grain Auger (ACRE Farm)
...
"ac41-"
...
AC42
...
Scales House (ACRE Farm)
...
"ac42-"
...
AC43
...
USDA Soybean Research Lab (ACRE Farm)
...
"ac43-"
...
AC44
...
USDA Rainulator Building Soil Erosion (ACRE Farm)
...
"ac44-"
...
AC45
...
Var Test Facility (ACRE Farm)
...
"ac45-"
...
AC46
...
Headquarters and Shop (ACRE Farm)
...
"ac46-"
...
AC51
...
Weather Facility (ACRE Farm)
...
"ac51-"
...
AC54
...
Crop Diagnostic Training Center (ACRE Farm)
...
"ac54-"
...
AF01
...
Aquaculture (ASREC Farm)
...
"af01-"
...
AFC
...
Anderson Flagship Center
...
"afc-"
...
AIDC
...
Agricultural Information Distribution Center
...
"aidc-"
...
ASB
...
Airport Service Building
...
"asb-"
...
B201
...
Swine Evaluation Headquarters (ASREC Farm)
...
"b201-"
...
B401
...
Poultry and Hatchery Facility (ASREC Farm)
...
"b401-"
...
B501
...
Sheep Research and Teaching Facility (ASREC Farm)
...
"b501-"
...
B602
...
Feed Mill (ASREC Farm)
...
"b602-"
...
B701
...
Swine Office Metabolism Facility (ASREC Farm)
...
"b701-"
...
B713
...
Environmental Research Facility (ASREC Farm)
...
"b713-"
...
B801
...
Farm Operations Shop and Headquarters (ASREC Farm)
...
"b801-"
...
B901
...
Teaching Center and Classroom (ASREC Farm)
...
"b901-"
...
BBCH
...
Purdue Baseball Clubhouse
...
"bbch-"
...
BBPB
...
Purdue Baseball Press Box
...
"bbpb-"
...
BECK
...
Beck Agricultural Center (ACRE Farm)
...
"beck-"
...
BTV
...
Boiler Television Building
...
"btv-"
...
CB10
...
Beef Building (ASREC Farm)
...
"cb10-"
...
COAL
...
Coal Handling Control/Fire Pump Building
...
"coal-"
...
FSHR
...
TAP Fishers Remote Site
...
"fshr"
...
GCMB
...
Golf Course Maintenance Barn
...
"gcmb-"
...
GMF
...
Grounds Maintenance Facility
...
"gmf-"
...
GMGF
...
Grounds Maintenance Greenhouse Facility
...
"gmgf-"
...
ICSC
...
Indiana Corn and Soybean Innovation Center (ACRE Farm)
...
"icsc-"
...
IDOT
...
Indiana Department of Transportation
...
"idot-"
...
INDY
...
Indianapolis External Site
...
"indy-"
...
INOK
...
Investments Warehouse
...
"inok-"
...
INSS
...
Intramural Storage Shed
...
"inss-"
...
KKM
...
TAP Kokomo Site
...
"kkm-"
...
LMSB
...
Laboratory Material Storage Building
...
"lmsb-"
...
NA
...
TAP New Albany Site
...
"na-"
...
NACC
...
Native American Educational and Cultural Center
...
"nacc-"
...
PAGE
...
Thomas A. Page Pavilion
...
"page-"
...
PWB
...
Purdue West Annex - Building B
...
"pwb-"
...
PWC
...
Purdue West - Building C
...
"pwc-"
...
RALR
...
Stadium Area - Visiting Team Locker Room
...
"ralr-"
...
SBCH
...
Purdue Softball Clubhouse
...
"sbch-"
...
SBPB
...
Purdue Softball Press Box
...
"sbpb-"
...
SCHO
...
Schowe House
...
"scho-"
...
SD02
...
Dairy Research Unit (ASREC Farm)
...
"sd02-"
...
SIA
...
Subaru of Indiana Automotive
...
"sia-"
...
SOCC
...
Purdue Women’s Soccer Building
...
"socc-"
...
SPUR
...
Spurgeon Golf Training Center
...
"spur-"
...
SWNA
...
State Wide New Albany
...
"swna-"
...
SWSB
...
State Wide South Bend
...
"swsb-"
...
TM02
...
Throckmorton Pesticide Building
...
"tm02-"
...
TM08
...
Throckmorton Meigs Building
...
"tm08-"
...
TM11
...
Throckmorton Fruit Barn
...
"tm11-"
...
TM36
...
Throckmorton Farm Crop Barn
...
"tm36-"
...
TPB
...
Rankin Track Press Box
...
"tpb-"
...
TURF
...
Intercollegiate Athletic Sports Turf Building
...
"turf-"
...
UNPD
...
University Police Department
...
"unpd-"
...
USDA
...
USDA Building 1 (ASREC Farm)
...
"usda-"
...
VCPR
...
Veterinary Center for Paralysis Research
...
"vcpr-"
...
VLAB
...
Veterinary Laboratory Animal Building
...
"vlab-"
...
VOIN
...
Voinoff (Samuel) Golf Pavilion
...
"voin-"
...
VTCH
...
Vision Technology 1 Building
...
"vtch-"
...
VPRB
...
Veterinary Pathobiology Research Building
...
"vprb-"
...
WH9
...
Well House 9
...
"wh9-"
...
WRIT
...
John S Wright Forestry Center
...
"writ-"
Information to Provide When Reporting Network Issues
For Squared Up alarms report the following information from Squared Up when reporting issue:
- Date/Time that alarm started (Example: October 26, 10:00 AM)
- Affected Device Name (Example: mrdh-285n-c2950-01)
- Last Ping (Example: 2017-11-01 11:14:43)
For network issues reported by I-Light / GigaPOP or the ITaP help desk, campus personnel, students, or visitors, report the following information:
- Date/Time that issue began or was first noticed (Example: October 26, 10:00 AM)
- Affected service (Examples: wireless/PAL, data PIC(s), or I-Light / GigaPOP call back)
- Location where problem is occurring (Examples: wireless on 2nd floor of Armstrong, all data PICs in the Forestry building, or I-Light / GigaPOP). Whenever possible obtain a specific building and nearest room number.
- Name and phone number (or at least email) of person experiencing/reporting problem
More details can be found on the attached document below:
ITIS-Data Network Incident Resolution Process.docx
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
|
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
This protocol also pertains to when they come in on the weekends for a key . Ask what they are working on and send email to ITI Data Center Management.Include the building, room, device name, and Physical Facilities technician name in the e-mail. |
...
Generator Test TEL Nodes
- ERHT
- LAMB
- LYNN
- TEL
Crac units Data Centers
- FREH G2 CRAC #1,2,3,4
- FREH G57 CRAC 1
- FREH G60 CRAC 1, ACG-2
- HAAS CRAC 1,2,3
- MATH B60 CRAC 1,2,3
- MATH G72 Chiller
- MATH G109 CRAC 5,6,13
- MATH G190 CRAC 1,12,32
TEL Nodes
- ERHT 5 CRAC 1,2
- LAMB 20 ACG-20,21
- LYNN G168 ACG-40, 41
- TEL 210 CRAC 1,2,3
...
Cameras
StruxureWare also includes monitoring functionality for the cameras in the data centers. This view in StruxureWare should be open at all times on the large screen.
- Please monitor the cameras from time to time to make sure nothing suspicious is going on. Check the Shift Log, Change and Release Management Workspace Calendar and your email for scheduled work. If it appears that the occupants do not belong in the room or are removing things during the night, notify PUPD (48221) to investigate. You will also need to call Todd Turner if you do call PUPD, or if you feel unsure.
- HAAS is considered a lights-out facility with several groups who have card access. The building fire panel is located in the datacenter. The Fire & Safety group works Midnight to 8am doing panel tests across campus, and you will see them or the PUFD in the room from time-to-time.
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
NOTE: There is no audio alarm for Siemens. If the Alarms button turns red, call Patrick Finnegan.
Attempt to call their cell first. If you cannot reach Patrick by any of those numbers, call Todd Turner at 68214.
If the IOC receives an automated call regarding the MATH back-up chiller:
|
...
| borderColor | black |
|---|---|
| borderStyle | solid |
| title | UC4 |
General Job Failure
Production Control must be contacted within 30 minutes.
- Between 08:00 – 17:00 on Monday – Friday (during non-holiday workdays), do not call the PCA; only send a report utilizing the shift log abend reporting tool. Follow-up emails are not necessary due to the log automatically sending out an email once the abend has been reported.
- Otherwise, call the PCA (Production Control Analyst) at the number listed within UC4’s console (60287).
Down Agent
- Immediately call the PCA listed within UC4’s console (60287) and send a follow-up email.
...
- Do not restart items with parentheses. These are child processes of the item that needs restarted.
- Click ‘Restart’ once.
- If it fails again, follow General Job Failure instructions.
Notifications
- React as necessary following any notifications or instructions.
- If needed, contact PCAs as with General Job Failures.
Further information regarding UC4’s configuration and operation can be found here.
...
| borderColor | black |
|---|---|
| borderStyle | solid |
| title | Outage Types with Unique Instructions |
Exchange Outage
Exchange outages are very similar to other outages with one exception. When Exchange goes down, Operations staff members are required to inform Purdue Police, Purdue Fire, and the CSC. Contact these groups both at the start, and end of the outage. It may be proper to provide updates, as the situation warrants.
- 24/7 call: Exchange, Purdue Police (48221), Purdue Fire (46919), CSC.
Mcafee ePO Updates
There may be times when various campus system admins need exceptions or policies changed in McAfee ePolicy Orchestrator. With the most recent ePO system we are limiting their permission sets, so in an emergency they need to be able to contact someone from the Security Engineering team to implement the changes for them. Follow on-call procedures when an admin calls the IOC with a request like this.
...
...
...
...