The IOC operator's duties are primarily ones of observation and proactive intervention. The job revolves around inputs from Xymon and Squared Up, as well as human feedback from sources such as the CSC, Classrooms, etc.
...
| Panel |
|---|
| borderColor | black |
|---|
| borderStyle | solid |
|---|
| title | Network Device Alerts |
|---|
|
| Note |
|---|
- Locate the night's planned maintenance in the Footprints Change and Release Management workspace calendar to ensure the device is not scheduled to be down.
- Action should be considered for any Squared Up alarm that shows up in the Normal Operations Network View for 20 minutes or longer in duration. The appropriate steps should be taken as outlined in Section II, Incident On-Call Process found elsewhere in this document.
- Action should be taken for network issues reported by I-Light (GigaPOP), the ITaP help desk, campus personnel, students, and/or visitors.
|
8:00 AM to 5:00 PM, Monday through Friday: | Issue | Action |
|---|
- Wireless/PAL issues that are being experienced by multiple clients.
- Reports of multiple data PIC service issues in an area.
- Report of single HIGH PRIORITY data PIC or wireless PAL outage.
- Call from I-Light / GigaPOP
| - Follow the normal on-call procedures
|
Outside of 8:00 AM to 5:00 PM, Monday through Friday: | Issue | Action |
|---|
- Wireless/PAL issues that are being experienced by multiple clients in different buildings.
| - Send an email to itns-pdnhlog-ext@lists.purdue.edu to notify Data Networking. If this issue seems to be high priority or widespread it may be justified to escalate to our normal on-call procedures.
| - Reports of multiple data PIC service issues in an area.
- Report of single HIGH PRIORITY data PIC or wireless PAL outage.
- Call from I-Light / GigaPOP
| - Follow the normal on-call procedures.
|
| Anchor |
|---|
| networkincident |
|---|
| networkincident |
|---|
|
Network Incident On-Call ProcessFor all SQUARED UP alarms, trouble calls, and StruxureWare alarms: ERHT 5A/5B, LAMB 20, LYNN B168, or TEL 210 follow the on-call triage process: For alarms originating from StruxureWare, DCM (Data Center Management) personnel should be notified in addition to proceeding with the Data Networking (ITIS) notification steps outlined below. - Verify that the building that is showing the alarm has electrical power. Pass this information to the on-call and in the follow up email.
To determine if a device is down on Squared Up due to power outage please start checking the middle screen that networking maintains. • If it is on this list do not call.
• If a building near the building listed on the power outage list is on Squared Up do not call. (Please only call if it is still showing on Squared Up once the power outage time listed is over.)
(You can look on Purdue Campus Map https://www.purdue.edu/campus_map/ to tell if where buildings are located) For alarms originating from StruxureWare, DCM personnel should be notified in addition to proceeding with the Data Networking notification steps outlined below. - Call Data Networking (ITIS) Primary On-Call phone (765-494-1591). If no answer, voice mail should be left on the phone and you should wait 5 minutes before proceeding.
- A Footprints ticket needs to be made and assigned to ITIS Networking. Put the ticket number in the follow up email. Contact info is the on-calls name.
- An email should be sent to itns-pdnhlog-ext@lists.purdue.edu. Justin McIntyre would like to be cc'd in the email also: mcintyrj@purdue.edu
You do not need to send another email when you try to reach someone. If you leave a message, include this sentence in the body of the email: "We will escalate the on-call process if there is no response within 5 minutes."
- Call Data Networking (ITIS) Secondary On-Call phone (765-494-1530). If no answer, voice mail should be left on the phone and you should wait 5 minutes before proceeding.
Call Justin McIntyre at 630-675-7640. If no answer, call Richard Letts 206-790-5837. Wait 5 minutes before proceeding to Step 7.
- In the event that none of the individuals from Steps 3 through 6 above has responded, repeat those steps until contact is made.
Special Notes:- Data Networking personnel who are contacted by IOC staff are responsible for providing issue status updates to IOC in a timely fashion. As a general rule this means that such feedback should be provided once you start investigating an issue, whenever an ETA for resolution has been determined and again when the issue has been resolved. Additional updates are always welcome as well, especially for extended outages.
- If Squared Up, itself, ever goes down for more than 20 mins make sure to let Data Networks know that we cannot monitor their devices while it is down. (If it is the Xymon part down do NOT call Data Networks as it does not pertain to their devices, only when "Network Devices" section is down). IT Service Management (ITSMO) 765-496-6390/ itcr-itsmo@lists.purdue.edu will be the on-call if Squared Up is down. (If it is the Xymon part down do NOT call Data Networks as it does not pertain to their devices.)
- Battery Alarms in Squared Up: Email Data Networks only. Do NOT call. Make FP ticket, assign to Networking and contact can be left blank or jpublic. Ex's: anything with "APC" in it.
Ignore the following list of buildings outside of normal business hours:
Cases where it is outside of business hours (as is also defined below), please do not call the data networks on-call staff for Squared Up alarms in any of the locations listed in the table down below. Essentially if the hostname value of the alarm contains a value from the “Squared Up Value” column of the list down below, it should be ignored. These values are typically fairly static but should any updates need to occur we’ll forward you an updated list. - Business Hours are Monday through Friday from 7:00 AM to 5:00 PM
- Official Purdue Holidays and Weekends Adjoining those Holiday Dates are Considered to be Non-Business Hours, Superseding the Monday through Friday Standard
Example: 5/25/19 through 5/27/19 (Memorial Day and the weekend adjoining it) would not be considered business hours. Name | Description | Squared Up Value | | 2550 | State Farm Building 2550 Northwestern Ave | "2550-" | AC20 | Superintendent Dwelling (ACRE Farm) | "ac20-" | AC22 | Field Research Facility (ACRE Farm) | "ac22-" | AC35 | Pest Lab and Storage Facility (ACRE Farm) | "ac35-" | AC41 | Grain Drying Complex - Grain Auger (ACRE Farm) | "ac41-" | AC42 | Scales House (ACRE Farm) | "ac42-" | AC43 | USDA Soybean Research Lab (ACRE Farm) | "ac43-" | AC44 | USDA Rainulator Building Soil Erosion (ACRE Farm) | "ac44-" | AC45 | Var Test Facility (ACRE Farm) | "ac45-" | AC46 | Headquarters and Shop (ACRE Farm) | "ac46-" | AC51 | Weather Facility (ACRE Farm) | "ac51-" | AF01 | Aquaculture (ASREC Farm) | "af01-"  | AFC | Anderson Flagship Center | "afc-" | AIDC | Agricultural Information Distribution Center | "aidc-" | B201 | Swine Evaluation Headquarters (ASREC Farm) | "b201-" | B401 | Poultry and Hatchery Facility (ASREC Farm) | "b401-" | B501 | Sheep Research and Teaching Facility (ASREC Farm) | "b501-" | B602 | Feed Mill (ASREC Farm) | "b602-" | B701 | Swine Office Metabolism Facility (ASREC Farm) | "b701-" | B713 | Environmental Research Facility (ASREC Farm) | "b713-" | B801 | Farm Operations Shop and Headquarters (ASREC Farm) | "b801-" | B901 | Teaching Center and Classroom (ASREC Farm) | "b901-" | BECK | Beck Agricultural Center (ACRE Farm) | "beck-" | CB10 | Beef Building (ASREC Farm) | "cb10-" | FHBC | Family Health Clinic of Burlington (Carroll County) | "fhbc-" | FHD2 | Family Health Clinic 2 of Delphi (Carroll County) | "fhd2-" | FHDC | Family Health Clinic 1 of Delphi (Carroll County) | "fhdc-" | FHMW | Family Health Clinic of Monon (White County) | "fhmw-" | FHWW | Family Health Clinic of Wolcott (White County) | "fhww-" | GRO | Government Relations Office in Indianapolis | "-gro-" | ICSC | Indiana Corn and Soybean Innovation Center | "icsc-" | IDOT | Indiana Department of Transportation | "idot-" | INOK | Investments Warehouse | "inok-" | INTECH10 | Technical Assistance Program in Fishers | "-intech10-" | KPNR | 1500 Kepner Drive Facility | "kpnr-" | SCHO | Schowe House | "scho-" | SD02 | Dairy Research Unit (ASREC Farm) | "sd02-" | SIA | Subaru of Indiana Automotive | "sia-" | SWSB | Statewide Tech South Bend Office #2 | "swsb-" | TAP | Technical Assistance Program in Indianapolis | "-tap-" | TM02 | Throckmorton Pesticide Building | "tm02-" | TM08 | Throckmorton Meigs Building | "tm08-" | TM36 | Throckmorton Farm Crop Barn | "tm36-" | TPB | Rankin Track Press Box | "tpb-100" | UNPD | University Police Department | "unpd-" | USDA | USDA Building 1 (ASREC Farm) | "usda-" | WH9 | Well House 9 | "wh9-" | WRIT | John S Wright Forestry Center | "writ-" |
For Squared Up alarms report the following information from Squared Up when reporting issue: - Date/Time that alarm started (Example: October 26, 10:00 AM)
- Affected Device Name (Example: mrdh-285n-c2950-01)
- Last Ping (Example: 2017-11-01 11:14:43)
For network issues reported by I-Light / GigaPOP or the ITaP help desk, campus personnel, students, or visitors, report the following information: - Date/Time that issue began or was first noticed (Example: October 26, 10:00 AM)
- Affected service (Examples: wireless/PAL, data PIC(s), or I-Light / GigaPOP call back)
- Location where problem is occurring (Examples: wireless on 2nd floor of Armstrong, all data PICs in the Forestry building, or I-Light / GigaPOP). Whenever possible obtain a specific building and nearest room number.
- Name and phone number (or at least email) of person experiencing/reporting problem
More details can be found on the attached document below: ITIS-Data Network Incident Resolution Process.docx
|
...