Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The IOC operator's duties are primarily ones of observation and proactive intervention. The job revolves around inputs from Xymon and Squared Up, as well as human feedback from sources such as the CSC, Classrooms, etc.

...

Panel
borderColorblack
borderStylesolid
titleNetwork Device Alerts


Note
  • Locate the night's planned maintenance in the Footprints Change and Release Management workspace calendar to ensure the device is not scheduled to be down.
  • Action should be considered for any Squared Up alarm that shows up in the Normal Operations Network View for 20 minutes or longer in duration.  The appropriate steps should be taken as outlined in Section II, Incident On-Call Process found elsewhere in this document.
  • Action should be taken for network issues reported by I-Light (GigaPOP), the ITaP help desk, campus personnel, students, and/or visitors.


8:00 AM to 5:00 PM, Monday through Friday:           
IssueAction
  • Wireless/PAL issues that are being experienced by multiple clients.
  • Reports of multiple data PIC service issues in an area.
  • Report of single HIGH PRIORITY data PIC or wireless PAL outage.
  • Call from I-Light / GigaPOP
  • Follow the normal on-call procedures
Outside of 8:00 AM to 5:00 PM, Monday through Friday:          
IssueAction
  • Wireless/PAL issues that are being experienced by multiple clients in different buildings.
  • Send an email to itns-pdnhlog-ext@lists.purdue.edu to notify Data Networking. If this issue seems to be high priority or widespread it may be justified to escalate to our normal on-call procedures.
  • Reports of multiple data PIC service issues in an area.
  • Report of single HIGH PRIORITY data PIC or wireless PAL outage.
  • Call from I-Light / GigaPOP
  • Follow the normal on-call procedures.

Anchor
networkincident
networkincident

Network Incident On-Call Process

For all SQUARED UP alarms and trouble calls follow the on-call triage process:

For alarms originating from StruxureWare, DCM (Data Center Management) personnel should be notified in addition to proceeding with the Data Networking (ITIS) notification steps outlined below.

  1. Verify that the building that is showing the alarm has electrical power. Pass this information to the on-call and in the follow up email.

    To determine if a device is down on Squared Up due to power outage please start checking the middle screen that networking maintains.

    • If it is on this list do not call.
    • If a building near the building listed on the power outage list is on Squared Up do not call. (Please only call if it is still showing on Squared Up once the power outage time listed is over.)
    (You can look on Purdue Campus Map https://www.purdue.edu/campus_map/ to tell if where buildings are located)

  2. For alarms originating from StruxureWare, DCM personnel should be notified in addition to proceeding with the Data Networking notification steps outlined below.

  3. Call Data Networking (ITIS) Primary On-Call phone (765-494-1591). If no answer, voice mail should be left on the phone and you should wait 5 minutes before proceeding.

  4. A Footprints ticket needs to be made and assigned to ITIS Networking. Put the ticket number in the follow up email. Contact info is the on-calls name.


  5. An email should be sent to itns-pdnhlog-ext@lists.purdue.edu. Justin McIntyre would like to be cc'd in the email also: mcintyrj@purdue.edu
    You do not need to send another email when you try to reach someone. If you leave a message, include this sentence in the body of the email: "We will escalate the on-call process if there is no response within 5 minutes."

  6. Call Data Networking (ITIS) Secondary On-Call phone (765-494-1530). If no answer, voice mail should be left on the phone and you should wait 5 minutes before proceeding.

  7. Call Justin McIntyre at 630-675-7640. If no answer, call Richard Letts 206-790-5837. Wait 5 minutes before proceeding to Step 8.


  8. In the event that none of the individuals from Steps 3 through 6 above has responded, repeat those steps until contact is made. 

Special Notes:

  • Data Networking personnel who are contacted by IOC Operations staff are responsible for providing issue status updates to IOC Operations in a timely fashion.  As a general rule this means that such feedback should be provided once you start investigating an issue, whenever an ETA for resolution has been determined and again when the issue has been resolved.  Additional updates are always welcome as well, especially for extended outages.
  • If If Squared Up, itself, ever goes down for more than 20 mins make sure to let Data Networks know that we cannot monitor their devices while it is down. Also send a ticket to the ITAP_Netwroking_II queue regarding it is down. Any part of it (Xymon, Network devices or whole page). Networking supports Squared Up now. Follow normal procedure after you call. i.e follow up email and log. Battery Alarms in Squared Up: Email Data Networks only. Do NOT call. Make FP ticket, assign to Networking and contact can be left blank or jpublic. Ex's: anything with
  • APC Alerts: Ignore all devices that have "APC" in it.
    Ignore the the title. DO NOT email or call about these alerts. If there is a Struxureware alert associated with it, please follow DCM guidelines.

    Ignore the following list of buildings only outside of normal business hours:

    Cases where it is outside of business hours (as is also defined below), please do not call the data networks on-call staff for Squared Up alarms in any of the locations listed in the table down below.  Essentially if the hostname value of the alarm contains a value from the “Squared Up Value” column of the list down below, it should be ignored.  These values are typically fairly static but should any updates need to occur we’ll forward you an updated list.

      • Business Hours – Monday through Friday – 7:00 AM to 5:00 PM
      • (Official Purdue Holidays and Weekends Adjoining those Holiday Dates are Considered to be Non-Business Hours, Superseding the Monday through Friday Standard)
      • Example: 5/25/19 through 5/27/19 (Memorial Day and the weekend adjoining it) would not be considered business hours


    Pest Lab and Storage -""vprb-"




    Building Short Name

    Building Long Name

    SquaredUp Value

    Reason for Ignoring

    -

    Any APC Devices

    "-apc******-"

    844S

    844 South River Road

    "844s-"

    AC22

    Field Research Facility (ACRE Farm)

    "ac22-"

    AC35

    Always ignore APCs in Squared Up

    -

    Any Device with the word "test"

    "test"

    Test devices are used for testing purposes and not on the production network.

    -Any tents"-tent****-"Tents are subject to tampering/environment issues and are hard to access res hall buildings after hours

    844S

    844 South River Road

    "844s-"

    No access, users aren't there after hours

    AC22

    Field Research Facility (ACRE Farm)

    "ac22-"

    No access, users aren't there after hours, feeds off of BECK

    AC35

    Pest Lab and Storage Facility (ACRE Farm)

    "ac35-"

    No access, users aren't there after hours, feeds off of BECK

    AC41

    Grain Drying Complex - Grain Auger (ACRE Farm)"ac41-"Farm)

    "ac41-"

    No access, users aren't there after hours, feeds off of BECK

    AC42

    Scales House (ACRE Farm)

    "ac42-"

    No access, users aren't there after hours, feeds off of BECK

    AC43

    USDA Soybean Research Lab (ACRE Farm)

    "ac43-"

    No access, users aren't there after hours, feeds off of BECK

    AC44

    USDA Rainulator Building Soil Erosion (ACRE Farm)

    "ac44-"

    No access, users aren't there after hours, feeds off of BECK

    AC45

    Var Test Facility (ACRE Farm)"ac45-"Farm)

    "ac45-"

    No access, users aren't there after hours, feeds off of BECK

    AC46

    Headquarters and Shop (ACRE Farm)

    "ac46-"

    No access, users aren't there after hours, feeds off of BECK

    AC51

    Weather Facility (ACRE Farm)

    "ac51-"

    No access, users aren't there after hours, feeds off of BECK

    AC54

    Crop Diagnostic Training Center (ACRE Farm)

    "ac54-"

    No access, users aren't there after hours, feeds off of BECK

    AF01

    Aquaculture (ASREC Farm)

    "af01-""af01-"

    No access, users aren't there after hours, feeds off of BECK

    AFC

    Anderson Flagship Center

    "afc-"

    Far remote site, no access

    AIDC

    Agricultural Information Distribution Center

    "aidc-"

    No access, users aren't there after hours

    ASB

    Airport Service Building

    "asb-"

    No access, airport hanger

    B201

    Swine Evaluation Headquarters (ASREC Farm)

    "b201-"

    No access, users aren't there after hours, feeds off of BECK

    B401

    Poultry and Hatchery Facility (ASREC Farm)

    "b401-"

    No access, users aren't there after hours, feeds off of BECK

    B501

    Sheep Research and Teaching Facility (ASREC Farm)"b501-"Farm)

    "b501-"

    No access, users aren't there after hours, feeds off of BECK

    B701

    Swine Office Metabolism Facility (ASREC Farm)

    "b701-"

    No access, users aren't there after hours, feeds off of BECK

    B713

    Environmental Research Facility (ASREC Farm)

    "b713-"

    No access, users aren't there after hours, feeds off of BECK

    B801

    Farm Operations Shop and Headquarters (ASREC Farm)

    "b801-"

    No access, users aren't there after hours, feeds off of BECK

    B901

    Teaching Center and Classroom (ASREC Farm)"b901-"

    "b901-"

    No access, users aren't there after hours, feeds off of BECK

    BBCH

    Purdue Baseball Clubhouse

    "bbch-"

    Seasonal

    BBPB

    Purdue Baseball Press Box

    "bbpb-"

    Seasonal

    BTV

    Boiler Television Building

    "btv-"

    No access

    CB10

    Beef Building (ASREC Farm)

    "cb10-"

    No access, users aren't there after hours, feeds off of BECK

    COAL

    Coal Handling Control/Fire Pump Building

    "coal-"

    No access

    FSHR

    TAP Fishers Remote Site

    "fshr"

    Charles Garwood, no access, remote site

    GCMB

    Golf Course Maintenance Barn

    "gcmb-"

    No access

    GMF

    Grounds Maintenance Facility

    "gmf-"

    No access

    GMGF

    Grounds Maintenance Greenhouse Facility

    "gmgf-"

    No access

    ICSC

    Indiana Corn and Soybean Innovation Center (ACRE Farm)

    "icsc-"

    No access, users aren't there after hours, feeds off of BECK

    IDOT

    Indiana Department of Transportation

    "idot

    -"

    Andy Sydelko says they can be ignored

    INDY

    Indianapolis External Site

    "indy-"

    Gro site - Susan Brock says ignore | Esk site - Mark Sharp says ignore

    INOK

    Investments Warehouse

    "inok-"

    Andy Sydelko says they can be ignored

    INSS

    Intramural Storage Shed

    "inss-"

    No access, only one user

    KKM

    TAP Kokomo Site

    "kkm-"

    Charles Garwood, no access, remote site

    LMSB

    Laboratory Material Storage Building

    "lmsb-"

    No access

    NA

    TAP New Albany Site

    "na-"

    Charles Garwood, no access, remote site

    NACC

    Native American Educational and Cultural Center

    "nacc-"

    No access

    PAGE

    Thomas A. Page Pavilion

    "page-"

    No access

    PWB

    Purdue West Annex - Building B"pwb-"

    "pwb-"

    Dennis Lord - Ignore outside of 7:30AM to 4:30 PM

    PWC

    Purdue West - Building C

    "pwc-"

    Dennis Lord - Ignore outside of 7:30AM to 4:30 PM

    RALR

    Stadium Area - Visiting Team Locker Room

    "ralr-"

    No access, Seasonal

    SBCH

    Purdue Softball Clubhouse

    "sbch-"

    Seasonal

    SBPB

    Purdue Softball Press Box

    "sbpb-"

    Seasonal

    SCHO

    Schowe House

    "scho-"

    No access

    SD02

    Dairy Research Unit (ASREC Farm)

    "sd02-"

    No access, users aren't there after hours, feeds off of BECK

    SIA

    Subaru of Indiana Automotive

    "sia-"

    No access

    SOCC

    Purdue Women’s Soccer Building

    "socc-"

    No access

    SPUR

    Spurgeon Golf Training Center

    "spur-"

    No access

    SWNA

    State Wide New Albany

    "swna-"

    Off campus

    SWSB

    State Wide South Bend

    "swsb-"

    Off campus

    TM02

    Throckmorton Pesticide Building

    "tm02-""

    No access, users aren't there after hours

    TM08

    Throckmorton Meigs Building

    "tm08-"

    No access, users aren't there after hours

    TM11

    Throckmorton Fruit Barn

    "tm11-"

    No access, users aren't there after hours

    TM36

    Throckmorton Farm Crop Barn

    "tm36-"

    No access, users aren't there after hours

    TPB

    Rankin Track Press Box

    "tpb-"

    Seasonal

    TURF

    Intercollegiate Athletic Sports Turf Building

    "turf-"

    No access, users aren't there after hours

    UNPD

    University Police Department

    "unpd-"

    This is a backup connection, Mick Kiefer says ignore outside business hours

    USDA

    USDA Building 1 (ASREC Farm)"usda-"Farm)

    "usda-"

    No access, users aren't there after hours, feeds off of BECK

    VCPR

    Veterinary Center for Paralysis Research

    "vcpr-"

    Dennis Barnett says ignore until business hours

    VLAB

    Veterinary Laboratory Animal Building

    "vlab-"

    Dennis Barnett says ignore until business hours

    VOIN

    Voinoff (Samuel) Golf Pavilion

    "voin-"

    Seasonal, no access

    VTCH

    Vision Technology 1 Building

    "vtch-"

    No access, not a Purdue building

    VPRB

    Veterinary Pathobiology Research Building

    "vprb-"

    Dennis Barnett says ignore until business hours

    WH9

    Well House 9

    "wh9-"

    One user, no access

    WRIT

    John S Wright Forestry Center

    "writ-"

    VPRB

    Veterinary Pathobiology Research Building

    No access, users aren't there after hours

    WH9

    Well House 9

    "wh9-"


    WRIT

    John S Wright Forestry Center

    "writ-"



Information to Provide When Reporting Network Issues

For Squared Up alarms report the following information from Squared Up when reporting issue:

  1. Date/Time that alarm started (Example: October 26, 10:00 AM)
  2. Affected Device Name (Example: mrdh-285n-c2950-01)
  3. Last Ping (Example: 2017-11-01 11:14:43)

For network issues reported by I-Light / GigaPOP or the ITaP help desk, campus personnel, students, or visitors, report the following information:

  1. Date/Time that issue began or was first noticed (Example: October 26, 10:00 AM)
  2. Affected service (Examples: wireless/PAL, data PIC(s), or I-Light / GigaPOP call back)
  3. Location where problem is occurring (Examples: wireless on 2nd floor of Armstrong, all data PICs in the Forestry building, or I-Light / GigaPOP). Whenever possible obtain a specific building and nearest room number.
  4. Name and phone number (or at least email) of person experiencing/reporting problem

More details can be found on the attached document below:

ITIS-Data Network Incident Resolution Process.docx

...