NGOP -- What is it?

• Scalable Monitoring System
• "Next Generation OPerations"
• Written at Fermilab
• Written in Python/C++
• Past HEPIX talks discussed architecture etc.
  
  

HEPIX 2002: NGOP Update Fermilab

What's happened lately?

• Monitoring still more stuff
• Smarter Agents
• Start/Stop standardization
• Performance Monitoring still on hold
• New User Interface code (more to come)
• Impending Lights Out Operation

HEPIX 2002: NGOP Update Fermilab

More Stuff

• Major Clusters:
  

  Farms	EMail	AFS
  Enstore	WWW	USCMS
  FNALU

• Totals:
    Hosts: ~1,100  Components: ~25,000

HEPIX 2002: NGOP Update Fermilab

More Stuff -- WWW

Common config include:

<SrvList Name="Central_Webservers">
<SrvList Name="Central_Webservers_inAFS">
  <Srv Name="boone">
    <IP Name="www-boone" />
    <MailTo Name="csi-group@fnal.gov,.../>
    <URLCk Name="boone_main">
      <URL Name="http://www-boone/" />
      <Ck4 Name=".*" />
    </URLCk>
  </Srv>

HEPIX 2002: NGOP Update Fermilab

More Stuff -- WWW

URL Agent config

<For Var="{%Srv}" Each="Srv" In="SrvList" 
      Name="Central_Webservers" Filename="webconfig.xml">
<For Var="{%IP}" Each="IP" In="Srv" Name="{%Srv}">
<System Name="{%Srv}" Cluster="WWW">
 <For Var="{%Ck}" Each="URLCk" In="Srv" Name="{%Srv}">
 <For Var="{%Ck4}" Each="Ck4" In="URLCk" Name="{%Ck}">
 <For Var="{%URL}" Each="URL" In="URLCk" Name="{%Ck}">
   <ME Name="{%Ck}" Host="{%IP}" Type="webpage">
    <URLRule href="{%URL}" RegExp="{%Ck4}" />
   </ME>
</For> </For> </For> </System> 

HEPIX 2002: NGOP Update Fermilab

More Stuff -- WWW

Plugins Agent config

<For Var="{%Srv}" Each="Srv" In="SrvList" 
      Name="Central_Webservers" Filename="webconfig.xml">
<For Var="{%IP}" Each="IP" In="Srv" Name="{%Srv}">
<System Name="{%Srv}" Cluster="WWW">
  <ME Name="{%Srv}_httpd" Host="{%Host}" Type="Daemon">
    <ConditionSet>
      <fn Name="plug_ins" RetVal="int:nservers" 
       Arg="set : `grep \^User .../{%Srv}/conf/httpd.conf`; 
            ps -fu $3 | grep httpd | wc -l" />
      <Condition State="0" SevLevel="0" Description="...">
        <apply><eq/><ci>nservers</ci><cn>0</cn> </apply>

HEPIX 2002: NGOP Update Fermilab

Smarter Agents/Servers

• Ping agent:
    route discovery, outage groups
• Swatch agent:
    combines repeated messages
• Delayed effect actions
  if condition persists...

HEPIX 2002: NGOP Update Fermilab

New User Interface:

• Split old GUI into:
  • GUI Front End
  • Status Engine(s)
  • Location Server
  
  
• Improves GUI responsiveness.

HEPIX 2002: NGOP Update Fermilab

New User Interface:

HEPIX 2002: NGOP Update Fermilab

Status Engine

• Determines "status" of entities
  -- individual and composite
• Triggers actions on status change
• Multiple GUI's -> One Status Engine
• One Status Engine per "role"
  -- operator, admin, etc.

HEPIX 2002: NGOP Update Fermilab

GUI Front Ends:

• java gui
  • Responsive, prettier
  • Bigger memory footprint
• web gui
  • FastCGI
  • 

HEPIX 2002: NGOP Update Fermilab

Demo

• Web GUI
• Web Admin Interface
• Newer Web Admin Interface
• Java GUI

HEPIX 2002: NGOP Update Fermilab

Near Future

• Continued growth (~ 2x) expected at Fermi
• Configuration issues --
    Full time administrator?
    Bigger Better Admin GUIs?
    XML classes?
  
• Other labs?

HEPIX 2002: NGOP Update Fermilab