Re: NOCEM



Bala & H.L.,

I spoke extensively with Jack on this one this morning.

Basically until we get to the 11/22 release we do not have sufficient logging capabilities in place to do any significant post-mortem analysis on this one.

HOWEVER, please note that this was a MAINTENANCE event and is NOT service-affecting in any way.

I'll be glad to take the time out to try to put something together on this, but we are talking a significant chunk of time to come up with anything more that what I have already given Jack on this one. To be frank, I do not think the time investment would be worth it, because the most likely result is still just as "fuzzy" as what you find below and as what Jack and I discussed this morning.

Just my $0.02 worth - please let me know whether to pursue this one any further. :-)

Thanks,

Bill

-----------------------

On 07/26/2005 11:41:49, jsprouls att com wrote:
Bala,

We've identified an MR relative to RAPID restorations being forwarded
to
NOCEM in a timely manner and would ask that your folks (Bill?) look in
to it.  Specifically, there was a 33 minute delay between the second
to
last and last INFO files being created and sent to NOCEM for RAPID
EVENT
280007. If there is an 'official' channel that I should be routing
this
MR request through, please let me know. Thanks.

Jack Sprouls
(732)420-8060

-----Original Message-----
From: William W. Austin [mailto:waustin alphga att com]
Sent: Tuesday, July 26, 2005 10:05 AM
To: Yankin, Denis S, ALABS
Cc: Pryor, Jodie L, NEO; NOC TRANSPORT; Sprouls, John (Jack), ALABS;
BalaPrasanna; Gowda, H L (HL), ALABS
Subject: Re: NOCEM


Denis,

Looking at the data from production, here is the flow of what occurred
here:
1.  Restoration times (from the report writer looking at the rapid
log):

RECAP RESTORATION EVENT: 280007
R/A PRI   CLLI                                   TIMESTAMP
PET
     TOT      PETSEC
--- ----- -------------------------------------- -------------------
-------- -------- -------
 R    809 1    T3         FNBGMDFB   LNMTCO01    07/25/2005 12:52:36
00:01:38 00:01:45      98
 R    801 1    T3         FNBGMDFB   STLSMO09    07/25/2005 12:52:53
00:01:55 00:02:02     115
 R    770 1    T3         CMDNNJCE   LNMTCO01    07/25/2005 12:53:09
00:02:11 00:02:18     131
 R    739 1    T3         AKRNOH25   DTRTMIBA    07/25/2005 12:53:27
00:02:29 00:02:36     149
 R    733 3    T3         NYCMNYBW   OKBRILOA    07/25/2005 12:53:47
00:02:49 00:02:56     169
 R    705 1201 TTT3       DTRTMIBAF03MNCHNHCOF19 07/25/2005 12:54:04
00:03:06 00:03:13     186
 R    705 99   TAT3       AKRNOH25DC8CLEVOH02DC8 07/25/2005 12:54:20
00:03:22 00:03:29     202
 R    705 1201 TTT3       AKRNOH25F31CLEVOH02F09 07/25/2005 12:54:36
00:03:38 00:03:45     218
 R    705 11   TAT3       AKRNOH25F31CLEVOH02DC8 07/25/2005 12:54:53
00:03:55 00:04:02     235
 R    690 1    T3         PITBPADGW10SCRMCA01    07/25/2005 12:55:07
00:04:09 00:04:16     249
 R    671 7    T3         AKRNOH25   TOLDOH21    07/25/2005 12:55:27
00:04:29 00:04:36     269
 R    637 102  T3         ASLDOHAT   TOLDOH21    07/25/2005 12:55:43
00:04:45 00:04:52     285
 R    835 1    T3         DYTNOH15T10WAYNPALA    07/25/2005 12:56:02
00:05:04 00:05:11     304
 R    787 2    T3         DTRTMIBA   YNTWOH02T10 07/25/2005 12:56:18
00:05:20 00:05:27     320
 R    767 1    T3         LNMTCO01   PITBPADGW10 07/25/2005 12:56:39
00:05:41 00:05:48     341
 R    753 4    T3         CLEVOH02S10CLMBOH11W03 07/25/2005 12:56:56
00:05:58 00:06:05     358
 R    760 1    T3         DTRTMIBA   PITBPADGW10 07/25/2005 12:57:13
00:06:15 00:06:22     375
 R    695 1    T3         DTRTMIBA   NWRKNJ02    07/25/2005 12:57:28
00:06:30 00:06:37     390
 R    690 1    T3         DTRTMIBA   YNTWOH02T10 07/25/2005 12:57:47
00:06:49 00:06:56     409
 R    690 6    T3         CLEVOH02S10XENIOH02    07/25/2005 12:58:02
00:07:04 00:07:11     424
 R    689 1    T3         DTRTMIBA   PHLAPASL    07/25/2005 12:58:19
00:07:21 00:07:28     441
 R    689 2    T3         CLEVOH02S10XENIOH02    07/25/2005 12:58:36
00:07:38 00:07:45     458
 R    683 1    T3         DYTNOH15T10PHLAPASL    07/25/2005 12:58:54
00:07:56 00:08:03     476
 R    655 1    T3         CHCGILCLW60WAYNPALA    07/25/2005 12:59:12
00:08:14 00:08:21     494
 R    671 6    T3         CLEVOH02S10CNCNOHWSW02 07/25/2005 12:59:30
00:08:32 00:08:39     512
------------------------------------------------------------------------
-----
   25 FACS,     25 KNOWN,    25 REST,     0 ABNDN,     0 UNREST,     0
XIDENT
========================================================================
=====
   PET = PROCESS ELAPSED TIME
   TOT = TOTAL OUTAGE TIME
PETSEC = PET IN SECONDS

So yes, the last t3 was restored at 12:59:30 (on a manual event such
as
this one,
a delay this long is not surprising).

And here is what was transmitted:

SENT 280007.e.T.rpt.info AT 2005-07-25 12:51:20 27 lines, 1482 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 12:53:25 27 lines, 1482 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 12:55:31 27 lines, 1482 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 12:57:36 27 lines, 1482 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 12:59:42 27 lines, 1482 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 13:01:47 41 lines, 2119 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 13:34:36 77 lines, 3757 chars

SENT 280007.e.facs       AT 2005-07-25125119    26 lines, 2407 chars

So at 13:01:47, the number of lines sent (indicating notificiation of
restored facilities) jumped at 13:01

It is clear from the RESTINFO log that between 12:51:14 and 13:34:36
(NWT),
there were 14 transmissions of the ".info" file, and that matches the
7
files
sent to each of 2 machines.  Unfortunately until we get the November
RAPID
release out, I won't have sufficient logging info to determine why
there
was
the gap from 13:01 until 13:34.

I will continue to look into this event, but without further enhanced
logging
information in RESTINFO, I have to rely on the RAPID log itself plus
system
accounting files (which are also somewhat vague) - and this is a fully
manual
operation, so it may take a little while...  and then the data may
still
not
be available.

As soon as I have more information on this item, I will let you know -
in the
mean time, please do not hesitate to ask if you have more questions.

- Bill
--
  william w. austin, ibm global services
    waustin att com (this address will change soon)
      770 750-6954

On 07/25/2005 22:33:37, dyankin att com wrote:
> Jodie, Jack,
>
> FYI
>
> I've checked NOCEM logs...  Until 15:34 there were only 7 facilities
> show as restored in NOCEM.
> At approx 15:34 we received a new info file with all 25 T3s shown as
> restored. So there was no discrepancy but a delay on RAPID side.
> NOCEM
> processed the file as soon as it was received.
>
> Thanks,
> Denis Yankin, MCSD.NET
> AT&T GEOLink,
> (770) 750-7288
>
> >  -----Original Message-----
> > From: 	Pryor, Jodie L, NEO
> > Sent:	Monday, July 25, 2005 3:24 PM
> > To:	NOC TRANSPORT
> > Cc:	Sprouls, John (Jack), ALABS; Yankin, Denis S, ALABS
> > Subject:	RE: NOCEM
> >
> > RAPID event 280007 @ 1250 NWT 7/25.
> >
> > First, I would like to suggest that any RAPID event with an mrXXXX
> in the Event Name be disregarded and not sent to NOCEM at all.
> >
> > Second, the last T3 in this event restored at 1259 NWT (many were
> restored a few minutes earlier).  It's now 1322 NWT, and only 7 out
> of
> the 25 DS3s are restored in NOCEM.
> >
> > Thanks, Jodie
> >
> >  -----Original Message-----
> > From: 	Misa, Tara D, NEO
> > Sent:	Thursday, July 21, 2005 11:29 AM
> > To:	NOC TRANSPORT
> > Subject:	NOCEM
> >
> > Please provide feedback on RAPID restores and on the FCC DS3
> calculations.
> >
> > There have been fixes deployed for these and need to be confirmed
> working as we need them to be.
> >
> > Thanks.
> > Tara
>







[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]