Re: NOCEM
- From: "William W. Austin" <waustin alphga att com>
- To: Balsa List <balsa-list gnome org>, "H.L. Gowda" <hlgowda att com>
- Cc:
- Subject: Re: NOCEM
- Date: Tue, 26 Jul 2005 18:28:23 +0000
Bala & H.L.,
I spoke extensively with Jack on this one this morning.
Basically until we get to the 11/22 release we do not have sufficient
logging capabilities in place to do any significant post-mortem
analysis on this one.
HOWEVER, please note that this was a MAINTENANCE event and is NOT
service-affecting in any way.
I'll be glad to take the time out to try to put something together on
this, but we are talking a significant chunk of time to come up with
anything more that what I have already given Jack on this one. To be
frank, I do not think the time investment would be worth it, because
the most likely result is still just as "fuzzy" as what you find below
and as what Jack and I discussed this morning.
Just my $0.02 worth - please let me know whether to pursue this one any
further. :-)
Thanks,
Bill
-----------------------
On 07/26/2005 11:41:49, jsprouls att com wrote:
Bala,
We've identified an MR relative to RAPID restorations being forwarded
to
NOCEM in a timely manner and would ask that your folks (Bill?) look in
to it. Specifically, there was a 33 minute delay between the second
to
last and last INFO files being created and sent to NOCEM for RAPID
EVENT
280007. If there is an 'official' channel that I should be routing
this
MR request through, please let me know. Thanks.
Jack Sprouls
(732)420-8060
-----Original Message-----
From: William W. Austin [mailto:waustin alphga att com]
Sent: Tuesday, July 26, 2005 10:05 AM
To: Yankin, Denis S, ALABS
Cc: Pryor, Jodie L, NEO; NOC TRANSPORT; Sprouls, John (Jack), ALABS;
BalaPrasanna; Gowda, H L (HL), ALABS
Subject: Re: NOCEM
Denis,
Looking at the data from production, here is the flow of what occurred
here:
1. Restoration times (from the report writer looking at the rapid
log):
RECAP RESTORATION EVENT: 280007
R/A PRI CLLI TIMESTAMP
PET
TOT PETSEC
--- ----- -------------------------------------- -------------------
-------- -------- -------
R 809 1 T3 FNBGMDFB LNMTCO01 07/25/2005 12:52:36
00:01:38 00:01:45 98
R 801 1 T3 FNBGMDFB STLSMO09 07/25/2005 12:52:53
00:01:55 00:02:02 115
R 770 1 T3 CMDNNJCE LNMTCO01 07/25/2005 12:53:09
00:02:11 00:02:18 131
R 739 1 T3 AKRNOH25 DTRTMIBA 07/25/2005 12:53:27
00:02:29 00:02:36 149
R 733 3 T3 NYCMNYBW OKBRILOA 07/25/2005 12:53:47
00:02:49 00:02:56 169
R 705 1201 TTT3 DTRTMIBAF03MNCHNHCOF19 07/25/2005 12:54:04
00:03:06 00:03:13 186
R 705 99 TAT3 AKRNOH25DC8CLEVOH02DC8 07/25/2005 12:54:20
00:03:22 00:03:29 202
R 705 1201 TTT3 AKRNOH25F31CLEVOH02F09 07/25/2005 12:54:36
00:03:38 00:03:45 218
R 705 11 TAT3 AKRNOH25F31CLEVOH02DC8 07/25/2005 12:54:53
00:03:55 00:04:02 235
R 690 1 T3 PITBPADGW10SCRMCA01 07/25/2005 12:55:07
00:04:09 00:04:16 249
R 671 7 T3 AKRNOH25 TOLDOH21 07/25/2005 12:55:27
00:04:29 00:04:36 269
R 637 102 T3 ASLDOHAT TOLDOH21 07/25/2005 12:55:43
00:04:45 00:04:52 285
R 835 1 T3 DYTNOH15T10WAYNPALA 07/25/2005 12:56:02
00:05:04 00:05:11 304
R 787 2 T3 DTRTMIBA YNTWOH02T10 07/25/2005 12:56:18
00:05:20 00:05:27 320
R 767 1 T3 LNMTCO01 PITBPADGW10 07/25/2005 12:56:39
00:05:41 00:05:48 341
R 753 4 T3 CLEVOH02S10CLMBOH11W03 07/25/2005 12:56:56
00:05:58 00:06:05 358
R 760 1 T3 DTRTMIBA PITBPADGW10 07/25/2005 12:57:13
00:06:15 00:06:22 375
R 695 1 T3 DTRTMIBA NWRKNJ02 07/25/2005 12:57:28
00:06:30 00:06:37 390
R 690 1 T3 DTRTMIBA YNTWOH02T10 07/25/2005 12:57:47
00:06:49 00:06:56 409
R 690 6 T3 CLEVOH02S10XENIOH02 07/25/2005 12:58:02
00:07:04 00:07:11 424
R 689 1 T3 DTRTMIBA PHLAPASL 07/25/2005 12:58:19
00:07:21 00:07:28 441
R 689 2 T3 CLEVOH02S10XENIOH02 07/25/2005 12:58:36
00:07:38 00:07:45 458
R 683 1 T3 DYTNOH15T10PHLAPASL 07/25/2005 12:58:54
00:07:56 00:08:03 476
R 655 1 T3 CHCGILCLW60WAYNPALA 07/25/2005 12:59:12
00:08:14 00:08:21 494
R 671 6 T3 CLEVOH02S10CNCNOHWSW02 07/25/2005 12:59:30
00:08:32 00:08:39 512
------------------------------------------------------------------------
-----
25 FACS, 25 KNOWN, 25 REST, 0 ABNDN, 0 UNREST, 0
XIDENT
========================================================================
=====
PET = PROCESS ELAPSED TIME
TOT = TOTAL OUTAGE TIME
PETSEC = PET IN SECONDS
So yes, the last t3 was restored at 12:59:30 (on a manual event such
as
this one,
a delay this long is not surprising).
And here is what was transmitted:
SENT 280007.e.T.rpt.info AT 2005-07-25 12:51:20 27 lines, 1482 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 12:53:25 27 lines, 1482 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 12:55:31 27 lines, 1482 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 12:57:36 27 lines, 1482 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 12:59:42 27 lines, 1482 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 13:01:47 41 lines, 2119 chars
SENT 280007.e.T.rpt.info AT 2005-07-25 13:34:36 77 lines, 3757 chars
SENT 280007.e.facs AT 2005-07-25125119 26 lines, 2407 chars
So at 13:01:47, the number of lines sent (indicating notificiation of
restored facilities) jumped at 13:01
It is clear from the RESTINFO log that between 12:51:14 and 13:34:36
(NWT),
there were 14 transmissions of the ".info" file, and that matches the
7
files
sent to each of 2 machines. Unfortunately until we get the November
RAPID
release out, I won't have sufficient logging info to determine why
there
was
the gap from 13:01 until 13:34.
I will continue to look into this event, but without further enhanced
logging
information in RESTINFO, I have to rely on the RAPID log itself plus
system
accounting files (which are also somewhat vague) - and this is a fully
manual
operation, so it may take a little while... and then the data may
still
not
be available.
As soon as I have more information on this item, I will let you know -
in the
mean time, please do not hesitate to ask if you have more questions.
- Bill
--
william w. austin, ibm global services
waustin att com (this address will change soon)
770 750-6954
On 07/25/2005 22:33:37, dyankin att com wrote:
> Jodie, Jack,
>
> FYI
>
> I've checked NOCEM logs... Until 15:34 there were only 7 facilities
> show as restored in NOCEM.
> At approx 15:34 we received a new info file with all 25 T3s shown as
> restored. So there was no discrepancy but a delay on RAPID side.
> NOCEM
> processed the file as soon as it was received.
>
> Thanks,
> Denis Yankin, MCSD.NET
> AT&T GEOLink,
> (770) 750-7288
>
> > -----Original Message-----
> > From: Pryor, Jodie L, NEO
> > Sent: Monday, July 25, 2005 3:24 PM
> > To: NOC TRANSPORT
> > Cc: Sprouls, John (Jack), ALABS; Yankin, Denis S, ALABS
> > Subject: RE: NOCEM
> >
> > RAPID event 280007 @ 1250 NWT 7/25.
> >
> > First, I would like to suggest that any RAPID event with an mrXXXX
> in the Event Name be disregarded and not sent to NOCEM at all.
> >
> > Second, the last T3 in this event restored at 1259 NWT (many were
> restored a few minutes earlier). It's now 1322 NWT, and only 7 out
> of
> the 25 DS3s are restored in NOCEM.
> >
> > Thanks, Jodie
> >
> > -----Original Message-----
> > From: Misa, Tara D, NEO
> > Sent: Thursday, July 21, 2005 11:29 AM
> > To: NOC TRANSPORT
> > Subject: NOCEM
> >
> > Please provide feedback on RAPID restores and on the FCC DS3
> calculations.
> >
> > There have been fixes deployed for these and need to be confirmed
> working as we need them to be.
> >
> > Thanks.
> > Tara
>
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]
