Pat,

I'm stuggling to get hold of our HPUX admin as we'r running Solaris :-)

--
Jarl Groneng
jarl@nextra.com



21.06.01 18:11:40, skrev Pat Ferger :

>**
>
>Jarl,
>
>It's not the total system memory that could be the problem, it may be the
>Per-Process-Memory-Limit feature that HP-UX has. HP-UX allows you to limit
>how much memory a process can take. The default limit is 67 MB, which may
>be too low for more sophisticated sites. 4.0.X didn't have this problem
>because on UNIX they were multi-process, meaning there were many processes
>with smaller memory footprints. But with 4.5, the AR Server has been
>consolidated into one process with multiple threads. This may put you over
>the limit in terms of process-memory size.
>
>Talk to your HPUX sysadmin about bumping up the per-process-memory limit.
>
>
>>From: Jarl Groneng
>>Reply-To: "Action Request System discussion list(ARSList)"
>>
>>To: ARSLIST@LISTSERV.VISTAIT.COM
>>Subject: Re: Remedy/Siebel
>>Date: Thu, 21 Jun 2001 07:48:37 +0200
>>
>>**
>>
>>Memory is not a problem, it hardly using swap.
>>
>>Memory: 1024M real, 756M free, 149M swap in use, 1661M swap free
>>
>>
>>Disk space is not a problem either.
>>
>>Filesystem kbytes used avail capacity Mounted on
>>/dev/dsk/c0t0d0s0 3588558 722568 2830105 21% /
>>swap 1699464 16 1699448 1% /var/run
>>swap 1702240 2792 1699448 1% /tmp
>>/dev/dsk/c0t0d0s5 4128246 1722165 2364799 43% /local
>>nfs1.nextel.no:/users
>> 2097152 1119256 977896 54% /home/nfs/users
>>
>>
>>
>>
>>--
>>Jarl Groneng
>>jarl@nextra.com
>>
>>
>>
>> > -----Original Message-----
>> > From: Action Request System discussion list(ARSList)
>> > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Pat Ferger
>> > Sent: 20. juni 2001 19:07
>> > To: ARSLIST@LISTSERV.VISTAIT.COM
>> > Subject: Re: Remedy/Siebel
>> >
>> >
>> > **
>> >
>> > Jarl,
>> >
>> > It sounds like you're running out of memory, or disk space in the /tmp
>> > directory. The "Failure with a temporary file" strongly suggests that
>>the
>> > server was unable to create a temp file, so you should
>> > double-check the size
>> > remaining on all your partitions (/var, /tmp, etc) to make sure you have
>> > enough space.
>> >
>> > The other problems are memory problems, so maybe you need to bump up
>>your
>> > process memory limits, also.
>> >
>> >
>> > >From: Jarl Groneng
>> > >Reply-To: "Action Request System discussion list(ARSList)"
>> > >
>> > >To: ARSLIST@LISTSERV.VISTAIT.COM
>> > >Subject: Re: Remedy/Siebel
>> > >Date: Wed, 20 Jun 2001 15:51:58 +0200
>> > >
>> > >**
>> > >
>> > >Hi,
>> > >
>> > >Very similar to the problem we has. Every 8-9 day, the cpu is crying
>>for
>> > >air. This happend around 06:20 in the morning(every time). No heavy
>>jobs
>> > >running this time. Opend a supportticket may 14, get two reply may 15.,
>> > >have
>> > >not heard anything from support after this date. Here is the last mail
>>I
>> > >got
>> > >from them:
>> > >
>> > >"Am I right to say that the problem has been resolved after
>> > server restart?
>> > >If so, do you want to close this ticket?"
>> > >
>> > >..close the ticket? ha, funny joke. I want my problem solved.
>> > >
>> > >
>> > >Got this errors on the client:
>> > >ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
>> > >occurred
>> > >
>> > >ARERR [316] Failure with a temporary file : No such file or directory
>> > >
>> > >
>> > >ARS 4.5.2 patch 1007
>> > >Solaris 8
>> > >Oracle 8.1.7 (on another server)
>> > >
>> > >Restart ars, and get this in the arerror logfile:
>> > >
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ErrorHandler
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > RollbackTransaction()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > BeginNetRespStatistics()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ARPostSwitch()
>> > >0 : AR System server terminated -- fatal error encountered (ARNOTE 21)
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ThreadCleanup()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ThreadCleanup()
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >
>> > >--
>> > >Jarl Groneng
>> > >jarl@nextra.com
>> > >
>> > >
>> > > > -----Original Message-----
>> > > > From: Action Request System discussion list(ARSList)
>> > > > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
>> > > > Sent: 20. juni 2001 01:27
>> > > > To: ARSLIST@LISTSERV.VISTAIT.COM
>> > > > Subject: Remedy/Siebel
>> > > >
>> > > >
>> > > > **
>> > > >
>> > > > I'm a new subscriber to arslist and I'm looking for help with
>> > > > performance issues
>> > > > in 4.5.2.
>> > > >
>> > > > We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
>> > > > two weeks ago
>> > > > to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
>> > > > our box we
>> > > > also have an interface with Siebel using MQ Series and Java. Our
>> > > > system has
>> > > > crashed three times in the past 48hrs, and we've had to do a
>> > > > power cycle on the
>> > > > box. Our CPU load average is 99% and our memory usage is 98%.
>> > > > We have 1gig of
>> > > > memory on the box and two cpu's.
>> > > >
>> > > > Here is a vmstat capture of what's going on on our box:
>> > > > $ vmstat -S 2 5
>> > > > procs memory page
>> > > > faults cpu
>> > > > r b w avm free si so pi po fr
>> > > > de sr in
>> > > > sy cs us sy id
>> > > > 7 0 0 41615 9211 1536 1536 25 4 122
>> > > > 0 39 686
>> > > > 9936 1339 52 30 18
>> > > > 7 0 0 41615 9014 0 0 21 0 61
>> > > > 0 0 385
>> > > > 7854 948 28 72 0
>> > > > 6 0 0 43620 7880 0 0 14 0 64
>> > > > 0 0 471
>> > > > 8778 933 43 55 2
>> > > > 6 0 0 43620 6542 0 0 9 0 69
>> > > > 0 0 426
>> > > > 7555 905 42 57 0
>> > > > 6 0 0 43620 5582 0 0 6 0 61
>> > > > 0 0 431
>> > > > 6988 869 44 56 0
>> > > >
>> > > > Here is a portion of the glance capture:
>> > > > B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
>> > > > Current Avg High
>> > > > ------------------------------------------------------------------
>> > > > ----------------------------------------------------------
>> > > >
>> > > > Cpu Util S SARU
>> > > > U |100%
>> > > > 100% 100%
>> > > > Disk Util F F
>> > > > | 55% 47% 67%
>> > > > Mem Util S SU
>> > > > UB B | 92% 91% 92%1
>> > > > Swap Util U UR R
>> > > > | 56% 56% 56%
>> > > > ------------------------------------------------------------------
>> > > > ----------------------------------------------------------
>> > > >
>> > > > PROCESS LIST
>> > > > Users= 7
>> > > > User CPU Util Cum Disk
>> > > > Thd
>> > > > Process Name PID PPID Pri Name ( 200% max) CPU IO
>> > > > Rate RSS Cnt
>> > > > ------------------------------------------------------------------
>> > > > --------------
>> > > > arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
>> > > > 2.3 43.7mb 25
>> > > > identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
>> > > > 0.7 388kb 1
>> > > > oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
>> > > > 48.7/17.1 10.7mb 1
>> > > > oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
>> > > > 34.1/10.2 10.7mb 1
>> > > > oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
>> > > > 6.3 10.7mb 1
>> > > > java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
>> > > > 2.4 17.3mb 10
>> > > > oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
>> > > > 3.7 10.8mb 1
>> > > > aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
>> > > > 0.0 7.0mb 1
>> > > > java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
>> > > > 1.5 12.6mb 9
>> > > > oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
>> > > > 8.4 10.7mb 1
>> > > > oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
>> > > > 0.2 11.0mb 1
>> > > > oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
>> > > > 0.0 10.8mb 1
>> > > >
>> > > >
>> > > > These are the errors that pop into the arerror.log before the
>> > > > system crashes:
>> > > > Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
>> > > > SQL database
>> > > > (ARERR 551)
>> > > > Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
>> > > > Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
>> > > > to the database
>> > > > (ARERR 552)
>> > > > Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef
>> > >extension
>> > > > doesn't exist
>> > > > (Oracle -1041)
>> > > > Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
>> > > > fatal error
>> > > > encountered (ARNOTE 21)
>> > > > Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
>> > > > storage. (ARERR
>> > > > 8747)
>> > > > Tue Jun 19 07:59:07 2001 ThreadCleanup()
>> > > > Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
>> > > > to the database
>> > > > (ARERR 552)
>> > > > Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef
>> > >extension
>> > > > doesn't exist
>> > > > (Oracle -1041)
>> > > > Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
>> > > > fatal error
>> > > > encountered (ARNOTE 21)
>> > > > Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
>> > > > storage. (ARERR
>> > > > 8747)
>> > > > Tue Jun 19 07:59:15 2001 ThreadCleanup()
>> > > > Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
>> > > > to the database
>> > > > (ARERR 552)
>> > > > Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef
>> > >extension
>> > > > doesn't exist
>> > > > (Oracle -1041)
>> > > > Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
>> > > > fatal error
>> > > > encountered (ARNOTE 21)
>> > > > Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
>> > > > storage. (ARERR
>> > > > 8747)
>> > > > Tue Jun 19 07:59:53 2001 ThreadCleanup()
>> > > >
>> > > > We apparently have the latest hp patch available. During the
>> > > > cutover we re-did
>> > > > all of our indexes and switched our macros to push fields. We've
>> > > > also reset our
>> > > > kernal parameters to the following:
>> > > > maxdsiz from 67108864 to 200000000
>> > > > maxssiz from 8388608 to 16000000
>> > > > maxtsiz from 67108864 to 200000000
>> > > > maxswapchunks from 1262 to 4096
>> > > > maxfiles from 60 to 2048
>> > > > nkthread 2048 to 2850
>> > > >
>> > > > We are still having performace issues, the load average is high
>> > > > and the cpu
>> > > > usage and memory usage are also very high. Does anyone have any
>> > > > suggestions or
>> > > > comments.
>> > > >
>> > > > ... Tracy
>> > > >
>> > > > ______
>> > > > _______________________
>> > > >
>> > > >
>> > > > You may also control your subscription options, including
>> > > > UNSUBSCRIBE, at www.ARSLIST.org
>> > > >
>> > > >
>> > >
>> > >_____
>>________________________
>> > >
>> > >
>> > >You may also control your subscription options, including UNSUBSCRIBE,
>>at
>> > >www.ARSLIST.org
>> >
>> >
>>_____________
>> > Get Your Private, Free E-mail from MSN Hotmail at
>>http://www.hotmail.com.
>> >
>> > ______
>> > _______________________
>> >
>> >
>> > You may also control your subscription options, including
>> > UNSUBSCRIBE, at www.ARSLIST.org
>> >
>> >
>>
>>_____________________________
>>
>>
>>You may also control your subscription options, including UNSUBSCRIBE, at
>>www.ARSLIST.org
>
>_____________
>Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.
>
>_____________________________
>
>
>
>
>

_____________________________

I am experiencing the same problem. No errors logged in the arerror.log. No
errors in the Oracle logs. Just start getting calls from the users as the
clients start getting strange errors - the same as Jarl

ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
>> > >occurred
>> > >
>> > >ARERR [316] Failure with a temporary file : No such file or directory
>> > >

Also getting "Connection reset by peer" ( I can't remember the error number)

Checked disk space - plenty of disk space, checked glance - plenty of
available memory and plenty of available CPU

Arserverd running okay - gaining time. Process also responds to the
/usr/ar/bin/arsystem stop command with no problem.

This has happened several times and a restart of the arserver fixes it. It
appears to happen when the arserverd process grows to over 200mg. Remedy
says Unix has a problem with never releasing memory until the deamon process
is restarted. The process does not crash - we have increased the process
limit to accommodate its growth. However arserverd seems to get 'weirded
out' (that technical talk for random events I can't explain) when it grows
much over 200 meg

HP-UX 11.0
Oracle 8.1.6
ARS 4.5.2 (no patches)
Aruser - various client OS and versions

Ann Mayes
Ciber, Inc



-----Original Message-----
From: Jarl Grøneng [mailto:rapid@ONLINE.NO]
Sent: June 21, 2001 11:16 AM
To: ARSLIST@LISTSERV.VISTAIT.COM
Subject: Re: Remedy/Siebel


**

Pat,

I'm stuggling to get hold of our HPUX admin as we'r running Solaris :-)

--
Jarl Groneng
jarl@nextra.com



21.06.01 18:11:40, skrev Pat Ferger :

>**
>
>Jarl,
>
>It's not the total system memory that could be the problem, it may be the
>Per-Process-Memory-Limit feature that HP-UX has. HP-UX allows you to limit
>how much memory a process can take. The default limit is 67 MB, which may
>be too low for more sophisticated sites. 4.0.X didn't have this problem
>because on UNIX they were multi-process, meaning there were many processes
>with smaller memory footprints. But with 4.5, the AR Server has been
>consolidated into one process with multiple threads. This may put you over
>the limit in terms of process-memory size.
>
>Talk to your HPUX sysadmin about bumping up the per-process-memory limit.
>
>
>>From: Jarl Groneng
>>Reply-To: "Action Request System discussion list(ARSList)"
>>
>>To: ARSLIST@LISTSERV.VISTAIT.COM
>>Subject: Re: Remedy/Siebel
>>Date: Thu, 21 Jun 2001 07:48:37 +0200
>>
>>**
>>
>>Memory is not a problem, it hardly using swap.
>>
>>Memory: 1024M real, 756M free, 149M swap in use, 1661M swap free
>>
>>
>>Disk space is not a problem either.
>>
>>Filesystem kbytes used avail capacity Mounted on
>>/dev/dsk/c0t0d0s0 3588558 722568 2830105 21% /
>>swap 1699464 16 1699448 1% /var/run
>>swap 1702240 2792 1699448 1% /tmp
>>/dev/dsk/c0t0d0s5 4128246 1722165 2364799 43% /local
>>nfs1.nextel.no:/users
>> 2097152 1119256 977896 54% /home/nfs/users
>>
>>
>>
>>
>>--
>>Jarl Groneng
>>jarl@nextra.com
>>
>>
>>
>> > -----Original Message-----
>> > From: Action Request System discussion list(ARSList)
>> > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Pat Ferger
>> > Sent: 20. juni 2001 19:07
>> > To: ARSLIST@LISTSERV.VISTAIT.COM
>> > Subject: Re: Remedy/Siebel
>> >
>> >
>> > **
>> >
>> > Jarl,
>> >
>> > It sounds like you're running out of memory, or disk space in the /tmp
>> > directory. The "Failure with a temporary file" strongly suggests that
>>the
>> > server was unable to create a temp file, so you should
>> > double-check the size
>> > remaining on all your partitions (/var, /tmp, etc) to make sure you
have
>> > enough space.
>> >
>> > The other problems are memory problems, so maybe you need to bump up
>>your
>> > process memory limits, also.
>> >
>> >
>> > >From: Jarl Groneng
>> > >Reply-To: "Action Request System discussion list(ARSList)"
>> > >
>> > >To: ARSLIST@LISTSERV.VISTAIT.COM
>> > >Subject: Re: Remedy/Siebel
>> > >Date: Wed, 20 Jun 2001 15:51:58 +0200
>> > >
>> > >**
>> > >
>> > >Hi,
>> > >
>> > >Very similar to the problem we has. Every 8-9 day, the cpu is crying
>>for
>> > >air. This happend around 06:20 in the morning(every time). No heavy
>>jobs
>> > >running this time. Opend a supportticket may 14, get two reply may
15.,
>> > >have
>> > >not heard anything from support after this date. Here is the last mail
>>I
>> > >got
>> > >from them:
>> > >
>> > >"Am I right to say that the problem has been resolved after
>> > server restart?
>> > >If so, do you want to close this ticket?"
>> > >
>> > >..close the ticket? ha, funny joke. I want my problem solved.
>> > >
>> > >
>> > >Got this errors on the client:
>> > >ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous
event
>> > >occurred
>> > >
>> > >ARERR [316] Failure with a temporary file : No such file or directory
>> > >
>> > >
>> > >ARS 4.5.2 patch 1007
>> > >Solaris 8
>> > >Oracle 8.1.7 (on another server)
>> > >
>> > >Restart ars, and get this in the arerror logfile:
>> > >
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ErrorHandler
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > RollbackTransaction()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > BeginNetRespStatistics()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ARPostSwitch()
>> > >0 : AR System server terminated -- fatal error encountered (ARNOTE 21)
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ThreadCleanup()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ThreadCleanup()
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >
>> > >--
>> > >Jarl Groneng
>> > >jarl@nextra.com
>> > >
>> > >
>> > > > -----Original Message-----
>> > > > From: Action Request System discussion list(ARSList)
>> > > > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
>> > > > Sent: 20. juni 2001 01:27
>> > > > To: ARSLIST@LISTSERV.VISTAIT.COM
>> > > > Subject: Remedy/Siebel
>> > > >
>> > > >
>> > > > **
>> > > >
>> > > > I'm a new subscriber to arslist and I'm looking for help with
>> > > > performance issues
>> > > > in 4.5.2.
>> > > >
>> > > > We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
>> > > > two weeks ago
>> > > > to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
>> > > > our box we
>> > > > also have an interface with Siebel using MQ Series and Java. Our
>> > > > system has
>> > > > crashed three times in the past 48hrs, and we've had to do a
>> > > > power cycle on the
>> > > > box. Our CPU load average is 99% and our memory usage is 98%.
>> > > > We have 1gig of
>> > > > memory on the box and two cpu's.
>> > > >
>> > > > Here is a vmstat capture of what's going on on our box:
>> > > > $ vmstat -S 2 5
>> > > > procs memory page
>> > > > faults cpu
>> > > > r b w avm free si so pi po fr
>> > > > de sr in
>> > > > sy cs us sy id
>> > > > 7 0 0 41615 9211 1536 1536 25 4 122
>> > > > 0 39 686
>> > > > 9936 1339 52 30 18
>> > > > 7 0 0 41615 9014 0 0 21 0 61
>> > > > 0 0 385
>> > > > 7854 948 28 72 0
>> > > > 6 0 0 43620 7880 0 0 14 0 64
>> > > > 0 0 471
>> > > > 8778 933 43 55 2
>> > > > 6 0 0 43620 6542 0 0 9 0 69
>> > > > 0 0 426
>> > > > 7555 905 42 57 0
>> > > > 6 0 0 43620 5582 0 0 6 0 61
>> > > > 0 0 431
>> > > > 6988 869 44 56 0
>> > > >
>> > > > Here is a portion of the glance capture:
>> > > > B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
>> > > > Current Avg High
>> > > > ------------------------------------------------------------------
>> > > > ----------------------------------------------------------
>> > > >
>> > > > Cpu Util S SARU
>> > > > U |100%
>> > > > 100% 100%
>> > > > Disk Util F F
>> > > > | 55% 47% 67%
>> > > > Mem Util S SU
>> > > > UB B | 92% 91% 92%1
>> > > > Swap Util U UR R
>> > > > | 56% 56% 56%
>> > > > ------------------------------------------------------------------
>> > > > ----------------------------------------------------------
>> > > >
>> > > > PROCESS
LIST
>> > > > Users= 7
>> > > > User CPU Util Cum Disk
>> > > > Thd
>> > > > Process Name PID PPID Pri Name ( 200% max) CPU IO
>> > > > Rate RSS Cnt
>> > > > ------------------------------------------------------------------
>> > > > --------------
>> > > > arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
>> > > > 2.3 43.7mb 25
>> > > > identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
>> > > > 0.7 388kb 1
>> > > > oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
>> > > > 48.7/17.1 10.7mb 1
>> > > > oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
>> > > > 34.1/10.2 10.7mb 1
>> > > > oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
>> > > > 6.3 10.7mb 1
>> > > > java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
>> > > > 2.4 17.3mb 10
>> > > > oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
>> > > > 3.7 10.8mb 1
>> > > > aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
>> > > > 0.0 7.0mb 1
>> > > > java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
>> > > > 1.5 12.6mb 9
>> > > > oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
>> > > > 8.4 10.7mb 1
>> > > > oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
>> > > > 0.2 11.0mb 1
>> > > > oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
>> > > > 0.0 10.8mb 1
>> > > >
>> > > >
>> > > > These are the errors that pop into the arerror.log before the
>> > > > system crashes:
>> > > > Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
>> > > > SQL database
>> > > > (ARERR 551)
>> > > > Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
>> > > > Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
>> > > > to the database
>> > > > (ARERR 552)
>> > > > Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef
>> > >extension
>> > > > doesn't exist
>> > > > (Oracle -1041)
>> > > > Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
>> > > > fatal error
>> > > > encountered (ARNOTE 21)
>> > > > Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
>> > > > storage. (ARERR
>> > > > 8747)
>> > > > Tue Jun 19 07:59:07 2001 ThreadCleanup()
>> > > > Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
>> > > > to the database
>> > > > (ARERR 552)
>> > > > Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef
>> > >extension
>> > > > doesn't exist
>> > > > (Oracle -1041)
>> > > > Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
>> > > > fatal error
>> > > > encountered (ARNOTE 21)
>> > > > Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
>> > > > storage. (ARERR
>> > > > 8747)
>> > > > Tue Jun 19 07:59:15 2001 ThreadCleanup()
>> > > > Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
>> > > > to the database
>> > > > (ARERR 552)
>> > > > Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef
>> > >extension
>> > > > doesn't exist
>> > > > (Oracle -1041)
>> > > > Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
>> > > > fatal error
>> > > > encountered (ARNOTE 21)
>> > > > Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
>> > > > storage. (ARERR
>> > > > 8747)
>> > > > Tue Jun 19 07:59:53 2001 ThreadCleanup()
>> > > >
>> > > > We apparently have the latest hp patch available. During the
>> > > > cutover we re-did
>> > > > all of our indexes and switched our macros to push fields. We've
>> > > > also reset our
>> > > > kernal parameters to the following:
>> > > > maxdsiz from 67108864 to 200000000
>> > > > maxssiz from 8388608 to 16000000
>> > > > maxtsiz from 67108864 to 200000000
>> > > > maxswapchunks from 1262 to 4096
>> > > > maxfiles from 60 to 2048
>> > > > nkthread 2048 to 2850
>> > > >
>> > > > We are still having performace issues, the load average is high
>> > > > and the cpu
>> > > > usage and memory usage are also very high. Does anyone have any
>> > > > suggestions or
>> > > > comments.
>> > > >
>> > > > ... Tracy
>> > > >
>> > > > ______
>> > > > _______________________
>> > > >
>> > > >
>> > > > You may also control your subscription options, including
>> > > > UNSUBSCRIBE, at www.ARSLIST.org
>> > > >
>> > > >
>> > >
>> > >_____
>>________________________
>> > >
>> > >
>> > >You may also control your subscription options, including UNSUBSCRIBE,
>>at
>> > >www.ARSLIST.org
>> >
>> >
>>_____________
>> > Get Your Private, Free E-mail from MSN Hotmail at
>>http://www.hotmail.com.
>> >
>> > ______
>> > _______________________
>> >
>> >
>> > You may also control your subscription options, including
>> > UNSUBSCRIBE, at www.ARSLIST.org
>> >
>> >
>>
>>______________
_______________
>>
>>
>>You may also control your subscription options, including UNSUBSCRIBE, at
>>www.ARSLIST.org
>
>_____________
>Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.
>
>_______________
______________
>

>You may also control your subscription options, including UNSUBSCRIBE, at
www.ARSLIST.org
>
>
>

________________
_____________


You may also control your subscription options, including UNSUBSCRIBE, at
www.ARSLIST.org

_____________________________

Jarl:

Fortunately we have enough space to allow logging for at least one day.
We review the log file about the time that the server drops and look at
what is happening. We then extract the log segment before and after the
drop, along with the arerror.log file and the system log file and I send
it to our system administration team to review. So far, we think the
problem is with folks who get a little too verbose and decide to fill up
a diary field. I wonder if any other folks here running on UNIX Oracle
are experiencing this problem.

James Mckenzie

Jarl Groneng wrote:
>
> **
>
> James,
>
> I created a small script that turn on the logging every morning and turn
> them off 3 hours later. But until now there has be no problems on the
> server.
>
> Hopefully I get an crach very soon ;-)
>
> --
> Jarl Groneng
> jarl@nextra.com
>
> > -----Original Message-----
> > From: Action Request System discussion list(ARSList)
> > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of James Mckenzie
> > Sent: 21. juni 2001 03:49
> > To: ARSLIST@LISTSERV.VISTAIT.COM
> > Subject: Re: Remedy/Siebel
> >
> >
> > **
> >
> > Jarl:
> >
> > Turn on logging (API and SQL) look for the server failing and see if you
> > get a -SE FAIL just before your failure. This should occur upon an
> > attempt to update a Diary type field. This is caused by the limit of
> > Oracle to 1MB (2^20) by default. You can raise this limit to up to 2GB
> > (unless you have a very large database pick a reasonable limit). The
> > parameter is listing in the Server Admin guide.
> >
> > BTW, what error are you experiencing in the log file when the server
> > crashes?
> >
> > James Mckenzie
> >
> > Jarl Groneng wrote:
> > >
> > > **
> > >
> > > Hi,
> > >
> > > Very similar to the problem we has. Every 8-9 day, the cpu is crying for
> > > air. This happend around 06:20 in the morning(every time). No heavy jobs
> > > running this time. Opend a supportticket may 14, get two reply
> > may 15., have
> > > not heard anything from support after this date. Here is the
> > last mail I got
> > > from them:
> > >
> > > "Am I right to say that the problem has been resolved after
> > server restart?
> > > If so, do you want to close this ticket?"
> > >
> > > ..close the ticket? ha, funny joke. I want my problem solved.
> > >
> > > Got this errors on the client:
> > > ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
> > > occurred
> > >
> > > ARERR [316] Failure with a temporary file : No such file or directory
> > >
> > > ARS 4.5.2 patch 1007
> > > Solaris 8
> > > Oracle 8.1.7 (on another server)
> > >
> > > Restart ars, and get this in the arerror logfile:
> > >
> > > 0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > > 0 : AR System server terminated when a signal/exception was
> > received by the
> > > server (ARNOTE 20)
> > > 15
> > > 0 : Unable to access thread local storage. (ARERR 8747)
> > > ErrorHandler
> > > 0 : Unable to access thread local storage. (ARERR 8747)
> > > RollbackTransaction()
> > > 0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > > 0 : Unable to access thread local storage. (ARERR 8747)
> > > BeginNetRespStatistics()
> > > 0 : Unable to access thread local storage. (ARERR 8747)
> > > ARPostSwitch()
> > > 0 : AR System server terminated -- fatal error encountered (ARNOTE 21)
> > > 0 : Unable to access thread local storage. (ARERR 8747)
> > > ThreadCleanup()
> > > 0 : Unable to access thread local storage. (ARERR 8747)
> > > ThreadCleanup()
> > > 0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > > 0 : AR System server terminated when a signal/exception was
> > received by the
> > > server (ARNOTE 20)
> > > 15
> > > 0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > > 0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > > 0 : AR System server terminated when a signal/exception was
> > received by the
> > > server (ARNOTE 20)
> > > 15
> > > 0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > > 0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > > 0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > > 0 : AR System server terminated when a signal/exception was
> > received by the
> > > server (ARNOTE 20)
> > > 15
> > > 0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > >
> > > --
> > > Jarl Groneng
> > > jarl@nextra.com
> > >
> > > > -----Original Message-----
> > > > From: Action Request System discussion list(ARSList)
> > > > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
> > > > Sent: 20. juni 2001 01:27
> > > > To: ARSLIST@LISTSERV.VISTAIT.COM
> > > > Subject: Remedy/Siebel
> > > >
> > > >
> > > > **
> > > >
> > > > I'm a new subscriber to arslist and I'm looking for help with
> > > > performance issues
> > > > in 4.5.2.
> > > >
> > > > We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
> > > > two weeks ago
> > > > to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
> > > > our box we
> > > > also have an interface with Siebel using MQ Series and Java. Our
> > > > system has
> > > > crashed three times in the past 48hrs, and we've had to do a
> > > > power cycle on the
> > > > box. Our CPU load average is 99% and our memory usage is 98%.
> > > > We have 1gig of
> > > > memory on the box and two cpu's.
> > > >
> > > > Here is a vmstat capture of what's going on on our box:
> > > > $ vmstat -S 2 5
> > > > procs memory page
> > > > faults cpu
> > > > r b w avm free si so pi po fr
> > > > de sr in
> > > > sy cs us sy id
> > > > 7 0 0 41615 9211 1536 1536 25 4 122
> > > > 0 39 686
> > > > 9936 1339 52 30 18
> > > > 7 0 0 41615 9014 0 0 21 0 61
> > > > 0 0 385
> > > > 7854 948 28 72 0
> > > > 6 0 0 43620 7880 0 0 14 0 64
> > > > 0 0 471
> > > > 8778 933 43 55 2
> > > > 6 0 0 43620 6542 0 0 9 0 69
> > > > 0 0 426
> > > > 7555 905 42 57 0
> > > > 6 0 0 43620 5582 0 0 6 0 61
> > > > 0 0 431
> > > > 6988 869 44 56 0
> > > >
> > > > Here is a portion of the glance capture:
> > > > B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
> > > > Current Avg High
> > > > ------------------------------------------------------------------
> > > > ----------------------------------------------------------
> > > >
> > > > Cpu Util S SARU
> > > > U |100%
> > > > 100% 100%
> > > > Disk Util F F
> > > > | 55% 47% 67%
> > > > Mem Util S SU
> > > > UB B | 92% 91% 92%1
> > > > Swap Util U UR R
> > > > | 56% 56% 56%
> > > > ------------------------------------------------------------------
> > > > ----------------------------------------------------------
> > > >
> > > > PROCESS LIST
> > > > Users= 7
> > > > User CPU Util Cum Disk
> > > > Thd
> > > > Process Name PID PPID Pri Name ( 200% max) CPU IO
> > > > Rate RSS Cnt
> > > > ------------------------------------------------------------------
> > > > --------------
> > > > arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
> > > > 2.3 43.7mb 25
> > > > identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
> > > > 0.7 388kb 1
> > > > oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
> > > > 48.7/17.1 10.7mb 1
> > > > oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
> > > > 34.1/10.2 10.7mb 1
> > > > oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
> > > > 6.3 10.7mb 1
> > > > java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
> > > > 2.4 17.3mb 10
> > > > oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
> > > > 3.7 10.8mb 1
> > > > aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
> > > > 0.0 7.0mb 1
> > > > java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
> > > > 1.5 12.6mb 9
> > > > oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
> > > > 8.4 10.7mb 1
> > > > oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
> > > > 0.2 11.0mb 1
> > > > oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
> > > > 0.0 10.8mb 1
> > > >
> > > >
> > > > These are the errors that pop into the arerror.log before the
> > > > system crashes:
> > > > Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
> > > > SQL database
> > > > (ARERR 551)
> > > > Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
> > > > Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
> > > > to the database
> > > > (ARERR 552)
> > > > Tue Jun 19 07:59:07 2001 ORA-01041: internal error.
> > hostdef extension
> > > > doesn't exist
> > > > (Oracle -1041)
> > > > Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
> > > > fatal error
> > > > encountered (ARNOTE 21)
> > > > Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
> > > > storage. (ARERR
> > > > 8747)
> > > > Tue Jun 19 07:59:07 2001 ThreadCleanup()
> > > > Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
> > > > to the database
> > > > (ARERR 552)
> > > > Tue Jun 19 07:59:15 2001 ORA-01041: internal error.
> > hostdef extension
> > > > doesn't exist
> > > > (Oracle -1041)
> > > > Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
> > > > fatal error
> > > > encountered (ARNOTE 21)
> > > > Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
> > > > storage. (ARERR
> > > > 8747)
> > > > Tue Jun 19 07:59:15 2001 ThreadCleanup()
> > > > Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
> > > > to the database
> > > > (ARERR 552)
> > > > Tue Jun 19 07:59:53 2001 ORA-01041: internal error.
> > hostdef extension
> > > > doesn't exist
> > > > (Oracle -1041)
> > > > Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
> > > > fatal error
> > > > encountered (ARNOTE 21)
> > > > Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
> > > > storage. (ARERR
> > > > 8747)
> > > > Tue Jun 19 07:59:53 2001 ThreadCleanup()
> > > >
> > > > We apparently have the latest hp patch available. During the
> > > > cutover we re-did
> > > > all of our indexes and switched our macros to push fields. We've
> > > > also reset our
> > > > kernal parameters to the following:
> > > > maxdsiz from 67108864 to 200000000
> > > > maxssiz from 8388608 to 16000000
> > > > maxtsiz from 67108864 to 200000000
> > > > maxswapchunks from 1262 to 4096
> > > > maxfiles from 60 to 2048
> > > > nkthread 2048 to 2850
> > > >
> > > > We are still having performace issues, the load average is high
> > > > and the cpu
> > > > usage and memory usage are also very high. Does anyone have any
> > > > suggestions or
> > > > comments.
> > > >
> > > > ... Tracy
> > > >
> > > > ______
> > > > _______________________
> > > >
> > > >
> > > > You may also control your subscription options, including
> > > > UNSUBSCRIBE, at www.ARSLIST.org
> > > >
> > > >
> > >
> > >
> > ______
> > _______________________
> > >
> >
> > > You may also control your subscription options, including
> > UNSUBSCRIBE, at www.ARSLIST.org
> >
> > ______
> > _______________________
> >
> >
> > You may also control your subscription options, including
> > UNSUBSCRIBE, at www.ARSLIST.org
> >
> >
>
> _____________________________
>
>

_____________________________

> -----Original Message-----
> From: Action Request System discussion list(ARSList)
> [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Mayes, Ann, NPONS
> Sent: 22. juni 2001 00:54
> To: ARSLIST@LISTSERV.VISTAIT.COM
> Subject: Re: Remedy/Siebel
>
> This has happened several times and a restart of the arserver fixes it. It
> appears to happen when the arserverd process grows to over 200mg. Remedy
> says Unix has a problem with never releasing memory until the
> deamon process
> is restarted.

Our old version was 4.02 patch 809, this was running for periods of 6-8
months without any problem. So I think its to easy to blame UNIX. Strange
they did't blame Oracle for this memory leak stuff.

Hopefylly we will install patch 1025 next week, maybe this solves some
issues.

We have some small memory leaks, but that is related to a coupe of API
programs.
--
Jarl

_____________________________

Hi Ann/Jarl,

Env: Solaris 8, Oracle8i [8.1.6] & ARS 4.5.2 svr/clients

ARERR 91 is a tricky one to resolve on the server/clients, the
error message makes reference to a mismatch between TCP/IP stacks on the
client/server but this is normally not the problem, in my experience I have
see the following error generated on the CLIENTS [NT workstations only, but
I imagine it happens on other o/s's] even if they have plenty of disk space:

"ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
occurred
ARERR [316] Failure with a temporary file : No such file or directory"

In the environment settings on the client they had TMP or TEMP set to a
directory that no longer existed.

For example, TMP & TEMP are set to C:\TEMP, but this directory no longer
exists as it was deleted by the user for whatever reason. If you updated
this/these values to a valid directory all would be OK. For example d:\temp
being a valid directory.


The error was generated only on opening of the main form.

It only affected one or two clients, so if it is across the board in your
cases this may not help.

On HP-UX as the server machine I have seen issues with /var/tmp [/usr/tmp]
filling up, as Remedy on HP I think was more prone to a process of chunking
data to disk during large and even sometimes small queries. I haven't see
any issues on Solaris Jarl with this process and it's easy enough to check
[but perhaps not at 6:20 in the morning manually, not in my case anyway!,
auto is the way to go]

Other than these pointers I would go about troublehsooting in a similar
manner to what you guys have suggested/done[logging db, remedy, os etc]
On Solaris I would also try and use truss [or sotruss] to help me along, as
sometimes it's output can be useful. I see Ann you have already used glance.


Best of Luck,
Kevin



-----Original Message-----
From: Mayes, Ann, NPONS [mailto:annmayes@ATT.COM]
Sent: 21 June 2001 23:5
To: ARSLIST@LISTSERV.VISTAIT.COM
Subject: Re: Remedy/Siebel


**

I am experiencing the same problem. No errors logged in the arerror.log. No
errors in the Oracle logs. Just start getting calls from the users as the
clients start getting strange errors - the same as Jarl

ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
>> > >occurred
>> > >
>> > >ARERR [316] Failure with a temporary file : No such file or directory
>> > >

Also getting "Connection reset by peer" ( I can't remember the error number)

Checked disk space - plenty of disk space, checked glance - plenty of
available memory and plenty of available CPU

Arserverd running okay - gaining time. Process also responds to the
/usr/ar/bin/arsystem stop command with no problem.

This has happened several times and a restart of the arserver fixes it. It
appears to happen when the arserverd process grows to over 200mg. Remedy
says Unix has a problem with never releasing memory until the deamon process
is restarted. The process does not crash - we have increased the process
limit to accommodate its growth. However arserverd seems to get 'weirded
out' (that technical talk for random events I can't explain) when it grows
much over 200 meg

HP-UX 11.0
Oracle 8.1.6
ARS 4.5.2 (no patches)
Aruser - various client OS and versions

Ann Mayes
Ciber, Inc



-----Original Message-----
From: Jarl Grøneng [mailto:rapid@ONLINE.NO]
Sent: June 21, 2001 11:16 AM
To: ARSLIST@LISTSERV.VISTAIT.COM
Subject: Re: Remedy/Siebel


**

Pat,

I'm stuggling to get hold of our HPUX admin as we'r running Solaris :-)

--
Jarl Groneng
jarl@nextra.com



21.06.01 18:11:40, skrev Pat Ferger :

>**
>
>Jarl,
>
>It's not the total system memory that could be the problem, it may be the
>Per-Process-Memory-Limit feature that HP-UX has. HP-UX allows you to limit
>how much memory a process can take. The default limit is 67 MB, which may
>be too low for more sophisticated sites. 4.0.X didn't have this problem
>because on UNIX they were multi-process, meaning there were many processes
>with smaller memory footprints. But with 4.5, the AR Server has been
>consolidated into one process with multiple threads. This may put you over
>the limit in terms of process-memory size.
>
>Talk to your HPUX sysadmin about bumping up the per-process-memory limit.
>
>
>>From: Jarl Groneng
>>Reply-To: "Action Request System discussion list(ARSList)"
>>
>>To: ARSLIST@LISTSERV.VISTAIT.COM
>>Subject: Re: Remedy/Siebel
>>Date: Thu, 21 Jun 2001 07:48:37 +0200
>>
>>**
>>
>>Memory is not a problem, it hardly using swap.
>>
>>Memory: 1024M real, 756M free, 149M swap in use, 1661M swap free
>>
>>
>>Disk space is not a problem either.
>>
>>Filesystem kbytes used avail capacity Mounted on
>>/dev/dsk/c0t0d0s0 3588558 722568 2830105 21% /
>>swap 1699464 16 1699448 1% /var/run
>>swap 1702240 2792 1699448 1% /tmp
>>/dev/dsk/c0t0d0s5 4128246 1722165 2364799 43% /local
>>nfs1.nextel.no:/users
>> 2097152 1119256 977896 54% /home/nfs/users
>>
>>
>>
>>
>>--
>>Jarl Groneng
>>jarl@nextra.com
>>
>>
>>
>> > -----Original Message-----
>> > From: Action Request System discussion list(ARSList)
>> > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Pat Ferger
>> > Sent: 20. juni 2001 19:07
>> > To: ARSLIST@LISTSERV.VISTAIT.COM
>> > Subject: Re: Remedy/Siebel
>> >
>> >
>> > **
>> >
>> > Jarl,
>> >
>> > It sounds like you're running out of memory, or disk space in the /tmp
>> > directory. The "Failure with a temporary file" strongly suggests that
>>the
>> > server was unable to create a temp file, so you should
>> > double-check the size
>> > remaining on all your partitions (/var, /tmp, etc) to make sure you
have
>> > enough space.
>> >
>> > The other problems are memory problems, so maybe you need to bump up
>>your
>> > process memory limits, also.
>> >
>> >
>> > >From: Jarl Groneng
>> > >Reply-To: "Action Request System discussion list(ARSList)"
>> > >
>> > >To: ARSLIST@LISTSERV.VISTAIT.COM
>> > >Subject: Re: Remedy/Siebel
>> > >Date: Wed, 20 Jun 2001 15:51:58 +0200
>> > >
>> > >**
>> > >
>> > >Hi,
>> > >
>> > >Very similar to the problem we has. Every 8-9 day, the cpu is crying
>>for
>> > >air. This happend around 06:20 in the morning(every time). No heavy
>>jobs
>> > >running this time. Opend a supportticket may 14, get two reply may
15.,
>> > >have
>> > >not heard anything from support after this date. Here is the last mail
>>I
>> > >got
>> > >from them:
>> > >
>> > >"Am I right to say that the problem has been resolved after
>> > server restart?
>> > >If so, do you want to close this ticket?"
>> > >
>> > >..close the ticket? ha, funny joke. I want my problem solved.
>> > >
>> > >
>> > >Got this errors on the client:
>> > >ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous
event
>> > >occurred
>> > >
>> > >ARERR [316] Failure with a temporary file : No such file or directory
>> > >
>> > >
>> > >ARS 4.5.2 patch 1007
>> > >Solaris 8
>> > >Oracle 8.1.7 (on another server)
>> > >
>> > >Restart ars, and get this in the arerror logfile:
>> > >
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ErrorHandler
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > RollbackTransaction()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > BeginNetRespStatistics()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ARPostSwitch()
>> > >0 : AR System server terminated -- fatal error encountered (ARNOTE 21)
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ThreadCleanup()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > ThreadCleanup()
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >0 : AR System server terminated normally (ARERR 32)
>> > > ARMonitor
>> > >0 : AR System server terminated when a signal/exception was
>> > received by the
>> > >server (ARNOTE 20)
>> > > 15
>> > >0 : Unable to access thread local storage. (ARERR 8747)
>> > > WriteWorkflowTraceLog()
>> > >
>> > >--
>> > >Jarl Groneng
>> > >jarl@nextra.com
>> > >
>> > >
>> > > > -----Original Message-----
>> > > > From: Action Request System discussion list(ARSList)
>> > > > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
>> > > > Sent: 20. juni 2001 01:27
>> > > > To: ARSLIST@LISTSERV.VISTAIT.COM
>> > > > Subject: Remedy/Siebel
>> > > >
>> > > >
>> > > > **
>> > > >
>> > > > I'm a new subscriber to arslist and I'm looking for help with
>> > > > performance issues
>> > > > in 4.5.2.
>> > > >
>> > > > We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
>> > > > two weeks ago
>> > > > to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
>> > > > our box we
>> > > > also have an interface with Siebel using MQ Series and Java. Our
>> > > > system has
>> > > > crashed three times in the past 48hrs, and we've had to do a
>> > > > power cycle on the
>> > > > box. Our CPU load average is 99% and our memory usage is 98%.
>> > > > We have 1gig of
>> > > > memory on the box and two cpu's.
>> > > >
>> > > > Here is a vmstat capture of what's going on on our box:
>> > > > $ vmstat -S 2 5
>> > > > procs memory page
>> > > > faults cpu
>> > > > r b w avm free si so pi po fr
>> > > > de sr in
>> > > > sy cs us sy id
>> > > > 7 0 0 41615 9211 1536 1536 25 4 122
>> > > > 0 39 686
>> > > > 9936 1339 52 30 18
>> > > > 7 0 0 41615 9014 0 0 21 0 61
>> > > > 0 0 385
>> > > > 7854 948 28 72 0
>> > > > 6 0 0 43620 7880 0 0 14 0 64
>> > > > 0 0 471
>> > > > 8778 933 43 55 2
>> > > > 6 0 0 43620 6542 0 0 9 0 69
>> > > > 0 0 426
>> > > > 7555 905 42 57 0
>> > > > 6 0 0 43620 5582 0 0 6 0 61
>> > > > 0 0 431
>> > > > 6988 869 44 56 0
>> > > >
>> > > > Here is a portion of the glance capture:
>> > > > B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
>> > > > Current Avg High
>> > > > ------------------------------------------------------------------
>> > > > ----------------------------------------------------------
>> > > >
>> > > > Cpu Util S SARU
>> > > > U |100%
>> > > > 100% 100%
>> > > > Disk Util F F
>> > > > | 55% 47% 67%
>> > > > Mem Util S SU
>> > > > UB B | 92% 91% 92%1
>> > > > Swap Util U UR R
>> > > > | 56% 56% 56%
>> > > > ------------------------------------------------------------------
>> > > > ----------------------------------------------------------
>> > > >
>> > > > PROCESS
LIST
>> > > > Users= 7
>> > > > User CPU Util Cum Disk
>> > > > Thd
>> > > > Process Name PID PPID Pri Name ( 200% max) CPU IO
>> > > > Rate RSS Cnt
>> > > > ------------------------------------------------------------------
>> > > > --------------
>> > > > arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
>> > > > 2.3 43.7mb 25
>> > > > identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
>> > > > 0.7 388kb 1
>> > > > oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
>> > > > 48.7/17.1 10.7mb 1
>> > > > oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
>> > > > 34.1/10.2 10.7mb 1
>> > > > oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
>> > > > 6.3 10.7mb 1
>> > > > java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
>> > > > 2.4 17.3mb 10
>> > > > oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
>> > > > 3.7 10.8mb 1
>> > > > aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
>> > > > 0.0 7.0mb 1
>> > > > java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
>> > > > 1.5 12.6mb 9
>> > > > oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
>> > > > 8.4 10.7mb 1
>> > > > oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
>> > > > 0.2 11.0mb 1
>> > > > oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
>> > > > 0.0 10.8mb 1
>> > > >
>> > > >
>> > > > These are the errors that pop into the arerror.log before the
>> > > > system crashes:
>> > > > Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
>> > > > SQL database
>> > > > (ARERR 551)
>> > > > Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
>> > > > Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
>> > > > to the database
>> > > > (ARERR 552)
>> > > > Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef
>> > >extension
>> > > > doesn't exist
>> > > > (Oracle -1041)
>> > > > Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
>> > > > fatal error
>> > > > encountered (ARNOTE 21)
>> > > > Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
>> > > > storage. (ARERR
>> > > > 8747)
>> > > > Tue Jun 19 07:59:07 2001 ThreadCleanup()
>> > > > Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
>> > > > to the database
>> > > > (ARERR 552)
>> > > > Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef
>> > >extension
>> > > > doesn't exist
>> > > > (Oracle -1041)
>> > > > Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
>> > > > fatal error
>> > > > encountered (ARNOTE 21)
>> > > > Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
>> > > > storage. (ARERR
>> > > > 8747)
>> > > > Tue Jun 19 07:59:15 2001 ThreadCleanup()
>> > > > Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
>> > > > to the database
>> > > > (ARERR 552)
>> > > > Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef
>> > >extension
>> > > > doesn't exist
>> > > > (Oracle -1041)
>> > > > Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
>> > > > fatal error
>> > > > encountered (ARNOTE 21)
>> > > > Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
>> > > > storage. (ARERR
>> > > > 8747)
>> > > > Tue Jun 19 07:59:53 2001 ThreadCleanup()
>> > > >
>> > > > We apparently have the latest hp patch available. During the
>> > > > cutover we re-did
>> > > > all of our indexes and switched our macros to push fields. We've
>> > > > also reset our
>> > > > kernal parameters to the following:
>> > > > maxdsiz from 67108864 to 200000000
>> > > > maxssiz from 8388608 to 16000000
>> > > > maxtsiz from 67108864 to 200000000
>> > > > maxswapchunks from 1262 to 4096
>> > > > maxfiles from 60 to 2048
>> > > > nkthread 2048 to 2850
>> > > >
>> > > > We are still having performace issues, the load average is high
>> > > > and the cpu
>> > > > usage and memory usage are also very high. Does anyone have any
>> > > > suggestions or
>> > > > comments.
>> > > >
>> > > > ... Tracy
>> > > >
>> > > > ______
>> > > > _______________________
>> > > >
>> > > >
>> > > > You may also control your subscription options, including
>> > > > UNSUBSCRIBE, at www.ARSLIST.org
>> > > >
>> > > >
>> > >
>> > >_____
>>________________________
>> > >
>> > >
>> > >You may also control your subscription options, including UNSUBSCRIBE,
>>at
>> > >www.ARSLIST.org
>> >
>> >
>>_____________
>> > Get Your Private, Free E-mail from MSN Hotmail at
>>http://www.hotmail.com.
>> >
>> > ______
>> > _______________________
>> >
>> >
>> > You may also control your subscription options, including
>> > UNSUBSCRIBE, at www.ARSLIST.org
>> >
>> >
>>
>>______________
_______________
>>
>>
>>You may also control your subscription options, including UNSUBSCRIBE, at
>>www.ARSLIST.org
>
>_____________
>Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.
>
>_______________
______________
>

>You may also control your subscription options, including UNSUBSCRIBE, at
www.ARSLIST.org
>
>
>

________________
_____________


You may also control your subscription options, including UNSUBSCRIBE, at
www.ARSLIST.org

________________
_____________


You may also control your subscription options, including UNSUBSCRIBE, at
www.ARSLIST.org


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

_____________________________