I'm a new subscriber to arslist and I'm looking for help with performance issues
in 4.5.2.

We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded two weeks ago
to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On our box we
also have an interface with Siebel using MQ Series and Java. Our system has
crashed three times in the past 48hrs, and we've had to do a power cycle on the
box. Our CPU load average is 99% and our memory usage is 98%. We have 1gig of
memory on the box and two cpu's.

Here is a vmstat capture of what's going on on our box:
$ vmstat -S 2 5
procs memory page
faults cpu
r b w avm free si so pi po fr de sr in
sy cs us sy id
7 0 0 41615 9211 1536 1536 25 4 122 0 39 686
9936 1339 52 30 18
7 0 0 41615 9014 0 0 21 0 61 0 0 385
7854 948 28 72 0
6 0 0 43620 7880 0 0 14 0 64 0 0 471
8778 933 43 55 2
6 0 0 43620 6542 0 0 9 0 69 0 0 426
7555 905 42 57 0
6 0 0 43620 5582 0 0 6 0 61 0 0 431
6988 869 44 56 0

Here is a portion of the glance capture:
B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
Current Avg High
----------------------------------------------------------------------------------------------------------------------------

Cpu Util S SARU
U |100% 100% 100%
Disk Util F F
| 55% 47% 67%
Mem Util S SU
UB B | 92% 91% 92%1
Swap Util U UR R
| 56% 56% 56%
----------------------------------------------------------------------------------------------------------------------------

PROCESS LIST
Users= 7
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name ( 200% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/ 2.3 43.7mb 25
identd 9052 892 154 bin 18.3/18.3 1.0 0.7/ 0.7 388kb 1
oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3 48.7/17.1 10.7mb 1
oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3 34.1/10.2 10.7mb 1
oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/ 6.3 10.7mb 1
java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/ 2.4 17.3mb 10
oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/ 3.7 10.8mb 1
aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/ 0.0 7.0mb 1
java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/ 1.5 12.6mb 9
oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/ 8.4 10.7mb 1
oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/ 0.2 11.0mb 1
oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/ 0.0 10.8mb 1


These are the errors that pop into the arerror.log before the system crashes:
Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with SQL database
(ARERR 551)
Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation to the database
(ARERR 552)
Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef extension
doesn't exist
(Oracle -1041)
Tue Jun 19 07:59:07 2001 390635 : AR System server terminated -- fatal error
encountered (ARNOTE 21)
Tue Jun 19 07:59:07 2001 0 : Unable to access thread local storage. (ARERR
8747)
Tue Jun 19 07:59:07 2001 ThreadCleanup()
Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation to the database
(ARERR 552)
Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef extension
doesn't exist
(Oracle -1041)
Tue Jun 19 07:59:15 2001 390635 : AR System server terminated -- fatal error
encountered (ARNOTE 21)
Tue Jun 19 07:59:15 2001 0 : Unable to access thread local storage. (ARERR
8747)
Tue Jun 19 07:59:15 2001 ThreadCleanup()
Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation to the database
(ARERR 552)
Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef extension
doesn't exist
(Oracle -1041)
Tue Jun 19 07:59:53 2001 390635 : AR System server terminated -- fatal error
encountered (ARNOTE 21)
Tue Jun 19 07:59:53 2001 0 : Unable to access thread local storage. (ARERR
8747)
Tue Jun 19 07:59:53 2001 ThreadCleanup()

We apparently have the latest hp patch available. During the cutover we re-did
all of our indexes and switched our macros to push fields. We've also reset our
kernal parameters to the following:
maxdsiz from 67108864 to 200000000
maxssiz from 8388608 to 16000000
maxtsiz from 67108864 to 200000000
maxswapchunks from 1262 to 4096
maxfiles from 60 to 2048
nkthread 2048 to 2850

We are still having performace issues, the load average is high and the cpu
usage and memory usage are also very high. Does anyone have any suggestions or
comments.

... Tracy

_____________________________

Tracy,

The main difference between Solaris and HP-UX in your case is that on HP-UX
they manually set the maximum amount of memory that a process can use. On
Solaris, you can use as much memory as you want. This has both advantages
and disadvantages.

As you pointed out, the default limit per process is 67 MB. Since you were
using Solaris, you probably don't know what the ceiling of memory use was
for both Oracle and AR Server, do you?

Setting the process memory limit to 200 MB may not be enough. I've seen
lots of problems on HP-UX vanish when you increase the process memory limit.

The difference between <4.5 and 4.5 is that the UNIX servers went from
multi-process to multi-threaded. This means that instead of having a bunch
of small processes, each with their own memory, the server has been
consolidated into one larger process with many threads. This is more
efficient, however it is more memory intensive.

How many forms, active links, filters, etc do you have? Is it a lot? How
big are the queries that are being generated by your users? All these affect
the amount of memory that arserverd requires.

I would set the memory limit to be as high as you can afford, possibly 1 GB
and see if this solves your crashing problem. Watch the memory
characteristics of your server, and then start scaling down accordingly.


>From: Tracy Bruneau
>Reply-To: "Action Request System discussion list(ARSList)"
>
>To: ARSLIST@LISTSERV.VISTAIT.COM
>Subject: Remedy/Siebel
>Date: Tue, 19 Jun 2001 16:26:54 -0700
>
>**
>
>I'm a new subscriber to arslist and I'm looking for help with performance
>issues
>in 4.5.2.
>
>We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded two weeks
>ago
>to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On our box
>we
>also have an interface with Siebel using MQ Series and Java. Our system
>has
>crashed three times in the past 48hrs, and we've had to do a power cycle on
>the
>box. Our CPU load average is 99% and our memory usage is 98%. We have
>1gig of
>memory on the box and two cpu's.
>
>Here is a vmstat capture of what's going on on our box:
>$ vmstat -S 2 5
> procs memory page
> faults cpu
> r b w avm free si so pi po fr de sr
> in
> sy cs us sy id
> 7 0 0 41615 9211 1536 1536 25 4 122 0 39
> 686
> 9936 1339 52 30 18
> 7 0 0 41615 9014 0 0 21 0 61 0 0
> 385
> 7854 948 28 72 0
> 6 0 0 43620 7880 0 0 14 0 64 0 0
> 471
> 8778 933 43 55 2
> 6 0 0 43620 6542 0 0 9 0 69 0 0
> 426
> 7555 905 42 57 0
> 6 0 0 43620 5582 0 0 6 0 61 0 0
> 431
> 6988 869 44 56 0
>
>Here is a portion of the glance capture:
>B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
> Current Avg High
>----------------------------------------------------------------------------------------------------------------------------
>
>Cpu Util S SARU
> U |100% 100%
>100%
>Disk Util F F
> | 55% 47% 67%
>Mem Util S SU
>UB B | 92% 91% 92%1
>Swap Util U UR R
> | 56% 56% 56%
>----------------------------------------------------------------------------------------------------------------------------
>
> PROCESS LIST
> Users= 7
> User CPU Util Cum Disk
> Thd
>Process Name PID PPID Pri Name ( 200% max) CPU IO Rate RSS
>Cnt
>--------------------------------------------------------------------------------
>arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/ 2.3 43.7mb
> 25
>identd 9052 892 154 bin 18.3/18.3 1.0 0.7/ 0.7 388kb
> 1
>oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3 48.7/17.1 10.7mb
> 1
>oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3 34.1/10.2 10.7mb
> 1
>oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/ 6.3 10.7mb
> 1
>java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/ 2.4 17.3mb
> 10
>oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/ 3.7 10.8mb
> 1
>aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/ 0.0 7.0mb
> 1
>java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/ 1.5 12.6mb
> 9
>oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/ 8.4 10.7mb
> 1
>oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/ 0.2 11.0mb
> 1
>oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/ 0.0 10.8mb
> 1
>
>
>These are the errors that pop into the arerror.log before the system
>crashes:
>Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with SQL
>database
>(ARERR 551)
>Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
>Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation to the
>database
>(ARERR 552)
>Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef extension
>doesn't exist
> (Oracle -1041)
>Tue Jun 19 07:59:07 2001 390635 : AR System server terminated -- fatal
>error
>encountered (ARNOTE 21)
>Tue Jun 19 07:59:07 2001 0 : Unable to access thread local storage. (ARERR
>8747)
>Tue Jun 19 07:59:07 2001 ThreadCleanup()
>Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation to the
>database
>(ARERR 552)
>Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef extension
>doesn't exist
> (Oracle -1041)
>Tue Jun 19 07:59:15 2001 390635 : AR System server terminated -- fatal
>error
>encountered (ARNOTE 21)
>Tue Jun 19 07:59:15 2001 0 : Unable to access thread local storage. (ARERR
>8747)
>Tue Jun 19 07:59:15 2001 ThreadCleanup()
>Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation to the
>database
>(ARERR 552)
>Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef extension
>doesn't exist
> (Oracle -1041)
>Tue Jun 19 07:59:53 2001 390635 : AR System server terminated -- fatal
>error
>encountered (ARNOTE 21)
>Tue Jun 19 07:59:53 2001 0 : Unable to access thread local storage. (ARERR
>8747)
>Tue Jun 19 07:59:53 2001 ThreadCleanup()
>
>We apparently have the latest hp patch available. During the cutover we
>re-did
>all of our indexes and switched our macros to push fields. We've also
>reset our
>kernal parameters to the following:
>maxdsiz from 67108864 to 200000000
>maxssiz from 8388608 to 16000000
>maxtsiz from 67108864 to 200000000
>maxswapchunks from 1262 to 4096
>maxfiles from 60 to 2048
>nkthread 2048 to 2850
>
>We are still having performace issues, the load average is high and the cpu
>usage and memory usage are also very high. Does anyone have any
>suggestions or
>comments.
>
>... Tracy
>
>_____________________________
>
>
>You may also control your subscription options, including UNSUBSCRIBE, at
>www.ARSLIST.org

_____________
Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.

_____________________________

Hi,

Very similar to the problem we has. Every 8-9 day, the cpu is crying for
air. This happend around 06:20 in the morning(every time). No heavy jobs
running this time. Opend a supportticket may 14, get two reply may 15., have
not heard anything from support after this date. Here is the last mail I got
from them:

"Am I right to say that the problem has been resolved after server restart?
If so, do you want to close this ticket?"

..close the ticket? ha, funny joke. I want my problem solved.


Got this errors on the client:
ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
occurred

ARERR [316] Failure with a temporary file : No such file or directory


ARS 4.5.2 patch 1007
Solaris 8
Oracle 8.1.7 (on another server)

Restart ars, and get this in the arerror logfile:

0 : AR System server terminated normally (ARERR 32)
ARMonitor
0 : AR System server terminated when a signal/exception was received by the
server (ARNOTE 20)
15
0 : Unable to access thread local storage. (ARERR 8747)
ErrorHandler
0 : Unable to access thread local storage. (ARERR 8747)
RollbackTransaction()
0 : Unable to access thread local storage. (ARERR 8747)
WriteWorkflowTraceLog()
0 : Unable to access thread local storage. (ARERR 8747)
BeginNetRespStatistics()
0 : Unable to access thread local storage. (ARERR 8747)
ARPostSwitch()
0 : AR System server terminated -- fatal error encountered (ARNOTE 21)
0 : Unable to access thread local storage. (ARERR 8747)
ThreadCleanup()
0 : Unable to access thread local storage. (ARERR 8747)
ThreadCleanup()
0 : AR System server terminated normally (ARERR 32)
ARMonitor
0 : AR System server terminated when a signal/exception was received by the
server (ARNOTE 20)
15
0 : AR System server terminated normally (ARERR 32)
ARMonitor
0 : AR System server terminated normally (ARERR 32)
ARMonitor
0 : AR System server terminated when a signal/exception was received by the
server (ARNOTE 20)
15
0 : Unable to access thread local storage. (ARERR 8747)
WriteWorkflowTraceLog()
0 : Unable to access thread local storage. (ARERR 8747)
WriteWorkflowTraceLog()
0 : AR System server terminated normally (ARERR 32)
ARMonitor
0 : AR System server terminated when a signal/exception was received by the
server (ARNOTE 20)
15
0 : Unable to access thread local storage. (ARERR 8747)
WriteWorkflowTraceLog()

--
Jarl Groneng
jarl@nextra.com


> -----Original Message-----
> From: Action Request System discussion list(ARSList)
> [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
> Sent: 20. juni 2001 01:27
> To: ARSLIST@LISTSERV.VISTAIT.COM
> Subject: Remedy/Siebel
>
>
> **
>
> I'm a new subscriber to arslist and I'm looking for help with
> performance issues
> in 4.5.2.
>
> We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
> two weeks ago
> to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
> our box we
> also have an interface with Siebel using MQ Series and Java. Our
> system has
> crashed three times in the past 48hrs, and we've had to do a
> power cycle on the
> box. Our CPU load average is 99% and our memory usage is 98%.
> We have 1gig of
> memory on the box and two cpu's.
>
> Here is a vmstat capture of what's going on on our box:
> $ vmstat -S 2 5
> procs memory page
> faults cpu
> r b w avm free si so pi po fr
> de sr in
> sy cs us sy id
> 7 0 0 41615 9211 1536 1536 25 4 122
> 0 39 686
> 9936 1339 52 30 18
> 7 0 0 41615 9014 0 0 21 0 61
> 0 0 385
> 7854 948 28 72 0
> 6 0 0 43620 7880 0 0 14 0 64
> 0 0 471
> 8778 933 43 55 2
> 6 0 0 43620 6542 0 0 9 0 69
> 0 0 426
> 7555 905 42 57 0
> 6 0 0 43620 5582 0 0 6 0 61
> 0 0 431
> 6988 869 44 56 0
>
> Here is a portion of the glance capture:
> B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
> Current Avg High
> ------------------------------------------------------------------
> ----------------------------------------------------------
>
> Cpu Util S SARU
> U |100%
> 100% 100%
> Disk Util F F
> | 55% 47% 67%
> Mem Util S SU
> UB B | 92% 91% 92%1
> Swap Util U UR R
> | 56% 56% 56%
> ------------------------------------------------------------------
> ----------------------------------------------------------
>
> PROCESS LIST
> Users= 7
> User CPU Util Cum Disk
> Thd
> Process Name PID PPID Pri Name ( 200% max) CPU IO
> Rate RSS Cnt
> ------------------------------------------------------------------
> --------------
> arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
> 2.3 43.7mb 25
> identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
> 0.7 388kb 1
> oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
> 48.7/17.1 10.7mb 1
> oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
> 34.1/10.2 10.7mb 1
> oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
> 6.3 10.7mb 1
> java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
> 2.4 17.3mb 10
> oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
> 3.7 10.8mb 1
> aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
> 0.0 7.0mb 1
> java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
> 1.5 12.6mb 9
> oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
> 8.4 10.7mb 1
> oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
> 0.2 11.0mb 1
> oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
> 0.0 10.8mb 1
>
>
> These are the errors that pop into the arerror.log before the
> system crashes:
> Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
> SQL database
> (ARERR 551)
> Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
> Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
> to the database
> (ARERR 552)
> Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef extension
> doesn't exist
> (Oracle -1041)
> Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
> fatal error
> encountered (ARNOTE 21)
> Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
> storage. (ARERR
> 8747)
> Tue Jun 19 07:59:07 2001 ThreadCleanup()
> Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
> to the database
> (ARERR 552)
> Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef extension
> doesn't exist
> (Oracle -1041)
> Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
> fatal error
> encountered (ARNOTE 21)
> Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
> storage. (ARERR
> 8747)
> Tue Jun 19 07:59:15 2001 ThreadCleanup()
> Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
> to the database
> (ARERR 552)
> Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef extension
> doesn't exist
> (Oracle -1041)
> Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
> fatal error
> encountered (ARNOTE 21)
> Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
> storage. (ARERR
> 8747)
> Tue Jun 19 07:59:53 2001 ThreadCleanup()
>
> We apparently have the latest hp patch available. During the
> cutover we re-did
> all of our indexes and switched our macros to push fields. We've
> also reset our
> kernal parameters to the following:
> maxdsiz from 67108864 to 200000000
> maxssiz from 8388608 to 16000000
> maxtsiz from 67108864 to 200000000
> maxswapchunks from 1262 to 4096
> maxfiles from 60 to 2048
> nkthread 2048 to 2850
>
> We are still having performace issues, the load average is high
> and the cpu
> usage and memory usage are also very high. Does anyone have any
> suggestions or
> comments.
>
> ... Tracy
>
> ______
> _______________________
>
>
> You may also control your subscription options, including
> UNSUBSCRIBE, at www.ARSLIST.org
>
>

_____________________________

Jarl,

It sounds like you're running out of memory, or disk space in the /tmp
directory. The "Failure with a temporary file" strongly suggests that the
server was unable to create a temp file, so you should double-check the size
remaining on all your partitions (/var, /tmp, etc) to make sure you have
enough space.

The other problems are memory problems, so maybe you need to bump up your
process memory limits, also.


>From: Jarl Groneng
>Reply-To: "Action Request System discussion list(ARSList)"
>
>To: ARSLIST@LISTSERV.VISTAIT.COM
>Subject: Re: Remedy/Siebel
>Date: Wed, 20 Jun 2001 15:51:58 +0200
>
>**
>
>Hi,
>
>Very similar to the problem we has. Every 8-9 day, the cpu is crying for
>air. This happend around 06:20 in the morning(every time). No heavy jobs
>running this time. Opend a supportticket may 14, get two reply may 15.,
>have
>not heard anything from support after this date. Here is the last mail I
>got
>from them:
>
>"Am I right to say that the problem has been resolved after server restart?
>If so, do you want to close this ticket?"
>
>..close the ticket? ha, funny joke. I want my problem solved.
>
>
>Got this errors on the client:
>ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
>occurred
>
>ARERR [316] Failure with a temporary file : No such file or directory
>
>
>ARS 4.5.2 patch 1007
>Solaris 8
>Oracle 8.1.7 (on another server)
>
>Restart ars, and get this in the arerror logfile:
>
>0 : AR System server terminated normally (ARERR 32)
> ARMonitor
>0 : AR System server terminated when a signal/exception was received by the
>server (ARNOTE 20)
> 15
>0 : Unable to access thread local storage. (ARERR 8747)
> ErrorHandler
>0 : Unable to access thread local storage. (ARERR 8747)
> RollbackTransaction()
>0 : Unable to access thread local storage. (ARERR 8747)
> WriteWorkflowTraceLog()
>0 : Unable to access thread local storage. (ARERR 8747)
> BeginNetRespStatistics()
>0 : Unable to access thread local storage. (ARERR 8747)
> ARPostSwitch()
>0 : AR System server terminated -- fatal error encountered (ARNOTE 21)
>0 : Unable to access thread local storage. (ARERR 8747)
> ThreadCleanup()
>0 : Unable to access thread local storage. (ARERR 8747)
> ThreadCleanup()
>0 : AR System server terminated normally (ARERR 32)
> ARMonitor
>0 : AR System server terminated when a signal/exception was received by the
>server (ARNOTE 20)
> 15
>0 : AR System server terminated normally (ARERR 32)
> ARMonitor
>0 : AR System server terminated normally (ARERR 32)
> ARMonitor
>0 : AR System server terminated when a signal/exception was received by the
>server (ARNOTE 20)
> 15
>0 : Unable to access thread local storage. (ARERR 8747)
> WriteWorkflowTraceLog()
>0 : Unable to access thread local storage. (ARERR 8747)
> WriteWorkflowTraceLog()
>0 : AR System server terminated normally (ARERR 32)
> ARMonitor
>0 : AR System server terminated when a signal/exception was received by the
>server (ARNOTE 20)
> 15
>0 : Unable to access thread local storage. (ARERR 8747)
> WriteWorkflowTraceLog()
>
>--
>Jarl Groneng
>jarl@nextra.com
>
>
> > -----Original Message-----
> > From: Action Request System discussion list(ARSList)
> > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
> > Sent: 20. juni 2001 01:27
> > To: ARSLIST@LISTSERV.VISTAIT.COM
> > Subject: Remedy/Siebel
> >
> >
> > **
> >
> > I'm a new subscriber to arslist and I'm looking for help with
> > performance issues
> > in 4.5.2.
> >
> > We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
> > two weeks ago
> > to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
> > our box we
> > also have an interface with Siebel using MQ Series and Java. Our
> > system has
> > crashed three times in the past 48hrs, and we've had to do a
> > power cycle on the
> > box. Our CPU load average is 99% and our memory usage is 98%.
> > We have 1gig of
> > memory on the box and two cpu's.
> >
> > Here is a vmstat capture of what's going on on our box:
> > $ vmstat -S 2 5
> > procs memory page
> > faults cpu
> > r b w avm free si so pi po fr
> > de sr in
> > sy cs us sy id
> > 7 0 0 41615 9211 1536 1536 25 4 122
> > 0 39 686
> > 9936 1339 52 30 18
> > 7 0 0 41615 9014 0 0 21 0 61
> > 0 0 385
> > 7854 948 28 72 0
> > 6 0 0 43620 7880 0 0 14 0 64
> > 0 0 471
> > 8778 933 43 55 2
> > 6 0 0 43620 6542 0 0 9 0 69
> > 0 0 426
> > 7555 905 42 57 0
> > 6 0 0 43620 5582 0 0 6 0 61
> > 0 0 431
> > 6988 869 44 56 0
> >
> > Here is a portion of the glance capture:
> > B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
> > Current Avg High
> > ------------------------------------------------------------------
> > ----------------------------------------------------------
> >
> > Cpu Util S SARU
> > U |100%
> > 100% 100%
> > Disk Util F F
> > | 55% 47% 67%
> > Mem Util S SU
> > UB B | 92% 91% 92%1
> > Swap Util U UR R
> > | 56% 56% 56%
> > ------------------------------------------------------------------
> > ----------------------------------------------------------
> >
> > PROCESS LIST
> > Users= 7
> > User CPU Util Cum Disk
> > Thd
> > Process Name PID PPID Pri Name ( 200% max) CPU IO
> > Rate RSS Cnt
> > ------------------------------------------------------------------
> > --------------
> > arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
> > 2.3 43.7mb 25
> > identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
> > 0.7 388kb 1
> > oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
> > 48.7/17.1 10.7mb 1
> > oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
> > 34.1/10.2 10.7mb 1
> > oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
> > 6.3 10.7mb 1
> > java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
> > 2.4 17.3mb 10
> > oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
> > 3.7 10.8mb 1
> > aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
> > 0.0 7.0mb 1
> > java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
> > 1.5 12.6mb 9
> > oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
> > 8.4 10.7mb 1
> > oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
> > 0.2 11.0mb 1
> > oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
> > 0.0 10.8mb 1
> >
> >
> > These are the errors that pop into the arerror.log before the
> > system crashes:
> > Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
> > SQL database
> > (ARERR 551)
> > Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
> > Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
> > to the database
> > (ARERR 552)
> > Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef
>extension
> > doesn't exist
> > (Oracle -1041)
> > Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
> > fatal error
> > encountered (ARNOTE 21)
> > Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
> > storage. (ARERR
> > 8747)
> > Tue Jun 19 07:59:07 2001 ThreadCleanup()
> > Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
> > to the database
> > (ARERR 552)
> > Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef
>extension
> > doesn't exist
> > (Oracle -1041)
> > Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
> > fatal error
> > encountered (ARNOTE 21)
> > Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
> > storage. (ARERR
> > 8747)
> > Tue Jun 19 07:59:15 2001 ThreadCleanup()
> > Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
> > to the database
> > (ARERR 552)
> > Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef
>extension
> > doesn't exist
> > (Oracle -1041)
> > Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
> > fatal error
> > encountered (ARNOTE 21)
> > Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
> > storage. (ARERR
> > 8747)
> > Tue Jun 19 07:59:53 2001 ThreadCleanup()
> >
> > We apparently have the latest hp patch available. During the
> > cutover we re-did
> > all of our indexes and switched our macros to push fields. We've
> > also reset our
> > kernal parameters to the following:
> > maxdsiz from 67108864 to 200000000
> > maxssiz from 8388608 to 16000000
> > maxtsiz from 67108864 to 200000000
> > maxswapchunks from 1262 to 4096
> > maxfiles from 60 to 2048
> > nkthread 2048 to 2850
> >
> > We are still having performace issues, the load average is high
> > and the cpu
> > usage and memory usage are also very high. Does anyone have any
> > suggestions or
> > comments.
> >
> > ... Tracy
> >
> > ______
> > _______________________
> >
> >
> > You may also control your subscription options, including
> > UNSUBSCRIBE, at www.ARSLIST.org
> >
> >
>
>_____________________________
>
>
>You may also control your subscription options, including UNSUBSCRIBE, at
>www.ARSLIST.org

_____________
Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.

_____________________________

Memory is not a problem, it hardly using swap.

Memory: 1024M real, 756M free, 149M swap in use, 1661M swap free


Disk space is not a problem either.

Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 3588558 722568 2830105 21% /
swap 1699464 16 1699448 1% /var/run
swap 1702240 2792 1699448 1% /tmp
/dev/dsk/c0t0d0s5 4128246 1722165 2364799 43% /local
nfs1.nextel.no:/users
2097152 1119256 977896 54% /home/nfs/users




--
Jarl Groneng
jarl@nextra.com



> -----Original Message-----
> From: Action Request System discussion list(ARSList)
> [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Pat Ferger
> Sent: 20. juni 2001 19:07
> To: ARSLIST@LISTSERV.VISTAIT.COM
> Subject: Re: Remedy/Siebel
>
>
> **
>
> Jarl,
>
> It sounds like you're running out of memory, or disk space in the /tmp
> directory. The "Failure with a temporary file" strongly suggests that the
> server was unable to create a temp file, so you should
> double-check the size
> remaining on all your partitions (/var, /tmp, etc) to make sure you have
> enough space.
>
> The other problems are memory problems, so maybe you need to bump up your
> process memory limits, also.
>
>
> >From: Jarl Groneng
> >Reply-To: "Action Request System discussion list(ARSList)"
> >
> >To: ARSLIST@LISTSERV.VISTAIT.COM
> >Subject: Re: Remedy/Siebel
> >Date: Wed, 20 Jun 2001 15:51:58 +0200
> >
> >**
> >
> >Hi,
> >
> >Very similar to the problem we has. Every 8-9 day, the cpu is crying for
> >air. This happend around 06:20 in the morning(every time). No heavy jobs
> >running this time. Opend a supportticket may 14, get two reply may 15.,
> >have
> >not heard anything from support after this date. Here is the last mail I
> >got
> >from them:
> >
> >"Am I right to say that the problem has been resolved after
> server restart?
> >If so, do you want to close this ticket?"
> >
> >..close the ticket? ha, funny joke. I want my problem solved.
> >
> >
> >Got this errors on the client:
> >ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
> >occurred
> >
> >ARERR [316] Failure with a temporary file : No such file or directory
> >
> >
> >ARS 4.5.2 patch 1007
> >Solaris 8
> >Oracle 8.1.7 (on another server)
> >
> >Restart ars, and get this in the arerror logfile:
> >
> >0 : AR System server terminated normally (ARERR 32)
> > ARMonitor
> >0 : AR System server terminated when a signal/exception was
> received by the
> >server (ARNOTE 20)
> > 15
> >0 : Unable to access thread local storage. (ARERR 8747)
> > ErrorHandler
> >0 : Unable to access thread local storage. (ARERR 8747)
> > RollbackTransaction()
> >0 : Unable to access thread local storage. (ARERR 8747)
> > WriteWorkflowTraceLog()
> >0 : Unable to access thread local storage. (ARERR 8747)
> > BeginNetRespStatistics()
> >0 : Unable to access thread local storage. (ARERR 8747)
> > ARPostSwitch()
> >0 : AR System server terminated -- fatal error encountered (ARNOTE 21)
> >0 : Unable to access thread local storage. (ARERR 8747)
> > ThreadCleanup()
> >0 : Unable to access thread local storage. (ARERR 8747)
> > ThreadCleanup()
> >0 : AR System server terminated normally (ARERR 32)
> > ARMonitor
> >0 : AR System server terminated when a signal/exception was
> received by the
> >server (ARNOTE 20)
> > 15
> >0 : AR System server terminated normally (ARERR 32)
> > ARMonitor
> >0 : AR System server terminated normally (ARERR 32)
> > ARMonitor
> >0 : AR System server terminated when a signal/exception was
> received by the
> >server (ARNOTE 20)
> > 15
> >0 : Unable to access thread local storage. (ARERR 8747)
> > WriteWorkflowTraceLog()
> >0 : Unable to access thread local storage. (ARERR 8747)
> > WriteWorkflowTraceLog()
> >0 : AR System server terminated normally (ARERR 32)
> > ARMonitor
> >0 : AR System server terminated when a signal/exception was
> received by the
> >server (ARNOTE 20)
> > 15
> >0 : Unable to access thread local storage. (ARERR 8747)
> > WriteWorkflowTraceLog()
> >
> >--
> >Jarl Groneng
> >jarl@nextra.com
> >
> >
> > > -----Original Message-----
> > > From: Action Request System discussion list(ARSList)
> > > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
> > > Sent: 20. juni 2001 01:27
> > > To: ARSLIST@LISTSERV.VISTAIT.COM
> > > Subject: Remedy/Siebel
> > >
> > >
> > > **
> > >
> > > I'm a new subscriber to arslist and I'm looking for help with
> > > performance issues
> > > in 4.5.2.
> > >
> > > We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
> > > two weeks ago
> > > to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
> > > our box we
> > > also have an interface with Siebel using MQ Series and Java. Our
> > > system has
> > > crashed three times in the past 48hrs, and we've had to do a
> > > power cycle on the
> > > box. Our CPU load average is 99% and our memory usage is 98%.
> > > We have 1gig of
> > > memory on the box and two cpu's.
> > >
> > > Here is a vmstat capture of what's going on on our box:
> > > $ vmstat -S 2 5
> > > procs memory page
> > > faults cpu
> > > r b w avm free si so pi po fr
> > > de sr in
> > > sy cs us sy id
> > > 7 0 0 41615 9211 1536 1536 25 4 122
> > > 0 39 686
> > > 9936 1339 52 30 18
> > > 7 0 0 41615 9014 0 0 21 0 61
> > > 0 0 385
> > > 7854 948 28 72 0
> > > 6 0 0 43620 7880 0 0 14 0 64
> > > 0 0 471
> > > 8778 933 43 55 2
> > > 6 0 0 43620 6542 0 0 9 0 69
> > > 0 0 426
> > > 7555 905 42 57 0
> > > 6 0 0 43620 5582 0 0 6 0 61
> > > 0 0 431
> > > 6988 869 44 56 0
> > >
> > > Here is a portion of the glance capture:
> > > B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
> > > Current Avg High
> > > ------------------------------------------------------------------
> > > ----------------------------------------------------------
> > >
> > > Cpu Util S SARU
> > > U |100%
> > > 100% 100%
> > > Disk Util F F
> > > | 55% 47% 67%
> > > Mem Util S SU
> > > UB B | 92% 91% 92%1
> > > Swap Util U UR R
> > > | 56% 56% 56%
> > > ------------------------------------------------------------------
> > > ----------------------------------------------------------
> > >
> > > PROCESS LIST
> > > Users= 7
> > > User CPU Util Cum Disk
> > > Thd
> > > Process Name PID PPID Pri Name ( 200% max) CPU IO
> > > Rate RSS Cnt
> > > ------------------------------------------------------------------
> > > --------------
> > > arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
> > > 2.3 43.7mb 25
> > > identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
> > > 0.7 388kb 1
> > > oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
> > > 48.7/17.1 10.7mb 1
> > > oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
> > > 34.1/10.2 10.7mb 1
> > > oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
> > > 6.3 10.7mb 1
> > > java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
> > > 2.4 17.3mb 10
> > > oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
> > > 3.7 10.8mb 1
> > > aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
> > > 0.0 7.0mb 1
> > > java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
> > > 1.5 12.6mb 9
> > > oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
> > > 8.4 10.7mb 1
> > > oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
> > > 0.2 11.0mb 1
> > > oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
> > > 0.0 10.8mb 1
> > >
> > >
> > > These are the errors that pop into the arerror.log before the
> > > system crashes:
> > > Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
> > > SQL database
> > > (ARERR 551)
> > > Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
> > > Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
> > > to the database
> > > (ARERR 552)
> > > Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef
> >extension
> > > doesn't exist
> > > (Oracle -1041)
> > > Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
> > > fatal error
> > > encountered (ARNOTE 21)
> > > Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
> > > storage. (ARERR
> > > 8747)
> > > Tue Jun 19 07:59:07 2001 ThreadCleanup()
> > > Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
> > > to the database
> > > (ARERR 552)
> > > Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef
> >extension
> > > doesn't exist
> > > (Oracle -1041)
> > > Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
> > > fatal error
> > > encountered (ARNOTE 21)
> > > Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
> > > storage. (ARERR
> > > 8747)
> > > Tue Jun 19 07:59:15 2001 ThreadCleanup()
> > > Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
> > > to the database
> > > (ARERR 552)
> > > Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef
> >extension
> > > doesn't exist
> > > (Oracle -1041)
> > > Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
> > > fatal error
> > > encountered (ARNOTE 21)
> > > Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
> > > storage. (ARERR
> > > 8747)
> > > Tue Jun 19 07:59:53 2001 ThreadCleanup()
> > >
> > > We apparently have the latest hp patch available. During the
> > > cutover we re-did
> > > all of our indexes and switched our macros to push fields. We've
> > > also reset our
> > > kernal parameters to the following:
> > > maxdsiz from 67108864 to 200000000
> > > maxssiz from 8388608 to 16000000
> > > maxtsiz from 67108864 to 200000000
> > > maxswapchunks from 1262 to 4096
> > > maxfiles from 60 to 2048
> > > nkthread 2048 to 2850
> > >
> > > We are still having performace issues, the load average is high
> > > and the cpu
> > > usage and memory usage are also very high. Does anyone have any
> > > suggestions or
> > > comments.
> > >
> > > ... Tracy
> > >
> > > ______
> > > _______________________
> > >
> > >
> > > You may also control your subscription options, including
> > > UNSUBSCRIBE, at www.ARSLIST.org
> > >
> > >
> >
> >_____
________________________
> >
> >
> >You may also control your subscription options, including UNSUBSCRIBE, at
> >www.ARSLIST.org
>
> _____________
> Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.
>
> ______
> _______________________
>
>
> You may also control your subscription options, including
> UNSUBSCRIBE, at www.ARSLIST.org
>
>

_____________________________

Jarl:

Turn on logging (API and SQL) look for the server failing and see if you
get a -SE FAIL just before your failure. This should occur upon an
attempt to update a Diary type field. This is caused by the limit of
Oracle to 1MB (2^20) by default. You can raise this limit to up to 2GB
(unless you have a very large database pick a reasonable limit). The
parameter is listing in the Server Admin guide.

BTW, what error are you experiencing in the log file when the server
crashes?

James Mckenzie

Jarl Groneng wrote:
>
> **
>
> Hi,
>
> Very similar to the problem we has. Every 8-9 day, the cpu is crying for
> air. This happend around 06:20 in the morning(every time). No heavy jobs
> running this time. Opend a supportticket may 14, get two reply may 15., have
> not heard anything from support after this date. Here is the last mail I got
> from them:
>
> "Am I right to say that the problem has been resolved after server restart?
> If so, do you want to close this ticket?"
>
> ..close the ticket? ha, funny joke. I want my problem solved.
>
> Got this errors on the client:
> ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
> occurred
>
> ARERR [316] Failure with a temporary file : No such file or directory
>
> ARS 4.5.2 patch 1007
> Solaris 8
> Oracle 8.1.7 (on another server)
>
> Restart ars, and get this in the arerror logfile:
>
> 0 : AR System server terminated normally (ARERR 32)
> ARMonitor
> 0 : AR System server terminated when a signal/exception was received by the
> server (ARNOTE 20)
> 15
> 0 : Unable to access thread local storage. (ARERR 8747)
> ErrorHandler
> 0 : Unable to access thread local storage. (ARERR 8747)
> RollbackTransaction()
> 0 : Unable to access thread local storage. (ARERR 8747)
> WriteWorkflowTraceLog()
> 0 : Unable to access thread local storage. (ARERR 8747)
> BeginNetRespStatistics()
> 0 : Unable to access thread local storage. (ARERR 8747)
> ARPostSwitch()
> 0 : AR System server terminated -- fatal error encountered (ARNOTE 21)
> 0 : Unable to access thread local storage. (ARERR 8747)
> ThreadCleanup()
> 0 : Unable to access thread local storage. (ARERR 8747)
> ThreadCleanup()
> 0 : AR System server terminated normally (ARERR 32)
> ARMonitor
> 0 : AR System server terminated when a signal/exception was received by the
> server (ARNOTE 20)
> 15
> 0 : AR System server terminated normally (ARERR 32)
> ARMonitor
> 0 : AR System server terminated normally (ARERR 32)
> ARMonitor
> 0 : AR System server terminated when a signal/exception was received by the
> server (ARNOTE 20)
> 15
> 0 : Unable to access thread local storage. (ARERR 8747)
> WriteWorkflowTraceLog()
> 0 : Unable to access thread local storage. (ARERR 8747)
> WriteWorkflowTraceLog()
> 0 : AR System server terminated normally (ARERR 32)
> ARMonitor
> 0 : AR System server terminated when a signal/exception was received by the
> server (ARNOTE 20)
> 15
> 0 : Unable to access thread local storage. (ARERR 8747)
> WriteWorkflowTraceLog()
>
> --
> Jarl Groneng
> jarl@nextra.com
>
> > -----Original Message-----
> > From: Action Request System discussion list(ARSList)
> > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
> > Sent: 20. juni 2001 01:27
> > To: ARSLIST@LISTSERV.VISTAIT.COM
> > Subject: Remedy/Siebel
> >
> >
> > **
> >
> > I'm a new subscriber to arslist and I'm looking for help with
> > performance issues
> > in 4.5.2.
> >
> > We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
> > two weeks ago
> > to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
> > our box we
> > also have an interface with Siebel using MQ Series and Java. Our
> > system has
> > crashed three times in the past 48hrs, and we've had to do a
> > power cycle on the
> > box. Our CPU load average is 99% and our memory usage is 98%.
> > We have 1gig of
> > memory on the box and two cpu's.
> >
> > Here is a vmstat capture of what's going on on our box:
> > $ vmstat -S 2 5
> > procs memory page
> > faults cpu
> > r b w avm free si so pi po fr
> > de sr in
> > sy cs us sy id
> > 7 0 0 41615 9211 1536 1536 25 4 122
> > 0 39 686
> > 9936 1339 52 30 18
> > 7 0 0 41615 9014 0 0 21 0 61
> > 0 0 385
> > 7854 948 28 72 0
> > 6 0 0 43620 7880 0 0 14 0 64
> > 0 0 471
> > 8778 933 43 55 2
> > 6 0 0 43620 6542 0 0 9 0 69
> > 0 0 426
> > 7555 905 42 57 0
> > 6 0 0 43620 5582 0 0 6 0 61
> > 0 0 431
> > 6988 869 44 56 0
> >
> > Here is a portion of the glance capture:
> > B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
> > Current Avg High
> > ------------------------------------------------------------------
> > ----------------------------------------------------------
> >
> > Cpu Util S SARU
> > U |100%
> > 100% 100%
> > Disk Util F F
> > | 55% 47% 67%
> > Mem Util S SU
> > UB B | 92% 91% 92%1
> > Swap Util U UR R
> > | 56% 56% 56%
> > ------------------------------------------------------------------
> > ----------------------------------------------------------
> >
> > PROCESS LIST
> > Users= 7
> > User CPU Util Cum Disk
> > Thd
> > Process Name PID PPID Pri Name ( 200% max) CPU IO
> > Rate RSS Cnt
> > ------------------------------------------------------------------
> > --------------
> > arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
> > 2.3 43.7mb 25
> > identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
> > 0.7 388kb 1
> > oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
> > 48.7/17.1 10.7mb 1
> > oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
> > 34.1/10.2 10.7mb 1
> > oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
> > 6.3 10.7mb 1
> > java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
> > 2.4 17.3mb 10
> > oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
> > 3.7 10.8mb 1
> > aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
> > 0.0 7.0mb 1
> > java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
> > 1.5 12.6mb 9
> > oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
> > 8.4 10.7mb 1
> > oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
> > 0.2 11.0mb 1
> > oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
> > 0.0 10.8mb 1
> >
> >
> > These are the errors that pop into the arerror.log before the
> > system crashes:
> > Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
> > SQL database
> > (ARERR 551)
> > Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
> > Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
> > to the database
> > (ARERR 552)
> > Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef extension
> > doesn't exist
> > (Oracle -1041)
> > Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
> > fatal error
> > encountered (ARNOTE 21)
> > Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
> > storage. (ARERR
> > 8747)
> > Tue Jun 19 07:59:07 2001 ThreadCleanup()
> > Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
> > to the database
> > (ARERR 552)
> > Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef extension
> > doesn't exist
> > (Oracle -1041)
> > Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
> > fatal error
> > encountered (ARNOTE 21)
> > Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
> > storage. (ARERR
> > 8747)
> > Tue Jun 19 07:59:15 2001 ThreadCleanup()
> > Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
> > to the database
> > (ARERR 552)
> > Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef extension
> > doesn't exist
> > (Oracle -1041)
> > Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
> > fatal error
> > encountered (ARNOTE 21)
> > Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
> > storage. (ARERR
> > 8747)
> > Tue Jun 19 07:59:53 2001 ThreadCleanup()
> >
> > We apparently have the latest hp patch available. During the
> > cutover we re-did
> > all of our indexes and switched our macros to push fields. We've
> > also reset our
> > kernal parameters to the following:
> > maxdsiz from 67108864 to 200000000
> > maxssiz from 8388608 to 16000000
> > maxtsiz from 67108864 to 200000000
> > maxswapchunks from 1262 to 4096
> > maxfiles from 60 to 2048
> > nkthread 2048 to 2850
> >
> > We are still having performace issues, the load average is high
> > and the cpu
> > usage and memory usage are also very high. Does anyone have any
> > suggestions or
> > comments.
> >
> > ... Tracy
> >
> > ______
> > _______________________
> >
> >
> > You may also control your subscription options, including
> > UNSUBSCRIBE, at www.ARSLIST.org
> >
> >
>
> _____________________________
>
>

_____________________________

Jarl:

You have a problem with the way that Remedy is talking to Oracle. I
cannot remember exactly what we did to fix the problem, but I think it
took patching Oracle to 8.1.6.2 to get it fixed as well as applying a
fix from Remedy. Look at the KB on supportweb for the hostdef error
while running on Oracle.

James Mckenzie

Jarl Groneng wrote:
>
> **
>
> Memory is not a problem, it hardly using swap.
>
> Memory: 1024M real, 756M free, 149M swap in use, 1661M swap free
>
> Disk space is not a problem either.
>
> Filesystem kbytes used avail capacity Mounted on
> /dev/dsk/c0t0d0s0 3588558 722568 2830105 21% /
> swap 1699464 16 1699448 1% /var/run
> swap 1702240 2792 1699448 1% /tmp
> /dev/dsk/c0t0d0s5 4128246 1722165 2364799 43% /local
> nfs1.nextel.no:/users
> 2097152 1119256 977896 54% /home/nfs/users
>
> --
> Jarl Groneng
> jarl@nextra.com
>
> > -----Original Message-----
> > From: Action Request System discussion list(ARSList)
> > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Pat Ferger
> > Sent: 20. juni 2001 19:07
> > To: ARSLIST@LISTSERV.VISTAIT.COM
> > Subject: Re: Remedy/Siebel
> >
> >
> > **
> >
> > Jarl,
> >
> > It sounds like you're running out of memory, or disk space in the /tmp
> > directory. The "Failure with a temporary file" strongly suggests that the
> > server was unable to create a temp file, so you should
> > double-check the size
> > remaining on all your partitions (/var, /tmp, etc) to make sure you have
> > enough space.
> >
> > The other problems are memory problems, so maybe you need to bump up your
> > process memory limits, also.
> >
> >
> > >From: Jarl Groneng
> > >Reply-To: "Action Request System discussion list(ARSList)"
> > >
> > >To: ARSLIST@LISTSERV.VISTAIT.COM
> > >Subject: Re: Remedy/Siebel
> > >Date: Wed, 20 Jun 2001 15:51:58 +0200
> > >
> > >**
> > >
> > >Hi,
> > >
> > >Very similar to the problem we has. Every 8-9 day, the cpu is crying for
> > >air. This happend around 06:20 in the morning(every time). No heavy jobs
> > >running this time. Opend a supportticket may 14, get two reply may 15.,
> > >have
> > >not heard anything from support after this date. Here is the last mail I
> > >got
> > >from them:
> > >
> > >"Am I right to say that the problem has been resolved after
> > server restart?
> > >If so, do you want to close this ticket?"
> > >
> > >..close the ticket? ha, funny joke. I want my problem solved.
> > >
> > >
> > >Got this errors on the client:
> > >ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
> > >occurred
> > >
> > >ARERR [316] Failure with a temporary file : No such file or directory
> > >
> > >
> > >ARS 4.5.2 patch 1007
> > >Solaris 8
> > >Oracle 8.1.7 (on another server)
> > >
> > >Restart ars, and get this in the arerror logfile:
> > >
> > >0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > >0 : AR System server terminated when a signal/exception was
> > received by the
> > >server (ARNOTE 20)
> > > 15
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > ErrorHandler
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > RollbackTransaction()
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > BeginNetRespStatistics()
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > ARPostSwitch()
> > >0 : AR System server terminated -- fatal error encountered (ARNOTE 21)
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > ThreadCleanup()
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > ThreadCleanup()
> > >0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > >0 : AR System server terminated when a signal/exception was
> > received by the
> > >server (ARNOTE 20)
> > > 15
> > >0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > >0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > >0 : AR System server terminated when a signal/exception was
> > received by the
> > >server (ARNOTE 20)
> > > 15
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > >0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > >0 : AR System server terminated when a signal/exception was
> > received by the
> > >server (ARNOTE 20)
> > > 15
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > >
> > >--
> > >Jarl Groneng
> > >jarl@nextra.com
> > >
> > >
> > > > -----Original Message-----
> > > > From: Action Request System discussion list(ARSList)
> > > > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
> > > > Sent: 20. juni 2001 01:27
> > > > To: ARSLIST@LISTSERV.VISTAIT.COM
> > > > Subject: Remedy/Siebel
> > > >
> > > >
> > > > **
> > > >
> > > > I'm a new subscriber to arslist and I'm looking for help with
> > > > performance issues
> > > > in 4.5.2.
> > > >
> > > > We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
> > > > two weeks ago
> > > > to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
> > > > our box we
> > > > also have an interface with Siebel using MQ Series and Java. Our
> > > > system has
> > > > crashed three times in the past 48hrs, and we've had to do a
> > > > power cycle on the
> > > > box. Our CPU load average is 99% and our memory usage is 98%.
> > > > We have 1gig of
> > > > memory on the box and two cpu's.
> > > >
> > > > Here is a vmstat capture of what's going on on our box:
> > > > $ vmstat -S 2 5
> > > > procs memory page
> > > > faults cpu
> > > > r b w avm free si so pi po fr
> > > > de sr in
> > > > sy cs us sy id
> > > > 7 0 0 41615 9211 1536 1536 25 4 122
> > > > 0 39 686
> > > > 9936 1339 52 30 18
> > > > 7 0 0 41615 9014 0 0 21 0 61
> > > > 0 0 385
> > > > 7854 948 28 72 0
> > > > 6 0 0 43620 7880 0 0 14 0 64
> > > > 0 0 471
> > > > 8778 933 43 55 2
> > > > 6 0 0 43620 6542 0 0 9 0 69
> > > > 0 0 426
> > > > 7555 905 42 57 0
> > > > 6 0 0 43620 5582 0 0 6 0 61
> > > > 0 0 431
> > > > 6988 869 44 56 0
> > > >
> > > > Here is a portion of the glance capture:
> > > > B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
> > > > Current Avg High
> > > > ------------------------------------------------------------------
> > > > ----------------------------------------------------------
> > > >
> > > > Cpu Util S SARU
> > > > U |100%
> > > > 100% 100%
> > > > Disk Util F F
> > > > | 55% 47% 67%
> > > > Mem Util S SU
> > > > UB B | 92% 91% 92%1
> > > > Swap Util U UR R
> > > > | 56% 56% 56%
> > > > ------------------------------------------------------------------
> > > > ----------------------------------------------------------
> > > >
> > > > PROCESS LIST
> > > > Users= 7
> > > > User CPU Util Cum Disk
> > > > Thd
> > > > Process Name PID PPID Pri Name ( 200% max) CPU IO
> > > > Rate RSS Cnt
> > > > ------------------------------------------------------------------
> > > > --------------
> > > > arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
> > > > 2.3 43.7mb 25
> > > > identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
> > > > 0.7 388kb 1
> > > > oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
> > > > 48.7/17.1 10.7mb 1
> > > > oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
> > > > 34.1/10.2 10.7mb 1
> > > > oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
> > > > 6.3 10.7mb 1
> > > > java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
> > > > 2.4 17.3mb 10
> > > > oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
> > > > 3.7 10.8mb 1
> > > > aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
> > > > 0.0 7.0mb 1
> > > > java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
> > > > 1.5 12.6mb 9
> > > > oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
> > > > 8.4 10.7mb 1
> > > > oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
> > > > 0.2 11.0mb 1
> > > > oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
> > > > 0.0 10.8mb 1
> > > >
> > > >
> > > > These are the errors that pop into the arerror.log before the
> > > > system crashes:
> > > > Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
> > > > SQL database
> > > > (ARERR 551)
> > > > Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
> > > > Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
> > > > to the database
> > > > (ARERR 552)
> > > > Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef
> > >extension
> > > > doesn't exist
> > > > (Oracle -1041)
> > > > Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
> > > > fatal error
> > > > encountered (ARNOTE 21)
> > > > Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
> > > > storage. (ARERR
> > > > 8747)
> > > > Tue Jun 19 07:59:07 2001 ThreadCleanup()
> > > > Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
> > > > to the database
> > > > (ARERR 552)
> > > > Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef
> > >extension
> > > > doesn't exist
> > > > (Oracle -1041)
> > > > Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
> > > > fatal error
> > > > encountered (ARNOTE 21)
> > > > Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
> > > > storage. (ARERR
> > > > 8747)
> > > > Tue Jun 19 07:59:15 2001 ThreadCleanup()
> > > > Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
> > > > to the database
> > > > (ARERR 552)
> > > > Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef
> > >extension
> > > > doesn't exist
> > > > (Oracle -1041)
> > > > Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
> > > > fatal error
> > > > encountered (ARNOTE 21)
> > > > Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
> > > > storage. (ARERR
> > > > 8747)
> > > > Tue Jun 19 07:59:53 2001 ThreadCleanup()
> > > >
> > > > We apparently have the latest hp patch available. During the
> > > > cutover we re-did
> > > > all of our indexes and switched our macros to push fields. We've
> > > > also reset our
> > > > kernal parameters to the following:
> > > > maxdsiz from 67108864 to 200000000
> > > > maxssiz from 8388608 to 16000000
> > > > maxtsiz from 67108864 to 200000000
> > > > maxswapchunks from 1262 to 4096
> > > > maxfiles from 60 to 2048
> > > > nkthread 2048 to 2850
> > > >
> > > > We are still having performace issues, the load average is high
> > > > and the cpu
> > > > usage and memory usage are also very high. Does anyone have any
> > > > suggestions or
> > > > comments.
> > > >
> > > > ... Tracy
> > > >
> > > > ______
> > > > _______________________
> > > >
> > > >
> > > > You may also control your subscription options, including
> > > > UNSUBSCRIBE, at www.ARSLIST.org
> > > >
> > > >
> > >
> > >_____
> ________________________
> > >
> > >
> > >You may also control your subscription options, including UNSUBSCRIBE, at
> > >www.ARSLIST.org
> >
> > _____________
> > Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.
> >
> > ______
> > _______________________
> >
> >
> > You may also control your subscription options, including
> > UNSUBSCRIBE, at www.ARSLIST.org
> >
> >
>
> _____________________________
>
>

_____________________________

James,

I dont have any problems regarding hostdef errors on Oracle, it was Tracy
Bruneau (check the bottom of the mail). I just reported that I have some
similar ARS error messages (not Oracle messages)

Anyway, we'r on 8.1.7

--
Jarl Groneng
jarl@nextra.com


> -----Original Message-----
> From: Action Request System discussion list(ARSList)
> [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of James Mckenzie
> Sent: 21. juni 2001 16:07
> To: ARSLIST@LISTSERV.VISTAIT.COM
> Subject: Re: Remedy/Siebel
>
>
> **
>
> Jarl:
>
> You have a problem with the way that Remedy is talking to Oracle. I
> cannot remember exactly what we did to fix the problem, but I think it
> took patching Oracle to 8.1.6.2 to get it fixed as well as applying a
> fix from Remedy. Look at the KB on supportweb for the hostdef error
> while running on Oracle.
>
> James Mckenzie
>
> Jarl Groneng wrote:
> >
> > **
> >
> > Memory is not a problem, it hardly using swap.
> >
> > Memory: 1024M real, 756M free, 149M swap in use, 1661M swap free
> >
> > Disk space is not a problem either.
> >
> > Filesystem kbytes used avail capacity Mounted on
> > /dev/dsk/c0t0d0s0 3588558 722568 2830105 21% /
> > swap 1699464 16 1699448 1% /var/run
> > swap 1702240 2792 1699448 1% /tmp
> > /dev/dsk/c0t0d0s5 4128246 1722165 2364799 43% /local
> > nfs1.nextel.no:/users
> > 2097152 1119256 977896 54% /home/nfs/users
> >
> > --
> > Jarl Groneng
> > jarl@nextra.com
> >
> > > -----Original Message-----
> > > From: Action Request System discussion list(ARSList)
> > > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Pat Ferger
> > > Sent: 20. juni 2001 19:07
> > > To: ARSLIST@LISTSERV.VISTAIT.COM
> > > Subject: Re: Remedy/Siebel
> > >
> > >
> > > **
> > >
> > > Jarl,
> > >
> > > It sounds like you're running out of memory, or disk space in the /tmp
> > > directory. The "Failure with a temporary file" strongly
> suggests that the
> > > server was unable to create a temp file, so you should
> > > double-check the size
> > > remaining on all your partitions (/var, /tmp, etc) to make
> sure you have
> > > enough space.
> > >
> > > The other problems are memory problems, so maybe you need to
> bump up your
> > > process memory limits, also.
> > >
> > >
> > > >From: Jarl Groneng
> > > >Reply-To: "Action Request System discussion list(ARSList)"
> > > >
> > > >To: ARSLIST@LISTSERV.VISTAIT.COM
> > > >Subject: Re: Remedy/Siebel
> > > >Date: Wed, 20 Jun 2001 15:51:58 +0200
> > > >
> > > >**
> > > >
> > > >Hi,
> > > >
> > > >Very similar to the problem we has. Every 8-9 day, the cpu
> is crying for
> > > >air. This happend around 06:20 in the morning(every time).
> No heavy jobs
> > > >running this time. Opend a supportticket may 14, get two
> reply may 15.,
> > > >have
> > > >not heard anything from support after this date. Here is the
> last mail I
> > > >got
> > > >from them:
> > > >
> > > >"Am I right to say that the problem has been resolved after
> > > server restart?
> > > >If so, do you want to close this ticket?"
> > > >
> > > >..close the ticket? ha, funny joke. I want my problem solved.
> > > >
> > > >
> > > >Got this errors on the client:
> > > >ARERR [91] RPC call failed : RPC: Unable to receive;
> Asynchronous event
> > > >occurred
> > > >
> > > >ARERR [316] Failure with a temporary file : No such file or directory
> > > >
> > > >
> > > >ARS 4.5.2 patch 1007
> > > >Solaris 8
> > > >Oracle 8.1.7 (on another server)
> > > >
> > > >Restart ars, and get this in the arerror logfile:
> > > >
> > > >0 : AR System server terminated normally (ARERR 32)
> > > > ARMonitor
> > > >0 : AR System server terminated when a signal/exception was
> > > received by the
> > > >server (ARNOTE 20)
> > > > 15
> > > >0 : Unable to access thread local storage. (ARERR 8747)
> > > > ErrorHandler
> > > >0 : Unable to access thread local storage. (ARERR 8747)
> > > > RollbackTransaction()
> > > >0 : Unable to access thread local storage. (ARERR 8747)
> > > > WriteWorkflowTraceLog()
> > > >0 : Unable to access thread local storage. (ARERR 8747)
> > > > BeginNetRespStatistics()
> > > >0 : Unable to access thread local storage. (ARERR 8747)
> > > > ARPostSwitch()
> > > >0 : AR System server terminated -- fatal error encountered
> (ARNOTE 21)
> > > >0 : Unable to access thread local storage. (ARERR 8747)
> > > > ThreadCleanup()
> > > >0 : Unable to access thread local storage. (ARERR 8747)
> > > > ThreadCleanup()
> > > >0 : AR System server terminated normally (ARERR 32)
> > > > ARMonitor
> > > >0 : AR System server terminated when a signal/exception was
> > > received by the
> > > >server (ARNOTE 20)
> > > > 15
> > > >0 : AR System server terminated normally (ARERR 32)
> > > > ARMonitor
> > > >0 : AR System server terminated normally (ARERR 32)
> > > > ARMonitor
> > > >0 : AR System server terminated when a signal/exception was
> > > received by the
> > > >server (ARNOTE 20)
> > > > 15
> > > >0 : Unable to access thread local storage. (ARERR 8747)
> > > > WriteWorkflowTraceLog()
> > > >0 : Unable to access thread local storage. (ARERR 8747)
> > > > WriteWorkflowTraceLog()
> > > >0 : AR System server terminated normally (ARERR 32)
> > > > ARMonitor
> > > >0 : AR System server terminated when a signal/exception was
> > > received by the
> > > >server (ARNOTE 20)
> > > > 15
> > > >0 : Unable to access thread local storage. (ARERR 8747)
> > > > WriteWorkflowTraceLog()
> > > >
> > > >--
> > > >Jarl Groneng
> > > >jarl@nextra.com
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Action Request System discussion list(ARSList)
> > > > > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
> > > > > Sent: 20. juni 2001 01:27
> > > > > To: ARSLIST@LISTSERV.VISTAIT.COM
> > > > > Subject: Remedy/Siebel
> > > > >
> > > > >
> > > > > **
> > > > >
> > > > > I'm a new subscriber to arslist and I'm looking for help with
> > > > > performance issues
> > > > > in 4.5.2.
> > > > >
> > > > > We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
> > > > > two weeks ago
> > > > > to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
> > > > > our box we
> > > > > also have an interface with Siebel using MQ Series and Java. Our
> > > > > system has
> > > > > crashed three times in the past 48hrs, and we've had to do a
> > > > > power cycle on the
> > > > > box. Our CPU load average is 99% and our memory usage is 98%.
> > > > > We have 1gig of
> > > > > memory on the box and two cpu's.
> > > > >
> > > > > Here is a vmstat capture of what's going on on our box:
> > > > > $ vmstat -S 2 5
> > > > > procs memory page
> > > > > faults cpu
> > > > > r b w avm free si so pi po fr
> > > > > de sr in
> > > > > sy cs us sy id
> > > > > 7 0 0 41615 9211 1536 1536 25 4 122
> > > > > 0 39 686
> > > > > 9936 1339 52 30 18
> > > > > 7 0 0 41615 9014 0 0 21 0 61
> > > > > 0 0 385
> > > > > 7854 948 28 72 0
> > > > > 6 0 0 43620 7880 0 0 14 0 64
> > > > > 0 0 471
> > > > > 8778 933 43 55 2
> > > > > 6 0 0 43620 6542 0 0 9 0 69
> > > > > 0 0 426
> > > > > 7555 905 42 57 0
> > > > > 6 0 0 43620 5582 0 0 6 0 61
> > > > > 0 0 431
> > > > > 6988 869 44 56 0
> > > > >
> > > > > Here is a portion of the glance capture:
> > > > > B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
> > > > > Current Avg High
> > > > > ------------------------------------------------------------------
> > > > > ----------------------------------------------------------
> > > > >
> > > > > Cpu Util S SARU
> > > > > U |100%
> > > > > 100% 100%
> > > > > Disk Util F F
> > > > > | 55% 47% 67%
> > > > > Mem Util S SU
> > > > > UB B | 92% 91% 92%1
> > > > > Swap Util U UR R
> > > > > | 56% 56% 56%
> > > > > ------------------------------------------------------------------
> > > > > ----------------------------------------------------------
> > > > >
> > > > >
> PROCESS LIST
> > > > > Users= 7
> > > > > User CPU Util Cum Disk
> > > > > Thd
> > > > > Process Name PID PPID Pri Name ( 200% max) CPU IO
> > > > > Rate RSS Cnt
> > > > > ------------------------------------------------------------------
> > > > > --------------
> > > > > arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
> > > > > 2.3 43.7mb 25
> > > > > identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
> > > > > 0.7 388kb 1
> > > > > oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
> > > > > 48.7/17.1 10.7mb 1
> > > > > oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
> > > > > 34.1/10.2 10.7mb 1
> > > > > oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
> > > > > 6.3 10.7mb 1
> > > > > java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
> > > > > 2.4 17.3mb 10
> > > > > oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
> > > > > 3.7 10.8mb 1
> > > > > aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
> > > > > 0.0 7.0mb 1
> > > > > java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
> > > > > 1.5 12.6mb 9
> > > > > oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
> > > > > 8.4 10.7mb 1
> > > > > oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
> > > > > 0.2 11.0mb 1
> > > > > oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
> > > > > 0.0 10.8mb 1
> > > > >
> > > > >
> > > > > These are the errors that pop into the arerror.log before the
> > > > > system crashes:
> > > > > Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
> > > > > SQL database
> > > > > (ARERR 551)
> > > > > Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
> > > > > Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
> > > > > to the database
> > > > > (ARERR 552)
> > > > > Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef
> > > >extension
> > > > > doesn't exist
> > > > > (Oracle -1041)
> > > > > Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
> > > > > fatal error
> > > > > encountered (ARNOTE 21)
> > > > > Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
> > > > > storage. (ARERR
> > > > > 8747)
> > > > > Tue Jun 19 07:59:07 2001 ThreadCleanup()
> > > > > Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
> > > > > to the database
> > > > > (ARERR 552)
> > > > > Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef
> > > >extension
> > > > > doesn't exist
> > > > > (Oracle -1041)
> > > > > Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
> > > > > fatal error
> > > > > encountered (ARNOTE 21)
> > > > > Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
> > > > > storage. (ARERR
> > > > > 8747)
> > > > > Tue Jun 19 07:59:15 2001 ThreadCleanup()
> > > > > Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
> > > > > to the database
> > > > > (ARERR 552)
> > > > > Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef
> > > >extension
> > > > > doesn't exist
> > > > > (Oracle -1041)
> > > > > Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
> > > > > fatal error
> > > > > encountered (ARNOTE 21)
> > > > > Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
> > > > > storage. (ARERR
> > > > > 8747)
> > > > > Tue Jun 19 07:59:53 2001 ThreadCleanup()
> > > > >
> > > > > We apparently have the latest hp patch available. During the
> > > > > cutover we re-did
> > > > > all of our indexes and switched our macros to push fields. We've
> > > > > also reset our
> > > > > kernal parameters to the following:
> > > > > maxdsiz from 67108864 to 200000000
> > > > > maxssiz from 8388608 to 16000000
> > > > > maxtsiz from 67108864 to 200000000
> > > > > maxswapchunks from 1262 to 4096
> > > > > maxfiles from 60 to 2048
> > > > > nkthread 2048 to 2850
> > > > >
> > > > > We are still having performace issues, the load average is high
> > > > > and the cpu
> > > > > usage and memory usage are also very high. Does anyone have any
> > > > > suggestions or
> > > > > comments.
> > > > >
> > > > > ... Tracy
> > > > >
> > > > > ______
> > > > > _______________________
> > > > >
> > > > >
> > > > > You may also control your subscription options, including
> > > > > UNSUBSCRIBE, at www.ARSLIST.org
> > > > >
> > > > >
> > > >
> > > >_____
> > ________________________
> > > >
> > > >
> > > >You may also control your subscription options, including
> UNSUBSCRIBE, at
> > > >www.ARSLIST.org
> > >
> > >
> _____________
> > > Get Your Private, Free E-mail from MSN Hotmail at
http://www.hotmail.com.
> >
> > ______
> > _______________________
> >
> >
> > You may also control your subscription options, including
> > UNSUBSCRIBE, at www.ARSLIST.org
> >
> >
>
>
________________
_____________
>

> You may also control your subscription options, including UNSUBSCRIBE, at
www.ARSLIST.org

________________
_____________


You may also control your subscription options, including UNSUBSCRIBE, at
www.ARSLIST.org

_____________________________

James,

I created a small script that turn on the logging every morning and turn
them off 3 hours later. But until now there has be no problems on the
server.

Hopefully I get an crach very soon ;-)

--
Jarl Groneng
jarl@nextra.com



> -----Original Message-----
> From: Action Request System discussion list(ARSList)
> [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of James Mckenzie
> Sent: 21. juni 2001 03:49
> To: ARSLIST@LISTSERV.VISTAIT.COM
> Subject: Re: Remedy/Siebel
>
>
> **
>
> Jarl:
>
> Turn on logging (API and SQL) look for the server failing and see if you
> get a -SE FAIL just before your failure. This should occur upon an
> attempt to update a Diary type field. This is caused by the limit of
> Oracle to 1MB (2^20) by default. You can raise this limit to up to 2GB
> (unless you have a very large database pick a reasonable limit). The
> parameter is listing in the Server Admin guide.
>
> BTW, what error are you experiencing in the log file when the server
> crashes?
>
> James Mckenzie
>
> Jarl Groneng wrote:
> >
> > **
> >
> > Hi,
> >
> > Very similar to the problem we has. Every 8-9 day, the cpu is crying for
> > air. This happend around 06:20 in the morning(every time). No heavy jobs
> > running this time. Opend a supportticket may 14, get two reply
> may 15., have
> > not heard anything from support after this date. Here is the
> last mail I got
> > from them:
> >
> > "Am I right to say that the problem has been resolved after
> server restart?
> > If so, do you want to close this ticket?"
> >
> > ..close the ticket? ha, funny joke. I want my problem solved.
> >
> > Got this errors on the client:
> > ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
> > occurred
> >
> > ARERR [316] Failure with a temporary file : No such file or directory
> >
> > ARS 4.5.2 patch 1007
> > Solaris 8
> > Oracle 8.1.7 (on another server)
> >
> > Restart ars, and get this in the arerror logfile:
> >
> > 0 : AR System server terminated normally (ARERR 32)
> > ARMonitor
> > 0 : AR System server terminated when a signal/exception was
> received by the
> > server (ARNOTE 20)
> > 15
> > 0 : Unable to access thread local storage. (ARERR 8747)
> > ErrorHandler
> > 0 : Unable to access thread local storage. (ARERR 8747)
> > RollbackTransaction()
> > 0 : Unable to access thread local storage. (ARERR 8747)
> > WriteWorkflowTraceLog()
> > 0 : Unable to access thread local storage. (ARERR 8747)
> > BeginNetRespStatistics()
> > 0 : Unable to access thread local storage. (ARERR 8747)
> > ARPostSwitch()
> > 0 : AR System server terminated -- fatal error encountered (ARNOTE 21)
> > 0 : Unable to access thread local storage. (ARERR 8747)
> > ThreadCleanup()
> > 0 : Unable to access thread local storage. (ARERR 8747)
> > ThreadCleanup()
> > 0 : AR System server terminated normally (ARERR 32)
> > ARMonitor
> > 0 : AR System server terminated when a signal/exception was
> received by the
> > server (ARNOTE 20)
> > 15
> > 0 : AR System server terminated normally (ARERR 32)
> > ARMonitor
> > 0 : AR System server terminated normally (ARERR 32)
> > ARMonitor
> > 0 : AR System server terminated when a signal/exception was
> received by the
> > server (ARNOTE 20)
> > 15
> > 0 : Unable to access thread local storage. (ARERR 8747)
> > WriteWorkflowTraceLog()
> > 0 : Unable to access thread local storage. (ARERR 8747)
> > WriteWorkflowTraceLog()
> > 0 : AR System server terminated normally (ARERR 32)
> > ARMonitor
> > 0 : AR System server terminated when a signal/exception was
> received by the
> > server (ARNOTE 20)
> > 15
> > 0 : Unable to access thread local storage. (ARERR 8747)
> > WriteWorkflowTraceLog()
> >
> > --
> > Jarl Groneng
> > jarl@nextra.com
> >
> > > -----Original Message-----
> > > From: Action Request System discussion list(ARSList)
> > > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
> > > Sent: 20. juni 2001 01:27
> > > To: ARSLIST@LISTSERV.VISTAIT.COM
> > > Subject: Remedy/Siebel
> > >
> > >
> > > **
> > >
> > > I'm a new subscriber to arslist and I'm looking for help with
> > > performance issues
> > > in 4.5.2.
> > >
> > > We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
> > > two weeks ago
> > > to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
> > > our box we
> > > also have an interface with Siebel using MQ Series and Java. Our
> > > system has
> > > crashed three times in the past 48hrs, and we've had to do a
> > > power cycle on the
> > > box. Our CPU load average is 99% and our memory usage is 98%.
> > > We have 1gig of
> > > memory on the box and two cpu's.
> > >
> > > Here is a vmstat capture of what's going on on our box:
> > > $ vmstat -S 2 5
> > > procs memory page
> > > faults cpu
> > > r b w avm free si so pi po fr
> > > de sr in
> > > sy cs us sy id
> > > 7 0 0 41615 9211 1536 1536 25 4 122
> > > 0 39 686
> > > 9936 1339 52 30 18
> > > 7 0 0 41615 9014 0 0 21 0 61
> > > 0 0 385
> > > 7854 948 28 72 0
> > > 6 0 0 43620 7880 0 0 14 0 64
> > > 0 0 471
> > > 8778 933 43 55 2
> > > 6 0 0 43620 6542 0 0 9 0 69
> > > 0 0 426
> > > 7555 905 42 57 0
> > > 6 0 0 43620 5582 0 0 6 0 61
> > > 0 0 431
> > > 6988 869 44 56 0
> > >
> > > Here is a portion of the glance capture:
> > > B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
> > > Current Avg High
> > > ------------------------------------------------------------------
> > > ----------------------------------------------------------
> > >
> > > Cpu Util S SARU
> > > U |100%
> > > 100% 100%
> > > Disk Util F F
> > > | 55% 47% 67%
> > > Mem Util S SU
> > > UB B | 92% 91% 92%1
> > > Swap Util U UR R
> > > | 56% 56% 56%
> > > ------------------------------------------------------------------
> > > ----------------------------------------------------------
> > >
> > > PROCESS LIST
> > > Users= 7
> > > User CPU Util Cum Disk
> > > Thd
> > > Process Name PID PPID Pri Name ( 200% max) CPU IO
> > > Rate RSS Cnt
> > > ------------------------------------------------------------------
> > > --------------
> > > arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
> > > 2.3 43.7mb 25
> > > identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
> > > 0.7 388kb 1
> > > oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
> > > 48.7/17.1 10.7mb 1
> > > oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
> > > 34.1/10.2 10.7mb 1
> > > oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
> > > 6.3 10.7mb 1
> > > java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
> > > 2.4 17.3mb 10
> > > oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
> > > 3.7 10.8mb 1
> > > aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
> > > 0.0 7.0mb 1
> > > java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
> > > 1.5 12.6mb 9
> > > oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
> > > 8.4 10.7mb 1
> > > oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
> > > 0.2 11.0mb 1
> > > oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
> > > 0.0 10.8mb 1
> > >
> > >
> > > These are the errors that pop into the arerror.log before the
> > > system crashes:
> > > Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
> > > SQL database
> > > (ARERR 551)
> > > Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
> > > Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
> > > to the database
> > > (ARERR 552)
> > > Tue Jun 19 07:59:07 2001 ORA-01041: internal error.
> hostdef extension
> > > doesn't exist
> > > (Oracle -1041)
> > > Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
> > > fatal error
> > > encountered (ARNOTE 21)
> > > Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
> > > storage. (ARERR
> > > 8747)
> > > Tue Jun 19 07:59:07 2001 ThreadCleanup()
> > > Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
> > > to the database
> > > (ARERR 552)
> > > Tue Jun 19 07:59:15 2001 ORA-01041: internal error.
> hostdef extension
> > > doesn't exist
> > > (Oracle -1041)
> > > Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
> > > fatal error
> > > encountered (ARNOTE 21)
> > > Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
> > > storage. (ARERR
> > > 8747)
> > > Tue Jun 19 07:59:15 2001 ThreadCleanup()
> > > Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
> > > to the database
> > > (ARERR 552)
> > > Tue Jun 19 07:59:53 2001 ORA-01041: internal error.
> hostdef extension
> > > doesn't exist
> > > (Oracle -1041)
> > > Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
> > > fatal error
> > > encountered (ARNOTE 21)
> > > Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
> > > storage. (ARERR
> > > 8747)
> > > Tue Jun 19 07:59:53 2001 ThreadCleanup()
> > >
> > > We apparently have the latest hp patch available. During the
> > > cutover we re-did
> > > all of our indexes and switched our macros to push fields. We've
> > > also reset our
> > > kernal parameters to the following:
> > > maxdsiz from 67108864 to 200000000
> > > maxssiz from 8388608 to 16000000
> > > maxtsiz from 67108864 to 200000000
> > > maxswapchunks from 1262 to 4096
> > > maxfiles from 60 to 2048
> > > nkthread 2048 to 2850
> > >
> > > We are still having performace issues, the load average is high
> > > and the cpu
> > > usage and memory usage are also very high. Does anyone have any
> > > suggestions or
> > > comments.
> > >
> > > ... Tracy
> > >
> > > ______
> > > _______________________
> > >
> > >
> > > You may also control your subscription options, including
> > > UNSUBSCRIBE, at www.ARSLIST.org
> > >
> > >
> >
> >
> ______
> _______________________
> >
>
> > You may also control your subscription options, including
> UNSUBSCRIBE, at www.ARSLIST.org
>
> ______
> _______________________
>
>
> You may also control your subscription options, including
> UNSUBSCRIBE, at www.ARSLIST.org
>
>

_____________________________

Jarl,

It's not the total system memory that could be the problem, it may be the
Per-Process-Memory-Limit feature that HP-UX has. HP-UX allows you to limit
how much memory a process can take. The default limit is 67 MB, which may
be too low for more sophisticated sites. 4.0.X didn't have this problem
because on UNIX they were multi-process, meaning there were many processes
with smaller memory footprints. But with 4.5, the AR Server has been
consolidated into one process with multiple threads. This may put you over
the limit in terms of process-memory size.

Talk to your HPUX sysadmin about bumping up the per-process-memory limit.


>From: Jarl Groneng
>Reply-To: "Action Request System discussion list(ARSList)"
>
>To: ARSLIST@LISTSERV.VISTAIT.COM
>Subject: Re: Remedy/Siebel
>Date: Thu, 21 Jun 2001 07:48:37 +0200
>
>**
>
>Memory is not a problem, it hardly using swap.
>
>Memory: 1024M real, 756M free, 149M swap in use, 1661M swap free
>
>
>Disk space is not a problem either.
>
>Filesystem kbytes used avail capacity Mounted on
>/dev/dsk/c0t0d0s0 3588558 722568 2830105 21% /
>swap 1699464 16 1699448 1% /var/run
>swap 1702240 2792 1699448 1% /tmp
>/dev/dsk/c0t0d0s5 4128246 1722165 2364799 43% /local
>nfs1.nextel.no:/users
> 2097152 1119256 977896 54% /home/nfs/users
>
>
>
>
>--
>Jarl Groneng
>jarl@nextra.com
>
>
>
> > -----Original Message-----
> > From: Action Request System discussion list(ARSList)
> > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Pat Ferger
> > Sent: 20. juni 2001 19:07
> > To: ARSLIST@LISTSERV.VISTAIT.COM
> > Subject: Re: Remedy/Siebel
> >
> >
> > **
> >
> > Jarl,
> >
> > It sounds like you're running out of memory, or disk space in the /tmp
> > directory. The "Failure with a temporary file" strongly suggests that
>the
> > server was unable to create a temp file, so you should
> > double-check the size
> > remaining on all your partitions (/var, /tmp, etc) to make sure you have
> > enough space.
> >
> > The other problems are memory problems, so maybe you need to bump up
>your
> > process memory limits, also.
> >
> >
> > >From: Jarl Groneng
> > >Reply-To: "Action Request System discussion list(ARSList)"
> > >
> > >To: ARSLIST@LISTSERV.VISTAIT.COM
> > >Subject: Re: Remedy/Siebel
> > >Date: Wed, 20 Jun 2001 15:51:58 +0200
> > >
> > >**
> > >
> > >Hi,
> > >
> > >Very similar to the problem we has. Every 8-9 day, the cpu is crying
>for
> > >air. This happend around 06:20 in the morning(every time). No heavy
>jobs
> > >running this time. Opend a supportticket may 14, get two reply may 15.,
> > >have
> > >not heard anything from support after this date. Here is the last mail
>I
> > >got
> > >from them:
> > >
> > >"Am I right to say that the problem has been resolved after
> > server restart?
> > >If so, do you want to close this ticket?"
> > >
> > >..close the ticket? ha, funny joke. I want my problem solved.
> > >
> > >
> > >Got this errors on the client:
> > >ARERR [91] RPC call failed : RPC: Unable to receive; Asynchronous event
> > >occurred
> > >
> > >ARERR [316] Failure with a temporary file : No such file or directory
> > >
> > >
> > >ARS 4.5.2 patch 1007
> > >Solaris 8
> > >Oracle 8.1.7 (on another server)
> > >
> > >Restart ars, and get this in the arerror logfile:
> > >
> > >0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > >0 : AR System server terminated when a signal/exception was
> > received by the
> > >server (ARNOTE 20)
> > > 15
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > ErrorHandler
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > RollbackTransaction()
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > BeginNetRespStatistics()
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > ARPostSwitch()
> > >0 : AR System server terminated -- fatal error encountered (ARNOTE 21)
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > ThreadCleanup()
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > ThreadCleanup()
> > >0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > >0 : AR System server terminated when a signal/exception was
> > received by the
> > >server (ARNOTE 20)
> > > 15
> > >0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > >0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > >0 : AR System server terminated when a signal/exception was
> > received by the
> > >server (ARNOTE 20)
> > > 15
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > >0 : AR System server terminated normally (ARERR 32)
> > > ARMonitor
> > >0 : AR System server terminated when a signal/exception was
> > received by the
> > >server (ARNOTE 20)
> > > 15
> > >0 : Unable to access thread local storage. (ARERR 8747)
> > > WriteWorkflowTraceLog()
> > >
> > >--
> > >Jarl Groneng
> > >jarl@nextra.com
> > >
> > >
> > > > -----Original Message-----
> > > > From: Action Request System discussion list(ARSList)
> > > > [mailto:ARSLIST@LISTSERV.VISTAIT.COM]On Behalf Of Tracy Bruneau
> > > > Sent: 20. juni 2001 01:27
> > > > To: ARSLIST@LISTSERV.VISTAIT.COM
> > > > Subject: Remedy/Siebel
> > > >
> > > >
> > > > **
> > > >
> > > > I'm a new subscriber to arslist and I'm looking for help with
> > > > performance issues
> > > > in 4.5.2.
> > > >
> > > > We were running ARS 3.2.1, oracle 7.3.4 and sun 2.6. We upgraded
> > > > two weeks ago
> > > > to ARS 4.5.2 patch 1043, oracle 8.1.6 and HP-Unix version 11. On
> > > > our box we
> > > > also have an interface with Siebel using MQ Series and Java. Our
> > > > system has
> > > > crashed three times in the past 48hrs, and we've had to do a
> > > > power cycle on the
> > > > box. Our CPU load average is 99% and our memory usage is 98%.
> > > > We have 1gig of
> > > > memory on the box and two cpu's.
> > > >
> > > > Here is a vmstat capture of what's going on on our box:
> > > > $ vmstat -S 2 5
> > > > procs memory page
> > > > faults cpu
> > > > r b w avm free si so pi po fr
> > > > de sr in
> > > > sy cs us sy id
> > > > 7 0 0 41615 9211 1536 1536 25 4 122
> > > > 0 39 686
> > > > 9936 1339 52 30 18
> > > > 7 0 0 41615 9014 0 0 21 0 61
> > > > 0 0 385
> > > > 7854 948 28 72 0
> > > > 6 0 0 43620 7880 0 0 14 0 64
> > > > 0 0 471
> > > > 8778 933 43 55 2
> > > > 6 0 0 43620 6542 0 0 9 0 69
> > > > 0 0 426
> > > > 7555 905 42 57 0
> > > > 6 0 0 43620 5582 0 0 6 0 61
> > > > 0 0 431
> > > > 6988 869 44 56 0
> > > >
> > > > Here is a portion of the glance capture:
> > > > B3692A GlancePlus C.02.30.00 08:46:40 rdyuxp01 9000/899
> > > > Current Avg High
> > > > ------------------------------------------------------------------
> > > > ----------------------------------------------------------
> > > >
> > > > Cpu Util S SARU
> > > > U |100%
> > > > 100% 100%
> > > > Disk Util F F
> > > > | 55% 47% 67%
> > > > Mem Util S SU
> > > > UB B | 92% 91% 92%1
> > > > Swap Util U UR R
> > > > | 56% 56% 56%
> > > > ------------------------------------------------------------------
> > > > ----------------------------------------------------------
> > > >
> > > > PROCESS LIST
> > > > Users= 7
> > > > User CPU Util Cum Disk
> > > > Thd
> > > > Process Name PID PPID Pri Name ( 200% max) CPU IO
> > > > Rate RSS Cnt
> > > > ------------------------------------------------------------------
> > > > --------------
> > > > arserverd 1747 1745 154 ARSystem 45.0/39.2 10.7 3.2/
> > > > 2.3 43.7mb 25
> > > > identd 9052 892 154 bin 18.3/18.3 1.0 0.7/
> > > > 0.7 388kb 1
> > > > oraclermdy 2114 1 214 oracle 10.9/ 4.8 1.3
> > > > 48.7/17.1 10.7mb 1
> > > > oraclermdy 2122 1 209 oracle 9.8/ 4.7 1.3
> > > > 34.1/10.2 10.7mb 1
> > > > oraclermdy 2082 1 154 oracle 8.9/ 3.0 0.8 29.8/
> > > > 6.3 10.7mb 1
> > > > java 4533 1 178 ARSystem 8.5/ 4.4 1.2 4.1/
> > > > 2.4 17.3mb 10
> > > > oraclermdy 2090 1 154 oracle 8.2/ 6.3 1.7 0.3/
> > > > 3.7 10.8mb 1
> > > > aruser 8935 8934 222 ARSystem 8.0/10.9 2.2 0.0/
> > > > 0.0 7.0mb 1
> > > > java 4534 1 168 ARSystem 5.3/ 6.6 1.8 2.1/
> > > > 1.5 12.6mb 9
> > > > oraclermdy 1983 1 154 oracle 4.4/ 5.2 1.4 0.7/
> > > > 8.4 10.7mb 1
> > > > oraclermdy 2050 1 154 oracle 3.1/ 2.0 0.6 0.0/
> > > > 0.2 11.0mb 1
> > > > oraclermdy 2009 1 154 oracle 2.2/ 1.5 0.4 0.0/
> > > > 0.0 10.8mb 1
> > > >
> > > >
> > > > These are the errors that pop into the arerror.log before the
> > > > system crashes:
> > > > Tue Jun 19 07:58:51 2001 390620 : Cannot initialize contact with
> > > > SQL database
> > > > (ARERR 551)
> > > > Tue Jun 19 07:58:51 2001 Thread 29 not handling connection
> > > > Tue Jun 19 07:59:07 2001 390635 : Failure during SQL operation
> > > > to the database
> > > > (ARERR 552)
> > > > Tue Jun 19 07:59:07 2001 ORA-01041: internal error. hostdef
> > >extension
> > > > doesn't exist
> > > > (Oracle -1041)
> > > > Tue Jun 19 07:59:07 2001 390635 : AR System server terminated --
> > > > fatal error
> > > > encountered (ARNOTE 21)
> > > > Tue Jun 19 07:59:07 2001 0 : Unable to access thread local
> > > > storage. (ARERR
> > > > 8747)
> > > > Tue Jun 19 07:59:07 2001 ThreadCleanup()
> > > > Tue Jun 19 07:59:15 2001 390635 : Failure during SQL operation
> > > > to the database
> > > > (ARERR 552)
> > > > Tue Jun 19 07:59:15 2001 ORA-01041: internal error. hostdef
> > >extension
> > > > doesn't exist
> > > > (Oracle -1041)
> > > > Tue Jun 19 07:59:15 2001 390635 : AR System server terminated --
> > > > fatal error
> > > > encountered (ARNOTE 21)
> > > > Tue Jun 19 07:59:15 2001 0 : Unable to access thread local
> > > > storage. (ARERR
> > > > 8747)
> > > > Tue Jun 19 07:59:15 2001 ThreadCleanup()
> > > > Tue Jun 19 07:59:53 2001 390635 : Failure during SQL operation
> > > > to the database
> > > > (ARERR 552)
> > > > Tue Jun 19 07:59:53 2001 ORA-01041: internal error. hostdef
> > >extension
> > > > doesn't exist
> > > > (Oracle -1041)
> > > > Tue Jun 19 07:59:53 2001 390635 : AR System server terminated --
> > > > fatal error
> > > > encountered (ARNOTE 21)
> > > > Tue Jun 19 07:59:53 2001 0 : Unable to access thread local
> > > > storage. (ARERR
> > > > 8747)
> > > > Tue Jun 19 07:59:53 2001 ThreadCleanup()
> > > >
> > > > We apparently have the latest hp patch available. During the
> > > > cutover we re-did
> > > > all of our indexes and switched our macros to push fields. We've
> > > > also reset our
> > > > kernal parameters to the following:
> > > > maxdsiz from 67108864 to 200000000
> > > > maxssiz from 8388608 to 16000000
> > > > maxtsiz from 67108864 to 200000000
> > > > maxswapchunks from 1262 to 4096
> > > > maxfiles from 60 to 2048
> > > > nkthread 2048 to 2850
> > > >
> > > > We are still having performace issues, the load average is high
> > > > and the cpu
> > > > usage and memory usage are also very high. Does anyone have any
> > > > suggestions or
> > > > comments.
> > > >
> > > > ... Tracy
> > > >
> > > > ______
> > > > _______________________
> > > >
> > > >
> > > > You may also control your subscription options, including
> > > > UNSUBSCRIBE, at www.ARSLIST.org
> > > >
> > > >
> > >
> > >_____
>________________________
> > >
> > >
> > >You may also control your subscription options, including UNSUBSCRIBE,
>at
> > >www.ARSLIST.org
> >
> >
>_____________
> > Get Your Private, Free E-mail from MSN Hotmail at
>http://www.hotmail.com.
> >
> > ______
> > _______________________
> >
> >
> > You may also control your subscription options, including
> > UNSUBSCRIBE, at www.ARSLIST.org
> >
> >
>
>_____________________________
>
>
>You may also control your subscription options, including UNSUBSCRIBE, at
>www.ARSLIST.org

_____________
Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.

_____________________________