Hi List,

Just wondering if anybody else has seen this occuring??

ARS 6.01 Patch 1380
MS W2K SP4
MS SQL 2000 SP3a (local)

Kind regards,

Dougal

-------------------------------------

Fri Jul 08 10:51:02 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:51:02 2005
Version: Jan 3 2005 23:03:43
Code: c0000005
Operation: read
Access Addr: 0000000C
Stack Begin:
Addr: 005E51B2
Addr: 62747261
Stack End:

Fri Jul 08 10:51:02 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:51:02 2005 0xc0000005
Fri Jul 08 10:51:02 2005 390620 : AR System server terminated -- fatal
error encountered (ARNOTE 21)
Fri Jul 08 10:51:03 2005 390635 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:51:03 2005
Version: Jan 3 2005 23:03:43
Code: c0000005
Operation: read
Access Addr: 0000000C
Stack Begin:
Addr: 004181FE
Addr: 22A068FF
Stack End:

Fri Jul 08 10:51:03 2005 390635 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:51:03 2005 0xc0000005
Fri Jul 08 10:51:03 2005 390635 : AR System server terminated -- fatal
error encountered (ARNOTE 21)
Fri Jul 08 10:52:48 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:52:48 2005
Version: Jan 3 2005 23:03:43
Code: c0000005
Operation: read
Access Addr: 0000000C
Stack Begin:
Addr: 005E51B2
Addr: 62747261
Stack End:

Fri Jul 08 10:52:48 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:52:48 2005 0xc0000005
Fri Jul 08 10:52:48 2005 390620 : AR System server terminated -- fatal
error encountered (ARNOTE 21)
Fri Jul 08 10:53:39 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:53:39 2005
Version: Jan 3 2005 23:03:43
Code: c0000005
Operation: read
Access Addr: 0000000C
Stack Begin:
Addr: 005E51B2
Addr: 62747261
Stack End:

Fri Jul 08 10:53:39 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:53:39 2005 0xc0000005
Fri Jul 08 10:53:39 2005 390620 : AR System server terminated -- fatal
error encountered (ARNOTE 21)


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

I am getting these as well. Suddenly started about 3 weeks ago and often
happen 2 times a day.


Our environment is:

ARS 5.1.2 patch 1346 on W2K sp4

Oracle 9i DB on Unix

-----Original Message-----
From: Action Request System discussion list(ARSList)
[mailto:arslist@ARSLIST.ORG] On Behalf Of Ballantyne, Dougal [OS-IE]
Sent: 08 July 2005 13:09
To: arslist@ARSLIST.ORG
Subject: Arserver.exe crashing

Hi List,

Just wondering if anybody else has seen this occuring??

ARS 6.01 Patch 1380
MS W2K SP4
MS SQL 2000 SP3a (local)

Kind regards,

Dougal

-------------------------------------

Fri Jul 08 10:51:02 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:51:02 2005
Version: Jan 3 2005 23:03:43
Code: c0000005
Operation: read
Access Addr: 0000000C
Stack Begin:
Addr: 005E51B2
Addr: 62747261
Stack End:

Fri Jul 08 10:51:02 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:51:02 2005 0xc0000005
Fri Jul 08 10:51:02 2005 390620 : AR System server terminated -- fatal
error encountered (ARNOTE 21)
Fri Jul 08 10:51:03 2005 390635 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:51:03 2005
Version: Jan 3 2005 23:03:43
Code: c0000005
Operation: read
Access Addr: 0000000C
Stack Begin:
Addr: 004181FE
Addr: 22A068FF
Stack End:

Fri Jul 08 10:51:03 2005 390635 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:51:03 2005 0xc0000005
Fri Jul 08 10:51:03 2005 390635 : AR System server terminated -- fatal
error encountered (ARNOTE 21)
Fri Jul 08 10:52:48 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:52:48 2005
Version: Jan 3 2005 23:03:43
Code: c0000005
Operation: read
Access Addr: 0000000C
Stack Begin:
Addr: 005E51B2
Addr: 62747261
Stack End:

Fri Jul 08 10:52:48 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:52:48 2005 0xc0000005
Fri Jul 08 10:52:48 2005 390620 : AR System server terminated -- fatal
error encountered (ARNOTE 21)
Fri Jul 08 10:53:39 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:53:39 2005
Version: Jan 3 2005 23:03:43
Code: c0000005
Operation: read
Access Addr: 0000000C
Stack Begin:
Addr: 005E51B2
Addr: 62747261
Stack End:

Fri Jul 08 10:53:39 2005 390620 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Fri Jul 08 10:53:39 2005 0xc0000005
Fri Jul 08 10:53:39 2005 390620 : AR System server terminated -- fatal
error encountered (ARNOTE 21)



UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

Does this coincide with an escalation running on your server that
generates a large number of errors? Also, do you have any escalations
running against view forms?

Axton


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

I have a 2 hourly escalation that queries AD (vendor form) and returns about
3000 rows. This populates a staging form. Filters then fire that update both
People and User.

I also have two separate hourly escalations that query a (view forms)remote
Oracle table. This also populates a staging form. Filters fire that then
update the SHR:Company form


-----Original Message-----
From: Action Request System discussion list(ARSList)
[mailto:arslist@ARSLIST.ORG] On Behalf Of Axton
Sent: 08 July 2005 13:59
To: arslist@ARSLIST.ORG
Subject: Re: Arserver.exe crashing

Does this coincide with an escalation running on your server that
generates a large number of errors? Also, do you have any escalations
running against view forms?

Axton



UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

Do you see errors generated by these escalations in the arerror.log
file preceding the crash?

Axton


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

I have no escalation errors or escalations against view forms. Also I have
checked the arerror.log and it has happen a couple of times before. Only
this morning it happened 4 times. Also the memory usage of arserver.exe
reached over 1GB.

I have since restarted the services and it has dropped back to 40MB. Also
got a script watching the mem usage.

Logged a ticket with Remedy but they wanted API,SQL,Filter,Escalation
logging combined and Thread logging enabled. Managed to write over 50MB in
2mins, so can't really leave it running...

-Dougal


-----Original Message-----
From: Action Request System discussion list(ARSList)
[mailto:arslist@ARSLIST.ORG] On Behalf Of Axton
Sent: 08 July 2005 13:59
To: arslist@ARSLIST.ORG
Subject: Re: Arserver.exe crashing

Does this coincide with an escalation running on your server that generates
a large number of errors? Also, do you have any escalations running against
view forms?

Axton



UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

The only reason I ask is because I had this problem with that version
of arserver where I used escalations on view forms. Do the timing of
the escalations coincide with the crashes to any degree?

Axton


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

Hi Dougal,

Search for cases with large attachments, we got a very similar problem
some time back and this was it. If any of the CLOBS (e.g. work log) is
large it may cause this to happen.

Thanks,
Firas

-----Original Message-----
From: Action Request System discussion list(ARSList)
[mailto:arslist@ARSLIST.ORG] On Behalf Of Ballantyne, Dougal [OS-IE]
Sent: Friday, July 08, 2005 10:14 AM
To: arslist@ARSLIST.ORG
Subject: Re: Arserver.exe crashing

I have no escalation errors or escalations against view forms. Also I
have checked the arerror.log and it has happen a couple of times before.
Only this morning it happened 4 times. Also the memory usage of
arserver.exe reached over 1GB.

I have since restarted the services and it has dropped back to 40MB.
Also got a script watching the mem usage.

Logged a ticket with Remedy but they wanted API,SQL,Filter,Escalation
logging combined and Thread logging enabled. Managed to write over 50MB
in 2mins, so can't really leave it running...

-Dougal


-----Original Message-----
From: Action Request System discussion list(ARSList)
[mailto:arslist@ARSLIST.ORG] On Behalf Of Axton
Sent: 08 July 2005 13:59
To: arslist@ARSLIST.ORG
Subject: Re: Arserver.exe crashing

Does this coincide with an escalation running on your server that
generates a large number of errors? Also, do you have any escalations
running against view forms?

Axton




UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)



UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

The memory used by the arserver.exe process is of concern. Once arserver.exe
starts to reach 900KB things start to go wrong. Using AR Server 5.0.1 I have
seen the problem with the error described in your first post. The first time
the error occurred all escalation processes shutdown when memory utilization
exceeded 1GB. The second time the error occurred escalations continued to
process, but I was unable to apply any changes through the Admin Tool when
memory utilization again exceeded 1GB.

2 changes were made in conjunction. 1 was applying the latest patch for
5.0.1. 2 was reducing the number of list and fast threads available to the
server. Since you are already on the latest patch, the 2nd option may be
your best bet. On your next restart enable thread logging to determine how
threads are being used, then adjust your minimum and maximum accordingly. I
would suggest reading the start of the log to determine if there is a large
number of threads opened on start (or within 15 minutes of start). The
minimum threads on startup should be derived from this number to lower the
additional overhead of opening new threads.

Making the 2 changes reduced the amount of memory used by AR System to 200MB
on average. This number will probably increase with the upgrade to 6.3 (just
because 6.3 uses more memory), but should not increase too much as more
memory issues are being resolved with a code cleanup.

Since you are connecting to a remote Oracle database, I would suggest
checking for patches on the Oracle client. Especially since 9i seems to have
so many memory leak issues.

Are your View Forms pointing to Views created from a Linked Server? If so,
you might want to consider having a DTS package move the data from the
Linked Server to a local table. When Remedy queries a View created from a
Linked Server, Remedy reads a View of the View which returns all records (at
least with a Linked Server that's not SQL Server). For every query, Remedy
will read every record in the table. For escalations this means a Remote
View with 2000 records will read all 2000 records, then read the details of
the 1st record, read all 2000 records, then read the details of the 2nd
record, and so on. By the time its all done Remedy has read 4,002,000
records.

Finally I would suggest monitoring Virtual Page Size. A lot of times when
looking at Memory the Page Size is ignored and only Physical Memory is
viewed. Most applications will reserve more memory in Virtual than in
Physical, even with Remedy. Often the Virtual Page Size used by an
application is 110% of the Physical. So if you're running 1GB Physical, then
1.1GB Virtual, a Page File with 3GB available will be quickly dissolved by
the remaining applications.

These are just starting points for investigating the memory problems. Any
positive changes to the system should be noted, as well as negative changes,
to determine the next course of action. I would suggest, and it is only a
suggestion, that once a method has been discovered for reducing memory
utilization that works for you, do not stop at just bringing down the memory
utilzation to a level where the errors won't occur (like around 800K), but
continue to find ways to bring memory utilization down even further, until
you reach a comfortable level.

- Daniel

-----Original Message-----
From: Ballantyne, Dougal [OS-IE] [mailto:DOUGAL.BALLANTYNE@SAIC.COM]
Sent: Friday, July 08, 2005 10:14 AM
To: arslist@ARSLIST.ORG
Subject: Re: Arserver.exe crashing


I have no escalation errors or escalations against view forms. Also I have
checked the arerror.log and it has happen a couple of times before. Only
this morning it happened 4 times. Also the memory usage of arserver.exe
reached over 1GB.

I have since restarted the services and it has dropped back to 40MB. Also
got a script watching the mem usage.

Logged a ticket with Remedy but they wanted API,SQL,Filter,Escalation
logging combined and Thread logging enabled. Managed to write over 50MB in
2mins, so can't really leave it running...

-Dougal


-----Original Message-----
From: Action Request System discussion list(ARSList)
[mailto:arslist@ARSLIST.ORG] On Behalf Of Axton
Sent: 08 July 2005 13:59
To: arslist@ARSLIST.ORG
Subject: Re: Arserver.exe crashing

Does this coincide with an escalation running on your server that generates
a large number of errors? Also, do you have any escalations running against
view forms?

Axton



UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)



UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)



===================================================================
NOTICE - CONFIDENTIAL AND PRIVILEGED - This e-mail may contain
privileged and confidential information and is intended only for the
addressee named above. If you received this message in error,
please immediately notify the sender by return e-mail and delete the
original message; any distribution, copying or use of this e-mail
by you is strictly prohibited and may be unlawful.


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

We had this problem several times per day on 6.0.1 patch 1368 and patch
1380. Remedy found a bug that was exposed when users of a 5.x client
retrieved a SQL menu. For some reason it caused some bad memory references
and crashed arserver. They gave us a special debug patch with the fix, but
that fix is supposedly included in patch 1420. I haven't tested that one
fully though to confirm though.

Chad Hall

On Fri, 8 Jul 2005 05:09:28 -0700, Ballantyne, Dougal [OS-IE]
wrote:

>Hi List,
>
>Just wondering if anybody else has seen this occuring??
>
>ARS 6.01 Patch 1380
>MS W2K SP4
>MS SQL 2000 SP3a (local)
>
>Kind regards,
>
>Dougal
>
>-------------------------------------
>
>Fri Jul 08 10:51:02 2005 390620 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:51:02 2005
> Version: Jan 3 2005 23:03:43
> Code: c0000005
> Operation: read
> Access Addr: 0000000C
> Stack Begin:
> Addr: 005E51B2
> Addr: 62747261
> Stack End:
>
>Fri Jul 08 10:51:02 2005 390620 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:51:02 2005 0xc0000005
>Fri Jul 08 10:51:02 2005 390620 : AR System server terminated -- fatal
>error encountered (ARNOTE 21)
>Fri Jul 08 10:51:03 2005 390635 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:51:03 2005
> Version: Jan 3 2005 23:03:43
> Code: c0000005
> Operation: read
> Access Addr: 0000000C
> Stack Begin:
> Addr: 004181FE
> Addr: 22A068FF
> Stack End:
>
>Fri Jul 08 10:51:03 2005 390635 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:51:03 2005 0xc0000005
>Fri Jul 08 10:51:03 2005 390635 : AR System server terminated -- fatal
>error encountered (ARNOTE 21)
>Fri Jul 08 10:52:48 2005 390620 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:52:48 2005
> Version: Jan 3 2005 23:03:43
> Code: c0000005
> Operation: read
> Access Addr: 0000000C
> Stack Begin:
> Addr: 005E51B2
> Addr: 62747261
> Stack End:
>
>Fri Jul 08 10:52:48 2005 390620 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:52:48 2005 0xc0000005
>Fri Jul 08 10:52:48 2005 390620 : AR System server terminated -- fatal
>error encountered (ARNOTE 21)
>Fri Jul 08 10:53:39 2005 390620 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:53:39 2005
> Version: Jan 3 2005 23:03:43
> Code: c0000005
> Operation: read
> Access Addr: 0000000C
> Stack Begin:
> Addr: 005E51B2
> Addr: 62747261
> Stack End:
>
>Fri Jul 08 10:53:39 2005 390620 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:53:39 2005 0xc0000005
>Fri Jul 08 10:53:39 2005 390620 : AR System server terminated -- fatal
>error encountered (ARNOTE 21)
>
>

>UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
>(Support: mailto:support@arslist.org)


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

Chad,

Thank you for this information. We have quite a large amount of 5.x clients
and SQL menus. I have checked the 1420 release notes and can't find it
referreced. Do you have a SW defect ID from Remedy?

Kind regards,

Dougal


-----Original Message-----
From: chad.hall@ACXIOM.COM [mailto:chad.hall@ACXIOM.COM]
Sent: 08 July 2005 22:25
To: ARSLIST@LISTSERV.RBUGS.COM; Dougal Ballantyne
Subject: Re: Arserver.exe crashing

We had this problem several times per day on 6.0.1 patch 1368 and patch
1380. Remedy found a bug that was exposed when users of a 5.x client
retrieved a SQL menu. For some reason it caused some bad memory references
and crashed arserver. They gave us a special debug patch with the fix, but
that fix is supposedly included in patch 1420. I haven't tested that one
fully though to confirm though.

Chad Hall

On Fri, 8 Jul 2005 05:09:28 -0700, Ballantyne, Dougal [OS-IE]
wrote:

>Hi List,
>
>Just wondering if anybody else has seen this occuring??
>
>ARS 6.01 Patch 1380
>MS W2K SP4
>MS SQL 2000 SP3a (local)
>
>Kind regards,
>
>Dougal
>
>-------------------------------------
>
>Fri Jul 08 10:51:02 2005 390620 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20) Fri Jul 08
>10:51:02 2005
> Version: Jan 3 2005 23:03:43
> Code: c0000005
> Operation: read
> Access Addr: 0000000C
> Stack Begin:
> Addr: 005E51B2
> Addr: 62747261
> Stack End:
>
>Fri Jul 08 10:51:02 2005 390620 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:51:02 2005 0xc0000005
>Fri Jul 08 10:51:02 2005 390620 : AR System server terminated -- fatal
>error encountered (ARNOTE 21) Fri Jul 08 10:51:03 2005 390635 : AR
>System server terminated when a signal/exception was received by the
>server (ARNOTE 20) Fri Jul 08 10:51:03 2005
> Version: Jan 3 2005 23:03:43
> Code: c0000005
> Operation: read
> Access Addr: 0000000C
> Stack Begin:
> Addr: 004181FE
> Addr: 22A068FF
> Stack End:
>
>Fri Jul 08 10:51:03 2005 390635 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:51:03 2005 0xc0000005
>Fri Jul 08 10:51:03 2005 390635 : AR System server terminated -- fatal
>error encountered (ARNOTE 21) Fri Jul 08 10:52:48 2005 390620 : AR
>System server terminated when a signal/exception was received by the
>server (ARNOTE 20) Fri Jul 08 10:52:48 2005
> Version: Jan 3 2005 23:03:43
> Code: c0000005
> Operation: read
> Access Addr: 0000000C
> Stack Begin:
> Addr: 005E51B2
> Addr: 62747261
> Stack End:
>
>Fri Jul 08 10:52:48 2005 390620 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:52:48 2005 0xc0000005
>Fri Jul 08 10:52:48 2005 390620 : AR System server terminated -- fatal
>error encountered (ARNOTE 21) Fri Jul 08 10:53:39 2005 390620 : AR
>System server terminated when a signal/exception was received by the
>server (ARNOTE 20) Fri Jul 08 10:53:39 2005
> Version: Jan 3 2005 23:03:43
> Code: c0000005
> Operation: read
> Access Addr: 0000000C
> Stack Begin:
> Addr: 005E51B2
> Addr: 62747261
> Stack End:
>
>Fri Jul 08 10:53:39 2005 390620 : AR System server terminated when a
>signal/exception was received by the server (ARNOTE 20)
>Fri Jul 08 10:53:39 2005 0xc0000005
>Fri Jul 08 10:53:39 2005 390620 : AR System server terminated -- fatal
>error encountered (ARNOTE 21)
>
>
>

>UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
>(Support: mailto:support@arslist.org)


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)