For several days now, arserverd has bounced every five to thirty minutes,
each time with a signal 11 error. There has been no correlation between
this error and any action in workflow logs. Remedy suggests it is a memory
issue, but even after having pored through a tusc log, they are unable to
point to a cause. So hardware, database, and workflow issues all have
seemingly been ruled out, although something has to be the culprit. Two
days into a critical ticket logged with Remedy and massive performance
degradation, we are now poised for an impromptu upgrade to 5.1.2 from
5.1.1 patch 1211, recommended by Remedy so they can at least build a debug
arserverd if necessary.

Does anyone have any stories of having experienced a similar problem in
the past?

Regards,
David

5.1.1 patch 1211 (soon to be on 5.1.2)
HP-UX 11.00
Oracle 8.1.7


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

Hello,
I was getting the same error a while back. It took Remedy a while to
determine the cause; eventually it appeared that it was caused by
running a report against a large subset of data returned from a search.
This was related to 5.1.2, but it may be the same error on your version.
Here's what they sent me:

Server crash SIGNAL 10 and SIGNAL 11 from report operation

1. DROP/BUILD BUG OCCURS IN (IF PRE-RELEASE)

v5.1.2 patch 1267

2. STEPS TO REPRODUCE:

- Create report which has sort order on one of the fields
- Run query to form returning large dataset (~4000 records)
- Select large subset of returned data
- Select any report option
Preview report


3. ACTUAL RESULTS:

SIGNAL 10 / SIGNAL 11 crash of server

4. EXPECTED RESULTS:

No crash

5. ADDITIONAL INFO:

Crash occurs when both a subset of data is selected for report and
report has sort order on at least one field

Crash of server reported on customer test and production server

Crash occurs with User tool v4.5.1 and v5.1.2 patch 1238 / patch
1267

Crash reproduced in-house on test server topaz

unqualified query returned 4114 records
report includes Request ID and Status field
sort on Status field

selecting subset of ~1500 records and greater crashes server


NOTE:
Doubled ulimit stack of 16384 had no impact on problem

SERVER PLATFORM: Solaris 2.8
DATABASE: Oracle

-----Original Message-----
From: David Harris [mailto:remedydave@NC.RR.COM]
Sent: Tuesday, December 28, 2004 2:29 PM
To: arslist@ARSLIST.ORG
Subject: Signal 11 errors - running 5.1.1 patch 1211


For several days now, arserverd has bounced every five to thirty
minutes, each time with a signal 11 error. There has been no correlation
between this error and any action in workflow logs. Remedy suggests it
is a memory issue, but even after having pored through a tusc log, they
are unable to point to a cause. So hardware, database, and workflow
issues all have seemingly been ruled out, although something has to be
the culprit. Two days into a critical ticket logged with Remedy and
massive performance degradation, we are now poised for an impromptu
upgrade to 5.1.2 from 5.1.1 patch 1211, recommended by Remedy so they
can at least build a debug arserverd if necessary.

Does anyone have any stories of having experienced a similar problem in
the past?

Regards,
David

5.1.1 patch 1211 (soon to be on 5.1.2)
HP-UX 11.00
Oracle 8.1.7



UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

John, thanks for the response. I've tried running some large reports along
the lines you suggested, but it is not causing any errors.


On Tue, 28 Dec 2004 14:55:55 -0800, John Hanson
wrote:

>Hello,
>I was getting the same error a while back. It took Remedy a while to
>determine the cause; eventually it appeared that it was caused by
>running a report against a large subset of data returned from a search.
>This was related to 5.1.2, but it may be the same error on your version.
>Here's what they sent me:
>
>Server crash SIGNAL 10 and SIGNAL 11 from report operation
>
>1. DROP/BUILD BUG OCCURS IN (IF PRE-RELEASE)
>
> v5.1.2 patch 1267
>
>2. STEPS TO REPRODUCE:
>
> - Create report which has sort order on one of the fields
> - Run query to form returning large dataset (~4000 records)
> - Select large subset of returned data
> - Select any report option
> Preview report
>
>
>3. ACTUAL RESULTS:
>
> SIGNAL 10 / SIGNAL 11 crash of server
>
>4. EXPECTED RESULTS:
>
> No crash
>
>5. ADDITIONAL INFO:
>
> Crash occurs when both a subset of data is selected for report and
>report has sort order on at least one field
>
> Crash of server reported on customer test and production server
>
> Crash occurs with User tool v4.5.1 and v5.1.2 patch 1238 / patch
>1267
>
> Crash reproduced in-house on test server topaz
>
> unqualified query returned 4114 records
> report includes Request ID and Status field
> sort on Status field
>
> selecting subset of ~1500 records and greater crashes server
>
>
>NOTE:
>Doubled ulimit stack of 16384 had no impact on problem
>


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)

I don't know if this will be of any help (or if it is located in the
same place on an HP-UX box), but on a Solaris box
the /usr/include/sys/iso/signaliso.h describes what the UNIX signal
errors that show up in the arerror.log file mean.

#define SIGSEGV 11 /* segmentation violation */

Check out the following URL which leads to an HP web site that talks
about "How to avoid SIGSEGV - segmentation violation(signal no 11)."
http://forums1.itrc.hp.com/service/forums/questionanswer.do?admit=716493758+1104378108457+28353475&threadId=703638

The problem typically ends up being a problem with the application
referencing an area of memory that it isn't supposed to. How are you
doing, from a memory perspective? If you run top on the box, what type
of memory utilization are you encountering? Do you have SQL logging on?
What types of queries are being executed by the users? Perhaps someone
is executing a large query that is eating up memory.

Good luck :-) I've been in that situation before ... where our
production AR System server just started crashing. When you have 3k+
named user accounts on the box and it goes down ... you get all kinds of
attention from all kinds of people.

Derek



Derek Berube
Wildstar Technologies, LLC.
1453 Riverview Run Lane
Suwanee, Georgia 30024
(404) 444-5283
http://www.wildstartech.com/

AIM Handle: BerubeDB
MSN Messenger: derekberube@hotmail.com
Yahoo! Messenger: derekberube

On Tue, 2004-12-28 at 17:29 -0500, David Harris wrote:

> For several days now, arserverd has bounced every five to thirty minutes,
> each time with a signal 11 error. There has been no correlation between
> this error and any action in workflow logs. Remedy suggests it is a memory
> issue, but even after having pored through a tusc log, they are unable to
> point to a cause. So hardware, database, and workflow issues all have
> seemingly been ruled out, although something has to be the culprit. Two
> days into a critical ticket logged with Remedy and massive performance
> degradation, we are now poised for an impromptu upgrade to 5.1.2 from
> 5.1.1 patch 1211, recommended by Remedy so they can at least build a debug
> arserverd if necessary.
>
> Does anyone have any stories of having experienced a similar problem in
> the past?
>
> Regards,
> David
>
> 5.1.1 patch 1211 (soon to be on 5.1.2)
> HP-UX 11.00
> Oracle 8.1.7
>
>
> UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
> (Support: mailto:support@arslist.org)


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)