Yesterday we discovered a new way to completely destroy the ability to
log in to a Remedy server using LDAP. I share it with you because it
points out a vulnerability in having multiple plugins running on the one
AR System Plug-In server, where function as critical as authentication
can be demolished by impaired performance in another, much less critical
plugin.

Our Novell shop was tinkering with their eDir servers over spring break,
which I think are actually Novell linux, and they managed to uncover /
trigger some bugs in the eDir software that basically crashed one of the
two server farms. We had to re-point our AREA configurations and sTunnel
connections supporting them to a different LDAP farm, which restored our
ability to service LDAP passwords. Unfortunately that farm does not
support our ARDBC integration so that plugin could not connect properly
and filled our error logs with ARERR 3375. We had lost our ability to
update email addresses in LDAP from the SHR:People form, no big deal.
Authentication was working, however, but slower.

The following day the farm we had been forced to switch to started
having problems of its own (the entire campus had switched everything
over to it the previous day), but they had rebuilt the original one with
three new servers overnight, so we switched the AREA configurations and
sTunnel connections back to the original one. All hell broke loose. The
logs filled with RPC timeouts (ARERR 91), '"AR System Plug-in server is
not responding" (ARERR 8939) errors, and "cannot establish a network
connection to the AR System Plug-In server" (ARERR 8760) errors. It took
the entire day to figure out that when they had copied the replicas of
the eDir directory from the working server to the three new ones, none
of the indexes had transferred with the data (they don't - they have to
be recreated manually). What that meant on our end was that when a
filter kicked off an update through ARDBC and a vendor form, it was
attempting a table scan of 320,000 records in ldap for the euid (unique
uid) on three out of the four LDAP servers. The AR System Plug-In server
would start the ARDBC transaction and hang there until it timed out.
While this was going on, AREA authentication attempts were being
rejected with the error: "The authentication service is not responding.
Cannot connect to the system at this time. Contact your AR System
Administrator for assistance." That would be the administrator who had
been pulling his hair out all day. At some point I figured out that
ARDBC was breaking AREA, disabled the filters trying to access LDAP, and
the authentication errors stopped. As SOON as the LDAP administrator
rebuilt the indexes and that process completed, LDAP authentication
immediately began working properly even after the ARDBC integration was
re-enabled.

Moral of the story - beware of low level integrations and other
non-critical applications that SHARE the AR System Plug-In server with
AREA authentication. When they become degraded or fail due to some sort
of problem completely external to your systems, the net effect on AREA
can be devastating. It didn't make me any happier that every other
system on campus that used the problem LDAP server farm was having the
same kinds of problems with authentication and data response times. What
a difference an index makes.

BTW, our helpdesk staff and many others who have set local passwords in
Remedy were unaffected by the entire fiasco. They had no problems
connecting at any point during what turned out to be a cascading
failure.

Christopher Strauss, MSLS
Remedy Database Administrator
University of North Texas Computing Center
http://remedy.unt.edu/
ITSM 5.5 p1243 (Help Desk 5.5.1 p1255,
SLA 5.5 p1239, Approval 5.1 p1293)
ARS 5.01.02 p1375, ARS Mail 5.1.2 p1303
Mid-Tier 5.1.2 p1337 and 6.3.0 p013
Rem-Mail 5.1.4.9 on BackOffice 2000:
Win2K Sp4/SQLSrvr2K Sp3/Exchg2K Sp3;IIS 5.0


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org

On 3/22/06, strauss wrote:
> Yesterday we discovered a new way to completely destroy the ability to
> log in to a Remedy server using LDAP. I share it with you because it
> points out a vulnerability in having multiple plugins running on the one
> AR System Plug-In server, where function as critical as authentication
> can be demolished by impaired performance in another, much less critical
> plugin.
>
> Our Novell shop was tinkering with their eDir servers over spring break,
> which I think are actually Novell linux, and they managed to uncover /
> trigger some bugs in the eDir software that basically crashed one of the
> two server farms. We had to re-point our AREA configurations and sTunnel
> connections supporting them to a different LDAP farm, which restored our
> ability to service LDAP passwords. Unfortunately that farm does not
> support our ARDBC integration so that plugin could not connect properly
> and filled our error logs with ARERR 3375. We had lost our ability to
> update email addresses in LDAP from the SHR:People form, no big deal.
> Authentication was working, however, but slower.
>
> The following day the farm we had been forced to switch to started
> having problems of its own (the entire campus had switched everything
> over to it the previous day), but they had rebuilt the original one with
> three new servers overnight, so we switched the AREA configurations and
> sTunnel connections back to the original one. All hell broke loose. The
> logs filled with RPC timeouts (ARERR 91), '"AR System Plug-in server is
> not responding" (ARERR 8939) errors, and "cannot establish a network
> connection to the AR System Plug-In server" (ARERR 8760) errors. It took
> the entire day to figure out that when they had copied the replicas of
> the eDir directory from the working server to the three new ones, none
> of the indexes had transferred with the data (they don't - they have to
> be recreated manually). What that meant on our end was that when a
> filter kicked off an update through ARDBC and a vendor form, it was
> attempting a table scan of 320,000 records in ldap for the euid (unique
> uid) on three out of the four LDAP servers. The AR System Plug-In server
> would start the ARDBC transaction and hang there until it timed out.
> While this was going on, AREA authentication attempts were being
> rejected with the error: "The authentication service is not responding.
> Cannot connect to the system at this time. Contact your AR System
> Administrator for assistance." That would be the administrator who had
> been pulling his hair out all day. At some point I figured out that
> ARDBC was breaking AREA, disabled the filters trying to access LDAP, and
> the authentication errors stopped. As SOON as the LDAP administrator
> rebuilt the indexes and that process completed, LDAP authentication
> immediately began working properly even after the ARDBC integration was
> re-enabled.
>
> Moral of the story - beware of low level integrations and other
> non-critical applications that SHARE the AR System Plug-In server with
> AREA authentication. When they become degraded or fail due to some sort
> of problem completely external to your systems, the net effect on AREA
> can be devastating. It didn't make me any happier that every other
> system on campus that used the problem LDAP server farm was having the
> same kinds of problems with authentication and data response times. What
> a difference an index makes.
>
> BTW, our helpdesk staff and many others who have set local passwords in
> Remedy were unaffected by the entire fiasco. They had no problems
> connecting at any point during what turned out to be a cascading
> failure.
>
> Christopher Strauss, MSLS
> Remedy Database Administrator
> University of North Texas Computing Center
> http://remedy.unt.edu/
> ITSM 5.5 p1243 (Help Desk 5.5.1 p1255,
> SLA 5.5 p1239, Approval 5.1 p1293)
> ARS 5.01.02 p1375, ARS Mail 5.1.2 p1303
> Mid-Tier 5.1.2 p1337 and 6.3.0 p013
> Rem-Mail 5.1.4.9 on BackOffice 2000:
> Win2K Sp4/SQLSrvr2K Sp3/Exchg2K Sp3;IIS 5.0
>
>
> UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
>
Chris:

Do you think the timeout with the AREA auth attempts were a result of
the LDAP servers being too busy servicing the ARDBC update attempts?

Were others using the LDAP servers in the same farm during this time,
and if so did they receive timeout errors?

Axton Grams


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org

They were monitoring all four of the LDAP servers all day, and even
without proper indexes they were servicing authentication requests for
other systems continuously, just slower than they should have been. Its
all new hardware. Data lookups were a different problem, because of the
lack of indexes. The LDAP servers did not appear to be overloading, but
the three that lacked indexes were just responding noticeably slower.
None of the other systems (WebCT, Checkin for the general access labs,
etc.) have both the data lookup component and the authentication
component running in the same service the way the AR System Plug-In
server does. Basically on our systems AREA was having to wait until
there were no ARDBC requests pending (they were really just awaiting
timeout - they were ALL failing the lookups) before it could service an
authentication request (bind - lookup user - bind as user). Remedy is
more vulnerable than most systems as a result of the architecture;
authentication shouldn't have to "share" or wait in line for anything
else.

Personal aside - formal dissertation proposal went to committee at 4 AM
this morning.

Christopher Strauss, MSLS
Remedy Database Administrator
University of North Texas Computing Center
http://remedy.unt.edu/

-----Original Message-----
From: Action Request System discussion list(ARSList)
[mailto:arslist@ARSLIST.ORG] On Behalf Of Axton
Sent: Wednesday, March 22, 2006 1:39 PM
To: arslist@ARSLIST.ORG
Subject: Re: AR System Plug-In server vulnerability - a cautionary tale

>
Chris:

Do you think the timeout with the AREA auth attempts were a result of
the LDAP servers being too busy servicing the ARDBC update attempts?

Were others using the LDAP servers in the same farm during this time,
and if so did they receive timeout errors?

Axton Grams



UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org