We have removed ethereal (and winpcap) because it was only authorized for
use for a limited time. Since then, we left a user on the webserver logged
in (and just locked the computer rather than logging out) and still, the
number of RPC bind errors was slim-to-none. Therefore, I conclude the
installation of ethereal/winpcap had no effect on RPC binds.

However, since leaving a user logged in isn't the greatest security
practice, we recently logged the user off and the RPC bind errors started
coming back with a rather high frequency.

The only variable that I can see now that is effecting the RPC bind issue is
a user being logged into the webserver.

Any thoughts?

Axton, to answer your question: yes, the mid-tier configuration has always
had a TCP port defined.

Andrew, to answer your questions: no, power management on the NIC on the
webserver is not enabled, therefore it should always be "on". I'll have to
ask the network guys about the settings on the switch however to verify that
it's port isn't powering down and that the speed/duplex settings match.
I'll attempt to log cpu utilization on both the webserver and application
server to see if the RPC binds correspond to these times (but won't this
actually inflate cpu utilization?).

Thanks for all of your help thus far, keep your suggestions coming...

Ben

-----Original Message-----
From: Action Request System discussion list(ARSList)
[mailto:arslist@ARSLIST.ORG] On Behalf Of Axton
Sent: Friday, December 23, 2005 8:00 PM
To: arslist@ARSLIST.ORG
Subject: Re: Frequent Remedy MidTier RPC Bind Errors

Try and define a tcp port in the mid-tier configuration page (thus bypassing
the use of the udp protol) and see if this resolves the problem.

I wonder if the winpcap libs installed by ethereal would impact this issue
as well. Try making the above change and uninstall ethereal and the bundled
winpcap.

Axton

On 12/23/05, Watson Benjamin A Contr AFWA/XOO
wrote:
> Andrew,
>
> I'm not sure about Windows 2003 Server, but I'm fairly certain that
> the NIC inactivity power-down setting on a default Windows (e.g. XP)
> install is disabled. However, it is a good idea to double check.
> Although, the NIC does get a fairly frequent ARP request which should
> keep it alive even if the power saving setting has been enabled.
>
> Axton,
>
> - When you stop ethereal and/or set the IF to non-promiscuous mode,
> does the problem start again?
>
> I haven't checked that, and due to the holiday, I doubt I'll have
> enough requests to measure, but it's worth a shot when people come back
next week.
>
> - Do you see the arserver/mid-tier communication using tcp or udp?
>
> I see a mix of TCP and UDP. By and large, the communication is TCP.
> If you drill down into the packets (TCP), in the data field you can
> see the text corresponding to active links being fired, etc. (really
> interesting actually). However, there are a few packets, whose
> protocol listed as Portmap (which I believe to be Portmapper) and is
> classified as UDP. These come during connection losses (although the
> connection losses in these particular cases are intentional (e.g. we've
restarted services)).
>
> - Is the firewall capable of connection/state tracking, and if so is
> it used for the protocol in use for the ars/mt communications?
>
> I would imagine that the firewall in place is stateful, however I'll
> have to confirm that with the network guys. I'll have to double check
> what rules they have set up.
>
> Thanks for the input thus far...
>
> -----Original Message-----
> From: Action Request System discussion list(ARSList)
> [mailto:arslist@ARSLIST.ORG] On Behalf Of Baxter, Andrew
> Sent: Friday, December 23, 2005 2:25 PM
> To: arslist@ARSLIST.ORG
> Subject: Re: Frequent Remedy MidTier RPC Bind Errors
>
> Just off the top of my head I would say that the speed of your nics
> has nothing to do with it, but I would suggest you check that power
> management has not been enabled on at least one system and when there
> is inactivity it is shutting off the nic. By using the sniffer you
> are keeping the nic active so it won't shut off. I would also verify
> your speed/duplex settings between the server and the switch to make
> sure they match. Other than that, it is possible one of the systems
> runs a job periodically that is using all available resources, but
> that seems unlikely since the problem has gone away with the
> installation of the sniffer software.
>
> I would start by logging out of the mid-tier instead of locking it to
> narrow down what "fixed" your problem.
>
> Thanks,
> Andrew Baxter
> Manager, Information Technology
> w. (781) 902-6026
> f. (781) 902-6002
> -----Original Message-----
> From: Action Request System discussion list(ARSList)
> [mailto:arslist@ARSLIST.ORG] On Behalf Of Watson Benjamin A Contr
> AFWA/XOO
> Sent: Friday, December 23, 2005 3:11 PM
> To: arslist@ARSLIST.ORG
> Subject: Frequent Remedy MidTier RPC Bind Errors
>
> Listers,
>
> For quite some time, we've (our users) that access our Remedy system
> via the web have reported frequent RPC Bind errors.
>
> When checking the MidTier logs, the error is visible with a message
> indicating that it (MidTier) has lost connection.
>
> See the "graphic" below for server configuration/layout:
>
> ---------------- ---------------- | ------------------
> |DB Server | |App Server | | |Web Server |
> |MS SQL 2000 SP4| |ARS 5.1.2 | | |MidTier 6.3 |
> |Win2k3 Ent. |<------>|patch 1375 |<---|---->|Patch 10 |
> |Plenty of H.W. | |Win2k3 Ent. | | |Win2k3 Ent. IIS 6|
> |"horsepower" | ---------------- | ------------------
> ---------------- |
> |firewall
>
> All servers are equipped with Gigabit NICs.
>
> This problem had also plagued us back when our MidTier was 5.1.2 and
> we hoped that the upgrade to 6.3 would help, but it has not.
>
> To perform some troubleshooting, we loaded Ethereal on our Web Server
> for some basic traffic analysis to see if we could hunt down the root
> of the problem.
>
> However, we noticed something; we haven't had a single RPC Bind error
> since Ethereal was loaded! The only changes that have been put in
> place due to loading Ethereal are:
>
> 1. The NIC on the Web Server was placed in promiscuous mode.
> 2. To keep the traffic capture alive, we do not log off of the machine
> (just lock it).
>
> The network guys tell us the firewall isn't blocking anything it
> shouldn't, and since we're on a switched network, I don't suspect
> promiscuous mode is helping out either.
>
> Have any of you had a problem with RPC Bind errors? If so, what was
> the resolution? Any ideas as to why the changes (1 and/or 2) above
> seem to mitigate this problem?
>
> Any help is much appreciated.
>
> //SIGNED//
> Benjamin A. Watson, Contractor, AFWA/XOO SAIC - Remedy Team
> Comm: (402) 294-8225
> DSN: 271-8225
> mailto:remedy.admin@afwa.af.mil
> mailto:benjamin.watson.ctr@afwa.af.mil
>
>
>
>
> UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
> (Support: mailto:support@arslist.org)
>
> ****This e-mail is sent by Hudson Highland Group, Inc., or one of its
> subsidiaries, and may contain information that is privileged or
> confidential. If you are not the intended recipient, please delete the
> e-mail and any attachments and notify us immediately.****
>
>
>
>
> UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
> (Support: mailto:support@arslist.org)
>
>
> UNSUBSCRIBE or access ARSlist Archives at
> http://www.ARSLIST.org
> (Support: mailto:support@arslist.org)
>



UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)


UNSUBSCRIBE or access ARSlist Archives at http://www.ARSLIST.org
(Support: mailto:support@arslist.org)