JBoss – Changing RMI Remote Client Callback Address

Recently during a JBoss production deployment (4.2.3.GA) that I had to carry on, I came across a problem with RMI Remoting (EJB3), which gave an exception when remote EJBs are invoked. The exception I got was ‘java.lang.IllegalArgumentException: port out of range:-1′, whenever a Remote EJB call was made.

12:57:58,354 WARN  [ServiceExceptionTranslatorAspect] Unable to Translate Exception : org.jboss.remoting.CannotConnectException
12:57:58,402 ERROR [[default]] Servlet.service() for servlet default threw exception
java.lang.IllegalArgumentException: port out of range:-1
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:118)
at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:183)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:827)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:569)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
at org.jboss.remoting.Client.invoke(Client.java:1634)
at org.jboss.remoting.Client.invoke(Client.java:548)
at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:74)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:107)

The application was working fine in our local environments, and this problem happened only in production. This deployment consisted of two JBoss AS instances running two applications, which communicates with each other via EJB Remoting.

Googling did not give me any positive lead, and I was stuck on this problem for a while. So I tried out the only option I was left with, trying to go to detailed log output to see what’s happening under the hood. After changing the log level for CONSOLE and org.jboss category to DEBUG level (if you are new to JBoss, you can change the logging mechanism by modifying jboss-log4j.xml in conf directory of server profile).

This gave me the following output.

12:21:41,049 DEBUG [AuthenticationContextInterceptor] AuthenticationContextInterceptor Intercepting EJB Invocation...
12:21:41,049 DEBUG [AuthenticationContextInterceptor] Setting the authenticated user details in Authentication Context: XXXX
12:21:41,061 DEBUG [AuthenticationContextInterceptor] AuthenticationContextInterceptor Intercepting EJB Invocation...
12:21:41,061 DEBUG [AuthenticationContextInterceptor] Setting the authenticated user details in Authentication Context: XXXX
12:21:43,994 DEBUG [MicroSocketClientInvoker] SocketClientInvoker[b874d2, socket://192.168.12.4:-1] constructed
12:21:43,994 DEBUG [MicroRemoteClientInvoker] SocketClientInvoker[b874d2, socket://192.168.12.4:-1] connecting
12:21:43,994 DEBUG [MicroSocketClientInvoker] Creating semaphore with size 50
12:21:43,994 DEBUG [MicroRemoteClientInvoker] SocketClientInvoker[b874d2, socket://192.168.12.4:-1] connected
12:21:43,995 DEBUG [InvokerRegistry] removed SocketClientInvoker[b874d2, socket://192.168.12.4:-1] from registry
12:21:43,995 DEBUG [MicroSocketClientInvoker] SocketClientInvoker[b874d2, socket://192.168.12.4:-1] disconnecting ...
12:21:44,946 ERROR [[default]] Servlet.service() for servlet default threw exception
java.lang.IllegalArgumentException: port out of range:-1

As the highlighted line shows, JBoss was trying to communicate to the remote EJB via port -1, which is of course invalid. Also, an interesting observation was that the IP address which JBoss tries to use is not the one I expected it to use. This, was actually due to the network setup of the production environment. The production server in question had two network cards, each having local IPs  192.168.12.4 and 192.168.12.5. The 192.168.12.4 NIC was used to expose the server to Internet via NAT. 192.168.12.5 was used to internally connect to the machine via VPN for maintenance, etc. Since 192.168.12.4 was used for NAT, it was restricted to HTTP traffic on port 8080 only.

So the problem that I was facing was that JBoss was using the 192.168.12.4 to refer to the remote EJB, where as I expected it to use 192.168.12.5 (note that we have to start JBoss bound to all addresses using -b 0.0.0.0 because we access it via multiple NICs). Since JBoss could not get a free port on 192.168.12.4, it was falling back to port -1.

So I wanted to find out a way to force JBoss to use the IP address I wanted for remote EJB calls (without binding it specifically to one IP using -b). I was already referring to the remote server’s JNDI Registry via the IP I expected (192.168.12.5). After trying out various options, finally I found the following in the deploy folder of JBoss.

File: <JBOSS_HOME>/server/xxxx/deploy/ejb3.deployer/META-INF/jboss-service.xml

<mbean code="org.jboss.remoting.transport.Connector"
name="jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3">
<depends>jboss.aop:service=AspectDeployer</depends>
<attribute name="InvokerLocator">socket://${jboss.bind.address}:13873</attribute>
<attribute name="Configuration">
<handlers>
<handler subsystem="AOP">org.jboss.aspects.remoting.AOPRemotingInvocationHandler</handler>
</handlers>
</attribute>
</mbean>

I changed it to the following, so that instead of dynamically resolving the IP address, JBoss will use the IP I wanted when remote client callbacks are created.

<mbean code="org.jboss.remoting.transport.Connector"
name="jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3">
<depends>jboss.aop:service=AspectDeployer</depends>
<attribute name="InvokerLocator">socket://192.168.12.5:13873</attribute>
<attribute name="Configuration">
<handlers>
<handler subsystem="AOP">org.jboss.aspects.remoting.AOPRemotingInvocationHandler</handler>
</handlers>
</attribute>
</mbean>

With this in place, the client was able to invoke the remote EJBs without any issues. This is a rare situation, but if anyone else face the same issue, I guess this post would help to get it sorted out.

4 comments

  1. Hi Yohan,

    thank you for your excellent article.
    The section you descripbe hier is commented ( inactivated), in the default JBoss configuration. Do you know how to activate it?
    If i comment it out i bekomme an excetion telling me that the ejb3 service is already registered

    Thank
    bate

    1. Hi Bate,

      Which version of JBoss are you using? Is it 4.2.3 GA?

      In 4.2.3.GA, the default (if I’m not mistaken) is that you have one

      You should change only the first one. If you have more than one entry for it, I guess you might end up with the exception that you have mentioned.

      Please feel free to contact me (via email / blog) if you need any help.

  2. Hi,

    I have the same problem described but the solution proposed will restrict the client who will connect to jboss servers

    1. Hi Ben,

      The solution outlined above will lead to a situation where the clients are able to connect to the JBoss server using only a particular IP Address of the server. For example, if you JBoss server has multiple network cards with multiple IP Addresses, you will be able to use only one.

      So in essence, this will not restrict the clients. Anyone can connect to the server. But they will be able to call the server using only one of the IP’s assigned to the server. This should be acceptable for majority of cases.

      But if you want to be able to call the server using any of it’s IPs, then yes, this will not work for you.

      Regards,
      Yohan.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>