Home > GLASSFISH / J2EE / JAVA PERFORMANCE > Glassfish – Vertical clustering with multiple domains

Glassfish – Vertical clustering with multiple domains


Introduction

There are two reasons why i started to try this out:

  1. It allows performing load balancing tests in a clustered system, using one single server (therefore hardware requirement reduces to one server instead of two)
  2. By using two domains under one glassfish instance, one can optimize the memory usage on a single server, having lots of RAM at disposal

The server’s hardware configuration i was running before doing this, was something like 8 GB RAM and 4 CPU’S. So i was thinking of using the 8 GB for two JVM’s instead of one. There are several reasons for working with a smaller JVM, first of them being the efficiency of Garbage Collection (other reasons include for example finding entries in large hashmaps, keeping redundant objects shorter in the memory, etc.)

Glassfish Horizontal Cluster

Glassfish Horizontal Cluster - ModJK balancing

We are currently using JMS for cache synchronization, so i had the option of using Glassfish’s embedded JMS Server (which i did), or a separate, stand alone JMS Server (which i also tried and works perfectly) This did not work out of the box, since there are some configuration items to be taken into consideration.

Ok, so here is the concept:

  1. Create, configure and start the first domainwith embedded JMS Server: ports, topics, jms configuration, jvm configuration, mod_jk, thread pools etc.
  2. Restart the first domain for configurations to apply
  3. Create the second domain
  4. Configure the second domain: ports, topics, jms configuration, jvm configuration, mod_jk, thread pools etc.
  5. Start the second domain using the first domain as JMS Server

When creating the first domain, there will be some default settings like:

  • http listener port: 8080
  • https listener port: 8081
  • JMS port 7676
  • jvm Xmx 512M
  • etc.

While this is very convenient if using one domain pro physical server, when creating the second domain, arbitrary ports will be assigned. Therefore you would like to have both domains follow a naming convention. In order to avoid any conflicts, following configuration items have to be kept synchronized (the ports have to be unique) The list contains the default configuration of a freshly set up domain:

Port configuration

<iiop-listener id="orb-listener-1" port="3700" address="0.0.0.0" lazy-init="true" />
<iiop-listener id="SSL" port="3820" address="0.0.0.0" security-enabled="true">
<iiop-listener id="SSL_MUTUALAUTH" port="3920" address="0.0.0.0" security-enabled="true">
<jmx-connector port="8686" address="0.0.0.0" security-enabled="false" name="system" auth-realm-name="admin-realm" />
<jvm-options>-Dosgi.shell.telnet.port=6666</jvm-options>
<network-listener port="8080" protocol="http-listener-1" transport="tcp" name="http-listener-1" thread-pool="http-thread-pool" />
<network-listener port="8181" protocol="http-listener-2" transport="tcp" name="http-listener-2" thread-pool="http-thread-pool" />
<network-listener port="4848" protocol="admin-listener" transport="tcp" name="admin-listener" thread-pool="http-thread-pool" />
<network-listener port="8010" protocol="jk-connector" transport="tcp" name="jk-connector" jk-enabled="true" thread-pool="http-thread-pool" />

I suggest a suffix based naming convention for the ports. Therefore, each configured port should end in the number of the domain. For example:

Domain’s 1 http port would be 8081 instead of 8080. Domain’s 2 port would then be 8082, and so on

This is how it would look like for two domains:

Glassfish Multiple Domain Port configuration

Glassfish Multiple Domain Port configuration

 JMS Configuration

Since the first Domain is running as Master JMS, we need to configure all other domains to use this JMS Server. There will be therefore a different JMS Configuration on the first domain on one hand, and all other domains on the other hand:

Domain 1 JMS  Configuration

<jms-service default-jms-host="default_JMS_host" type="LOCAL">
<jms-host host="0.0.0.0" name="default_JMS_host" />
</jms-service>

Domain 2 JMS Configuration
<jms-service default-jms-host="default_JMS_host" type="REMOTE">
<jms-host host="127.0.0.1" name="default_JMS_host" />
</jms-service>

Creating and configuring Domain 1

There is nothing easier creating a domain in Glassfish. This will create all default options for you to start and get it running. We will configure it directly to use 4841 as administration port, instead of 4848.
asadmin create-domain --adminport 4841 domain1
In order to be able to reconfigure the default settings, we need to start the domain
asadmin start-domain domain1

We will want to do the following:

  • delete the default memory settings and re-create them with our configured values
  • reset the JMS Service configuration
  • create a JMS topic
  • reset the default ports to the ones following the naming convention

Reconfigure Memory Settings

asadmin delete-jvm-options "-client:-Xmx512m:-XX\:NewRatio=2:-XX\:MaxPermSize=192m"

Since we do not know the Telnet port of the second domain (it will be arbitrarly assigned), we have to be able to delete this configuration in a dynamic way:

asadmin delete-jvm-options `asadmin get server.* | grep -o '\-Dosgi.shell.telnet.port=[0-9]*'`
Now we can re-create the memory settings, and configure the garbage collection according to our needs. The configuration below deals with a 3.5GB JVM, and a CMS Garbage Collector. More details are available by searching this blog:
asadmin create-jvm-options "-server:-Dosgi\.shell\.telnet\.port=6661:-DjvmRoute=lb1:-XX\:MaxPermSize=512m:-Xmx3550m:-Xms3550m:-XX\:NewSize=1500m:-XX\:MaxNewSize=1500m:-XX\:ParallelGCThreads=2:-XX\:\+UseConcMarkSweepGC:-XX\:\+UseParNewGC:-XX\:SurvivorRatio=3:-XX\:TargetSurvivorRatio=90:-XX\:MaxTenuringThreshold=4:-XX\:\+CMSParallelRemarkEnabled:-XX\:\+CMSPermGenSweepingEnabled:-XX\:\+CMSClassUnloadingEnabled:-XX\:\+PrintGCDetails:-Xloggc\:\${com.sun.aas.instanceRoot}/logs/jgc.log:-Xss128k"
We have now configured the JVM options, and the telnet port. Let's proceed with configuring JMS

Reset the JMS Service Configuration

As mentioned before, the JMS Server is acting as Master. We need to set it as LOCAL Type.

asadmin -p 4841 -W /opt/glassfish/passwords set server.jms-service.type=LOCAL
asadmin -p 4841 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.lazy-init=false
asadmin -p 4841 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.host=0.0.0.0
asadmin -p 4841 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.port=7676

Create a JMS Topic


asadmin -p 4841 -W /opt/glassfish/passwords create-jmsdest --desttype topic "Sample"
asadmin -p 4841 -W /opt/glassfish/passwords create-jms-resource --restype javax.jms.Topic --property Name=Sample "jms/SampleTopic"
asadmin -p 4841 -W /opt/glassfish/passwords create-jms-resource --restype javax.jms.TopicConnectionFactory --property transaction-support=NoTransaction "jms/SampleTopicFactory"

We are now ready for the last step, configuring the remaining ports

Reset the default ports to the ones following the naming convention

Configure the http listeners

asadmin -p 4841 -W /opt/glassfish/passwords set server.network-config.network-listeners.network-listener.http-listener-1.port=8081
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.compression=off
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.compressable-mime-type=text/html,text/xml,text/plain,text/javascript,text/css
asadmin -p 4841 -W /opt/glassfish/passwords set server.network-config.network-listeners.network-listener.http-listener-2.port=8181
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.compression=off
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.compressable-mime-type=text/html,text/xml,text/plain,text/javascript,text/css

Configure the thread pool (default is minimum of 2 and max of 5)

asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=200
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.min-thread-pool-size=100

Configure ModJK Connector (this needs its own listener)

asadmin -p 4841 -W /opt/glassfish/passwords create-http-listener --listenerport 8011 --listeneraddress 0.0.0.0 --defaultvs server jk-connector
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true

Configure JMX Connector

asadmin -p 4841 -W /opt/glassfish/passwords set server.admin-service.jmx-connector.system.port=8681

Configure other Domain Ports

asadmin -p 4841 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.SSL.port=3821
asadmin -p 4841 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.SSL_MUTUALAUTH.port=3921
asadmin -p 4841 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.orb-listener-1.port=3701

We are now ready to restart the domain
asadmin -p 4841 -W /opt/glassfish/passwords restart-domain domain1

Creating and configuring Domain 2

asadmin create-domain --adminport 4842 domain2
In order to be able to reconfigure the default settings, we need to start the domain
asadmin start-domain domain2

We will want to do the following:

  • delete the default memory settings and re-create them with our configured values
  • reset the JMS Service configuration, and configure it to use the Domain 1 JMS Server
  • create a JMS topic
  • reset the default ports to the ones following the naming convention

Reconfigure Memory Settings

asadmin -p 4842 -W /opt/glassfish/passwords delete-jvm-options "-client:-Xmx512m:-XX\:NewRatio=2:-XX\:MaxPermSize=192m"

Since we do not know the Telnet port of the second domain (it will be arbitrarly assigned), we have to be able to delete this configuration in a dynamic way:

asadmin -p 4842 -W /opt/glassfish/passwords delete-jvm-options `asadmin get server.* | grep -o '\-Dosgi.shell.telnet.port=[0-9]*'`
Now we can re-create the memory settings, and configure the garbage collection according to our needs. The configuration for the second domain has been adapted to a smaller JVM. Watch out how you name your jvmRoute Parameter, since this will be used when configuring the Apache ModJK workers (in this case, lb2 – load balancer 2):
asadmin -p 4842 -W /opt/glassfish/passwords create-jvm-options "-server:-Dosgi\.shell\.telnet\.port=6662:-DjvmRoute=lb2:-XX\:MaxPermSize=512m:-Xmx2500m:-Xms2500m:-XX\:NewSize=1500m:-XX\:MaxNewSize=1500m:-XX\:ParallelGCThreads=2:-XX\:\+UseConcMarkSweepGC:-XX\:\+UseParNewGC:-XX\:SurvivorRatio=3:-XX\:TargetSurvivorRatio=90:-XX\:MaxTenuringThreshold=4:-XX\:\+CMSParallelRemarkEnabled:-XX\:\+CMSPermGenSweepingEnabled:-XX\:\+CMSClassUnloadingEnabled:-XX\:\+PrintGCDetails:-Xloggc\:\${com.sun.aas.instanceRoot}/logs/jgc.log:-Xss128k"
We have now configured the JVM options, and the telnet port. Let's proceed with configuring JMS

Reset the JMS Service Configuration

As mentioned before, the JMS Server is acting as slave. We need to set it as REMOTE Type.

asadmin -p 4842 -W /opt/glassfish/passwords set server.jms-service.type=REMOTE
asadmin -p 4842 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.lazy-init=false
asadmin -p 4842 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.host=127.0.0.1
asadmin -p 4842 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.port=7676

Create a JMS Topic

The destination has previously been created on domain 1 , so we do not need to create it again (if you try that, you will get an error). We will therefore skip step1, and proceed with creating only the jms resources.

asadmin -p 4842 -W /opt/glassfish/passwords create-jms-resource –restype javax.jms.Topic –property Name=Sample “jms/SampleTopic”
asadmin -p 4842 -W /opt/glassfish/passwords create-jms-resource –restype javax.jms.TopicConnectionFactory –property transaction-support=NoTransaction “jms/SampleTopicFactory”

We are now ready for the last step, configuring the remaining ports

Reset the default ports to the ones following the naming convention

Configure the http listeners

asadmin -p 4842 -W /opt/glassfish/passwords set server.network-config.network-listeners.network-listener.http-listener-1.port=8082
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.compression=off
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.compressable-mime-type=text/html,text/xml,text/plain,text/javascript,text/css
asadmin -p 4842 -W /opt/glassfish/passwords set server.network-config.network-listeners.network-listener.http-listener-2.port=8182
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.compression=off
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.compressable-mime-type=text/html,text/xml,text/plain,text/javascript,text/css

Configure the thread pool (default is minimum of 2 and max of 5)

asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=200
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.min-thread-pool-size=100

Configure ModJK Connector (this needs its own listener)

asadmin -p 4842 -W /opt/glassfish/passwords create-http-listener --listenerport 8012 --listeneraddress 0.0.0.0 --defaultvs server jk-connector
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true

Configure JMX Connector

asadmin -p 4842 -W /opt/glassfish/passwords set server.admin-service.jmx-connector.system.port=8682

Configure other Domain Ports

asadmin -p 4842 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.SSL.port=3822
asadmin -p 4842 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.SSL_MUTUALAUTH.port=3922
asadmin -p 4842 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.orb-listener-1.port=3702

We are now ready to restart the domain
asadmin -p 4842 -W /opt/glassfish/passwords restart-domain domain2

MOD JK Workers Configuration

I suggest configuring your Workers using templates. It will be easier to handle, since you will update the configuration in one single place.

Let’s define a worker template

worker.template.type=ajp13
worker.template.lbfactor=50
worker.template.socket_keepalive=false
worker.template.socket_connect_timeout=300
worker.template.ping_mode=A
worker.template.ping_timeout=1000
worker.template.connection_pool_size=32
worker.template.connection_pool_timeout=600

We can now define our two workers, each linked to one Glassfish Domain. Pay attention to the name of the workers, and the port, which have to be synchronized with the names you defined in the JVM Options of the Domains (jvmRoute Parameter)

Configure worker for Domain 1

worker.lb1.reference=worker.template
worker.lb1.host=glassfish_server
worker.lb1.port=8011
worker.lb1.lbfactor=50
worker.lb1.socket_keepalive=1

Configure worker for Domain 2
worker.lb2.reference=worker.template
worker.lb2.host=glassfish_server
worker.lb2.port=8012
worker.lb2.lbfactor=50
worker.lb2.socket_keepalive=1

Configure the load balancer

worker.lb.type=lb
worker.lb.balance_workers=lb1,lb2
worker.lb.sticky_session=True
worker.lb.sticky_session_force=false
worker.lb.method=B


# list workers visible to apache
worker.list=lb

Domains running – Check processes

You should now have two root processes:

  • domain 1 glassfish, with embedded JMS
  • domain 2 glassfish

This should look like:

14031 ?        Sl     2:07 /usr/java/jdk1.6.0_23/bin/java -cp /opt/glassfishv3/glassfish/modules/glassfish.jar -XX:+UnlockDiagnosticVMOptions -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:MaxPermSize=51
12394 ?        Sl     3:16 /usr/java/jdk1.6.0_23/bin/java -cp /opt/glassfishv3/glassfish/modules/glassfish.jar -XX:+UnlockDiagnosticVMOptions -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:MaxPermSize=51
12446 ?        S      0:00  \_ /bin/sh /opt/glassfishv3/mq/bin/imqbrokerd -javahome /usr/java/jdk1.6.0_23/jre -Dimq.cluster.nowaitForMasterBroker=true -varhome /opt/glassfishv3/glassfish/domains/domain1/imq -
12480 ?        Sl     0:04      \_ /usr/java/jdk1.6.0_23/jre/bin/java -cp /opt/glassfishv3/mq/bin/../lib/imqbroker.jar:/opt/glassfishv3/mq/bin/../lib/imqutil.jar:/opt/glassfishv3/mq/bin/../lib/jsse.jar:/opt/g

You can now check if the workers are configured properly, by running some http requests over your server. You should see where the request has been directed to:

“GET /home” 200 2601 14984 “https://myserver/&#8221; “Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.8)
Gecko/20100722 Firefox/3.6.8″ lb 0.014809 F<lb2 0 OK> L<lb2 0 OK> – the request has been directed to the second server

“GET /home” 200 2601 14984 “https://myserver/&#8221; “Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.8)
Gecko/20100722 Firefox/3.6.8″ lb 0.014809 F<lb1 0 OK> L<lb1 0 OK>- the request has been directed to the first server

Performance Impact

I ran the same test by directing all requests first to the domain 1, and then to domain 2.

Let’s review the JVM and Garbage Collection Strategy for the two domains, and then review the results by comparing response times, and garbage collection statistics.

Domain 1 JVM Configuration – Bigger JVM


<jvm-options>-DjvmRoute=lb1</jvm-options>
<jvm-options>-XX:MaxPermSize=512m</jvm-options>
<jvm-options>-Xmx3550m</jvm-options>
<jvm-options>-Xms3550m</jvm-options>
<jvm-options>-XX:NewSize=1500m</jvm-options>
<jvm-options>-XX:MaxNewSize=1500m</jvm-options>
<jvm-options>-XX:ParallelGCThreads=2</jvm-options>
<jvm-options>-XX:+UseConcMarkSweepGC</jvm-options>
<jvm-options>-XX:+UseParNewGC</jvm-options>
<jvm-options>-XX:SurvivorRatio=3</jvm-options>
<jvm-options>-XX:TargetSurvivorRatio=90</jvm-options>
<jvm-options>-XX:MaxTenuringThreshold=4</jvm-options>
<jvm-options>-XX:+CMSParallelRemarkEnabled</jvm-options>
<jvm-options>-XX:+CMSPermGenSweepingEnabled</jvm-options>
<jvm-options>-XX:+CMSClassUnloadingEnabled</jvm-options>
<jvm-options>-XX:+PrintGCDetails</jvm-options>

Response times for Domain 1

Request

90% response time

Homepage 688
Login 1375
Add transaction 243
Logout 555

Domain 2 JVM Configuration – Smaller JVM


<jvm-options>-DjvmRoute=tap2w2</jvm-options>
<jvm-options>-XX:MaxPermSize=512m</jvm-options>
<jvm-options>-Xmx2500m</jvm-options>
<jvm-options>-Xms2500m</jvm-options>
<jvm-options>-XX:NewSize=1500m</jvm-options>
<jvm-options>-XX:MaxNewSize=1500m</jvm-options>
<jvm-options>-XX:ParallelGCThreads=2</jvm-options>
<jvm-options>-XX:+UseConcMarkSweepGC</jvm-options>
<jvm-options>-XX:+UseParNewGC</jvm-options>
<jvm-options>-XX:SurvivorRatio=3</jvm-options>
<jvm-options>-XX:TargetSurvivorRatio=90</jvm-options>
<jvm-options>-XX:MaxTenuringThreshold=4</jvm-options>
<jvm-options>-XX:+CMSParallelRemarkEnabled</jvm-options>
<jvm-options>-XX:+CMSPermGenSweepingEnabled</jvm-options>
<jvm-options>-XX:+CMSClassUnloadingEnabled</jvm-options>
<jvm-options>-XX:+PrintGCDetails</jvm-options>

Response Times for Domain 2

Request

90% response time

Homepage 338
Login 491
Add transaction 156
Logout 219

Now, let’s do a head to head compare also by adding the number of performed transactions, and the performance improvement from the big domain 1 configuration, to the smaller domain 2 configuration

# of Requests Request 90% response time Performance Improvement
Domain 1 Domain 2
150 Homepage 688 338 103 %
150 Login 1375 491 180 %
15 000 Add transaction 243 156 55 %
150 Logout 555 219 153 %

As you can see, there is a major improvement, from 55 % to 180 %, depending on the business transaction. Having taken all other outside variables into consideration, i can conclude there is definitely place for improvement in a smaller jvm.

Let’s take a look at some garbage collection statistics:

Garbage Collection duration:

Garbage Collection Duration statistics

Garbage Collection Duration statistics

Garbage Collection Time Statistics – Head to Head compare

Statistics Domain 1 Domain 2
Percentage of time in GC 0.217 % 0.098 %
Time spent in Full GC 15.877 s 4.758 s
Percentage of time in Full GC 0.118 % 0.035 %
Average duration Parallel Scavenge 0.16 s 0.113 s

Final

This post was about vertical clustering in glassfish. It can help individuals who, in lack of hardware, can develop and test in a clustered system, using one single Glassfish Instance, and multiple domains. I have reviewed the pros and cons, the things to take care of when configuring multiple domains, and the performance improvement by running some load tests against the two configurations.

I hope you can use this in your deployment scenarios, and that this post will clear any or at least some unanswered question relating to this topic.Feel free to ask me if you still have questions. I will try answering them as soon as possible.

Cheers,

Alex

About these ads
  1. KB
    February 1, 2012 at 9:00 am

    Hi,

    interesting article. What tools did you use to set up the requests and measure the throughput?

  2. February 1, 2012 at 9:21 am
  3. March 16, 2012 at 3:49 pm

    Reset the JMS Service Configuration?

  4. June 8, 2012 at 8:43 am

    Very interesting feedback !

  5. Alex
    September 18, 2012 at 9:44 am

    Hi Alex,

    Out of interest what are you using to read in and graph the Cumulative GC duration?

  6. September 18, 2012 at 9:45 am

    Hi Alex,

    Out of interest what are you using to read in the log and output to graphs for the Cumulative GC Duration?

    Thanks

    • September 19, 2012 at 3:01 pm

      Hey,
      i am using HPJmeter. It’s an open source tool, you can download it freely from HP website.

  7. August 21, 2013 at 4:37 am

    Hi Alex,

    We posted an article an week about how to run multiple domains with Glassfish

    http://blog.jelastic.com/2013/08/19/running-multiple-custom-domains-on-glassfish-server/

    Looking forward to your feedback :)

  1. February 3, 2012 at 7:20 am
  2. February 7, 2012 at 6:44 am
  3. June 2, 2012 at 1:10 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 32 other followers

%d bloggers like this: