Introduction

There are two reasons why i started to try this out:

1. It allows performing load balancing tests in a clustered system, using one single server (therefore hardware requirement reduces to one server instead of two)
2. By using two domains under one glassfish instance, one can optimize the memory usage on a single server, having lots of RAM at disposal

The server’s hardware configuration i was running before doing this, was something like 8 GB RAM and 4 CPU’S. So i was thinking of using the 8 GB for two JVM’s instead of one. There are several reasons for working with a smaller JVM, first of them being the efficiency of Garbage Collection (other reasons include for example finding entries in large hashmaps, keeping redundant objects shorter in the memory, etc.)

We are currently using JMS for cache synchronization, so i had the option of using Glassfish’s embedded JMS Server (which i did), or a separate, stand alone JMS Server (which i also tried and works perfectly) This did not work out of the box, since there are some configuration items to be taken into consideration.

Ok, so here is the concept:

1. Create, configure and start the first domainwith embedded JMS Server: ports, topics, jms configuration, jvm configuration, mod_jk, thread pools etc.
2. Restart the first domain for configurations to apply
3. Create the second domain
4. Configure the second domain: ports, topics, jms configuration, jvm configuration, mod_jk, thread pools etc.
5. Start the second domain using the first domain as JMS Server

When creating the first domain, there will be some default settings like:

• http listener port: 8080
• https listener port: 8081
• JMS port 7676
• jvm Xmx 512M
• etc.

While this is very convenient if using one domain pro physical server, when creating the second domain, arbitrary ports will be assigned. Therefore you would like to have both domains follow a naming convention. In order to avoid any conflicts, following configuration items have to be kept synchronized (the ports have to be unique) The list contains the default configuration of a freshly set up domain:

Port configuration

<iiop-listener id="orb-listener-1" port="3700" address="0.0.0.0" lazy-init="true" />
<iiop-listener id="SSL" port="3820" address="0.0.0.0" security-enabled="true">
<iiop-listener id="SSL_MUTUALAUTH" port="3920" address="0.0.0.0" security-enabled="true">
<jmx-connector port="8686" address="0.0.0.0" security-enabled="false" name="system" auth-realm-name="admin-realm" />
<jvm-options>-Dosgi.shell.telnet.port=6666</jvm-options>
<network-listener port="8080" protocol="http-listener-1" transport="tcp" name="http-listener-1" thread-pool="http-thread-pool" />
<network-listener port="8181" protocol="http-listener-2" transport="tcp" name="http-listener-2" thread-pool="http-thread-pool" />
<network-listener port="4848" protocol="admin-listener" transport="tcp" name="admin-listener" thread-pool="http-thread-pool" />
<network-listener port="8010" protocol="jk-connector" transport="tcp" name="jk-connector" jk-enabled="true" thread-pool="http-thread-pool" />

I suggest a suffix based naming convention for the ports. Therefore, each configured port should end in the number of the domain. For example:

Domain’s 1 http port would be 8081 instead of 8080. Domain’s 2 port would then be 8082, and so on

This is how it would look like for two domains:

 JMS Configuration

Since the first Domain is running as Master JMS, we need to configure all other domains to use this JMS Server. There will be therefore a different JMS Configuration on the first domain on one hand, and all other domains on the other hand:

Domain 1 JMS  Configuration

<jms-service default-jms-host="default_JMS_host" type="LOCAL">
<jms-host host="0.0.0.0" name="default_JMS_host" />
</jms-service>

Domain 2 JMS Configuration
<jms-service default-jms-host="default_JMS_host" type="REMOTE">
<jms-host host="127.0.0.1" name="default_JMS_host" />
</jms-service>

Creating and configuring Domain 1

There is nothing easier creating a domain in Glassfish. This will create all default options for you to start and get it running. We will configure it directly to use 4841 as administration port, instead of 4848.
asadmin create-domain --adminport 4841 domain1
In order to be able to reconfigure the default settings, we need to start the domain
asadmin start-domain domain1

We will want to do the following:

• delete the default memory settings and re-create them with our configured values
• reset the JMS Service configuration
• create a JMS topic
• reset the default ports to the ones following the naming convention

Reconfigure Memory Settings

asadmin delete-jvm-options "-client:-Xmx512m:-XX\:NewRatio=2:-XX\:MaxPermSize=192m"

Since we do not know the Telnet port of the second domain (it will be arbitrarly assigned), we have to be able to delete this configuration in a dynamic way:

asadmin delete-jvm-options `asadmin get server.* | grep -o '\-Dosgi.shell.telnet.port=[0-9]*'`
Now we can re-create the memory settings, and configure the garbage collection according to our needs. The configuration below deals with a 3.5GB JVM, and a CMS Garbage Collector. More details are available by searching this blog:
asadmin create-jvm-options "-server:-Dosgi\.shell\.telnet\.port=6661:-DjvmRoute=lb1:-XX\:MaxPermSize=512m:-Xmx3550m:-Xms3550m:-XX\:NewSize=1500m:-XX\:MaxNewSize=1500m:-XX\:ParallelGCThreads=2:-XX\:\+UseConcMarkSweepGC:-XX\:\+UseParNewGC:-XX\:SurvivorRatio=3:-XX\:TargetSurvivorRatio=90:-XX\:MaxTenuringThreshold=4:-XX\:\+CMSParallelRemarkEnabled:-XX\:\+CMSPermGenSweepingEnabled:-XX\:\+CMSClassUnloadingEnabled:-XX\:\+PrintGCDetails:-Xloggc\:\${com.sun.aas.instanceRoot}/logs/jgc.log:-Xss128k"
We have now configured the JVM options, and the telnet port. Let's proceed with configuring JMS

Reset the JMS Service Configuration

As mentioned before, the JMS Server is acting as Master. We need to set it as LOCAL Type.

asadmin -p 4841 -W /opt/glassfish/passwords set server.jms-service.type=LOCAL
asadmin -p 4841 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.lazy-init=false
asadmin -p 4841 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.host=0.0.0.0
asadmin -p 4841 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.port=7676


Create a JMS Topic


asadmin -p 4841 -W /opt/glassfish/passwords create-jmsdest --desttype topic "Sample"
asadmin -p 4841 -W /opt/glassfish/passwords create-jms-resource --restype javax.jms.Topic --property Name=Sample "jms/SampleTopic"
asadmin -p 4841 -W /opt/glassfish/passwords create-jms-resource --restype javax.jms.TopicConnectionFactory --property transaction-support=NoTransaction "jms/SampleTopicFactory"

We are now ready for the last step, configuring the remaining ports

Reset the default ports to the ones following the naming convention

Configure the http listeners

asadmin -p 4841 -W /opt/glassfish/passwords set server.network-config.network-listeners.network-listener.http-listener-1.port=8081
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.compression=off
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.compressable-mime-type=text/html,text/xml,text/plain,text/javascript,text/css
asadmin -p 4841 -W /opt/glassfish/passwords set server.network-config.network-listeners.network-listener.http-listener-2.port=8181
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.compression=off
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.compressable-mime-type=text/html,text/xml,text/plain,text/javascript,text/css

Configure the thread pool (default is minimum of 2 and max of 5)

asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=200
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.min-thread-pool-size=100

Configure ModJK Connector (this needs its own listener)

asadmin -p 4841 -W /opt/glassfish/passwords create-http-listener --listenerport 8011 --listeneraddress 0.0.0.0 --defaultvs server jk-connector
asadmin -p 4841 -W /opt/glassfish/passwords set configs.config.server-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true

Configure JMX Connector

asadmin -p 4841 -W /opt/glassfish/passwords set server.admin-service.jmx-connector.system.port=8681

Configure other Domain Ports

asadmin -p 4841 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.SSL.port=3821
asadmin -p 4841 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.SSL_MUTUALAUTH.port=3921
asadmin -p 4841 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.orb-listener-1.port=3701

We are now ready to restart the domain
asadmin -p 4841 -W /opt/glassfish/passwords restart-domain domain1

Creating and configuring Domain 2

asadmin create-domain --adminport 4842 domain2
In order to be able to reconfigure the default settings, we need to start the domain
asadmin start-domain domain2

We will want to do the following:

• delete the default memory settings and re-create them with our configured values
• reset the JMS Service configuration, and configure it to use the Domain 1 JMS Server
• create a JMS topic
• reset the default ports to the ones following the naming convention

Reconfigure Memory Settings

asadmin -p 4842 -W /opt/glassfish/passwords delete-jvm-options "-client:-Xmx512m:-XX\:NewRatio=2:-XX\:MaxPermSize=192m"

Since we do not know the Telnet port of the second domain (it will be arbitrarly assigned), we have to be able to delete this configuration in a dynamic way:

asadmin -p 4842 -W /opt/glassfish/passwords delete-jvm-options `asadmin get server.* | grep -o '\-Dosgi.shell.telnet.port=[0-9]*'`
Now we can re-create the memory settings, and configure the garbage collection according to our needs. The configuration for the second domain has been adapted to a smaller JVM. Watch out how you name your jvmRoute Parameter, since this will be used when configuring the Apache ModJK workers (in this case, lb2 – load balancer 2):
asadmin -p 4842 -W /opt/glassfish/passwords create-jvm-options "-server:-Dosgi\.shell\.telnet\.port=6662 jvmRoute=lb2:-XX\:MaxPermSize=512m:-Xmx2500m:-Xms2500m:-XX\:NewSize=1500m:-XX\:MaxNewSize=1500m:-XX\:ParallelGCThreads=2:-XX\:\+UseConcMarkSweepGC:-XX\:\+UseParNewGC:-XX\:SurvivorRatio=3:-XX\:TargetSurvivorRatio=90:-XX\:MaxTenuringThreshold=4:-XX\:\+CMSParallelRemarkEnabled:-XX\:\+CMSPermGenSweepingEnabled:-XX\:\+CMSClassUnloadingEnabled:-XX\:\+PrintGCDetails:-Xloggc\:\${com.sun.aas.instanceRoot}/logs/jgc.log:-Xss128k"
We have now configured the JVM options, and the telnet port. Let's proceed with configuring JMS

Reset the JMS Service Configuration

As mentioned before, the JMS Server is acting as slave. We need to set it as REMOTE Type.

asadmin -p 4842 -W /opt/glassfish/passwords set server.jms-service.type=REMOTE
asadmin -p 4842 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.lazy-init=false
asadmin -p 4842 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.host=127.0.0.1
asadmin -p 4842 -W /opt/glassfish/passwords set server.jms-service.jms-host.default_JMS_host.port=7676


Create a JMS Topic

The destination has previously been created on domain 1 , so we do not need to create it again (if you try that, you will get an error). We will therefore skip step1, and proceed with creating only the jms resources.

asadmin -p 4842 -W /opt/glassfish/passwords create-jms-resource –restype javax.jms.Topic –property Name=Sample “jms/SampleTopic”
asadmin -p 4842 -W /opt/glassfish/passwords create-jms-resource –restype javax.jms.TopicConnectionFactory –property transaction-support=NoTransaction “jms/SampleTopicFactory”

We are now ready for the last step, configuring the remaining ports

Reset the default ports to the ones following the naming convention

Configure the http listeners

asadmin -p 4842 -W /opt/glassfish/passwords set server.network-config.network-listeners.network-listener.http-listener-1.port=8082
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.compression=off
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.compressable-mime-type=text/html,text/xml,text/plain,text/javascript,text/css
asadmin -p 4842 -W /opt/glassfish/passwords set server.network-config.network-listeners.network-listener.http-listener-2.port=8182
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.compression=off
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.compressable-mime-type=text/html,text/xml,text/plain,text/javascript,text/css

Configure the thread pool (default is minimum of 2 and max of 5)

asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=200
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.min-thread-pool-size=100

Configure ModJK Connector (this needs its own listener)

asadmin -p 4842 -W /opt/glassfish/passwords create-http-listener --listenerport 8012 --listeneraddress 0.0.0.0 --defaultvs server jk-connector
asadmin -p 4842 -W /opt/glassfish/passwords set configs.config.server-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true

Configure JMX Connector

asadmin -p 4842 -W /opt/glassfish/passwords set server.admin-service.jmx-connector.system.port=8682

Configure other Domain Ports

asadmin -p 4842 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.SSL.port=3822
asadmin -p 4842 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.SSL_MUTUALAUTH.port=3922
asadmin -p 4842 -W /opt/glassfish/passwords set server.iiop-service.iiop-listener.orb-listener-1.port=3702

We are now ready to restart the domain
asadmin -p 4842 -W /opt/glassfish/passwords restart-domain domain2

MOD JK Workers Configuration

I suggest configuring your Workers using templates. It will be easier to handle, since you will update the configuration in one single place.

Let’s define a worker template

worker.template.type=ajp13
worker.template.lbfactor=50
worker.template.socket_keepalive=false
worker.template.socket_connect_timeout=300
worker.template.ping_mode=A
worker.template.ping_timeout=1000
worker.template.connection_pool_size=32
worker.template.connection_pool_timeout=600

We can now define our two workers, each linked to one Glassfish Domain. Pay attention to the name of the workers, and the port, which have to be synchronized with the names you defined in the JVM Options of the Domains (jvmRoute Parameter)

Configure worker for Domain 1

worker.lb1.reference=worker.template
worker.lb1.host=glassfish_server
worker.lb1.port=8011
worker.lb1.lbfactor=50
worker.lb1.socket_keepalive=1

Configure worker for Domain 2
worker.lb2.reference=worker.template
worker.lb2.host=glassfish_server
worker.lb2.port=8012
worker.lb2.lbfactor=50
worker.lb2.socket_keepalive=1

Configure the load balancer

worker.lb.type=lb
worker.lb.balance_workers=lb1,lb2
worker.lb.sticky_session=True
worker.lb.sticky_session_force=false
worker.lb.method=B

# list workers visible to apache
worker.list=lb

Domains running – Check processes

You should now have two root processes:

• domain 1 glassfish, with embedded JMS
• domain 2 glassfish

This should look like:

14031 ?        Sl     2:07 /usr/java/jdk1.6.0_23/bin/java -cp /opt/glassfishv3/glassfish/modules/glassfish.jar -XX:+UnlockDiagnosticVMOptions -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:MaxPermSize=51
12394 ?        Sl     3:16 /usr/java/jdk1.6.0_23/bin/java -cp /opt/glassfishv3/glassfish/modules/glassfish.jar -XX:+UnlockDiagnosticVMOptions -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:MaxPermSize=51
12446 ?        S      0:00  \_ /bin/sh /opt/glassfishv3/mq/bin/imqbrokerd -javahome /usr/java/jdk1.6.0_23/jre -Dimq.cluster.nowaitForMasterBroker=true -varhome /opt/glassfishv3/glassfish/domains/domain1/imq -
12480 ?        Sl     0:04      \_ /usr/java/jdk1.6.0_23/jre/bin/java -cp /opt/glassfishv3/mq/bin/../lib/imqbroker.jar:/opt/glassfishv3/mq/bin/../lib/imqutil.jar:/opt/glassfishv3/mq/bin/../lib/jsse.jar:/opt/g

You can now check if the workers are configured properly, by running some http requests over your server. You should see where the request has been directed to:

“GET /home” 200 2601 14984 “https://myserver/” “Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.8)
Gecko/20100722 Firefox/3.6.8″ lb 0.014809 F<lb2 0 OK> L<lb2 0 OK> – the request has been directed to the second server

“GET /home” 200 2601 14984 “https://myserver/&#8221; “Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.8)
Gecko/20100722 Firefox/3.6.8″ lb 0.014809 F<lb1 0 OK> L<lb1 0 OK>- the request has been directed to the first server

Performance Impact

I ran the same test by directing all requests first to the domain 1, and then to domain 2.

Let’s review the JVM and Garbage Collection Strategy for the two domains, and then review the results by comparing response times, and garbage collection statistics.

Domain 1 JVM Configuration – Bigger JVM


<jvm-options>-DjvmRoute=lb1</jvm-options>
<jvm-options>-XX:MaxPermSize=512m</jvm-options>
<jvm-options>-Xmx3550m</jvm-options>
<jvm-options>-Xms3550m</jvm-options>
<jvm-options>-XX:NewSize=1500m</jvm-options>
<jvm-options>-XX:MaxNewSize=1500m</jvm-options>
<jvm-options>-XX:ParallelGCThreads=2</jvm-options>
<jvm-options>-XX:+UseConcMarkSweepGC</jvm-options>
<jvm-options>-XX:+UseParNewGC</jvm-options>
<jvm-options>-XX:SurvivorRatio=3</jvm-options>
<jvm-options>-XX:TargetSurvivorRatio=90</jvm-options>
<jvm-options>-XX:MaxTenuringThreshold=4</jvm-options>
<jvm-options>-XX:+CMSParallelRemarkEnabled</jvm-options>
<jvm-options>-XX:+CMSPermGenSweepingEnabled</jvm-options>
<jvm-options>-XX:+CMSClassUnloadingEnabled</jvm-options>
<jvm-options>-XX:+PrintGCDetails</jvm-options>

Response times for Domain 1

Request	90% response time
Homepage	688
Login	1375
Add transaction	243
Logout	555

Domain 2 JVM Configuration – Smaller JVM


<jvm-options>-DjvmRoute=tap2w2</jvm-options>
<jvm-options>-XX:MaxPermSize=512m</jvm-options>
<jvm-options>-Xmx2500m</jvm-options>
<jvm-options>-Xms2500m</jvm-options>
<jvm-options>-XX:NewSize=1500m</jvm-options>
<jvm-options>-XX:MaxNewSize=1500m</jvm-options>
<jvm-options>-XX:ParallelGCThreads=2</jvm-options>
<jvm-options>-XX:+UseConcMarkSweepGC</jvm-options>
<jvm-options>-XX:+UseParNewGC</jvm-options>
<jvm-options>-XX:SurvivorRatio=3</jvm-options>
<jvm-options>-XX:TargetSurvivorRatio=90</jvm-options>
<jvm-options>-XX:MaxTenuringThreshold=4</jvm-options>
<jvm-options>-XX:+CMSParallelRemarkEnabled</jvm-options>
<jvm-options>-XX:+CMSPermGenSweepingEnabled</jvm-options>
<jvm-options>-XX:+CMSClassUnloadingEnabled</jvm-options>
<jvm-options>-XX:+PrintGCDetails</jvm-options>

Response Times for Domain 2

Request	90% response time
Homepage	338
Login	491
Add transaction	156
Logout	219

Now, let’s do a head to head compare also by adding the number of performed transactions, and the performance improvement from the big domain 1 configuration, to the smaller domain 2 configuration

# of Requests	Request	90% response time		Performance Improvement
		Domain 1	Domain 2
150	Homepage	688	338	103 %
150	Login	1375	491	180 %
15 000	Add transaction	243	156	55 %
150	Logout	555	219	153 %

As you can see, there is a major improvement, from 55 % to 180 %, depending on the business transaction. Having taken all other outside variables into consideration, i can conclude there is definitely place for improvement in a smaller jvm.

Let’s take a look at some garbage collection statistics:

Garbage Collection duration:

Garbage Collection Time Statistics – Head to Head compare

Statistics	Domain 1	Domain 2
Percentage of time in GC	0.217 %	0.098 %
Time spent in Full GC	15.877 s	4.758 s
Percentage of time in Full GC	0.118 %	0.035 %
Average duration Parallel Scavenge	0.16 s	0.113 s

Final

This post was about vertical clustering in glassfish. It can help individuals who, in lack of hardware, can develop and test in a clustered system, using one single Glassfish Instance, and multiple domains. I have reviewed the pros and cons, the things to take care of when configuring multiple domains, and the performance improvement by running some load tests against the two configurations.

I hope you can use this in your deployment scenarios, and that this post will clear any or at least some unanswered question relating to this topic.Feel free to ask me if you still have questions. I will try answering them as soon as possible.

Cheers,

Alex