Archive
Performance Tuning for Apache Worker Model with Glassfish Application Server
There is a lot of material and information on the subject of how to realize the two tier architecture, with the Apache Web Server in front of several application servers, like Tomcat, JBoss, Glassfish and so on.
On the other hand, although the information is there, the number of configuration and fine tuning possibilities are so high, that it takes quite a lot of try and err to sort, understand and implement them.
The aim of this post is to try to take not all, but maybe some hidden, or not so detailed variables in consideration for configuring the web and application server not only for normal workload, but for some extra load, and even for some spikes.
Online businesses, based on high transactional processing systems, need to be able to serve users fast, safe and reliable. It is not enough to be prepared for the normal workload. Real users will always behave differently and, with no exception, unpredictable to a level sometime hard to believe.
The posting deals with the following aspects:
- The concept of backward tuning
This is the starting point for determining the system capacity that we need to handle with the two dimensional web server – application server architecture. We will define key metrics like throughput, memory footprint, maximum number of users, etc. Although the concept is further applicable to the third tier component, the database, this is not part of this post - Building the Apache Workload Model
- Apache configuration and tuning using the worker model
- Glassfish configuration and tuning for the Workers Model
1. BACKWARD TUNING
- First determine how much and at what costs your last tier component – the application server – can handle.
- Build your user model based on throughput and estimations, and limit any more than that entering your system.
- Synchronize the first tier component – the web server – with the application server so that only the designated number of users can use the system
- Start by determining the maximum capacity and throughput of your application server
- Determine how many concurrent users can you handle on your application server, and what is their memory footprint (memory used by one user session for the duration of using your solution)
- Determine the number of connections that you need to maintain open on the web server side in order to accommodate all users above
- Tune the corresponding thread pools (web server and application server) accordingly, so that only a determined number of connections can be opened on the web server, and that only a determined number of users can make it to the application server
- Determine the average number of business requests a user will execute on your system
- Tune the backlog of the web server so that you can prepare for spikes, and that you can determine when to send the surplus users to another web server or waiting server
- Tune the operating systems for handling the configured number of connections – network, memory, open files, etc.
By now, you should have gone through the process of load and performance testing of your application. After this step, you should be in the position of knowing what resources you would need to reach maximum throughput. You should therefore also have configured and tuned your HEAP and decided on the Garbage Collection strategy. These are critical aspects that are derived after testing your application in an isolated one tier architecture, the application server.
These are the statistics that we need:
- Number of transactions / second with a CPU load of max 80 % ( we need to leave an additional 20 % for workload spikes) that your application server can process
- Average think time between transactions how do users interact with your application? What is the average think time before firing the next transaction. Together with the “number of transactions / second” this will give you the maximum number of “business users” / application server
- Transaction Distribution Model: percentage of users executing business transactions vs “just visitors”, users just surfing your web page. How many of your visitors are just surfing, and how many are calling business requests? The distribution model will tell you how many resources you will need, both on the web server and the application server
- User Memory Footprint: how much heap memory does a user executing transactions (to be called registered user) occupy vs a user just visiting the page – in other words, business session vs anonymous session. Depending on this information, you allow a maximum number of users to enter your system
- Transaction model: how are the business transactions being executed, how many HTTP connections are being opened for executing business transactions. Is your application an interactive application or a static resource intensive application? Typical browsers these days try to serve the content to the users as soon as possible, therefore opening several persistent connections to the web server at once (Firefox for example opens between 6 and 15 connections) How many connections are you using when triggering a business request? 1,2 or all 6 of them?
- Long and very long loading and response times – nowadays any webpage loading in more than 3 to 5 seconds is annoying. Dirk, an ex colleague of mine was telling me that an increase of 0.1 seconds in the loading time of Amazon costs a fortune…
- White pages and never loading page – This is even worse. A web page loading very slowly is still a loading web page. Yet, any new user landing on your product page, and experiencing a white page as his first experience will unlikely come again very soon.
- be processed immediately
- wait for a free working thread, and be processed with an acceptable delay
- be redirected to another web server
- be redirected to a wait server
Application Server User Memory footprint
- memory required / registered user: 1.5 MB
- memory required / unregistered user (simple visitor): 0.5 MB
Application Server Throughput
Application Server Workload model
- A business transaction is executed by a user every approximately 3 seconds
- The average response time for a transaction is between half a second and a second
- Worse case scenario (is a strategy I often use): x users / (3 seconds think time + 1 second response time) = 600 transactions =>> x = 2400 concurrent “business” users (users executing business transactions)
- Light transactions (like users just visiting your webpage) are executed in a rate of 30 users / seconds, so that after one minute, you will have 60 seconds * 30 users = 1800 anonymous users
- 1800 anonymous users, needing 1800*0.5 MB= 900 MB
- 2400 registered users, needing 2700*1.5 MB = 3600 MB
Application Server HEAP structuring
Application Server Transaction Model
- business transactions – executing some work in the database, persisting/modifying specific information, executing application logic with a result that is to be persisted
- light transactions – users just surfing on your page, not actually producing any workload on your database
- business transactions are being processed using one single persistent connection ( in other words, keep alive connection). That means that several business transactions of the same user will use the one and same keep alive connection (of course, if such thing as keep alive is configured and activated)
- light transactions are being processed using between 6 and 15 persistent connections (depending on the browser, FF uses for example between 6 and 15)
- total number of connections needed: 2400 registered users * 1 persistent connection + 1800 anonymous users * 6 persistent connections =13200 connections on the web server.
2. Building the Apache Web Server Model
- It costs less to spawn a large number of threads than to spawn new processes, as in the prefork model
- The memory is shared between all threads belonging to a process
- The number of threads can be dynamically increased using graceful restart of the web server
- The number of threads used in the application server is somehow compressed, meaning that for each 2, 3, 4 apache threads, there will be one application server thread doing the application business. The application server thread will be at some point shared between web server threads. This way you can define a larger number of apache threads, with a lower number of application server threads
Apache Worker Model - Request Workflow
- how long will these users hang on this connection?
- what if the user sends one request, and than does not send any additional request? When will the connection be closed and the thread returned to the pool?
- what if the user opens the connection and regularly sends requests forever, keeping the connection open forever (Denial of service)
- KeepAlive (do we allow persistent connections at all)
- KeepAliveTimeout (how long should we wait on the client sending an additional request over the connection before closing the connection and returning the thread to the thread pool)
- MaxKeepAliveRequests (how many requests is a client allowed to send over one persistent connection before the server closes the connection and returns the thread to the thread pool)
- If one user sends one request over a persistent connection, and then waits 59 seconds before sending the next request, he will reuse the same connection, and no other connection (apache thread) will be created. But:
- If one user sends a request over a persistent connection, and then waits longer than 60 seconds before sending the next one, or even worse, will never send an additional request, the connection will be kept open with no reason, and will be dropped without serving any more than one request. Not that optimal, is it?
Using this configuration and a time-scaled representation, after 60 seconds you would have the following scenario:
Apache Connections and Keep Alive
- 60 % (18 users) will continue just executing business requests, needing one single persistent connection / user
- 40 % (12 users) will continue just surfing, needing 6 persistent connection / user
Apache Connections Keep Alive
Compared to the first model, where we had to maintain 10 800 connections, you now have to only maintain 6840 connections!
But, one target was to be also prepared for spikes. What if we have more users / second than 30 ? What do we do with them, how do we react to that, and how do we prevent the scenarios described in the beginning (white page, long waiting times, etc.)
Well, since we know we can accommodate 1800 users, using the 6840 connections, we can prepare for an extra 50 % workload. That would mean we would get to about 10 000 connections (6840 + 3420) I will just use 10 000 for simplicity.
Right. But…what will happen with the connection request 10001 ?
Since we know that we have a loading time of about 6 seconds, no user should wait longer than 3 seconds to get a connection (so that the total response time still falls under 10 seconds)
With a workload of 30 users / second, we can accommodate:
( 30 users * 6 connections * 3 seconds ) requests in 3 seconds= 540 requests
But we just spoke of spikes, so let us increase the backlog with 50 % to 810 connections.
That means:
- First 10000 requests will get their threads
- requests between 10001 and 10810 will wait in the backlog for a free connection
- request 10811 will be either sent to a wait server or to another scaled web server
Tuning Apache Workers for Load - The connection and backlog model
Now that we decided on what we want to handle, let’s get our servers ready for this.
3. Configuring and tuning APACHE using Workers for performance and high number of connections
We need to take care both of APACHE configuration, and of configuration of the operating system where Apache resides
APACHE WebServer Configuration
We are ready to configure APACHE for 10 000 connections. Since it is more efficient to spawn threads than processes, let’s use a higher number of threads per process.
This is how a configuration could look like. With this configuration we will:
- start with 25 servers, each spawning 200 threads, resulting 5000 threads
- use about 50 k / thread, resulting in about 500 MB reserved memory
- expand to maximum 50 servers, each spawning 200 threads, resulting 10000 threads. This is also the value used by APACHE as MaxClients (maximum number of connections)
- start shrinking the number of threads once the load has decreased, and keep no less than 7500 threads in the pool
- allow no more than 100 000 requests / process, which results in an average of 500 requests / child thread. We use this in order to avoid nasty users holding to the process forever, and we also avoid memory leaks, by destroying the process after the number of requests has been reached
<IfModule worker.c> # initial number of server processes to start # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers StartServers 25 # highest possible MaxClients setting for the lifetime of the Apache process. # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit ServerLimit 50 # minimum number of worker threads which are kept spare # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#minsparethreads MinSpareThreads 1000 # maximum number of worker threads which are kept spare # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxsparethreads MaxSpareThreads 7500 # upper limit on the configurable number of threads per child process # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadlimit ThreadLimit 200 # maximum number of simultaneous client connections # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients MaxClients 10000 # number of worker threads created by each child process # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadsperchild ThreadsPerChild 200 # maximum number of requests a server process serves # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild MaxRequestsPerChild 150 </IfModule> KeepAlive On MaxKeepAliveRequests 1000 KeepAliveTimeout 10 ListenBackLog 810
A couple of things to consider:
- MaxClients: If the allocated MaxClients is higher than ServerLimit * ThreadLimit Apache will automatically reduce MaxClients to the value of ServerLimit * ThreadLimit
In this case, you will receive the following message:WARNING: MaxClients of 10000 would require 200 servers, and would exceed the ServerLimit value of 100. Automatically lowering MaxClients to 5000. To increase, please see the ServerLimit directive.
- Apache Memory / Process and Threads: Check if you have enough memory to handle the configured number of threads. Use the following scripts to monitor the Apache Servers, the number of threads, and the memory reserved:
- The formula used for configuring your maximum number of clients is:
maxclients = total ram / ram per process - List all Apache processes and number of threads per process
for pid in `ps U wwwrun | grep httpd | grep -v grep | awk '{ print $1 }'`;
do echo Apache Worker Server $pid has `ps ms -p $pid | wc -l` threads;
done
This will output:
Apache Worker Server 7528 has 3 threads
Apache Worker Server 7583 has 3 threads
Apache Worker Server 7587 has 3 threads
Apache Worker Server 7596 has 204 threads
Apache Worker Server 7601 has 204 threads
Apache Worker Server 7610 has 204 threads
Apache Worker Server 7618 has 204 threads
Apache Worker Server 7628 has 204 threads
Apache Worker Server 7640 has 204 threads
Apache Worker Server 7651 has 204 threads
Apache Worker Server 7668 has 204 threads
............ - List the reserved memory per APACHE Server and threads
servers=0;threads=0;space=0;
for pid in `ps U wwwrun | grep httpd | grep -v grep | awk '{ print $1 }'`;
do process_threads=`ps ms -p $pid | wc -l`;
process_memory=`ps -ylC -p $pid | grep -v PID | awk '{ print $8}'`;
echo Apache Worker Server with pid $pid has $process_threads threads occupying $process_memory bytes;
servers=`expr $servers + 1`;threads=`expr $process_threads + $threads`;space=`expr $process_memory + $space`; done;
echo "-------------"; echo Total Apache Servers: $servers \| Total threads: $threads \| Total memory reserved: $space
This will output:
.......
Apache Worker Server with pid 1733 has 204 threads occupying 34644 bytes
Apache Worker Server with pid 1759 has 204 threads occupying 34612 bytes
Apache Worker Server with pid 1788 has 204 threads occupying 34620 bytes
Apache Worker Server with pid 1816 has 204 threads occupying 34648 bytes
Apache Worker Server with pid 1844 has 204 threads occupying 34648 bytes
Apache Worker Server with pid 1875 has 204 threads occupying 34648 bytes
Apache Worker Server with pid 1907 has 204 threads occupying 34684 bytes
Apache Worker Server with pid 1940 has 204 threads occupying 34676 bytes
-------------
Total Apache Servers: 26 | Total threads: 5103 | Total memory reserved: 885272
- The formula used for configuring your maximum number of clients is:
- ThreadsPerchild vs ThreadLimit: There is a difference between ThreadsPerchild and ThreadLimit. ThreadsPerChild defines the initial number of threads spawned per worker process. Defining a larger number as ThreadLimit allows you to modify the ThreadsPerChild up to ThreadLimit, without needing to do a hard restart of Apache. A graceful restart will suffice.For example:
ThreadsPerChild 100 ThreadLimit 500
This will start the apache processes with 100 threads each. If you feel that your webserver cannot handle the current load with the current configuration, you just need to increase the ThreadsPerChild, (in this case to maximum 500) and execute a graceful restart:
ThreadsPerChild 300 ThreadLimit 500
Yet, it is very important to remember that:
- ThreadLimit will allocate memory in advance, so check in advance if you can handle the ServerLimit * ThreadsPerChild with your current RAM
- If you increase the ThreadsPerChild up to a value that the APACHE Server will consider it cannot allocate the needed memory, APACHE will exit!
- ServerLimit vs ThreadLimit: You cannot increase the maximum number of server processes ServerLimit without hard restart of APACHE, so take this into consideration when planning the baseline configuration. If you want to be prepared for a higher load, configure the ThreadLimit higher, which will allow you to increase the capacity of your server with a graceful restart
- Number of Apache Threads spawned: Make sure that the number of threads spawned is the one you expect. Use the following to check the number of threads spawned.”Sl” is for Interruptable sleep, multithreaded
ps axsm | grep -c 'Sl'
- Denial of Service and Keep Alive Parameter: Be prepared for Denial of Service Attacks. Do not allow any user to hold on to a persistent connection forever. Decide on the average number of transactions that a user will execute while using your system, and add it as parameter:
KeepAlive On MaxKeepAliveRequests 1000 KeepAliveTimeout 10
- Backlog for extra spikes:Be prepared for those extra spikes, where all your threads are busy, and you need a “waiting room” for your users. Use the ListenBackLog Directive, which we configured to 810:
ListenBackLog 810
- ONE OF THE MOST IMPORTANT:
The only way you can control the life of an APACHE server, and its children, is using this two parameters. Set them too high, and they will never die, therefore, the number of Threads in your Application Server will increase continously, eventually reaching the maximum configured number of threads, where it will stop servicing requests.
Now, there are two parameters that control the behaviour of the apache servers and their children, and therefore the behaviour of the thread pool on the application server side. MaxRequestsPerChild refers to the maximum number of NEW requests (meaning a new client, opening a completely new Keep Alive Connection on this child) that can be sent over one Apache child. If for example, you had one client, opening a connection, and sending 500 requests, one after the other, on the same keep alive connection, than this parameter would have the value 1 (only the first request over a new keep alive connection will be counted) On the other hand, you do not want one client to hold on forever on his connection (remember the DOS), so you control that by setting the maximum, total number of requests a client can send over a SINGLE keep alive connection. When one of the two happens, the client will be assigned a new child thread, and the thread having served will be marked as “dead”MaxRequestsPerChild 150 MaxKeepAliveRequests 1000
APACHE Operating System Configuration
Apache Tuning – Max Open Files
Since every socket in Unix is actually a file, we need to tune the maximum allowed number of open files.
This specifies the number of open files that are supported. The default setting is typically sufficient for most applications. If the value set for this parameter is too low, a file open error, memory allocation failure, or connection establishment error might be displayed.
More on this here: IBM Websphere Linux Tuning
Apache Tuning – TCP Settings
Keep in mind the seven layer architecture (also known as OSI model) Both Glassfish and Apache control the transport layer, which sits further on the network layer.
OSI Model - Seven layer architecture
Image courtesy: Novell - http://www.novell.com/info/primer/prim05.html
You can configure and limit GLASSFISH and APACHE resources, as long as you do not exceed the network layer settings. That is why we need to take that also into consideration, and tune them accordingly, so that we can plug in the Glassfish or Apache without exceeding any configured resources
Apache Tuning – Backlog and Maximum Connections
Change the following parameters when you prepare for a high rate of incoming connections. Keep in mind that this setting is shared between all Apache Servers, so that if you want to host several web servers, you need to be prepared to accommodate the connections for all servers:
- Number of connections in backlog: cat /proc/sys/net/core/netdev_max_backlog Default is 1000, modify this according to the maximum expected number of connections that you want to accommodate in your backlog (keep them waiting for a connection)
- Size of the listen queue, in bytes, for accepting new tcp connections: cat /proc/sys/net/core/somaxconn Default is 128, modify this according to the size of the backlog you would need for the maximum expected number of connections that you want to accommodate in your backlog
Apache Tuning – Operating System – Keep Alive Settings
Although Unix has built-in support for KeepAlive, this is not the default behavior in Linux. Programs must request keepalive control for their sockets using the setsockoptinterface. http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.htmlThere are three “tunable” parameters:
- tcp_keepalive_time
- the interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe; after the connection is marked to need keepalive, this counter is not used any further
- tcp_keepalive_intvl
- the interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime
- tcp_keepalive_probes
- the number of unacknowledged probes to send before considering the connection dead and notifying the application layer
As long as the keep alive is high enough (higher than your configured Apache Server) you should not worry, since Apache will take care of managing the connections. Just make sure that the settings are higher than the ones configured in your web and application server. You can check all three by issuing the following command (i have also listed the default values for SUSE Linux Enterprise Server 11 ):
- cat /proc/sys/net/ipv4/tcp_keepalive_intvl : 75
- cat /proc/sys/net/ipv4/tcp_keepalive_probes: 9
- cat /proc/sys/net/ipv4/tcp_keepalive_time: 7200 (seconds)
Apache Tuning – KEEP ALIVE SETTINGS
As defined above, these are the three configuration parameters you’d have to take care of when configuring for keep-alive connections (repeated, see above, chapter 2):
- KeepAlive (do we allow persistent connections at all)
- KeepAliveTimeout (how long should we wait on the client sending an additional request over the connection before closing the connection and returning the thread to the thread pool)
- MaxKeepAliveRequests (how many requests is a client allowed to send over one persistent connection before the server closes the connection and returns the thread to the thread pool)
Apache Tuning – Operating System – CONNECTION MANAGEMENT
tcp_fin_timeout:
This basically holds a connection in a TIME_WAIT mode (after the keep alive timeout expired), waiting for the same client to reinitiate the communication. Basically it says that it is cheaper to reactivate a sleeping connection, than to build a new one. Maintaining a connection open, after the user has not sent any additional request in the KeepAliveTimeout timeout defined may be expensive, since other users may just well sit in the backlog, waiting for exactly that one connection to be freed. If this is the case, you should set this to a level as low as possible, so that after a small timeout, the connection can be finally destroyed, and rebuilt with another id, for another user.
“This determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. This interval between closure and release is known as the TIME_WAIT state or twice the maximum segment lifetime (2MSL) state. During this time, reopening the connection to the client and server cost less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, providing more resources for new connections. Adjust this parameter if the running application requires rapid release, the creation of new connections, and a low throughput due to many connections sitting in the TIME_WAIT state.”
More on this here: IBM WebSphere Tuning
A simple way to see how many connections you have in TIME_WAIT is using netstat:
watch --interval=1 "netstat | grep -c TIME_WAIT"
You can view and set this parameter as follows:
- View:
cat /proc/sys/net/ipv4/tcp_fin_timeout: 60 (default on SUSE Linux Enterprise Server 11) - Set: echo 10 > /proc/sys/net/ipv4/tcp_fin_timeout
Apache Tuning – Compression
Most modern browsers can handle compressed content, decompressing it upon receiving it. It is a great idea to compress content, which will save you a lot of bandwidth. Of course, this comes with a price – CPU is needed for compressing the files. Since you do not want your application server to use resources for compression, leave it to the web server. And since Apache has caching mechanisms, this should not be a problem, since the file will be zipped once, and then saved and served in/from cache
Most compression algorithms, when applied to a plain-text file, can reduce its size by 70% or more, depending on the content in the file. When using compression algorithms, the difference between standard and maximum compression levels is small, especially when you consider the extra CPU time necessary to process these extra compression passes. This is quite important when dynamically compressing Web content. Most software content compression techniques use a compression level of 6 (out of 9 levels) to conserve CPU cycles. The file size difference between level 6 and level 9 is usually so small as to be not worth the extra time involved.
http://www.linuxjournal.com/article/6802
There are two modules for using compression in Apache:
- mod_gzip
- mod_deflate
- type of files you need to compress (it does not make sense compressing already compressed files like pdf, jpg, etc.):
AddOutputFilterByType DEFLATE text/html text/plain text/xml
Apache Tuning – Caching
Related to the point above, you can benefit of both compressing and caching content. You can either use a disk based cache store manager or a faster, memory based store manager
mod_disk_cache-
implements a disk based storage manager.
implements a memory based storage manager.
mod_mem_cachecan be configured to operate in two modes: caching open file descriptors or caching objects in heap storage.mod_mem_cachecan be used to cache locally generated content or to cache backend server content formod_proxywhen configured usingProxyPass(aka reverse proxy)
More on this here: http://httpd.apache.org/docs/2.2/mod/mod_cache.html
4. Configuring and tuning the Glassfish Application Server to work with the Workers Model
One thing you need to understand is that in the worker model, APACHE controls the lifecycle of working threads in Glassfish. Therefore, we need to configure Glassfish knowing that APACHE takes care of things like:
- thread removal
- keep alive timeouts
- maximum number of requests / connection
- compression
- etc
Most UNIX distributions come with an out of the box configuration, that is prepared for normal workload, but definitely not for performance. We need to be aware of network, memory and other system resource settings that we have to reconfigure in order to gain and prepare for performance.
Glassfish Tuning
Glassfish Tuning – Max Open Files
Once again, since every socket in Unix is actually a file, we need to tune the maximum allowed number of open files.
You can of course set it to unlimited, but since we spoke of a APACHE Thread Pool -> Glassfish Thread Pool compression, i would set this to half the number of threads apache can spawn + extra 5000 for all other processes
ulimit -n 10000
Yet, be aware that if you have multiple Glassfish Domains Application Server, they will all share the number of open files. Keep this in mind when calculating the number of possible open threads / server.
Glassfish Tuning – Max Threads
Since your system can perform up to about 600 transactions / second, you should configure your maximum thread pool according to this number. Allow about 50 % more threads for those extra spikes, even if that comes with longer response times.
I would therefore set the maximum thread pool size to about (rounded) 1000:
<thread-pool max-thread-pool-size="1024" name="http-thread-pool" />
Glassfish Tuning – TCP Settings
Keep in mind the seven layer architecture (also known as OSI model) Both Glassfish and Apache control the transport layer, which sits further on the network layer (see above, in chapter 3, Apache Tuning – TCP Settings)
You can configure and limit GLASSFISH and APACHE resources, as long as you do not exceed the network layer settings. That is why we need to take that also into consideration, and tune them accordingly, so that we can plug in the Glassfish or Apache without exceeding any configured resources
Glassfish Tuning – Backlog and Maximum Connections
Change the following parameters when you prepare for a high rate of incoming connections. Keep in mind that this setting is shared between all Glassfish Servers respectively Apache Servers:
- Number of connections in backlog:
/proc/sys/net/core/netdev_max_backlog
Default is 1000, modify this according to the maximum expected number of connections that you want to accommodate in your backlog (keep them waiting for a connection). Remember that these connections are the ones opened by APACHE, so you may want to tune this according to the maximum number of threads you will allow Apache to start inside the application server - Size of the listen queue, in bytes, for accepting new tcp connections:
/proc/sys/net/core/somaxconn
Default is 128, modify this according to the size of the backlog you would need for the maximum expected number of connections that you want to accommodate in your backlog
Glassfish Tuning – Operating System – Keep Alive Settings
Although Unix has built-in support for KeepAlive, this is not the default behavior in Linux. Programs must request keepalive control for their sockets using the setsockoptinterface.
http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html
There are three “tunable” parameters:
- tcp_keepalive_time
- the interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe; after the connection is marked to need keepalive, this counter is not used any further
- tcp_keepalive_intvl
- the interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime
- tcp_keepalive_probes
- the number of unacknowledged probes to send before considering the connection dead and notifying the application layer
As long as the keep alive is high enough (higher than your configured Glassfish or Apache Server) you should not worry, since Apache or Glassfish will take care of managing the connections. Just make sure that the settings are higher than the ones configured in your web and application server.
You can check all three by issuing the following command:
- cat /proc/sys/net/ipv4/tcp_keepalive_intvl
- cat /proc/sys/net/ipv4/tcp_keepalive_probes
- cat /proc/sys/net/ipv4/tcp_keepalive_time
Glassfish Tuning – KEEP ALIVE SETTINGS
As mentioned above, Apache and Glassfish regulates this and other options at the protocol level. Interesting at this point are:
- timeout-seconds: this is the time that GLASSFISH waits for a new request, before closing the connection. Since the internal thread behavior will be controlled by APACHE (it is APACHE’s thread after all), you should set it to unlimited, “-1″ (default ist 30)
- max-connections: this is the maximum number of requests a client can send over a connection before GLASSFISH closes the connection. Since the internal thread behavior will be controlled by APACHE (it is APACHE’s thread after all), you should set it to unlimited, “-1″, and let APACHE clean the opened threads (the ones opened in Glassfish) and connections by itself (default is 256)
<protocol name="http-protocol"> <http xpowered-by="false" timeout-seconds="-1" max-connections="-1" default-virtual-server="server" compressable-mime-type="text/html, text/xml,text/plain,text/javascript,text/css" compression="on" server-name=""> <file-cache enabled="false" /> </http> </protocol>
Glassfish Tuning – Operating System – CONNECTION MANAGEMENT
tcp_fin_timeout:
If maintaining a connection for you is expensive, you should reconfigure this parameter. This basically holds a connection in a TIME_WAIT mode (after the keep alive timeout expired), waiting for the same client to reinitiate the communication. Basically it says that it is cheaper to reactivate a sleeping connection, than to build a new one.
A simple way to see how many connections you have in TIME_WAIT is using netstat:
watch –interval=1 “netstat | grep -c TIME_WAIT”
“This determinesthe time that must elapse before TCP/IP can release a closed connection and reuse its resources. This interval between closure and release is known as the TIME_WAIT state or twice the maximum segment lifetime (2MSL) state. During this time, reopening the connection to the client and server cost less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, providing more resources for new connections. Adjust this parameter if the running application requires rapid release, the creation of new connections, and a low throughput due to many connections sitting in the TIME_WAIT state.”
More on this here:
You can view and set this parameter as follows:
View: cat /proc/sys/net/ipv4/tcp_fin_timeout
Set: echo 10 > /proc/sys/net/ipv4/tcp_fin_timeout
Glassfish Tuning – Compression
Most modern browsers can handle compressed content, decompressing it upon receiving it. It is a great idea to compress content, which will save you a lot of bandwidth.
Of course, this comes with a price – CPU is needed for compressing the files. Since you do not want your application server to use resources for compression, leave it to the web server. I have detailed compression in the web server in chapter 3, Configuring Apache.
If you still think that compression is a good idea, or you just want to play with it, these are the parameters you can change
compression-min-size-bytes – defines the minimum size of files where compression will be applied. Everything equal or greater than this will be compressed and sent compressed to the client.
compressable-mime-type – defines which extensions will be taken into consideration for compressing
compression – sets the compression on or off
<protocol name="http-protocol"> <http ......compression-min-size-bytes="4096"compressable-mime-type="text/html, text/xml,text/plain,text/javascript,text/css" compression="on" server-name=""> <file-cache enabled="false" /> </http> </protocol>
Glassfish Tuning – Chunking
No point reinventing the wheel…:) I will just quote wiki on that :
” Chunked transfer encoding is a data transfer mechanism in version 1.1 of the Hypertext Transfer Protocol (HTTP) in which a web server serves content in a series of chunks. It uses the Transfer-Encoding HTTP response header in place of the Content-Length header, which the protocol would otherwise require. Because the Content-Length header is not used, the server does not need to know the length of the content before it starts transmitting a response to the client (usually a web browser). Web servers can begin transmitting responses with dynamically-generated content before knowing the total size of that content.
The size of each chunk is sent right before the chunk itself so that a client can tell when it has finished receiving data for that chunk. The data transfer is terminated by a final chunk of length zero.
The introduction of chunked encoding into HTTP 1.1 provided a number of benefits:
- Chunked transfer encoding allows a server to maintain a HTTP persistent connection for dynamically generated content. Normally, persistent connections require the server to send a Content-Length field in the header before starting to send the entity body, but for dynamically generated content this is usually not known before the content is created.[1]
- Chunked encoding allows the sender to send additional header fields after the message body. This is important in cases where values of a field cannot be known until the content has been produced such as when the content of the message must be digitally signed. Without chunked encoding, the sender would have to buffer the content until it was complete in order to calculate a field value and send it before the content.
- HTTP servers sometimes use compression (gzip) or deflate methods to optimize transmission. Chunked transfer encoding can be used to delimit parts of the compressed object. In this case the chunks are not individually compressed. Instead, the complete payload is compressed and the output of the compression process is chunk encoded. In the case of compression, chunked encoding has the benefit that the compression can be performed on the fly while the data is delivered, as opposed to completing the compression process beforehand to determine the final size.”
Why disable a great feature? Chunking comes enabled with Glassfish. Leave it as is.
Logical renaming of thread pools for better resource management and monitoring
The default configuration of Glassfish comes with three configured listeners:
- http-listener-1 – for http requests
- http-listener-2 – for https secure requests
- admin-listener – for admin purposes (opening the web administration console)
The problem is that all of these listeners are configured to work on a single thread pool, thread-pool-1:
<network-listener port="8080" protocol="http-listener-1" transport="tcp" name="http-listener-1" thread-pool="http-thread-pool" /> <network-listener port="8181" protocol="http-listener-2" transport="tcp" name="http-listener-2" thread-pool="http-thread-pool" /> <network-listener port="4848" protocol="admin-listener" transport="tcp" name="admin-listener" thread-pool="http-thread-pool" />
This is not that optimal… If you find yourselve in the position that all threads of a thread pool are busy, you cannot even log in to your administration console.I do not agree to this “shared thread pool” model, so i recommend the logical organization and splitting of listeners, thread pools and protocols. This enables also a better monitoring of the threads, since each of them will appear with it’s own name, as in the example below:
And this is how the configuration would look like in the domain.xml file:
<network-config>
<protocols>
<protocol name=”http-protocol”>
<http xpowered-by=”false” max-connections=”-1″ default-virtual-server=”server” compressable-mime-type=”text/html,text/xml,text/plain,text/javascript,text/css” server-name=”"><file-cache enabled=”false” />
</http>
</protocol>
<protocol security-enabled=”true” name=”secure-protocol”>
<http xpowered-by=”false” default-virtual-server=”server” compressable-mime-type=”text/html,text/xml,text/plain,text/javascript,text/css” server-name=”">
<file-cache enabled=”false” />
</http>
<ssl ssl3-enabled=”false” cert-nickname=”s1as” />
</protocol>
<protocol name=”admin-protocol”>
<http default-virtual-server=”__asadmin” server-name=”">
<file-cache enabled=”false” />
</http>
</protocol>
<protocol name=”jk-protocol”>
<http xpowered-by=”false” max-connections=”-1″ default-virtual-server=”server” compressable-mime-type=”text/html,text/xml,text/plain,text/javascript,text/css” server-name=”">
<file-cache enabled=”false” />
</http>
</protocol>
</protocols>
<network-listeners>
<network-listener port=”8080″ protocol=”http-protocol” transport=”tcp” name=”http-listener” thread-pool=”http-thread-pool” />
<network-listener port=”8081″ protocol=”secure-protocol” transport=”tcp” name=”secure-listener” thread-pool=”secure-thread-pool” />
<network-listener port=”4848″ protocol=”admin-protocol” transport=”tcp” name=”admin-listener” thread-pool=”admin-thread-pool” />
<network-listener port=”8009″ protocol=”jk-protocol” transport=”tcp” name=”jk-main-listener-1″ jk-enabled=”true” thread-pool=”jk-main-thread-pool1″ />
</network-listeners>
<transports>
<transport name=”tcp” />
</transports>
</network-config>
<thread-pools>
<thread-pool max-thread-pool-size=”10″ name=”http-thread-pool” />
<thread-pool max-thread-pool-size=”10″ name=”admin-thread-pool” />
<thread-pool max-thread-pool-size=”50″ name=”secure-thread-pool” />
<thread-pool max-thread-pool-size=”1024″ name=”jk-main-thread-pool-1″ />
<thread-pool max-thread-pool-size=”200″ name=”thread-pool-1″ />
</thread-pools>
Conclusion
There are of course a lot of other options and “tunable” parameters. I have not tried to cover all of them, since this would be out of my power, knowledge, and before of all, time. It is probably the blog post for which i have been working most, so i would be delighted if this will be used/tried/implemented by some of you.
I would be also amazingly happy to receive remarks/critics or comments, since this is the result of a lot of brainstorming and try/err experiments.
I will update this blog post as i go along and find new and interesting things that have to be taken care of when configuring Apache and Glassfish for performance.
Thanks to Niels and Mario for their creativity, flexibility and agility. Together we are strong! 
All best,
Alex
Java EE – Load & Performance – Testing and reporting using open source tools – Part 2 – Configuring Load Generators – Generating load with JMeter, storing, filtering and preparing results for importing
CONFIGURING LOAD GENERATORS – JMETER
.
Load testing generators – Getting your application under load with distributed JMeter
- CONFIGURING JMETER DISTRIBUTED MODE
JMeter is the number one open source tool for generating load. It simulates multiple virtual users, by running the test for each virtual user in a separate thread. It is built on Java Platform, hence it provides high portability. It provides a straight-forward graphical interface for designing and running tests, which turns to be very helpful for debugging purposes. Additionally, it allows running testplans in command line interface, eliminating therefore the resource consumption overhead of the GUI.Configured in Distributed Mode, JMeter allows using multiple instances of JMeter as agents (slaves), using the RMI protocol and configurable ports. Thank to this, any new JMeter Agent can be “plugged in”at any time by simply starting the agent, and adding it to the list of slaves.It is of course best to keep your JMeter environment in a separate subnetwork, in order to avoid any outer factors like network delays due to traffic in the main network. I suggest using the JMeter Server as a gateway to the JMeter load subnetwork, so that the JMeter server has two network interfaces:- One for communicating with the main network, where the database and reporting server will reside
- One for communicating and controlling the load on the JMeter Agents residing in the JMeter load subnetworkA simple graphical representation of this structure looks like below:
- JMeter Distributed Testing Example
- .
Now, JMeter is a very flexible solution, and it is designed to get your tests up and running in no time. In order to run in in distributed mode, one of your servers has to be the controller. I suggest keeping this server as a controller only, and not using it also for generating load, but for collecting results, starting transformations, running scripts,etc. Think of it like a “Test Commander“
There are two steps that have to be performed in order to get your distributed environment running:
- Configure the JMeter Master Controller
This is quite simple. You need to modify one single property file, residing in the bin folder of your JMeter installation folder. The file to be modified is “jmeter.properties“
Look for the following block:
#---------------------------------------------------------------------------
# Remote hosts and RMI configuration
#---------------------------------------------------------------------------
# Remote Hosts - comma delimited
remote_hosts=127.0.0.1
#remote_hosts:localhost:1099,localhost:2010This is where you have to add your JMeter servers, as remote_hosts, separated by comma, like this:remote_hosts=192.168.1.2,192.168.1.3,192.168.1.4
I do suggest adding the JMeter agents as a list of IP’s instead of hostnames, to avoid any DNS problems due to faulty network configurations.- Configure the JMeter slaves
The only thing left to do is starting the JMeter agents in “server mode”. I suggest creating a service in your startup folder (etc/init.d or a windows service) so that the JMeter agent starts up at every server restart.
In order to start JMeter in server mode you have to run the following file, residing in the same “bin” folder of the JMeter installation folder on the JMeter agents:
bin/jmeter-server - GENERATING LOAD AND STORING THE RESULTS
The solution to which this series or articles is referring to is using XML as test results output. There are two reasons behind choosing XML for output:
- Preparing the test results for data transformation
XML is preferred because of the simple xpath extractor. It will get very handy when we want to group and export additional information like test date, build number, software version, etc.
- Preparing for the Hudson JMeter plugin which expects xml files for reporting
The Hudson plugin comes very handy when analyzing trends between several runs over different software versions. It expects an xml file which it will parse and build the performance report and trend. The output looks like this:
JMeter Hudson Performance Plugin
.
.
We will use one JMeter Master Controller called “master” and three slave agents that we will call “agent1″, “agent2″, “agent3″. The JMeter Master Controller is the server that we will use as test controller. It is the server where we store the test plans, test scenarios, function files, test results. It is the location which will hold the entire file structure needed for running a load test. It is the “Test commander”, and will only be used for generating load, exporting data, and running additional functions. What it will definitely not fulfill is the function of generating load, where the load will only be generated by the JMeter agents (slaves).
Besides this, the JMeter agents only need to be started in “server mode”. The file structure (repository) of the JMeter Master Contoller does not need to be manually synchronized on the agents. When starting a test, the agent will retrieve a temporary copy of the test plan from the JMeter Master Contoller which it will hold onto for the time of running the test
- Starting a simple test in distributed mode and storing the results locally
With JMeter properly set up and configured, we are now ready to start a simple test in distributed mode. Let us suppose we now have a testplan called “DemoDistributedTest.jmx“, stored in the /tmp/jmeter/testplans folder. Since we are only interested in running the test in command line , this is the command template needed:
jmeter -n -t /tmp/jmeter/testplans/DemoDistributedTest.jmx -Jagent1,agent2,agent3 -l /tmp/jmeter/testresults/DemoDistributedTest.xml
This will start the test in non-gui mode (-n flag) indicating the location of the testplan in the Repository (the folder structure from above) (-t flag), choosing the JMeter agents to run the test with (-J flag) and storing the results into a selected file
- Adding a jmeter agent to the load configuration
As mentioned above, you only need to set up JMeter on an additional agent server, which we will call agent4. After adding the new agent to the list of JMeter agents in the Master configuration file, we are ready to repeat the test with 4 JMeter agents:
jmeter -n -t /tmp/jmeter/testplans/DemoDistributedTest.jmx -Jagent1,agent2,agent3,agent4 -l /tmp/jmeter/testresults/DemoDistributedTest.xml
- Preparing for automation and dynamic parameters
We have now run a test on multiple agents, by defining the testplan, the agents and the output file using the command line. Going on from here, we need to provide all this information in a dynamic way. We need to be able to run the command above by using some sort of controlling scripts. I will use some sort of pseudo codes for expressing what i want to achieve.
PSEUDOCODE: RUN TESTPLAN testplan_name ON AGENTS agent_list STORE THE RESULTS IN result_file
This translates into the following command line:
jmeter -n -t ${TESTPLAN} -J${AGENT_LIST} -l ${RESULT_FILE}
Additionally, we need to be able to specify not only test configuration parameters, but also application test parameters, like usernames, passwords, number of virtual users, pause times and so on:
PSEUDOCODE:RUN TESTPLAN testplan_name ON AGENTS agent_list WITH THE FOLLOWING PARAMS runtime_parameters STORE THE RESULTS IN result_file
This translates into:
jmeter -n -t ${TESTPLAN} -J${AGENT_LIST} -l ${RESULT_FILE} - parameters
We are now ready to create our first test script. There are three sections that each and every test script will contain:
- header - this will contain test configuration information: location of the testplans, agent list as well as the file containing the functions for controlling the test. This information will be stored in variables in the header.
- body - this is the part where the functions controlling the run of the test will reside, together with other monitoring tasks that need to be running while the test is running
- footer - this is where we will end the testplan, and generate the reports. Since we want the test plans and test results to reside under a logical file structure, we can now build the structure of folders needed. This looks like:
-
- testplans
- scripts
- results
- includes {contains files with functions and global variables needed by every test}
- propfiles {contains application test specific variables, will be discussed later}
By now our template test script looks like this:
# META
# Created: 28 July 2009
# Owner: Alexandru Ersenie
# Description: Description of what the test does
# HEADER
JMETER_TEST_FOLDER="/home/testing/jmeter/"
. ${JMETER_TEST_FOLDER}includes/testconfiguration.properties
TESTPLAN_SUBFOLDER="homepage/login/"
TESTPLAN_FILE_NAME="Tplan_Login.jmx"
JMETER_AGENTS="agent1,agent2,agent3"
# BODY
jmeter -n -t ${JMETER_TESTPLANS}${TESTPLAN_SUBFOLDER}${TESTPLAN_FILE_NAME} -R ${JMETER_AGENTS} -l ${JMETER_RESULTS}${TESTPLAN_FILE_NAME}.xml
- Function file and global variables
Running a test is not enough. We want to be able not only to pass runtime variables to the test, but also collect all kind of application server statistics, control if the test is going to store the results in the database and generate a report, if it uses a warm up or not,etc. We need to be able to clear the application server log files before starting the test, and moving them after running the test into a location where we can reactively analyse them, and so on.
In order to achieve this, there is a need for some sort of functions controlling the set up, runtime, tear down and export phases.
It will then be very easy to create a script template which will call the functions that reside in this file. This is how a test script will look like in the end. The functions are explained below:
# META
# Created: 28 July 2009
# Owner: Alexandru Ersenie
# Description: Description of what the test does
# HEADER
JMETER_TEST_FOLDER="/home/testing/jmeter/"
. ${JMETER_TEST_FOLDER}includes/testconfiguration.properties
TESTPLAN_SUBFOLDER="homepage/login/"
TESTPLAN_FILE_NAME="Tplan_Login.jmx"
JMETER_AGENTS="agent1,agent2,agent3"
# BODY
log_test_config
get_db_info
empty_temp_files
configure_appserver
run_scenario_warmup
trace_jms_usage &
trace_class_loading &
trace_gf_statistics
run_scenario
wait_for_user_input
#FOOTER
collect_logs
write_testInfo
log_heap_config
check_reporting
log_test_end
exit
SET UP PHASE
The set up phase contains functions needed to get the test running. This includes: configuring the application server, removing server log files, removing temporary files, retrieving actual status of the database, etc.
- function log_test_config: logs information regarding the test run to the console and into a log file
function log_test_config
{
mkdir -p ${JMETER_RESULTS}
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Test Configuration: Test will be performed on deployed version: "$APPNAME" - Build Number: "$APP_VERSION" - Revision Number: "$REVISION_NUMBER"" | tee ${JMETER_RESULTS}/run.log
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Test Configuration: Test will be performed on server: "${HOST}" port "${PORT}" protocol "${PROTOCOL}""
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Test Configuration: "$VIRTUAL_USERS" virtual users starting in "$RAMP_TIME" seconds " | tee -a ${JMETER_RESULTS}/run.log
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Test Configuration: "$TRANSACTIONS" transactions will be generated, waiting between "$PAUSE_TIME" and "`expr $PAUSE_TIME + $PAUSE_TIME_DEV`" ms between requests" | tee -a ${JMETER_RESULTS}/run.log
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Test Configuration: Repeating the scenario "$REPETITIONS" times, increasing the load "$LOOPS" times with "$ADDEDVU" users / repetition" | tee -a ${JMETER_RESULTS}/run.log}
- function get_db_info: retrieves the number of specific records in the database at the start of the test by running a JMeter test containing a JDBC Select Request. The testplan will store the total number of records into an output file, which will be grepped after. The best part here is that you can add as many requests in the jmeter test plan, and then just extract them one by one from the output file and use them further on in your report.
function get_db_info
{
jmeter -n -t ${JMETER_TESTPLANS}extra/Tplan-DBInfo.jmx -JRESULT_FOLDER=${JMETER_RESULTS}
DBINFO_RECORDSINDB=`grep -E '[0-9]' ${JMETER_RESULTS}/db_info*`
echo "Number of records in Database:" ${DBINFO_RECORDSINDB}
}
- function empty_log: connects via ssh to the application server and empties the log files. It will clear the application server log file, the garbage collection and java virtual machine log file.
function empty_log
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Removing old Glassfish and JVM logs" | tee -a ${JMETER_RESULTS}/run.log
ssh ${DOMAIN_SERVER} sudo -u glassfish /bin/bash -c "'rm ${GLASSFISH_PATH}/${DOMAIN_NAME}/logs/server.log_*'"
ssh ${DOMAIN_SERVER} sudo -u glassfish /bin/bash -c "'cat /dev/null > ${GLASSFISH_PATH}/${DOMAIN_NAME}/logs/jvm.log'"
ssh ${DOMAIN_SERVER} sudo -u glassfish /bin/bash -c "'cat /dev/null > ${GLASSFISH_PATH}/${DOMAIN_NAME}/logs/jgc.log'"
}
- function empty_temp_files: clears the temporary files used by the monitoring and exporting functions (jms monitoring, glassfish statistics, etc.)
function empty_temp_files
{
cat /dev/null > /tmp/running
cat /dev/null > /tmp/jms_monitoring
cat /dev/null > /tmp/jmap_objects.log
cat /dev/null > /tmp/glassfish_stats
}
- function configure_appserver: configures the desired number of threads in JDBC and HTTP Thread pool. This is useful when you want to increase the number of threads in the thread pool by keeping the same load
function configure_appserver
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Server Configuration: Configuring server minimum and maximum number of threads" | tee -a ${JMETER_RESULTS}/run.log
ssh ${APPSERVER} "sudo -u glassfish /bin/bash -c '/opt/glassfish/bin/asadmin --port '${DOMAIN_ADMIN_PORT}' -u admin --passwordfile /opt/glassfish/passwords set server.thread-pools.thread-pool.http-thread-pool.min-thread-pool-size=${GF_THREADS_MIN}'"
ssh${APPSERVER}"sudo -u glassfish /bin/bash -c '/opt/glassfish/bin/asadmin --port '${DOMAIN_ADMIN_PORT}' -u admin --passwordfile /opt/glassfish/passwords set server.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=${GF_THREADS_MAX}'"
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Set up: Starting the test with a threadpool of minimum ${GF_THREADS_MIN} and maximum ${GF_THREADS_MAX} threads" | tee -a ${JMETER_RESULTS}/run.log
#echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - This test runs with ${VIRTUAL_USERS} Virtual Users / Agent, on following agents: ${JMETER_AGENTS}"
}
RUNTIME PHASE
- run_scenario_warmup: the function takes a command line input parameter which controls if the test is going to be run with a warm-up first or not. It will start a test with a reduced number of virtual users in order to warm up the application server. The test will output the results into a simple log file (not xml), so that the warm up results will not be taken into consideration when importing the test results generated by the load test.
function run_scenario_warmup
{
if [ "$WARMUP" != "yes" ];
then
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Set up: No server warm up. Test will be started without warming up the server" | tee -a ${JMETER_RESULTS}/run.log
else
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Set up: Warming up server by running the scenario with "${WARMUP_THREAD_USERS}" users on one agent executing exactly 50 payment transactions" | tee -a ${JMETER_RESULTS}/run.log
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Running: Warming up started" | tee -a ${JMETER_RESULTS}/run.log
jmeter -n -t ${JMETER_TESTPLANS}${TESTPLAN_SUBFOLDER}${TESTPLAN_FILE_NAME} -R agent1 -l ${JMETER_RESULTS}${TESTPLAN_FILE_NAME}.log-Ggroup1.hostname=${HOST} -Ggroup1.port=${PORT} -Ggroup1.protocol=${PROTOCOL} -Ggroup1.fullhost=${FULLHOST} -Ggroup2.threads=${threads_start} -Ggroup2.ramptime=${rampup_start} -Ggroup2.users=${THREAD_USERS} -Ggroup2.startid=${START_ID} -Ggroup2.synctimer=${SYNC_TIMER} -Ggroup3.pay_transactions=50-Ggroup4.pausetimeconst=${PAUSE_TIME} -Ggroup4.pausetimerandom=${PAUSE_TIME_DEV} -G ${JMETER_PROPFILES}${TESTPLAN_SUBFOLDER}${TESTPLAN_FILE_NAME}.properties &
wait_for_plan '${TESTPLAN_FILE_NAME}'
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Tear down: Warming up ended, sleeping for 10 seconds" | tee -a ${JMETER_RESULTS}/run.log
sleep 10
fi
}
- trace_class_loading: this function will trace memory usage statistics every 10 seconds, using jmap. It will connect to the application server and retrieve a map of objects periodically, which will be stored into an export file, that will be processed in the end. The type of objects can be extended by simply adding a regular expression. Currently it retrieves hashmaps,vectors, eclipse persistence objects, javascript objects etc. The function uses a temporary file called “running” which resides in the “/tmp” folder. While the file exists, the collection of statistics keeps going, and stops once the test stops. (when the test stops, the temporary file is removed, so that in the next check, the statistics function will not find the file and will exit). In the end, the file is processed and exported into a final jmap_objects file, which will be used for exporting memory information in the performance database
function trace_class_loading
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Tracing: Starting to collect Heap Usage statistics with a refresh of 10 seconds" | tee -a ${JMETER_RESULTS}/run.log
glassfish_pid=`ssh ${DOMAIN_SERVER} sudo -u glassfish /bin/bash -c '/usr/java/jdk1.6.0_24/bin/jps -q | grep ASMain ' | awk '{print$1}'`
echo "glassfih pid is " $glassfish_pid
status=`ls /tmp | grep running`
# As long as the script is running perform Jmap histogram on the server every x seconds, where x is defined in sleep command
while [ "$status" != "" ];
do
JMAP_TIMESTAMP=`date +%H-%M-%S`
ssh ${DOMAIN_SERVER} "sudo -u glassfish /bin/bash -c '${JAVA_BIN}jmap -histo ${glassfish_pid}'" | egrep -i -e '\[[I,B,C]+' -e 'myobjects' -e 'java.util.Tree' -e 'java.util.Hash' -e 'java.util.Vector' -e 'Klass' -e 'org.mozilla' -e 'com.sun.tools.javac.zip.ZipFileIndexEntry' -e 'java.lang' -e 'java.util.concurrent' -e 'org.eclipse.persistence' | sed -e '/[0-9]/s/$/',"${JMAP_TIMESTAMP}"'/' >> /tmp/jmap_objects.log
sleep 10
status=`ls /tmp | grep running`
done
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Tracing: Collection of Heap Usage Statistics ended" | tee -a ${JMETER_RESULTS}/run.log
# After the script has ended perform a full garbage collector. Comment this line if you do not want a FULL GC
ssh ${DOMAIN_SERVER} "sudo -u glassfish /bin/bash -c '${JAVA_BIN}jmap -histo:live ${glassfish_pid}'" | egrep -i -e '\[[I,B,C]+' -e 'myobjects' -e 'java.util.Tree' -e 'java.util.Hash' -e 'java.util.Vector' -e 'Klass' >> /tmp/jmap_objects.log
# Move the jmap log in the results folder for further processing
mv /tmp/jmap_objects.log ${JMETER_RESULTS}/jmap_objects.log
#Process jmap log for removing spaces - old and very slow
awk '{printf("%s,%s,%s,%s,%s\n",$1,$2,$3,$4,$5);}' ${JMETER_RESULTS}/jmap_objects.log | awk '{gsub(/:/,"");print}' > ${JMETER_RESULTS}/processed_jmap_objects.log
cp ${JMETER_RESULTS}/processed_jmap_objects.log ${JMETER_TRANSFORMATION}processed_jmap_objects.log}
- trace_app_statistics: this function uses glassfish’s rest monitoring interface and curl commands for retrieving information such as jdbc connections, http thread usage, number of specific beans in cache, etc. This function is very flexible, as all data is written in the database in the form of: test_id, timestamp, label, value. Thank to this, any new monitoring item is just a new label in the database, and can be retrieved in a separate report by building a simple query. I will post here a sample of this function, which collects information regarding JDBC Connection usage. It can be easily extended by simply adding a new monitoring item, for example HTTP_THREAD_POOL_THREAD_COUNT
function trace_app_statistics
{
status=`ls /tmp | grep glassfish_stats`
while [ "$status" != "" ];
do
MONITOR_TIMESTAMP=`date +%H-%M-%S`
#JDBC Monitoring
JDBC_CONN_USED=`curl -s -u user:password http://glassfish:4848/monitoring/domain/server/resources/EocPool | grep numconnused | grep -o -E '"current":[0-9]*' | sed 's/["]*[a-z]*["][:]*//'`
echo $MONITOR_TIMESTAMP":JDBC - Connections used:"$JDBC_CONN_USED >> ${JMETER_RESULTS}/glassfish_stats.log
sleep $CMD_PARAM_INTERVAL
status=`ls /tmp | grep glassfish_stats`
done
cp ${JMETER_RESULTS}/glassfish_stats.log ${JMETER_TRANSFORMATION}glassfish_stats.log
}
- run_scenario: this is the main function controlling the run of the test. It takes several parameters as input, such as number of repetitions, number of virtual users, number of loops inside a repetition, number of virtual users to be increased in a loop, ramp up time, timers, etc. It sets the basis for running the test scenario in the following form:
./PAYMENT_SCENARIO 'users=150 ramptime=150 pausetime=2000 pausetimedev=200 pay_transactions=500 repeats=1 loopsinrepeat=1 waitforuserinput=no minthreads=150 maxthreads=200 generatereport=no warmup=no startid=100' 'hostname=myappserver port=8080 protocol=http'
It will repeat the test for the desired number of times, increasing the number of users progressively by the defined number and so on.
function run_scenario
{
scenario_counter=$REPETITIONS
cool_down=0
while [ "$scenario_counter" != "0" ];
do
# Repeat test for a defined number of times (counter)
counter_start=$LOOPS
counter_end=$LOOPS
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Set up: Starting the load test" | tee -a ${JMETER_RESULTS}/run.log
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Set up: Starting iteration number " `expr $REPETITIONS - $scenario_counter + 1` "out of " $REPETITIONS "iterations" | tee -a ${JMETER_RESULTS}/run.log
threads_start=${VIRTUAL_USERS}
rampup_start=${RAMP_TIME}
while [ "$counter_start" != "0" ];
do
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Running: Running with a number of " ${threads_start} " virtual users / agent" | tee -a ${JMETER_RESULTS}/run.log
counter_start=`expr $counter_start - 1`
jmeter -n -t ${JMETER_TESTPLANS}${TESTPLAN_SUBFOLDER}${TESTPLAN_FILE_NAME} -R ${JMETER_AGENTS} -l ${JMETER_RESULTS}/${TESTPLAN_FILE_NAME}.xml -Ggroup1.hostname=${HOST} -Ggroup1.port=${PORT} -Ggroup1.protocol=${PROTOCOL} -Ggroup1.fullhost=${FULLHOST} -Ggroup2.threads=${threads_start} -Ggroup2.ramptime=${rampup_start} -Ggroup2.users=${THREAD_USERS} -Ggroup2.startid=${START_ID} -Ggroup2.synctimer=${SYNC_TIMER} -Ggroup3.pay_transactions=${TRANSACTIONS} -Ggroup4.pausetimeconst=${PAUSE_TIME} -Ggroup4.pausetimerandom=${PAUSE_TIME_DEV} -G ${JMETER_PROPFILES}${TESTPLAN_SUBFOLDER}${TESTPLAN_FILE_NAME}.properties &
wait_for_plan '${TESTPLAN_FILE_NAME}'
threads_start=`expr $threads_start + $ADDEDVU`
if [ "$counter_start" != "0" ];
then echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Running: Increasing number of virtual users by " ${ADDEDVU} "users / agent" | tee -a ${JMETER_RESULTS}/run.log
fi
done
# Cool Down
scenario_counter=`expr $scenario_counter - 1`
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Teardown: Load test ended, sleeping for 30 seconds" | tee -a ${JMETER_RESULTS}/run.log
sleep 10
done
}
- wait_for_user_input: this function waits for the users input in order to stop the test scenario, and to start importing the results. It is useful when you want to keep collecting memory statistics until the test is ended, and you want for example to do a full garbage collection, so that, before importing, you have a list of all objects that are left after the test run, and after a major collection. These will be good candidates for memory leaks. The function takes a command line parameter as input. Once the wait is over, all temporary files are erased, and all functions that run by checking the existence of such temporary files will exit, allowing the main scenario to end, and start the processing of test results and importing into the performance database
function wait_for_user_input
{
if [ "$USERINPUT" != "no" ];
then
while : ; do
read -t 2 && break
done
echo exited while loop
rm -r /tmp/running
rm -r /tmp/jms_monitoring
rm -r /tmp/glassfish_stats
else
rm -r /tmp/running
rm -r /tmp/jms_monitoring
rm -r /tmp/glassfish_stats
fi
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Analysis Services: Sleeping for 30 seconds to allow jms and performance monitor to stop gracefully" | tee -a ${JMETER_RESULTS}/run.log
sleep 30
}
TEAR DOWN
- collect_logs
function collect_logs
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Analysis Services: Collecting and compressing application server logs (jvm,jgc,jms,server)" | tee -a ${JMETER_RESULTS}/run.log
cd ${JMETER_RESULTS}
# Collect Glassfish logs on Server
scp ${DOMAIN_SERVER}:${GLASSFISH_PATH}/${DOMAIN_NAME}/logs/server.log* ${JMETER_RESULTS} || on_error "${LOG_TIMESTAMP}: ERROR - Could not perform Glassfish log collecting via SCP on server ${DOMAIN_SERVER} "
zip -j ${JMETER_RESULTS}/${testrun_id}-server.zip ${JMETER_RESULTS}/server.log*
rm server.log*
# Collect JVM logs on server
scp ${DOMAIN_SERVER}:${GLASSFISH_PATH}/${DOMAIN_NAME}/logs/jvm.log* ${JMETER_RESULTS}/${testrun_id}-jvm-original.log || on_error "${LOG_TIMESTAMP}: ERROR - Could not perform JVM log collecting via SCP on server ${DOMAIN_SERVER} "
sed 's/\x0//g' ${JMETER_RESULTS}/${testrun_id}-jvm-original.log > ${JMETER_RESULTS}/${testrun_id}-jvm.log
rm ${JMETER_RESULTS}/${testrun_id}-jvm-original.log
zip -j ${JMETER_RESULTS}/${testrun_id}-server.zip ${JMETER_RESULTS}/${testrun_id}-jvm.log*
# Collect Java Garbage collection output on server
scp ${DOMAIN_SERVER}:${GLASSFISH_PATH}/${DOMAIN_NAME}/logs/jgc.log* ${JMETER_RESULTS}/${testrun_id}-jgc-original.log || on_error "${LOG_TIMESTAMP}: ERROR - Could not perform JGC log collecting via SCP on server ${DOMAIN_SERVER} "
sed 's/\x0//g' ${JMETER_RESULTS}/${testrun_id}-jgc-original.log > ${JMETER_RESULTS}/${testrun_id}-jgc.log
rm ${JMETER_RESULTS}/${testrun_id}-jgc-original.log
zip -j ${JMETER_RESULTS}/${testrun_id}-server.zip ${JMETER_RESULTS}/${testrun_id}-jgc.log*
# Collect Java Garbage collection output on server
scp ${DOMAIN_SERVER}:${GLASSFISH_PATH}/${DOMAIN_NAME}/logs/jms.log* ${JMETER_RESULTS}/${testrun_id}-jms-original.log || on_error "${LOG_TIMESTAMP}: ERROR - Could not perform JGC log collecting via SCP on server ${DOMAIN_SERVER} "
sed 's/\x0//g' ${JMETER_RESULTS}/${testrun_id}-jms-original.log > ${JMETER_RESULTS}/${testrun_id}-jms.log
rm ${JMETER_RESULTS}/${testrun_id}-jms-original.log
# Calculate peak number of jms messages while test was running
TOTAL_JMS_MSG=`sort -r -t ' ' +1 -n ${JMETER_RESULTS}/${testrun_id}-jms.log | awk '{print $1}' | sed q`
TOTAL_JMS_BYTES=`sort -r -t ' ' +3 -n ${JMETER_RESULTS}/${testrun_id}-jms.log | awk '{print $3}' | sed q`
PEAK_JMS_MSG=`sort -r -t ' ' +6 -n ${JMETER_RESULTS}/${testrun_id}-jms.log | awk '{print $6}' | sed q`
PEAK_JMS_TOTAL_BYTES=`sort -r -t ' ' +9 -n ${JMETER_RESULTS}/${testrun_id}-jms.log | awk '{print $9}' | sed q`
zip -j ${JMETER_RESULTS}/${testrun_id}-server.zip ${JMETER_RESULTS}/${testrun_id}-jms.log*
return
}
- write_test_info: this function writes the necessary test configuration information in an xml file. It generates an increasing unique id for the testrun_id, and writes the testinformation from multiple sources that were provided as command line parameters, or as results of other functions called in the set up or run time phase. It will write all information needed to identify the load configuration and the server configuration at the time of test run: number of users, ramp up, number of processing threads, garbage collection strategy, jvm heap configuration, jdbc pool configuration, etc.
function write_testInfo
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Analysis Services: Writing Test Summary for importing in Test Results" | tee -a ${JMETER_RESULTS}/run.log
# Get unique ID for the script,in order to store in DB. ID is formed of the last digits of IP address and an id that is incremented
# Get last digits of IP address
typeset -i test_id
local_ip=`/sbin/ifconfig | grep ‘inet addr:’| grep -v ’127.0.0.1′ | grep -v ’192.168′ | cut -d: -f2 | cut -d. -f4 | awk ‘{ print $1}’`
test_id=$(cat ${ID_FILE})
new_id=`expr $test_id + 1`
echo $new_id >${ID_FILE}
testrun_id=$local_ip$new_id
# Getting testdescription and server configuration
echo $threadnum
echo $threadwarm
# Copying the glassfish server logs to a path configured to be the logging path for the reporting server
cp ${JMETER_RESULTS}/${testrun_id}-server.zip ${JMETER_SERVER_LOGS}${testrun_id}_server.zip
# Writing xml test information
CODE_500=`grep -c ‘rc=”5′ ${JMETER_RESULTS}/*.xml`
CODE_400=`grep -c ‘rc=”4′ ${JMETER_RESULTS}/*.xml`
CODE_200=`grep -c ‘rc=”2′ ${JMETER_RESULTS}/*.xml`
echo '<?xml version="1.0" encoding="UTF-8"?><testInfo build="'$REVISION_NUMBER'" test_id="'$testrun_id'" portal_version="'$PORTAL_VERSION'" revision_number="'$REVISION_NUMBER'" scenario_id="'${SCRIPT_NAME}'" architecture_id="'$ARCHITECTURE_ID'" test_date="'$LOG_RUNDATE $LOG_TIMESTAMP'" db_recordsindb="'$DBINFO_RECORDSINDB'"><test-configuration threads="'$VIRTUAL_USERS'" rampup="'$RAMP_TIME'" payment_transactions="'$TRANSACTIONS'" waittime="'$PAUSE_TIME'" waitdeviation="'$PAUSE_TIME_DEV'"></test-configuration><glassfish-configuration server_threads_min="'$GF_THREADS_MIN'" server_threads_max="'$GF_THREADS_MAX'" jdbc_thread_pool_min="'$JDBC_THREAD_POOL_MIN'" jdbc_thread_pool_max="'$JDBC_THREAD_POOL_MAX'" jdbc_statement_cache="'$JDBC_STATEMENT_CACHE'"></glassfish-configuration><jvm-options gc_strategy_param1="'$GC_STRATEGY_PARAM1'" total_initial_heap="'$TOTAL_INITIAL_HEAP'" total_maximum_heap="'$TOTAL_MAXIMUM_HEAP'" newgen_initial_heap="'$NEWGEN_INITIAL_HEAP'" newgen_maximum_heap="'$NEWGEN_MAXIMUM_HEAP'" survivor_ratio="'$SURVIVOR_RATIO'" log_level="'$LOG_LEVEL'"></jvm-options><jms-usage>total_jms_msg="'$TOTAL_JMS_MSG'" total_jms_bytes="'$TOTAL_JMS_BYTES'" peak_jms_msg="'$PEAK_JMS_MSG'" peak_jms_total_bytes="'$PEAK_JMS_TOTAL_BYTES'"</jms-usage><statusCodes>code_500="'$CODE_500'" code_400="'$CODE_400'" code_200="'$CODE_200'" </statusCodes></testInfo>' > ${JMETER_RESULTS}/testInfo.xml
return
}
- log_heap_config: this function analysis the jvm configuration on the server by using “jmap” with “-heap” parameter
function log_heap_config
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Analysis Services: Collecting Glassfish heap configuration" | tee -a ${JMETER_RESULTS}/run.log
glassfish_pid=`ssh ${DOMAIN_SERVER} sudo -u glassfish /bin/bash -c '${JAVA_BIN}jps -q | grep ASMain ' | awk '{print$1}'`
ssh ${DOMAIN_SERVER} "sudo -u glassfish /bin/bash -c '${JAVA_BIN}jmap -heap ${glassfish_pid}'" > ${JMETER_RESULTS}/${testrun_id}-heapconfig.txt
}
- check_reporting: this function checks if the import of testresults is wished after the test run. This is controlled by a command line parameter called “REPORTING“. If no reporting is wished, then just a small processing is performed, for hudson analysis. Otherwise, the “data_transformation” function will be called, which will start the exporting of data in the database.
function check_reporting
{
if [ "$REPORTING" == "yes" ];
then
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Analysis Services: Sleeping for 30 seconds before starting data transformation services" | tee -a ${JMETER_RESULTS}/run.log
data_transformation
else
hudson_transformation
fi
}
- log_test_end: this function signalises the end of the test by outputing a log message to the console and in the test run log file.
function log_test_end
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Test Finish: Test Scenario Ended" | tee -a ${JMETER_RESULTS}/run.log
}
.
In the second article i discussed how to build an integrated load solution with JMeter. I covered topics such as: dynamic test plans, test scripts, test repository, shell functions, monitoring functions, logging functions, etc. I talked about the structure of a test script template, and detailed the shell functions called in it. It is now time to dive into the third part, one of the most interesting maybe in this series of articles:
ETL – EXTRACT TRANSFORM AND LOAD – DATA PROCESSING AND IMPORTING INTO THE DATABASE
.
Latest comments