Qants – Tips in the field of Java Performance & Software Testing

• Start by determining the maximum capacity and throughput of your application server
• Determine how many concurrent users can you handle on your application server, and what is their memory footprint (memory used by one user session for the duration of using your solution)
• Determine the number of connections that you need to maintain open on the web server side in order to accommodate all users above
• Tune the corresponding thread pools (web server and application server) accordingly, so that only a determined number of connections can be opened on the web server, and that only a determined number of users can make it to the application server
• Determine the average number of business requests a user will execute on your system
• Tune the backlog of the web server so that you can prepare for spikes, and that you can determine when to send the surplus users to another web server or waiting server
• Tune the operating systems  for handling the configured number of connections – network, memory, open files, etc.

1. Long and very long loading and response times – nowadays any webpage loading in more than 3 to 5 seconds is annoying. Dirk, an ex colleague of mine was telling me that an increase of 0.1 seconds in the loading time of Amazon costs a fortune…
2. White pages and never loading page – This is even worse. A web page loading very slowly is still a loading web page. Yet, any new user landing on your product page, and experiencing a white page as his first experience will unlikely come again very soon.

• be processed immediately
• wait for a free working thread, and be processed with an acceptable delay
• be redirected to another web server
• be redirected to a wait server

• memory required / registered user: 1.5 MB
• memory required / unregistered user (simple visitor): 0.5 MB

Application Server Throughput

Application Server Workload model

• A business transaction is executed by a user every approximately 3 seconds
• The average response time for a transaction is between half a second and a second
• Worse case scenario (is a strategy I often use): x users / (3 seconds think time + 1 second response time) = 600 transactions  =>> x = 2400 concurrent “business” users (users executing business transactions)
• Light transactions (like users just visiting your webpage) are executed in a rate of 30 users / seconds, so that after one minute, you will have 60 seconds * 30 users = 1800 anonymous users

• 1800 anonymous users, needing 1800*0.5 MB= 900 MB
• 2400 registered users, needing 2700*1.5 MB = 3600 MB

Application Server HEAP structuring

• business transactions – executing some work in the database, persisting/modifying specific information, executing application logic with a result that is to be persisted
• light transactions – users just surfing on your page, not actually producing any workload on your database

2. Building the Apache Web Server Model

1. It costs less to spawn a large number of threads than to spawn new processes, as in the prefork model
2. The memory is shared between all threads belonging to a process
3. The number of threads can be dynamically increased using graceful restart of the web server
4. The number of threads used in the application server is somehow compressed, meaning that for each 2, 3, 4 apache threads, there will be one application server thread doing the application business. The application server thread will be at some point shared between web server threads. This way you can define a larger number of apache threads, with a lower number of application server threads

• how long will these users hang on this connection?
• what if the user sends one request, and than does not send any additional request? When will the connection be closed and the thread returned to the pool?
• what if the user opens the connection and regularly sends requests forever, keeping the connection open forever (Denial of service)

• If one user sends one request over a persistent connection, and then waits 59 seconds before sending the next request, he will reuse the same connection, and no other connection (apache thread) will be created. But:
• If one user sends a request over a persistent connection, and then waits longer than 60 seconds before sending the next one, or even worse, will never send an additional request, the connection will be kept open with no reason, and will be dropped without serving any more than one request. Not that optimal, is it?

Using this configuration and a time-scaled representation, after 60 seconds you would have the following scenario:

• 60 % (18 users) will continue just executing business requests, needing one single persistent connection / user
• 40 % (12 users) will continue just surfing, needing 6 persistent connection / user

Apache Connections Keep Alive

Compared to the first model, where we had to maintain 10 800 connections, you now have to only maintain 6840 connections!

But, one target was to be also prepared for spikes. What if we have more users / second than 30 ? What do we do with them, how do we react to that, and how do we prevent the scenarios described in the beginning (white page, long waiting times, etc.)

Well, since we know we can accommodate 1800 users, using the 6840 connections, we can prepare for an extra 50 % workload. That would mean we would get to about 10 000 connections (6840 + 3420) I will just use 10 000 for simplicity.

Right. But…what will happen with the connection request 10001 ?

Since we know that we have a loading time of about 6 seconds, no user should wait longer than 3 seconds to get a connection (so that the total response time still falls under 10 seconds)

With a workload of 30 users / second, we can accommodate:

( 30 users * 6 connections * 3 seconds ) requests in 3 seconds= 540 requests

But we just spoke of spikes, so let us increase the backlog with 50 % to  810 connections.

That means:

• First 10000 requests will get their threads
• requests between 10001 and 10810 will wait in the backlog for a free connection
• request 10811 will be either sent to a wait server or to another scaled web server

Tuning Apache Workers for Load - The connection and backlog model

Now that we decided on what we want to handle, let’s get our servers ready for this.

3. Configuring and tuning APACHE using Workers for performance and high number of connections

We need to take care both of APACHE configuration, and of configuration of the operating system where Apache resides

APACHE WebServer Configuration

We are ready to configure APACHE for 10 000  connections. Since it is more efficient to spawn threads than processes, let’s use a higher number of threads per process.

This is how a configuration could look like. With this configuration we will:

• start with 25 servers, each spawning 200 threads, resulting 5000 threads
• use about 50 k / thread, resulting in about 500 MB reserved memory
• expand to maximum 50 servers, each spawning 200 threads, resulting 10000 threads. This is also the value used by APACHE as MaxClients (maximum number of connections)
• start shrinking the number of threads once the load has decreased, and keep no less than 7500 threads in the pool
• allow no more than 100 000 requests / process, which results in an average of 500 requests / child thread. We use this in order to avoid nasty users holding to the process forever, and we also avoid memory leaks, by destroying the process after the number of requests has been reached

<IfModule worker.c>
# initial number of server processes to start
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers
StartServers        25
# highest possible MaxClients setting for the lifetime of the Apache process.
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit
ServerLimit         50
# minimum number of worker threads which are kept spare
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#minsparethreads
MinSpareThreads     1000
# maximum number of worker threads which are kept spare
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxsparethreads
MaxSpareThreads    7500
# upper limit on the configurable number of threads per child process
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadlimit
ThreadLimit        200
# maximum number of simultaneous client connections
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
MaxClients         10000
# number of worker threads created by each child process
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadsperchild
ThreadsPerChild     200
# maximum number of requests a server process serves
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild
MaxRequestsPerChild  150
</IfModule>
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 10
ListenBackLog 810

A couple of things to consider:

1. MaxClients: If the allocated MaxClients is higher than ServerLimit * ThreadLimit Apache will automatically reduce MaxClients to the value of ServerLimit * ThreadLimit
   In this case, you will receive the following message:

   WARNING: MaxClients of 10000 would require 200 servers,
    and would exceed the ServerLimit value of 100.
    Automatically lowering MaxClients to 5000.  To increase,
    please see the ServerLimit directive.

2. Apache Memory / Process and Threads: Check if you have enough memory to handle the configured number of threads. Use the following scripts to monitor the Apache Servers, the number of threads, and the memory reserved:
   • The formula used for configuring your maximum number of clients is:
     
maxclients = total ram / ram per process
   • List all Apache processes and number of threads per process
     
for pid in `ps U wwwrun | grep httpd | grep -v grep | awk '{ print $1 }'`;
do echo Apache Worker Server $pid has `ps ms -p $pid | wc -l` threads;
done

     This will output:
     
Apache Worker Server 7528 has 3 threads
Apache Worker Server 7583 has 3 threads
Apache Worker Server 7587 has 3 threads
Apache Worker Server 7596 has 204 threads
Apache Worker Server 7601 has 204 threads
Apache Worker Server 7610 has 204 threads
Apache Worker Server 7618 has 204 threads
Apache Worker Server 7628 has 204 threads
Apache Worker Server 7640 has 204 threads
Apache Worker Server 7651 has 204 threads
Apache Worker Server 7668 has 204 threads
............
   • List the reserved memory per APACHE Server and threads
     
servers=0;threads=0;space=0;
for pid in `ps U wwwrun | grep httpd | grep -v grep | awk '{ print $1 }'`;
do process_threads=`ps ms -p $pid | wc -l`;
process_memory=`ps -ylC -p $pid | grep -v PID | awk '{ print $8}'`;
echo Apache Worker Server with pid $pid has $process_threads threads occupying $process_memory bytes;
servers=`expr $servers + 1`;threads=`expr $process_threads + $threads`;space=`expr $process_memory + $space`; done;
echo "-------------"; echo Total Apache Servers: $servers \| Total threads: $threads \| Total memory reserved: $space

     This will output:
     
.......
Apache Worker Server with pid 1733 has 204 threads occupying 34644 bytes
Apache Worker Server with pid 1759 has 204 threads occupying 34612 bytes
Apache Worker Server with pid 1788 has 204 threads occupying 34620 bytes
Apache Worker Server with pid 1816 has 204 threads occupying 34648 bytes
Apache Worker Server with pid 1844 has 204 threads occupying 34648 bytes
Apache Worker Server with pid 1875 has 204 threads occupying 34648 bytes
Apache Worker Server with pid 1907 has 204 threads occupying 34684 bytes
Apache Worker Server with pid 1940 has 204 threads occupying 34676 bytes
-------------
Total Apache Servers: 26 | Total threads: 5103 | Total memory reserved: 885272
3. ThreadsPerchild vs ThreadLimit: There is a difference between ThreadsPerchild and ThreadLimit. ThreadsPerChild defines the initial number of threads spawned per worker process. Defining a larger number as ThreadLimit allows you to modify the ThreadsPerChild up to ThreadLimit, without needing to do a hard restart of Apache. A graceful restart will suffice.For example:

   ThreadsPerChild 100
   ThreadLimit 500

   This will start the apache processes with 100 threads each. If you feel that your webserver cannot handle the current load with the current configuration, you just need to increase the ThreadsPerChild, (in this case to maximum  500) and execute a graceful restart:

   ThreadsPerChild 300
   ThreadLimit 500

   Yet, it is very important to remember that:

1. ThreadLimit will allocate memory in advance, so check in advance if you can handle the ServerLimit * ThreadsPerChild with your current RAM
2. If you increase the ThreadsPerChild up to a value that the APACHE Server will consider it cannot allocate the needed memory, APACHE will exit!
4. ServerLimit vs ThreadLimit: You cannot increase the maximum number of server processes ServerLimit without hard restart of APACHE, so take this into consideration when planning the baseline configuration. If you want to be prepared for a higher load, configure the ThreadLimit higher, which will allow you to increase the capacity of your server with a graceful restart
5. Number of Apache Threads spawned: Make sure that the number of threads spawned is the one you expect. Use the following to check the number of threads spawned.”Sl” is for Interruptable sleep, multithreaded

   ps axsm | grep -c 'Sl'

6. Denial of Service and Keep Alive Parameter: Be prepared for Denial of Service Attacks. Do not allow any user to hold on to a persistent connection forever. Decide on the average number of transactions that a user will execute while using your system, and add it as parameter:

   KeepAlive On
   MaxKeepAliveRequests 1000
   KeepAliveTimeout 10

7. Backlog for extra spikes:Be prepared for those extra spikes, where all your threads are busy, and you need a “waiting room” for your users. Use the ListenBackLog Directive, which we configured to 810:

   ListenBackLog 810

8. ONE OF THE MOST IMPORTANT:
   The only way you can control the life of an APACHE server, and its children, is using this two parameters. Set them too high, and they will never die, therefore, the number of Threads in your Application Server will increase continously, eventually reaching the maximum configured number of threads, where it will stop servicing requests.
   Now, there are two parameters that control the behaviour of the apache servers and their children, and therefore the behaviour of the thread pool on the application server side. MaxRequestsPerChild refers to the maximum number of NEW requests (meaning a new client, opening a completely new Keep Alive Connection on this child) that can be sent over one Apache child. If for example, you had one client, opening a connection, and sending 500 requests, one after the other, on the same keep alive connection, than this parameter would have the value 1 (only the first request over a new keep alive connection will be counted) On the other hand, you do not want one client to hold on forever on his connection (remember the DOS), so you control that by setting the maximum, total number of requests a client can send over a SINGLE keep alive connection. When one of the two happens, the client will be assigned a new child thread, and the thread having served will be marked as “dead”

   MaxRequestsPerChild  150
   MaxKeepAliveRequests 1000
   
   

APACHE Operating System Configuration

Apache Tuning – Max Open Files

Since every socket in Unix is actually a file, we need to tune the maximum allowed number of open files.

This specifies the number of open files that are supported.
The default setting is typically sufficient for most applications. 
If the value set for this parameter is too low, a file open error,
memory allocation failure, or connection establishment error might
be displayed.

More on this here: IBM Websphere Linux Tuning

Apache Tuning – TCP Settings

Keep in mind the seven layer architecture (also known as OSI model) Both Glassfish and Apache control the transport layer, which sits further on the network layer.

OSI Model - Seven layer architecture

Image courtesy: Novell - http://www.novell.com/info/primer/prim05.html

You can configure and limit GLASSFISH and APACHE resources, as long as you do not exceed the network layer settings. That is why we need to take that also into consideration, and tune them accordingly, so that we can plug in the Glassfish or Apache without exceeding any configured resources

Apache Tuning – Backlog and Maximum Connections

Change the following parameters when you prepare for a high rate of incoming connections. Keep in mind that this setting is shared between all Apache Servers, so that if you want to host several web servers, you need to be prepared to accommodate the connections for all servers:

• Number of connections in backlog: cat /proc/sys/net/core/netdev_max_backlog Default is 1000, modify this according to the maximum expected number of connections that you want to accommodate in your backlog (keep them waiting for a connection)
• Size of the listen queue, in bytes, for accepting new tcp connections: cat /proc/sys/net/core/somaxconn Default is 128, modify this according to the size of the backlog you would need for the maximum expected number of connections that you want to accommodate in your backlog

Apache Tuning – Operating System – Keep Alive Settings

Although Unix has built-in support for KeepAlive, this is not the default behavior in Linux. Programs must request keepalive control for their sockets using the setsockoptinterface. http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.htmlThere are three “tunable” parameters:

tcp_keepalive_time

the interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe; after the connection is marked to need keepalive, this counter is not used any further

tcp_keepalive_intvl

the interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime

tcp_keepalive_probes

the number of unacknowledged probes to send before considering the connection dead and notifying the application layer

As long as the keep alive is high enough (higher than your configured Apache Server) you should not worry, since Apache will take care of managing the connections. Just make sure that the settings are higher than the ones configured in your web and application server. You can check all three by issuing the following command (i have also listed the default values for SUSE Linux Enterprise Server 11 ):

• cat /proc/sys/net/ipv4/tcp_keepalive_intvl  : 75
• cat /proc/sys/net/ipv4/tcp_keepalive_probes: 9
• cat /proc/sys/net/ipv4/tcp_keepalive_time: 7200 (seconds)

Apache Tuning – KEEP ALIVE SETTINGS

As defined above, these are the three configuration parameters you’d have to take care of when configuring for keep-alive connections (repeated, see above, chapter 2):

• KeepAlive (do we allow persistent connections at all)
• KeepAliveTimeout (how long should we wait on the client sending an additional request over the connection before closing the connection and returning the thread to the thread pool)
• MaxKeepAliveRequests (how many requests is a client allowed to send over one persistent connection before the server closes the connection and returns the thread to the thread pool)

Apache Tuning – Operating System – CONNECTION MANAGEMENT

tcp_fin_timeout:

This basically holds a connection in a   TIME_WAIT mode (after the keep alive timeout expired), waiting for the same client to reinitiate the communication. Basically it says that it is cheaper to reactivate a sleeping connection, than to build a new one. Maintaining a connection open, after the user has not sent any additional request in the KeepAliveTimeout timeout defined may be expensive, since other users may just well sit in the backlog, waiting for exactly that one connection to be freed. If this is the case, you should set this to a level as low as possible, so that after a small timeout, the connection can be finally destroyed, and rebuilt with another id, for another user.

“This determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. This interval between closure and release is known as the TIME_WAIT state or twice the maximum segment lifetime (2MSL) state. During this time, reopening the connection to the client and server cost less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, providing more resources for new connections. Adjust this parameter if the running application requires rapid release, the creation of new connections, and a low throughput due to many connections sitting in the TIME_WAIT state.”

More on this here: IBM WebSphere Tuning

A simple way to see how many connections you have in TIME_WAIT is using netstat:

• watch --interval=1 "netstat | grep -c TIME_WAIT"

You can view and set this parameter as follows:

• View: cat /proc/sys/net/ipv4/tcp_fin_timeout : 60 (default on SUSE Linux Enterprise Server 11) 
• Set: echo 10 > /proc/sys/net/ipv4/tcp_fin_timeout

Apache Tuning – Compression

Most modern browsers can handle compressed content, decompressing it upon receiving it. It is a great idea to compress content, which will save you a lot of bandwidth. Of course, this comes with a price – CPU is needed for compressing the files. Since you do not want your application server to use resources for compression, leave it to the web server. And since Apache has caching mechanisms, this should not be a problem, since the file will be zipped once, and then saved and served in/from cache

Most compression algorithms, when applied to a plain-text file, can reduce its size by 70% or more, depending on the content in the file. When using compression algorithms, the difference between standard and maximum compression levels is small, especially when you consider the extra CPU time necessary to process these extra compression passes. This is quite important when dynamically compressing Web content. Most software content compression techniques use a compression level of 6 (out of 9 levels) to conserve CPU cycles. The file size difference between level 6 and level 9 is usually so small as to be not worth the extra time involved.

http://www.linuxjournal.com/article/6802

There are two modules for using compression in Apache:

• mod_gzip
• mod_deflate

Just be aware of the compression directives:

• type of files you need to compress (it does not make sense compressing already compressed files like pdf, jpg, etc.): AddOutputFilterByType DEFLATE text/html text/plain text/xml

Apache Tuning – Caching

Related to the point above, you can benefit of both compressing and caching content. You can either use a disk based cache store manager or a faster, memory based store manager

mod_cache implements an RFC 2616 compliant HTTP content cache that can be used to cache either local or proxied content. mod_cache requires the services of one or more storage management modules. Two storage management modules are included in the base Apache distribution:

mod_mem_cache

implements a memory based storage manager. mod_mem_cache can be configured to operate in two modes: caching open file descriptors or caching objects in heap storage. mod_mem_cache can be used to cache locally generated content or to cache backend server content for mod_proxy when configured using ProxyPass (aka reverse proxy)

More on this here: http://httpd.apache.org/docs/2.2/mod/mod_cache.html

4. Configuring and tuning the Glassfish Application Server to work with the Workers Model

One thing you need to understand is that in the worker model, APACHE controls the lifecycle of working threads in Glassfish. Therefore, we need to configure Glassfish knowing that APACHE takes care of things like:

• thread removal
• keep alive timeouts
• maximum number of requests / connection
• compression
• etc

Most UNIX distributions come with an out of the box configuration, that is prepared for normal workload, but definitely not for performance. We need to be aware of network, memory and other system resource settings that we have to reconfigure in order to gain and prepare for performance.

Glassfish Tuning

Glassfish Tuning – Max Open Files

Once again, since every socket in Unix is actually a file, we need to tune the maximum allowed number of open files.

You can of course set it to unlimited, but since we spoke of a APACHE Thread Pool -> Glassfish Thread Pool compression, i would set this to half the number of threads apache can spawn + extra 5000 for all other processes

ulimit -n 10000

Yet, be aware that if you have multiple Glassfish Domains Application Server, they will all share the number of open files. Keep this in mind when calculating the number of possible open threads / server.

Glassfish Tuning – Max Threads

Since your system can perform up to about 600 transactions / second, you should configure your maximum thread pool according to this number. Allow about 50 % more threads for those extra spikes, even if that comes with longer response times.

I would therefore set the maximum thread pool size to about (rounded) 1000:

<thread-pool max-thread-pool-size="1024" name="http-thread-pool" />

Glassfish Tuning – TCP Settings

Keep in mind the seven layer architecture (also known as OSI model) Both Glassfish and Apache control the transport layer, which sits further on the network layer (see above, in chapter 3, Apache Tuning – TCP Settings)

You can configure and limit GLASSFISH and APACHE resources, as long as you do not exceed the network layer settings. That is why we need to take that also into consideration, and tune them accordingly, so that we can plug in the Glassfish or Apache without exceeding any configured resources

Glassfish Tuning – Backlog and Maximum Connections

Change the following parameters when you prepare for a high rate of incoming connections. Keep in mind that this setting is shared between all Glassfish Servers respectively Apache Servers:

• Number of connections in backlog:
  /proc/sys/net/core/netdev_max_backlog
  Default is 1000, modify this according to the maximum expected number of connections that you want to accommodate in your backlog (keep them waiting for a connection). Remember that these connections are the ones opened by APACHE, so you may want to tune this according to the maximum number of threads you will allow Apache to start inside the application server
• Size of the listen queue, in bytes, for accepting new tcp connections:
  /proc/sys/net/core/somaxconn
  Default is 128, modify this according to the size of the backlog you would need for the maximum expected number of connections that you want to accommodate in your backlog

Glassfish Tuning – Operating System – Keep Alive Settings

Although Unix has built-in support for KeepAlive, this is not the default behavior in Linux. Programs must request keepalive control for their sockets using the setsockoptinterface.

http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html

There are three “tunable” parameters:

As long as the keep alive is high enough (higher than your configured Glassfish or Apache Server) you should not worry, since Apache or Glassfish will take care of managing the connections. Just make sure that the settings are higher than the ones configured in your web and application server.

You can check all three by issuing the following command:

• cat /proc/sys/net/ipv4/tcp_keepalive_intvl

• cat /proc/sys/net/ipv4/tcp_keepalive_probes

• cat /proc/sys/net/ipv4/tcp_keepalive_time

Glassfish Tuning – KEEP ALIVE SETTINGS

As mentioned above, Apache and Glassfish regulates this and other options at the protocol level. Interesting at this point are:

• timeout-seconds: this is the time that GLASSFISH waits for a new request, before closing the connection. Since the internal thread behavior will be controlled by APACHE (it is APACHE’s thread after all), you should set it to unlimited, “-1″ (default ist 30)
• max-connections: this is the maximum number of requests a client can send over a connection before GLASSFISH closes the connection. Since the internal thread behavior will be controlled by APACHE (it is APACHE’s thread after all), you should set it to unlimited, “-1″, and let APACHE clean the opened  threads (the ones opened in Glassfish) and connections by itself (default is 256)

<protocol name="http-protocol">
<http xpowered-by="false" timeout-seconds="-1" max-connections="-1"
default-virtual-server="server" compressable-mime-type="text/html,
text/xml,text/plain,text/javascript,text/css" compression="on"
server-name="">
<file-cache enabled="false" />
</http>
</protocol>