• Start by determining the maximum capacity and throughput of your application server
• Determine how many concurrent users can you handle on your application server, and what is their memory footprint (memory used by one user session for the duration of using your solution)
• Determine the number of connections that you need to maintain open on the web server side in order to accommodate all users above
• Tune the corresponding thread pools (web server and application server) accordingly, so that only a determined number of connections can be opened on the web server, and that only a determined number of users can make it to the application server
• Determine the average number of business requests a user will execute on your system
• Tune the backlog of the web server so that you can prepare for spikes, and that you can determine when to send the surplus users to another web server or waiting server
• Tune the operating systems  for handling the configured number of connections – network, memory, open files, etc.

1. Long and very long loading and response times – nowadays any webpage loading in more than 3 to 5 seconds is annoying. Dirk, an ex colleague of mine was telling me that an increase of 0.1 seconds in the loading time of Amazon costs a fortune…
2. White pages and never loading page – This is even worse. A web page loading very slowly is still a loading web page. Yet, any new user landing on your product page, and experiencing a white page as his first experience will unlikely come again very soon.

• be processed immediately
• wait for a free working thread, and be processed with an acceptable delay
• be redirected to another web server
• be redirected to a wait server

• memory required / registered user: 1.5 MB
• memory required / unregistered user (simple visitor): 0.5 MB

Application Server Throughput

Application Server Workload model

• A business transaction is executed by a user every approximately 3 seconds
• The average response time for a transaction is between half a second and a second
• Worse case scenario (is a strategy I often use): x users / (3 seconds think time + 1 second response time) = 600 transactions  =>> x = 2400 concurrent “business” users (users executing business transactions)
• Light transactions (like users just visiting your webpage) are executed in a rate of 30 users / seconds, so that after one minute, you will have 60 seconds * 30 users = 1800 anonymous users

• 1800 anonymous users, needing 1800*0.5 MB= 900 MB
• 2400 registered users, needing 2700*1.5 MB = 3600 MB

Application Server HEAP structuring

• business transactions – executing some work in the database, persisting/modifying specific information, executing application logic with a result that is to be persisted
• light transactions – users just surfing on your page, not actually producing any workload on your database

2. Building the Apache Web Server Model

1. It costs less to spawn a large number of threads than to spawn new processes, as in the prefork model
2. The memory is shared between all threads belonging to a process
3. The number of threads can be dynamically increased using graceful restart of the web server
4. The number of threads used in the application server is somehow compressed, meaning that for each 2, 3, 4 apache threads, there will be one application server thread doing the application business. The application server thread will be at some point shared between web server threads. This way you can define a larger number of apache threads, with a lower number of application server threads

• how long will these users hang on this connection?
• what if the user sends one request, and than does not send any additional request? When will the connection be closed and the thread returned to the pool?
• what if the user opens the connection and regularly sends requests forever, keeping the connection open forever (Denial of service)

• If one user sends one request over a persistent connection, and then waits 59 seconds before sending the next request, he will reuse the same connection, and no other connection (apache thread) will be created. But:
• If one user sends a request over a persistent connection, and then waits longer than 60 seconds before sending the next one, or even worse, will never send an additional request, the connection will be kept open with no reason, and will be dropped without serving any more than one request. Not that optimal, is it?

Using this configuration and a time-scaled representation, after 60 seconds you would have the following scenario:

• 60 % (18 users) will continue just executing business requests, needing one single persistent connection / user
• 40 % (12 users) will continue just surfing, needing 6 persistent connection / user

Apache Connections Keep Alive

Compared to the first model, where we had to maintain 10 800 connections, you now have to only maintain 6840 connections!

But, one target was to be also prepared for spikes. What if we have more users / second than 30 ? What do we do with them, how do we react to that, and how do we prevent the scenarios described in the beginning (white page, long waiting times, etc.)

Well, since we know we can accommodate 1800 users, using the 6840 connections, we can prepare for an extra 50 % workload. That would mean we would get to about 10 000 connections (6840 + 3420) I will just use 10 000 for simplicity.

Right. But…what will happen with the connection request 10001 ?

Since we know that we have a loading time of about 6 seconds, no user should wait longer than 3 seconds to get a connection (so that the total response time still falls under 10 seconds)

With a workload of 30 users / second, we can accommodate:

( 30 users * 6 connections * 3 seconds ) requests in 3 seconds= 540 requests

But we just spoke of spikes, so let us increase the backlog with 50 % to  810 connections.

That means:

• First 10000 requests will get their threads
• requests between 10001 and 10810 will wait in the backlog for a free connection
• request 10811 will be either sent to a wait server or to another scaled web server

Tuning Apache Workers for Load - The connection and backlog model

Now that we decided on what we want to handle, let’s get our servers ready for this.

3. Configuring and tuning APACHE using Workers for performance and high number of connections

We need to take care both of APACHE configuration, and of configuration of the operating system where Apache resides

APACHE WebServer Configuration

We are ready to configure APACHE for 10 000  connections. Since it is more efficient to spawn threads than processes, let’s use a higher number of threads per process.

This is how a configuration could look like. With this configuration we will:

• start with 25 servers, each spawning 200 threads, resulting 5000 threads
• use about 50 k / thread, resulting in about 500 MB reserved memory
• expand to maximum 50 servers, each spawning 200 threads, resulting 10000 threads. This is also the value used by APACHE as MaxClients (maximum number of connections)
• start shrinking the number of threads once the load has decreased, and keep no less than 7500 threads in the pool
• allow no more than 100 000 requests / process, which results in an average of 500 requests / child thread. We use this in order to avoid nasty users holding to the process forever, and we also avoid memory leaks, by destroying the process after the number of requests has been reached

<IfModule worker.c>
# initial number of server processes to start
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers
StartServers        25
# highest possible MaxClients setting for the lifetime of the Apache process.
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit
ServerLimit         50
# minimum number of worker threads which are kept spare
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#minsparethreads
MinSpareThreads     1000
# maximum number of worker threads which are kept spare
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxsparethreads
MaxSpareThreads    7500
# upper limit on the configurable number of threads per child process
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadlimit
ThreadLimit        200
# maximum number of simultaneous client connections
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
MaxClients         10000
# number of worker threads created by each child process
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadsperchild
ThreadsPerChild     200
# maximum number of requests a server process serves
# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild
MaxRequestsPerChild  150
</IfModule>
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 10
ListenBackLog 810

A couple of things to consider:

1. MaxClients: If the allocated MaxClients is higher than ServerLimit * ThreadLimit Apache will automatically reduce MaxClients to the value of ServerLimit * ThreadLimit
   In this case, you will receive the following message:

   WARNING: MaxClients of 10000 would require 200 servers,
    and would exceed the ServerLimit value of 100.
    Automatically lowering MaxClients to 5000.  To increase,
    please see the ServerLimit directive.

2. Apache Memory / Process and Threads: Check if you have enough memory to handle the configured number of threads. Use the following scripts to monitor the Apache Servers, the number of threads, and the memory reserved:
   • The formula used for configuring your maximum number of clients is:
     
maxclients = total ram / ram per process
   • List all Apache processes and number of threads per process
     
for pid in `ps U wwwrun | grep httpd | grep -v grep | awk '{ print $1 }'`;
do echo Apache Worker Server $pid has `ps ms -p $pid | wc -l` threads;
done

     This will output:
     
Apache Worker Server 7528 has 3 threads
Apache Worker Server 7583 has 3 threads
Apache Worker Server 7587 has 3 threads
Apache Worker Server 7596 has 204 threads
Apache Worker Server 7601 has 204 threads
Apache Worker Server 7610 has 204 threads
Apache Worker Server 7618 has 204 threads
Apache Worker Server 7628 has 204 threads
Apache Worker Server 7640 has 204 threads
Apache Worker Server 7651 has 204 threads
Apache Worker Server 7668 has 204 threads
............
   • List the reserved memory per APACHE Server and threads
     
servers=0;threads=0;space=0;
for pid in `ps U wwwrun | grep httpd | grep -v grep | awk '{ print $1 }'`;
do process_threads=`ps ms -p $pid | wc -l`;
process_memory=`ps -ylC -p $pid | grep -v PID | awk '{ print $8}'`;
echo Apache Worker Server with pid $pid has $process_threads threads occupying $process_memory bytes;
servers=`expr $servers + 1`;threads=`expr $process_threads + $threads`;space=`expr $process_memory + $space`; done;
echo "-------------"; echo Total Apache Servers: $servers \| Total threads: $threads \| Total memory reserved: $space

     This will output:
     
.......
Apache Worker Server with pid 1733 has 204 threads occupying 34644 bytes
Apache Worker Server with pid 1759 has 204 threads occupying 34612 bytes
Apache Worker Server with pid 1788 has 204 threads occupying 34620 bytes
Apache Worker Server with pid 1816 has 204 threads occupying 34648 bytes
Apache Worker Server with pid 1844 has 204 threads occupying 34648 bytes
Apache Worker Server with pid 1875 has 204 threads occupying 34648 bytes
Apache Worker Server with pid 1907 has 204 threads occupying 34684 bytes
Apache Worker Server with pid 1940 has 204 threads occupying 34676 bytes
-------------
Total Apache Servers: 26 | Total threads: 5103 | Total memory reserved: 885272
3. ThreadsPerchild vs ThreadLimit: There is a difference between ThreadsPerchild and ThreadLimit. ThreadsPerChild defines the initial number of threads spawned per worker process. Defining a larger number as ThreadLimit allows you to modify the ThreadsPerChild up to ThreadLimit, without needing to do a hard restart of Apache. A graceful restart will suffice.For example:

   ThreadsPerChild 100
   ThreadLimit 500

   This will start the apache processes with 100 threads each. If you feel that your webserver cannot handle the current load with the current configuration, you just need to increase the ThreadsPerChild, (in this case to maximum  500) and execute a graceful restart:

   ThreadsPerChild 300
   ThreadLimit 500

   Yet, it is very important to remember that:

1. ThreadLimit will allocate memory in advance, so check in advance if you can handle the ServerLimit * ThreadsPerChild with your current RAM
2. If you increase the ThreadsPerChild up to a value that the APACHE Server will consider it cannot allocate the needed memory, APACHE will exit!
4. ServerLimit vs ThreadLimit: You cannot increase the maximum number of server processes ServerLimit without hard restart of APACHE, so take this into consideration when planning the baseline configuration. If you want to be prepared for a higher load, configure the ThreadLimit higher, which will allow you to increase the capacity of your server with a graceful restart
5. Number of Apache Threads spawned: Make sure that the number of threads spawned is the one you expect. Use the following to check the number of threads spawned.”Sl” is for Interruptable sleep, multithreaded

   ps axsm | grep -c 'Sl'

6. Denial of Service and Keep Alive Parameter: Be prepared for Denial of Service Attacks. Do not allow any user to hold on to a persistent connection forever. Decide on the average number of transactions that a user will execute while using your system, and add it as parameter:

   KeepAlive On
   MaxKeepAliveRequests 1000
   KeepAliveTimeout 10

7. Backlog for extra spikes:Be prepared for those extra spikes, where all your threads are busy, and you need a “waiting room” for your users. Use the ListenBackLog Directive, which we configured to 810:

   ListenBackLog 810

8. ONE OF THE MOST IMPORTANT:
   The only way you can control the life of an APACHE server, and its children, is using this two parameters. Set them too high, and they will never die, therefore, the number of Threads in your Application Server will increase continously, eventually reaching the maximum configured number of threads, where it will stop servicing requests.
   Now, there are two parameters that control the behaviour of the apache servers and their children, and therefore the behaviour of the thread pool on the application server side. MaxRequestsPerChild refers to the maximum number of NEW requests (meaning a new client, opening a completely new Keep Alive Connection on this child) that can be sent over one Apache child. If for example, you had one client, opening a connection, and sending 500 requests, one after the other, on the same keep alive connection, than this parameter would have the value 1 (only the first request over a new keep alive connection will be counted) On the other hand, you do not want one client to hold on forever on his connection (remember the DOS), so you control that by setting the maximum, total number of requests a client can send over a SINGLE keep alive connection. When one of the two happens, the client will be assigned a new child thread, and the thread having served will be marked as “dead”

   MaxRequestsPerChild  150
   MaxKeepAliveRequests 1000
   

APACHE Operating System Configuration

Apache Tuning – Max Open Files

Since every socket in Unix is actually a file, we need to tune the maximum allowed number of open files.

This specifies the number of open files that are supported.
The default setting is typically sufficient for most applications. 
If the value set for this parameter is too low, a file open error,
memory allocation failure, or connection establishment error might
be displayed.

More on this here: IBM Websphere Linux Tuning

Apache Tuning – TCP Settings

Keep in mind the seven layer architecture (also known as OSI model) Both Glassfish and Apache control the transport layer, which sits further on the network layer.

OSI Model - Seven layer architecture

Image courtesy: Novell - http://www.novell.com/info/primer/prim05.html

You can configure and limit GLASSFISH and APACHE resources, as long as you do not exceed the network layer settings. That is why we need to take that also into consideration, and tune them accordingly, so that we can plug in the Glassfish or Apache without exceeding any configured resources

Apache Tuning – Backlog and Maximum Connections

Change the following parameters when you prepare for a high rate of incoming connections. Keep in mind that this setting is shared between all Apache Servers, so that if you want to host several web servers, you need to be prepared to accommodate the connections for all servers:

• Number of connections in backlog: cat /proc/sys/net/core/netdev_max_backlog Default is 1000, modify this according to the maximum expected number of connections that you want to accommodate in your backlog (keep them waiting for a connection)
• Size of the listen queue, in bytes, for accepting new tcp connections: cat /proc/sys/net/core/somaxconn Default is 128, modify this according to the size of the backlog you would need for the maximum expected number of connections that you want to accommodate in your backlog

Apache Tuning – Operating System – Keep Alive Settings

Although Unix has built-in support for KeepAlive, this is not the default behavior in Linux. Programs must request keepalive control for their sockets using the setsockoptinterface. http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.htmlThere are three “tunable” parameters:

tcp_keepalive_time

the interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe; after the connection is marked to need keepalive, this counter is not used any further

tcp_keepalive_intvl

the interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime

tcp_keepalive_probes

the number of unacknowledged probes to send before considering the connection dead and notifying the application layer

As long as the keep alive is high enough (higher than your configured Apache Server) you should not worry, since Apache will take care of managing the connections. Just make sure that the settings are higher than the ones configured in your web and application server. You can check all three by issuing the following command (i have also listed the default values for SUSE Linux Enterprise Server 11 ):

• cat /proc/sys/net/ipv4/tcp_keepalive_intvl  : 75
• cat /proc/sys/net/ipv4/tcp_keepalive_probes: 9
• cat /proc/sys/net/ipv4/tcp_keepalive_time: 7200 (seconds)

Apache Tuning – KEEP ALIVE SETTINGS

As defined above, these are the three configuration parameters you’d have to take care of when configuring for keep-alive connections (repeated, see above, chapter 2):

• KeepAlive (do we allow persistent connections at all)
• KeepAliveTimeout (how long should we wait on the client sending an additional request over the connection before closing the connection and returning the thread to the thread pool)
• MaxKeepAliveRequests (how many requests is a client allowed to send over one persistent connection before the server closes the connection and returns the thread to the thread pool)

Apache Tuning – Operating System – CONNECTION MANAGEMENT

tcp_fin_timeout:

This basically holds a connection in a   TIME_WAIT mode (after the keep alive timeout expired), waiting for the same client to reinitiate the communication. Basically it says that it is cheaper to reactivate a sleeping connection, than to build a new one. Maintaining a connection open, after the user has not sent any additional request in the KeepAliveTimeout timeout defined may be expensive, since other users may just well sit in the backlog, waiting for exactly that one connection to be freed. If this is the case, you should set this to a level as low as possible, so that after a small timeout, the connection can be finally destroyed, and rebuilt with another id, for another user.

“This determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. This interval between closure and release is known as the TIME_WAIT state or twice the maximum segment lifetime (2MSL) state. During this time, reopening the connection to the client and server cost less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, providing more resources for new connections. Adjust this parameter if the running application requires rapid release, the creation of new connections, and a low throughput due to many connections sitting in the TIME_WAIT state.”

More on this here: IBM WebSphere Tuning

A simple way to see how many connections you have in TIME_WAIT is using netstat:

• watch --interval=1 "netstat | grep -c TIME_WAIT"

You can view and set this parameter as follows:

• View: cat /proc/sys/net/ipv4/tcp_fin_timeout : 60 (default on SUSE Linux Enterprise Server 11) 
• Set: echo 10 > /proc/sys/net/ipv4/tcp_fin_timeout

Apache Tuning – Compression

Most modern browsers can handle compressed content, decompressing it upon receiving it. It is a great idea to compress content, which will save you a lot of bandwidth. Of course, this comes with a price – CPU is needed for compressing the files. Since you do not want your application server to use resources for compression, leave it to the web server. And since Apache has caching mechanisms, this should not be a problem, since the file will be zipped once, and then saved and served in/from cache

Most compression algorithms, when applied to a plain-text file, can reduce its size by 70% or more, depending on the content in the file. When using compression algorithms, the difference between standard and maximum compression levels is small, especially when you consider the extra CPU time necessary to process these extra compression passes. This is quite important when dynamically compressing Web content. Most software content compression techniques use a compression level of 6 (out of 9 levels) to conserve CPU cycles. The file size difference between level 6 and level 9 is usually so small as to be not worth the extra time involved.

http://www.linuxjournal.com/article/6802

There are two modules for using compression in Apache:

• mod_gzip
• mod_deflate

Just be aware of the compression directives:

• type of files you need to compress (it does not make sense compressing already compressed files like pdf, jpg, etc.): AddOutputFilterByType DEFLATE text/html text/plain text/xml

Apache Tuning – Caching

Related to the point above, you can benefit of both compressing and caching content. You can either use a disk based cache store manager or a faster, memory based store manager

mod_cache implements an RFC 2616 compliant HTTP content cache that can be used to cache either local or proxied content. mod_cache requires the services of one or more storage management modules. Two storage management modules are included in the base Apache distribution:

mod_mem_cache

implements a memory based storage manager. mod_mem_cache can be configured to operate in two modes: caching open file descriptors or caching objects in heap storage. mod_mem_cache can be used to cache locally generated content or to cache backend server content for mod_proxy when configured using ProxyPass (aka reverse proxy)

More on this here: http://httpd.apache.org/docs/2.2/mod/mod_cache.html

4. Configuring and tuning the Glassfish Application Server to work with the Workers Model

One thing you need to understand is that in the worker model, APACHE controls the lifecycle of working threads in Glassfish. Therefore, we need to configure Glassfish knowing that APACHE takes care of things like:

• thread removal
• keep alive timeouts
• maximum number of requests / connection
• compression
• etc

Most UNIX distributions come with an out of the box configuration, that is prepared for normal workload, but definitely not for performance. We need to be aware of network, memory and other system resource settings that we have to reconfigure in order to gain and prepare for performance.

Glassfish Tuning

Glassfish Tuning – Max Open Files

Once again, since every socket in Unix is actually a file, we need to tune the maximum allowed number of open files.

You can of course set it to unlimited, but since we spoke of a APACHE Thread Pool -> Glassfish Thread Pool compression, i would set this to half the number of threads apache can spawn + extra 5000 for all other processes

ulimit -n 10000

Yet, be aware that if you have multiple Glassfish Domains Application Server, they will all share the number of open files. Keep this in mind when calculating the number of possible open threads / server.

Glassfish Tuning – Max Threads

Since your system can perform up to about 600 transactions / second, you should configure your maximum thread pool according to this number. Allow about 50 % more threads for those extra spikes, even if that comes with longer response times.

I would therefore set the maximum thread pool size to about (rounded) 1000:

<thread-pool max-thread-pool-size="1024" name="http-thread-pool" />

Glassfish Tuning – TCP Settings

Keep in mind the seven layer architecture (also known as OSI model) Both Glassfish and Apache control the transport layer, which sits further on the network layer (see above, in chapter 3, Apache Tuning – TCP Settings)

You can configure and limit GLASSFISH and APACHE resources, as long as you do not exceed the network layer settings. That is why we need to take that also into consideration, and tune them accordingly, so that we can plug in the Glassfish or Apache without exceeding any configured resources

Glassfish Tuning – Backlog and Maximum Connections

Change the following parameters when you prepare for a high rate of incoming connections. Keep in mind that this setting is shared between all Glassfish Servers respectively Apache Servers:

• Number of connections in backlog:
  /proc/sys/net/core/netdev_max_backlog
  Default is 1000, modify this according to the maximum expected number of connections that you want to accommodate in your backlog (keep them waiting for a connection). Remember that these connections are the ones opened by APACHE, so you may want to tune this according to the maximum number of threads you will allow Apache to start inside the application server
• Size of the listen queue, in bytes, for accepting new tcp connections:
  /proc/sys/net/core/somaxconn
  Default is 128, modify this according to the size of the backlog you would need for the maximum expected number of connections that you want to accommodate in your backlog

Glassfish Tuning – Operating System – Keep Alive Settings

Although Unix has built-in support for KeepAlive, this is not the default behavior in Linux. Programs must request keepalive control for their sockets using the setsockoptinterface.

http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html

There are three “tunable” parameters:

As long as the keep alive is high enough (higher than your configured Glassfish or Apache Server) you should not worry, since Apache or Glassfish will take care of managing the connections. Just make sure that the settings are higher than the ones configured in your web and application server.

You can check all three by issuing the following command:

• cat /proc/sys/net/ipv4/tcp_keepalive_intvl

• cat /proc/sys/net/ipv4/tcp_keepalive_probes

• cat /proc/sys/net/ipv4/tcp_keepalive_time

Glassfish Tuning – KEEP ALIVE SETTINGS

As mentioned above, Apache and Glassfish regulates this and other options at the protocol level. Interesting at this point are:

• timeout-seconds: this is the time that GLASSFISH waits for a new request, before closing the connection. Since the internal thread behavior will be controlled by APACHE (it is APACHE’s thread after all), you should set it to unlimited, “-1″ (default ist 30)
• max-connections: this is the maximum number of requests a client can send over a connection before GLASSFISH closes the connection. Since the internal thread behavior will be controlled by APACHE (it is APACHE’s thread after all), you should set it to unlimited, “-1″, and let APACHE clean the opened  threads (the ones opened in Glassfish) and connections by itself (default is 256)

<protocol name="http-protocol">
<http xpowered-by="false" timeout-seconds="-1" max-connections="-1"
default-virtual-server="server" compressable-mime-type="text/html,
text/xml,text/plain,text/javascript,text/css" compression="on"
server-name="">
<file-cache enabled="false" />
</http>
</protocol>

• run_scenario_warmup: the function takes a command line input parameter which controls if the test is going to be run with a warm-up first or not. It will start a test with a reduced number of virtual users in order to warm up the application server. The test will output the results into a simple log file (not xml), so that the warm up results will not be taken into consideration when importing the test results generated by the load test.
  
function run_scenario_warmup
{
if [ "$WARMUP" != "yes" ];
then
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Set up: No server warm up. Test will be started without warming up the server" | tee -a ${JMETER_RESULTS}/run.log
else
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Set up: Warming up server by running the scenario with "${WARMUP_THREAD_USERS}" users on one agent executing exactly 50 payment transactions" | tee -a ${JMETER_RESULTS}/run.log
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Running: Warming up started" | tee -a ${JMETER_RESULTS}/run.log
jmeter -n -t ${JMETER_TESTPLANS}${TESTPLAN_SUBFOLDER}${TESTPLAN_FILE_NAME} -R agent1 -l ${JMETER_RESULTS}${TESTPLAN_FILE_NAME}.log -Ggroup1.hostname=${HOST} -Ggroup1.port=${PORT} -Ggroup1.protocol=${PROTOCOL} -Ggroup1.fullhost=${FULLHOST} -Ggroup2.threads=${threads_start} -Ggroup2.ramptime=${rampup_start} -Ggroup2.users=${THREAD_USERS} -Ggroup2.startid=${START_ID} -Ggroup2.synctimer=${SYNC_TIMER} -Ggroup3.pay_transactions=50-Ggroup4.pausetimeconst=${PAUSE_TIME} -Ggroup4.pausetimerandom=${PAUSE_TIME_DEV} -G ${JMETER_PROPFILES}${TESTPLAN_SUBFOLDER}${TESTPLAN_FILE_NAME}.properties &
wait_for_plan '${TESTPLAN_FILE_NAME}'
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Tear down: Warming up ended, sleeping for 10 seconds" | tee -a ${JMETER_RESULTS}/run.log
sleep 10
fi
}

• trace_class_loading: this function will trace memory usage statistics every 10 seconds, using jmap. It will connect to the application server and retrieve a map of objects periodically, which will be stored into an export file, that will be processed in the end. The type of objects can be extended by simply adding a regular expression. Currently it retrieves hashmaps,vectors, eclipse persistence objects, javascript objects etc. The function uses a temporary file called “running” which resides in the “/tmp” folder. While the file exists, the collection of statistics keeps going, and stops once the test stops. (when the test stops, the temporary file is removed, so that in the next check, the statistics function will not find the file and will exit). In the end, the file is processed and exported into a final jmap_objects file, which will be used for exporting memory information in the performance database
  function trace_class_loading
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Tracing: Starting to collect Heap Usage statistics with a refresh of 10 seconds" | tee -a ${JMETER_RESULTS}/run.log
glassfish_pid=`ssh ${DOMAIN_SERVER} sudo -u glassfish /bin/bash -c '/usr/java/jdk1.6.0_24/bin/jps -q | grep ASMain ' | awk '{print$1}'`
echo "glassfih pid is " $glassfish_pid
status=`ls /tmp | grep running`
# As long as the script is running perform Jmap histogram on the server every x seconds, where x is defined in sleep command


while [ "$status" != "" ];
do
JMAP_TIMESTAMP=`date +%H-%M-%S`
ssh ${DOMAIN_SERVER} "sudo -u glassfish /bin/bash -c '${JAVA_BIN}jmap -histo ${glassfish_pid}'" | egrep -i -e '\[[I,B,C]+' -e 'myobjects' -e 'java.util.Tree' -e 'java.util.Hash' -e 'java.util.Vector' -e 'Klass' -e 'org.mozilla' -e 'com.sun.tools.javac.zip.ZipFileIndexEntry' -e 'java.lang' -e 'java.util.concurrent' -e 'org.eclipse.persistence' | sed -e '/[0-9]/s/$/',"${JMAP_TIMESTAMP}"'/' >> /tmp/jmap_objects.log
sleep 10
status=`ls /tmp | grep running`
done

echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Tracing: Collection of Heap Usage Statistics ended" | tee -a ${JMETER_RESULTS}/run.log

# After the script has ended perform a full garbage collector. Comment this line if you do not want a FULL GC

ssh ${DOMAIN_SERVER} "sudo -u glassfish /bin/bash -c '${JAVA_BIN}jmap -histo:live ${glassfish_pid}'" | egrep -i -e '\[[I,B,C]+' -e 'myobjects' -e 'java.util.Tree' -e 'java.util.Hash' -e 'java.util.Vector' -e 'Klass' >> /tmp/jmap_objects.log

# Move the jmap log in the results folder for further processing

mv /tmp/jmap_objects.log ${JMETER_RESULTS}/jmap_objects.log

#Process jmap log for removing spaces - old and very slow

awk '{printf("%s,%s,%s,%s,%s\n",$1,$2,$3,$4,$5);}' ${JMETER_RESULTS}/jmap_objects.log | awk '{gsub(/:/,"");print}' > ${JMETER_RESULTS}/processed_jmap_objects.log
cp ${JMETER_RESULTS}/processed_jmap_objects.log ${JMETER_TRANSFORMATION}processed_jmap_objects.log}

  
• trace_app_statistics: this function uses glassfish’s rest monitoring interface and curl commands for retrieving information such as jdbc connections, http thread usage, number of specific beans in cache, etc. This function is very flexible, as all data is written in the database in the form of: test_id, timestamp, label, value. Thank to this, any new monitoring item is just a new label in the database, and can be retrieved in a separate report by building a simple query. I will post here a sample of this function, which collects information regarding JDBC Connection usage. It can be easily extended by simply adding a new monitoring item, for example HTTP_THREAD_POOL_THREAD_COUNT
  
function trace_app_statistics
{
status=`ls /tmp | grep glassfish_stats`
while [ "$status" != "" ];
do
MONITOR_TIMESTAMP=`date +%H-%M-%S`
#JDBC Monitoring
JDBC_CONN_USED=`curl -s -u user:password http://glassfish:4848/monitoring/domain/server/resources/EocPool | grep numconnused | grep -o -E '"current":[0-9]*' | sed 's/["]*[a-z]*["][:]*//'`

echo $MONITOR_TIMESTAMP":JDBC - Connections used:"$JDBC_CONN_USED >> ${JMETER_RESULTS}/glassfish_stats.log

sleep $CMD_PARAM_INTERVAL
status=`ls /tmp | grep glassfish_stats`
done
cp ${JMETER_RESULTS}/glassfish_stats.log ${JMETER_TRANSFORMATION}glassfish_stats.log
}


• run_scenario: this is the main function controlling the run of the test. It takes several parameters as input, such as number of repetitions, number of virtual users, number of loops inside a repetition, number of virtual users to be increased in a loop, ramp up time, timers, etc. It sets the basis for running the test scenario in the following form:
  
./PAYMENT_SCENARIO 'users=150 ramptime=150 pausetime=2000 pausetimedev=200 pay_transactions=500 repeats=1 loopsinrepeat=1 waitforuserinput=no minthreads=150 maxthreads=200 generatereport=no warmup=no startid=100' 'hostname=myappserver port=8080 protocol=http'

  It will repeat the test for the desired number of times, increasing the number of users progressively by the defined number and so on.
  
function run_scenario
{
scenario_counter=$REPETITIONS
cool_down=0
while [ "$scenario_counter" != "0" ];
do
# Repeat test for a defined number of times (counter)
counter_start=$LOOPS
counter_end=$LOOPS
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Set up: Starting the load test" | tee -a ${JMETER_RESULTS}/run.log
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Set up: Starting iteration number " `expr $REPETITIONS - $scenario_counter + 1` "out of " $REPETITIONS "iterations" | tee -a ${JMETER_RESULTS}/run.log
threads_start=${VIRTUAL_USERS}
rampup_start=${RAMP_TIME}
while [ "$counter_start" != "0" ];
do
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Running: Running with a number of " ${threads_start} " virtual users / agent" | tee -a ${JMETER_RESULTS}/run.log
counter_start=`expr $counter_start - 1`
jmeter -n -t ${JMETER_TESTPLANS}${TESTPLAN_SUBFOLDER}${TESTPLAN_FILE_NAME} -R ${JMETER_AGENTS} -l ${JMETER_RESULTS}/${TESTPLAN_FILE_NAME}.xml -Ggroup1.hostname=${HOST} -Ggroup1.port=${PORT} -Ggroup1.protocol=${PROTOCOL} -Ggroup1.fullhost=${FULLHOST} -Ggroup2.threads=${threads_start} -Ggroup2.ramptime=${rampup_start} -Ggroup2.users=${THREAD_USERS} -Ggroup2.startid=${START_ID} -Ggroup2.synctimer=${SYNC_TIMER} -Ggroup3.pay_transactions=${TRANSACTIONS} -Ggroup4.pausetimeconst=${PAUSE_TIME} -Ggroup4.pausetimerandom=${PAUSE_TIME_DEV} -G ${JMETER_PROPFILES}${TESTPLAN_SUBFOLDER}${TESTPLAN_FILE_NAME}.properties &
wait_for_plan '${TESTPLAN_FILE_NAME}'
threads_start=`expr $threads_start + $ADDEDVU`
if [ "$counter_start" != "0" ];
then echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Running: Increasing number of virtual users by " ${ADDEDVU} "users / agent" | tee -a ${JMETER_RESULTS}/run.log
fi
done
# Cool Down
scenario_counter=`expr $scenario_counter - 1`
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Teardown: Load test ended, sleeping for 30 seconds" | tee -a ${JMETER_RESULTS}/run.log
sleep 10
done
}


• wait_for_user_input: this function waits for the users input in order to stop the test scenario, and to start importing the results. It is useful when you want to keep collecting memory statistics until the test is ended, and you want for example to do a full garbage collection, so that, before importing, you have a list of all objects that are left after the test run, and after a major collection. These will be good candidates for memory leaks. The function takes a command line parameter as input. Once the wait is over, all temporary files are erased, and all functions that run by checking the existence of such temporary files will exit, allowing the main scenario to end, and start the processing of test results and importing into the performance database
  
function wait_for_user_input
{
if [ "$USERINPUT" != "no" ];
then
while : ; do
read -t 2 && break
done
echo exited while loop
rm -r /tmp/running
rm -r /tmp/jms_monitoring
rm -r /tmp/glassfish_stats
else
rm -r /tmp/running
rm -r /tmp/jms_monitoring
rm -r /tmp/glassfish_stats
fi
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Analysis Services: Sleeping for 30 seconds to allow jms and performance monitor to stop gracefully" | tee -a ${JMETER_RESULTS}/run.log
sleep 30
}

TEAR DOWN

• collect_logs
  
function collect_logs
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Analysis Services: Collecting and compressing application server logs (jvm,jgc,jms,server)" | tee -a ${JMETER_RESULTS}/run.log
cd ${JMETER_RESULTS}
# Collect Glassfish logs on Server
scp ${DOMAIN_SERVER}:${GLASSFISH_PATH}/${DOMAIN_NAME}/logs/server.log* ${JMETER_RESULTS} || on_error "${LOG_TIMESTAMP}: ERROR - Could not perform Glassfish log collecting via SCP on server ${DOMAIN_SERVER} "
zip -j ${JMETER_RESULTS}/${testrun_id}-server.zip ${JMETER_RESULTS}/server.log*
rm server.log*
# Collect JVM logs on server
scp ${DOMAIN_SERVER}:${GLASSFISH_PATH}/${DOMAIN_NAME}/logs/jvm.log* ${JMETER_RESULTS}/${testrun_id}-jvm-original.log || on_error "${LOG_TIMESTAMP}: ERROR - Could not perform JVM log collecting via SCP on server ${DOMAIN_SERVER} "
sed 's/\x0//g' ${JMETER_RESULTS}/${testrun_id}-jvm-original.log > ${JMETER_RESULTS}/${testrun_id}-jvm.log
rm ${JMETER_RESULTS}/${testrun_id}-jvm-original.log
zip -j ${JMETER_RESULTS}/${testrun_id}-server.zip ${JMETER_RESULTS}/${testrun_id}-jvm.log*
# Collect Java Garbage collection output on server
scp ${DOMAIN_SERVER}:${GLASSFISH_PATH}/${DOMAIN_NAME}/logs/jgc.log* ${JMETER_RESULTS}/${testrun_id}-jgc-original.log || on_error "${LOG_TIMESTAMP}: ERROR - Could not perform JGC log collecting via SCP on server ${DOMAIN_SERVER} "
sed 's/\x0//g' ${JMETER_RESULTS}/${testrun_id}-jgc-original.log > ${JMETER_RESULTS}/${testrun_id}-jgc.log
rm ${JMETER_RESULTS}/${testrun_id}-jgc-original.log
zip -j ${JMETER_RESULTS}/${testrun_id}-server.zip ${JMETER_RESULTS}/${testrun_id}-jgc.log*
# Collect Java Garbage collection output on server
scp ${DOMAIN_SERVER}:${GLASSFISH_PATH}/${DOMAIN_NAME}/logs/jms.log* ${JMETER_RESULTS}/${testrun_id}-jms-original.log || on_error "${LOG_TIMESTAMP}: ERROR - Could not perform JGC log collecting via SCP on server ${DOMAIN_SERVER} "
sed 's/\x0//g' ${JMETER_RESULTS}/${testrun_id}-jms-original.log > ${JMETER_RESULTS}/${testrun_id}-jms.log
rm ${JMETER_RESULTS}/${testrun_id}-jms-original.log
# Calculate peak number of jms messages while test was running
TOTAL_JMS_MSG=`sort -r -t ' ' +1 -n ${JMETER_RESULTS}/${testrun_id}-jms.log | awk '{print $1}' | sed q`
TOTAL_JMS_BYTES=`sort -r -t ' ' +3 -n ${JMETER_RESULTS}/${testrun_id}-jms.log | awk '{print $3}' | sed q`
PEAK_JMS_MSG=`sort -r -t ' ' +6 -n ${JMETER_RESULTS}/${testrun_id}-jms.log | awk '{print $6}' | sed q`
PEAK_JMS_TOTAL_BYTES=`sort -r -t ' ' +9 -n ${JMETER_RESULTS}/${testrun_id}-jms.log | awk '{print $9}' | sed q`
zip -j ${JMETER_RESULTS}/${testrun_id}-server.zip ${JMETER_RESULTS}/${testrun_id}-jms.log*
  return
  }

• write_test_info: this function writes the necessary test configuration information in an xml file. It generates an increasing unique id for the testrun_id, and writes the testinformation from multiple sources that were provided as command line parameters, or as results of other functions called in the set up or run time phase. It will write all information needed to identify the load configuration and the server configuration at the time of test run: number of users, ramp up, number of processing threads, garbage collection strategy, jvm heap configuration, jdbc pool configuration, etc.
  
  
function write_testInfo
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Analysis Services: Writing Test Summary for importing in Test Results" | tee -a ${JMETER_RESULTS}/run.log
  # Get unique ID for the script,in order to store in DB. ID is formed of the last digits of IP address and an id that is incremented
  # Get last digits of IP address
  typeset -i test_id
  local_ip=`/sbin/ifconfig | grep ‘inet addr:’| grep -v ’127.0.0.1′ | grep -v ’192.168′ | cut -d: -f2 | cut -d. -f4 | awk ‘{ print $1}’`
  test_id=$(cat ${ID_FILE})
  new_id=`expr $test_id + 1`
  echo $new_id >${ID_FILE}
  testrun_id=$local_ip$new_id
  # Getting testdescription and server configuration
  echo $threadnum
  echo $threadwarm
  # Copying the glassfish server logs to a path configured to be the logging path for the reporting server
  cp ${JMETER_RESULTS}/${testrun_id}-server.zip ${JMETER_SERVER_LOGS}${testrun_id}_server.zip
  # Writing xml test information
  CODE_500=`grep -c ‘rc=”5′ ${JMETER_RESULTS}/*.xml`
  CODE_400=`grep -c ‘rc=”4′ ${JMETER_RESULTS}/*.xml`
  CODE_200=`grep -c ‘rc=”2′ ${JMETER_RESULTS}/*.xml`
  
  
echo '<?xml version="1.0" encoding="UTF-8"?><testInfo build="'$REVISION_NUMBER'" test_id="'$testrun_id'" portal_version="'$PORTAL_VERSION'" revision_number="'$REVISION_NUMBER'" scenario_id="'${SCRIPT_NAME}'" architecture_id="'$ARCHITECTURE_ID'" test_date="'$LOG_RUNDATE $LOG_TIMESTAMP'" db_recordsindb="'$DBINFO_RECORDSINDB'"><test-configuration threads="'$VIRTUAL_USERS'" rampup="'$RAMP_TIME'" payment_transactions="'$TRANSACTIONS'" waittime="'$PAUSE_TIME'" waitdeviation="'$PAUSE_TIME_DEV'"></test-configuration><glassfish-configuration server_threads_min="'$GF_THREADS_MIN'" server_threads_max="'$GF_THREADS_MAX'" jdbc_thread_pool_min="'$JDBC_THREAD_POOL_MIN'" jdbc_thread_pool_max="'$JDBC_THREAD_POOL_MAX'" jdbc_statement_cache="'$JDBC_STATEMENT_CACHE'"></glassfish-configuration><jvm-options gc_strategy_param1="'$GC_STRATEGY_PARAM1'" total_initial_heap="'$TOTAL_INITIAL_HEAP'" total_maximum_heap="'$TOTAL_MAXIMUM_HEAP'" newgen_initial_heap="'$NEWGEN_INITIAL_HEAP'" newgen_maximum_heap="'$NEWGEN_MAXIMUM_HEAP'" survivor_ratio="'$SURVIVOR_RATIO'" log_level="'$LOG_LEVEL'"></jvm-options><jms-usage>total_jms_msg="'$TOTAL_JMS_MSG'" total_jms_bytes="'$TOTAL_JMS_BYTES'" peak_jms_msg="'$PEAK_JMS_MSG'" peak_jms_total_bytes="'$PEAK_JMS_TOTAL_BYTES'"</jms-usage><statusCodes>code_500="'$CODE_500'" code_400="'$CODE_400'" code_200="'$CODE_200'" </statusCodes></testInfo>' > ${JMETER_RESULTS}/testInfo.xml
return
}

• log_heap_config: this function analysis the jvm configuration on the server by using “jmap” with “-heap” parameter
  
  
function log_heap_config
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Analysis Services: Collecting Glassfish heap configuration" | tee -a ${JMETER_RESULTS}/run.log
glassfish_pid=`ssh ${DOMAIN_SERVER} sudo -u glassfish /bin/bash -c '${JAVA_BIN}jps -q | grep ASMain ' | awk '{print$1}'`
ssh ${DOMAIN_SERVER} "sudo -u glassfish /bin/bash -c '${JAVA_BIN}jmap -heap ${glassfish_pid}'" > ${JMETER_RESULTS}/${testrun_id}-heapconfig.txt
}


• check_reporting: this function checks if the import of testresults is wished after the test run. This is controlled by a command line parameter called “REPORTING“. If no reporting is wished, then just a small processing is performed, for hudson analysis. Otherwise, the “data_transformation” function will be called, which will start the exporting of data in the database.
  
function check_reporting
{
if [ "$REPORTING" == "yes" ];
then
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Analysis Services: Sleeping for 30 seconds before starting data transformation services" | tee -a ${JMETER_RESULTS}/run.log
data_transformation
else
hudson_transformation
fi
}


• log_test_end: this function signalises the end of the test by outputing a log message to the console and in the test run log file.
  
  
function log_test_end
{
echo "${PLACEHOLDER}`date +%H-%M-%S`: LOGGING - Test Finish: Test Scenario Ended" | tee -a ${JMETER_RESULTS}/run.log
}

.

In the second article i discussed how to build an integrated load solution with JMeter. I covered topics such as: dynamic test plans, test scripts, test repository, shell functions, monitoring functions, logging functions, etc. I talked about the structure of a test script template, and detailed the shell functions called in it. It is now time to dive into the third part, one of the most interesting maybe in this series of articles:

ETL – EXTRACT TRANSFORM AND LOAD – DATA PROCESSING AND IMPORTING INTO THE DATABASE

Database specfic information:

• number of records in the database before starting the test and after running the test (records of interest to the test)

You can download the pdf (BI version) of this report from the following location:

http://qants.files.wordpress.com/2011/05/template-1-full-report.pdf

Jasper Report – Template 2 - Performance report / request – Drill down reporting

If we wanted to see the behaviour of one request involved in the test, we would need to be able to select a single request from the main report, and drill down into it, analyzing the above information only for the selected request. The final report will look almost identical to the main report, the only difference being that all other requests will be filtered (not shown and not taken into consideration) This allows us to take an isolated look on the behaviour of a specific request with regard to the behaviour of the test in general

This is how the header will look like for the “Home” transaction, after drilling down from the report above. The drill down will select a transaction, and rebuild the report only for that transaction

Drill down performance report

Surely, the chart will also be filtered (this is only a sample of it, since it does not match the size of the image), and will follow exactly the same structure as Template 1, the main report

Jasper Report – Template 3 – Performance report sampling – time filtering

If we wanted to see the behaviour of the application under test at a specific point in time, we would need to be able to generate the report only for a selected timeframe. If for example we notice a sudden drop down in the throughput (number of transactions / second), we need to take a deeper look in the timeframe when this happened. The filtering of the timestamp will be reflected in all information, from response times, to the chart. (response times, average transactions, min and max times, etc.)

This is also very useful to see when the application is “on top”: all users have logged in, and the maximum number of transactions / second can be acchieved at this moment. You can therefore use such report to identify the “maximum number of transactions on top” and the corresponding response times

This is how the chart will look like after filtering for one minute ( the selected timeframe) of the test. As one can see, the data is shown only for the selected timeframe.

Jasper Report – Template 4 – Performance Report Memory usage

Template 4.1 – Application specific – Application own objects

We need to be able to monitor specific objects over time. We can have multiple reports, each representing different memory performance areas. The report below is a graphical representation of usage of objects belonging to the application under test, over time. Since the number of objects used in an application can be very high, the chart needs to be extended horizontally, therefore the sample below is a zoomed out version of the chart. As one can see, each object is represented with a different color, over time, so one can detect increases or decreases in number of objects over time.

The right side displays the legend, showing all object names and corresponding colors and forms used in representing in the chart.

This is a sample of the report above, which demonstrates how the chart looks when zoomed in

Template 4.2 – Application specific – Application own objects – Filtered object

We need to drill down in the above report, by selecting the object for which we want a detailed report. This is how the report will look like. The upper part denotes the number of objects, the lower one the storage needed.

Template 4.3 – Application specific – Java Base Type objects

This report is using the same concept as the one above, but only the java base types are displayed (chars, integers, arrays, double arrays, etc.) The upper part displays the number of objects in memory at the time of sample, the one below displays the total number of bytes occupied in memory.

As in the chart above, the history is displayed on the right side

Basically, as it will be explained in the details section, the memory analysis reporting can be extended to any pattern of objects by simply adding them to the function collecting the memory usage information. In the end it is a question of creating a separate query for each of the type of objects we need, and use it in a separate dataset or report. I have for example created different reports for eclipselink objects, javascript objects, vectors lists and arrays, etc.

I discussed about the outputs, the performance reports. Now i will focus myself on how to build the integrated load testing solution, from generating the load to presenting it in graphical representations, and integrating them into continuous integration systems, or automating them as individual test suites.

The rest of the items following will be presented using the following structure:

CONFIGURING LOAD GENERATORS

1. Load testing generators – Getting your application under load with distributed JMeter
   • basic concept of jmeter in distributed mode and network architecturing
   • detail of the concept to which this white paper is referring
   • detail of the folder structure needed (results, temporary folders, transformation folders,classes, scripts and testplan folders,property files)
   • configuring the server and the agents
2. Test plan structure
   • configuring the testplan to use dynamic generated parameters (runtime parameters and property files)
3. Test script structure
   • name of the testplan
   • location of the testplan
   • jmeter agents to use in the load test
   • workflow controlling : defining what and the order of the functions called (starting the test, collecting memory information, collecting jms information, warming up, collecting logs data transformation, generating performance report, etc.) thereby controlling the run of the test
   • runtime parameters – virtual users, ramp up time, pause times, number of repetions, number of loops / repetition, application specific parameters, etc.)
4. Function file
   • detail of the file containing all functions needed for controlling the load test, and of the functions in the file
5. Storing and processing the test results
   • selecting what information to store
   • transforming the results file into a fully xml compliant file for further ETL processing – detail of processing functions
6. Storing and processing additional collected data
   • use of shell functions for collecting application server information such as memory, jms usage, ejb cache usage and so on
   • detailing of tools used to collect application server information:
   • jmap for memory
   • imqcmd for jms
   • vmstat for cpu
   • glassfish rest monitoring for app server resources

ETL – EXTRACT TRANSFORM AND LOAD – DATA PROCESSING AND IMPORTING INTO THE DATABASE

1. Database structure:
   • detail of the structure needed for performance reporting
   • ER Diagram
2. Importing test results into the database using Pentaho Data Integration Jobs
   • import job for response times and throughput
   • import job for memory footprinting

REPORTING

1. Jasper Reports – Generating and storing the performance reports
   
   • using Jasper Server for reporting server
   • using IReport for designing reports
   • detailing of reports based on real life examples
2. Continuous integration – Automating and Integrating the tests into CI Systems
   • integrating with hudson: concept, implementation
   • jmeter plugin description and configuration
   • automatically generating reports with use of JMeter testplans