Java EE 6 Glassfish 3.1.1 Cluster with Load-Balancer Installation

Posted: décembre 21st, 2011 | Author: | Filed under: Dev, Java, Java EE, Tutorial | Tags: , , , , , , , , , , , , | 12 Comments »

In this tutorial we’ll see how to install a Glassfish cluster with this architecture

 
architecture
 

Assumptions : there’s nothing on servers (either java, xauth or whatever you want)

 

Prepare Node 2 and Node 1


 

Add the glassfish user (with the same password on each) and choose the installation path

adduser glassfish mkdir /opt/glassfish3
chown glassfish:glassfish
/opt/glassfish3

Add non-free in sources.list for sun jdk installation

# vi /etc/apt/sources.list
deb http://mirror.ovh.net/debian/ squeeze main non-free
deb-src http://mirror.ovh.net/debian/ squeeze main non-free

Install sun jdk

apt-get update apt-get install sun-java6-jdk

 

Prepare LB


 

Add the glassfish user (still with the same password)

adduser glassfish

 

On Node 1


  • Install xauth

The simpliest way is to install glassfish in graphical mode so we need xauth for x forwarding with ssh.

apt-get install xauth

Logout and Log in with -X option.

  • Glassfish install

Download the GlassFish Server 3.1.1 Open Source Edition Full Platform

wget http://download.java.net/glassfish/3.1.1/release/glassfish-3.1.1-unix.sh
chmod +x glassfish-3.1.1-unix.sh ./glassfish-3.1.1-unix.sh

Choose custom, install and configure, choose the installation path (/opt/glassfish3 for us)

chown -R glassfish:glassfish 

Add the path to <installation path>/bin to glassfish user’s PATH (.bashrc .profile …)
With this, glassfish user could use asadmin command line tool

PATH="/opt/glassfish3/glassfish/bin:$PATH"
  • Glassfish config

To know what we could do with asadmin

asadmin list-commands

So let’s go

asadmin login
asadmin enable-secure-admin asadmin restart-domain
#If you forget the domain name : <strong>asadmin list-domains
</strong>
  • Installing Node 2 from Node 1

asadmin setup-ssh --generatekey=true node-2 lb-1
asadmin install-node node-2
  •  Create cluster

First, create the cluster config (based on the existing one),  all cluster nodes will share the same config.

asadmin copy-config default-config cluster-config

Now we could create the nodes

asadmin create-node-ssh --nodehost localhost node-1-ssh
asadmin create-node-ssh --sshuser glassfish --sshkeyfile /home/glassfish/.ssh/id_rsa --nodehost node-2 node-2-ssh

And finally the cluster and the instances

asadmin create-cluster --config cluster-config usul-cluster
asadmin create-instance --cluster usul-cluster --node node-1-ssh gf-1
asadmin create-instance --cluster usul-cluster --node node-2-ssh gf-2
# Verifying
asadmin list-instances --long=true
# Start the cluster
asadmin start-cluster usul-cluster

Load-Balancer setup


  • Install xauth because lb installer needs GUI

Edit : As John said in the comments : The Load Balancer installer requires a GUI on first install, but you can save the LB configuration to a file and subsequent installs can be done in a headless manner.

yum install xauth

We also need the JDK, so download java 6 sdk bin installer on java.sun.com (the bin one, not rpm.bin)

wget http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-x64.bin

chmod +x jdk-6u30-linux-x64.bin
./jdk-6u30-linux-x64.bin
mv jdk1.6.0_30 /usr/local && cd /usr/local && ln -s jdk1.6.0_30 jdk
# Install in the system
# 1 because there's no java on the server, put 2 or 3 if you already have another java version)
# and launch alternatives --config
java alternatives --install /usr/bin/java java /usr/local/jdk/bin/java 1

# Verifying
java -version
  • Install and prepare Apache

yum install httpd mod_ssl openssl

For cert generation, be careful to put your server name in Common Name

# Generate private key
openssl genrsa -out ca.key 1024
# Generate CSR
openssl req -new -key ca.key -out ca.csr
# Generate Self Signed Key
openssl x509 -req -days 365 -in ca.csr -signkey ca.key -out ca.crt
# Copy the files to the correct locations
cp ca.crt /etc/pki/tls/certs
cp ca.key /etc/pki/tls/private/ca.key
cp ca.csr /etc/pki/tls/private/ca.csr

Edit /httpd/conf/ssl.conf and set the correct paths

SSLCertificateFile /etc/pki/tls/certs/ca.crt
SSLCertificateKeyFile /etc/pki/tls/private/ca.key

Now we need to tweak this installation because the load balancer installer needs some files and paths

# Folder for the glassfish DAS certificate
mkdir /etc/httpd/conf/ssl.crt
# Installer will look for files in $apache_home/conf/extra
ln-s /etc/httpd/conf.d /etc/httpd/conf/extra
touch /etc/httpd/conf/extra/httpd-ssl.conf
touch /etc/httpd/conf/extra/httpd-vhosts.conf
touch/etc/httpd/conf/extra/httpd-mpm.conf
# Installer will look for bin/apachectl and bin/envvars
mkdir /etc/httpd/bin
touch /etc/httpd/bin/envvars
touch /etc/httpd/bin/apachectl

Edit Mpm config file

PidFile "logs/httpd.pid"
LockFile "logs/accept.lock"

<IfModule mpm_prefork_module>
 StartServers 5
 MinSpareServers 5
 MaxSpareServers 10
 MaxClients 150
 MaxRequestsPerChild 0
</IfModule>

<IfModule mpm_worker_module>
 StartServers 2
 MaxClients 150
 MinSpareThreads 25
 MaxSpareThreads 75
 ThreadsPerChild 25
 MaxRequestsPerChild 0
</IfModule>

Edit apachectl

#!/bin/bash
/usr/sbin/apachectl -v

 

chmod 755 /etc/httpd/bin/apachectl
  • Export the DAS certificate

The load balancer needs an xml file to be updated : loadbalancer.xml By putting the certificate on the LB, we could update it from the DAS with apply-http-lb-changes

#Export crt
keytool \
 -export \
 -rfc \
 -alias s1as \
 -keystore <installPath>/glassfish/domains/usul/config/keystore.jks \
 -file ./glassfish.crt \
 -storepass changeit

# Send it on the LB
scp glassfish.crt root@lb-1:/etc/httpd/conf/

# The installer will put the crt into ssl.crt later
  • Install the load-balancer

Download loadbalancer : http://www.oracle.com/technetwork/middleware/glassfish/downloads/index.html

# Don't forget to ssh -X because of GUI intall
java -jar glassfish-lbconfigurator-3_1_1.jar

Choose apache and put the installation dir (/etc/httpd)

Choose the DAS cert in /etc/httpd/conf/

When finished, edit /etc/init.d/httpd and add

 if [ -f /etc/httpd/bin/envvars]; then
 . /etc/httpd/bin/envvars
 fi
  • Configure the ssl access

keytool -printcert -file /etc/httpd/conf/ssl.cert/glassfish.crt

Copy Serial number (UPPER CASE !!!) and Organization (O) and Organization Unit (OU)
Edit /etc/httpd/conf/extra/httpd-ssl.conf and edit both SSLRequire at the end of file with O OU and Serial (in Upper Case)
You will have something like this

 SSLVerifyClient require
 SSLVerifyDepth 1
 SSLRequireSSL
 SSLCACertificateFile /etc/httpd/conf/ssl.crt/glassfish.crt
 SSLRequire ( %{SSL_CIPHER} !~ m/^(EXP|NULL)-/ \
 and %{SSL_CLIENT_S_DN_O} eq "Oracle Corporation" \
 and %{SSL_CLIENT_S_DN_OU} eq "GlassFish" \
 and %{SSL_CLIENT_M_SERIAL} eq "4EEB6172" )

 SSLVerifyClient require
 SSLVerifyDepth 1
 SSLRequireSSL
 SSLCACertificateFile /etc/httpd/conf/ssl.crt/glassfish.crt
 SSLRequire ( %{SSL_CIPHER} !~ m/^(EXP|NULL)-/ \
 and %{SSL_CLIENT_S_DN_O} eq "Oracle Corporation" \
 and %{SSL_CLIENT_S_DN_OU} eq "GlassFish" \
 and %{SSL_CLIENT_M_SERIAL} eq "4EEB6172" )

Edit /etc/httpd/conf/extra/httpd-vhosts.conf

#Be sure to have :
NameVirtualHost *:80
<VirtualHost *:80>
 ServerName lb-1
 DocumentRoot "/var/www"
 #  The document root is not important, it must exist to avoid warnings or errors in logs but this is not used
</VirtualHost>

Raise the MaxClients in /etc/httpd/conf/extra/httpd-mpm.conf because installer put 1 that is not enough

chown -R apache:apache /etc/httpd
# just check before that apache is the user for apache in httpd.conf
/etc/init.d/httpd restart

Create the load-balancer in the DAS

asadmin create-http-lb --devicehost lb-1 --deviceport 443 --target usul-cluster lb-1

asadmin apply-http-lb-changes lb-1
 

  • http://Website Bcode

    Why was it necessary to install glassfish in graphical mode? It was always just unzip the archive.

    • usul

      Hi,
      for the server, yes you could just unzip, but the graphical installer creates the startup scripts and you could choose more options than just unzip.
      On the other side, the load-balancer only exists in GUI installer mode.

  • http://Website John Clingan

    The Load Balancer installer requires a GUI on first install, but you can save the LB configuration to a file and subsequent installs can be done in a headless manner.

    • usul

      Yes, thanks, I forgot to mention it but i’ll add it

  • Pingback: GlassFish v3 by tvibes - Pearltrees

  • http://Website Lee

    Does this setup have the fault tolerance for the load balancer as well?

    • usul

      You can install another load-balancer and so you’ll have it.
      In this case the best practice is to have a physical load-balancer in front of your software ones.

  • http://Website Lee

    Are you saying having two identical software load balancer? and have the physical load balancer on top of it?

    Or just remove software load balancer and use the physical load balancer on top of the clustered glassfish?

    • usul

      Yes the first one, 2 software load-balancers and 1 physical.
      You could not add the physical one but you’ll have 2 entry points

  • http://Website Lee

    Wouldn’t that be redundant? and more complex to set up?

    What benefit does that have over having two fail over load balancer (physical)?

    Cost?

    • usul

      I just said that because if you have 2 load-balancers, you will have 2 entry points, that’s all ;)

  • Pingback: Blog: From Zero to Load-Balanced | Oracle | Syngu