Java EE 6 Glassfish 3.1.1 Cluster with Load-Balancer Installation

Posted: décembre 21st, 2011 | Author: usul | Filed under: Dev, Java, Java EE, Tutorial | Tags: Centos, Cluster, Debian, EJB, Glassfish, HowTo, Java, Java EE, Load-balancer, Oracle, Server, Setup, Tutorial | 12 Comments »

In this tutorial we’ll see how to install a Glassfish cluster with this architecture

 

 

Assumptions : there’s nothing on servers (either java, xauth or whatever you want)

 

Prepare Node 2 and Node 1

---

 

Add the glassfish user (with the same password on each) and choose the installation path

adduser glassfish mkdir /opt/glassfish3
chown glassfish:glassfish
/opt/glassfish3

Add non-free in sources.list for sun jdk installation

# vi /etc/apt/sources.list
deb http://mirror.ovh.net/debian/ squeeze main non-free
deb-src http://mirror.ovh.net/debian/ squeeze main non-free

Install sun jdk

apt-get update apt-get install sun-java6-jdk

 

Prepare LB

---

 

Add the glassfish user (still with the same password)

adduser glassfish

 

On Node 1

---

• Install xauth

The simpliest way is to install glassfish in graphical mode so we need xauth for x forwarding with ssh.

apt-get install xauth

Logout and Log in with -X option.

• Glassfish install

Download the GlassFish Server 3.1.1 Open Source Edition Full Platform

wget http://download.java.net/glassfish/3.1.1/release/glassfish-3.1.1-unix.sh
chmod +x glassfish-3.1.1-unix.sh ./glassfish-3.1.1-unix.sh

Choose custom, install and configure, choose the installation path (/opt/glassfish3 for us)

chown -R glassfish:glassfish 

Add the path to <installation path>/bin to glassfish user’s PATH (.bashrc .profile …)
With this, glassfish user could use asadmin command line tool

PATH="/opt/glassfish3/glassfish/bin:$PATH"

• Glassfish config

To know what we could do with asadmin

asadmin list-commands

So let’s go

asadmin login
asadmin enable-secure-admin asadmin restart-domain
#If you forget the domain name : <strong>asadmin list-domains
</strong>

• Installing Node 2 from Node 1

asadmin setup-ssh --generatekey=true node-2 lb-1
asadmin install-node node-2

•  Create cluster

First, create the cluster config (based on the existing one),  all cluster nodes will share the same config.

asadmin copy-config default-config cluster-config

Now we could create the nodes

asadmin create-node-ssh --nodehost localhost node-1-ssh
asadmin create-node-ssh --sshuser glassfish --sshkeyfile /home/glassfish/.ssh/id_rsa --nodehost node-2 node-2-ssh

And finally the cluster and the instances

asadmin create-cluster --config cluster-config usul-cluster
asadmin create-instance --cluster usul-cluster --node node-1-ssh gf-1
asadmin create-instance --cluster usul-cluster --node node-2-ssh gf-2
# Verifying
asadmin list-instances --long=true
# Start the cluster
asadmin start-cluster usul-cluster

Load-Balancer setup

---

• Install xauth because lb installer needs GUI

Edit : As John said in the comments : The Load Balancer installer requires a GUI on first install, but you can save the LB configuration to a file and subsequent installs can be done in a headless manner.

yum install xauth

We also need the JDK, so download java 6 sdk bin installer on java.sun.com (the bin one, not rpm.bin)

wget http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-x64.bin

chmod +x jdk-6u30-linux-x64.bin
./jdk-6u30-linux-x64.bin
mv jdk1.6.0_30 /usr/local && cd /usr/local && ln -s jdk1.6.0_30 jdk
# Install in the system
# 1 because there's no java on the server, put 2 or 3 if you already have another java version)
# and launch alternatives --config
java alternatives --install /usr/bin/java java /usr/local/jdk/bin/java 1

# Verifying
java -version

• Install and prepare Apache

yum install httpd mod_ssl openssl

For cert generation, be careful to put your server name in Common Name

# Generate private key
openssl genrsa -out ca.key 1024
# Generate CSR
openssl req -new -key ca.key -out ca.csr
# Generate Self Signed Key
openssl x509 -req -days 365 -in ca.csr -signkey ca.key -out ca.crt
# Copy the files to the correct locations
cp ca.crt /etc/pki/tls/certs
cp ca.key /etc/pki/tls/private/ca.key
cp ca.csr /etc/pki/tls/private/ca.csr

Edit /httpd/conf/ssl.conf and set the correct paths

SSLCertificateFile /etc/pki/tls/certs/ca.crt
SSLCertificateKeyFile /etc/pki/tls/private/ca.key

Now we need to tweak this installation because the load balancer installer needs some files and paths

# Folder for the glassfish DAS certificate
mkdir /etc/httpd/conf/ssl.crt
# Installer will look for files in $apache_home/conf/extra
ln-s /etc/httpd/conf.d /etc/httpd/conf/extra
touch /etc/httpd/conf/extra/httpd-ssl.conf
touch /etc/httpd/conf/extra/httpd-vhosts.conf
touch/etc/httpd/conf/extra/httpd-mpm.conf
# Installer will look for bin/apachectl and bin/envvars
mkdir /etc/httpd/bin
touch /etc/httpd/bin/envvars
touch /etc/httpd/bin/apachectl

Edit Mpm config file

PidFile "logs/httpd.pid"
LockFile "logs/accept.lock"

<IfModule mpm_prefork_module>
 StartServers 5
 MinSpareServers 5
 MaxSpareServers 10
 MaxClients 150
 MaxRequestsPerChild 0
</IfModule>

<IfModule mpm_worker_module>
 StartServers 2
 MaxClients 150
 MinSpareThreads 25
 MaxSpareThreads 75
 ThreadsPerChild 25
 MaxRequestsPerChild 0
</IfModule>

Edit apachectl

#!/bin/bash
/usr/sbin/apachectl -v

 

chmod 755 /etc/httpd/bin/apachectl

• Export the DAS certificate

The load balancer needs an xml file to be updated : loadbalancer.xml By putting the certificate on the LB, we could update it from the DAS with apply-http-lb-changes

#Export crt
keytool \
 -export \
 -rfc \
 -alias s1as \
 -keystore <installPath>/glassfish/domains/usul/config/keystore.jks \
 -file ./glassfish.crt \
 -storepass changeit

# Send it on the LB
scp glassfish.crt root@lb-1:/etc/httpd/conf/

# The installer will put the crt into ssl.crt later

• Install the load-balancer

Download loadbalancer : http://www.oracle.com/technetwork/middleware/glassfish/downloads/index.html

# Don't forget to ssh -X because of GUI intall
java -jar glassfish-lbconfigurator-3_1_1.jar

Choose apache and put the installation dir (/etc/httpd)

Choose the DAS cert in /etc/httpd/conf/

When finished, edit /etc/init.d/httpd and add

 if [ -f /etc/httpd/bin/envvars]; then
 . /etc/httpd/bin/envvars
 fi

• Configure the ssl access

keytool -printcert -file /etc/httpd/conf/ssl.cert/glassfish.crt

Copy Serial number (UPPER CASE !!!) and Organization (O) and Organization Unit (OU)
Edit /etc/httpd/conf/extra/httpd-ssl.conf and edit both SSLRequire at the end of file with O OU and Serial (in Upper Case)
You will have something like this

 SSLVerifyClient require
 SSLVerifyDepth 1
 SSLRequireSSL
 SSLCACertificateFile /etc/httpd/conf/ssl.crt/glassfish.crt
 SSLRequire ( %{SSL_CIPHER} !~ m/^(EXP|NULL)-/ \
 and %{SSL_CLIENT_S_DN_O} eq "Oracle Corporation" \
 and %{SSL_CLIENT_S_DN_OU} eq "GlassFish" \
 and %{SSL_CLIENT_M_SERIAL} eq "4EEB6172" )

 SSLVerifyClient require
 SSLVerifyDepth 1
 SSLRequireSSL
 SSLCACertificateFile /etc/httpd/conf/ssl.crt/glassfish.crt
 SSLRequire ( %{SSL_CIPHER} !~ m/^(EXP|NULL)-/ \
 and %{SSL_CLIENT_S_DN_O} eq "Oracle Corporation" \
 and %{SSL_CLIENT_S_DN_OU} eq "GlassFish" \
 and %{SSL_CLIENT_M_SERIAL} eq "4EEB6172" )

Edit /etc/httpd/conf/extra/httpd-vhosts.conf

#Be sure to have :
NameVirtualHost *:80
<VirtualHost *:80>
 ServerName lb-1
 DocumentRoot "/var/www"
 #  The document root is not important, it must exist to avoid warnings or errors in logs but this is not used
</VirtualHost>

Raise the MaxClients in /etc/httpd/conf/extra/httpd-mpm.conf because installer put 1 that is not enough

chown -R apache:apache /etc/httpd
# just check before that apache is the user for apache in httpd.conf
/etc/init.d/httpd restart

Create the load-balancer in the DAS

asadmin create-http-lb --devicehost lb-1 --deviceport 443 --target usul-cluster lb-1

asadmin apply-http-lb-changes lb-1

---