Snails In Noble Software : File based Coherence configuration for Oracle SOA Suite

Oracle SOA Suite comes with embedded Oracle Coherence - data grid software used to improve communication in the cluster. Customer is not intended to change configured internal caches, and cannot use this Coherence software for his own purposes. On the other hand, customer may need to update cluster layer technical parameters to adjust them to the environment. One of recommended configuration steps is switching from default multicast, UDP based, cluster discovery and communication protocol, to TCP based, unicast mode. Enabling Well Known Addresses (WKA) is a technical activity that does the switch. This document describes how to do this with receipt of moving configuration to files.

To enable WKA in SOA Suite environment one should use set of Java options to define environment properties, that will be picked up by Coherence start-up classes to configure cluster layer. The first step is to define unicast listener by specifying its IP address and port. The second part of configuration is definition of WKA member list. Both are done via arguments defining Java environment properties.

On each  node it's needed to specify TCP address and port. This section is unique for each node.
-Dtangosol.coherence.localhost=192.168.0.100
-Dtangosol.coherence.localport=8088
-Dtangosol.coherence.localport.adjust=false

It's needed to define list of all WKA nodes. For small SOA Suite cluster (in sense of Coherence cluster size) each node should be listed here:
-Dtangosol.coherence.wka1=192.168.0.100
-Dtangosol.coherence.wka2=192.168.0.101
-Dtangosol.coherence.wka3=192.168.0.102
-Dtangosol.coherence.wka4=192.168.0.103
-Dtangosol.coherence.wka1.port=8088
-Dtangosol.coherence.wka2.port=8088
-Dtangosol.coherence.wka3.port=8088
-Dtangosol.coherence.wka4.port=8088

That's all.

Let's ask how it technically works. Coherence contains quite smart configuration engine based on xml files. The root file is shipped with Coherence itself and embedded in coherence.jar, speaking precisely Coherence is packaged with two files chained together. It's the default configuration. To override defaults one should prepare tangosol-coherence-override.xml file and put in the classpath.

Before going further let's check how Coherence chains configuration files. The best description is in headers of standard files:

tangosol-coherence.xml:
<coherence  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
  xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"
  xml-override="{tangosol.coherence.override /tangosol-coherence-override-{mode}.xml}">

tangosol-coherence-override-dev.xml:
<coherence  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
  xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"
  xml-override="/tangosol-coherence-override.xml">

This configuration file informs, by xml-override attribute, that after its load, additional file will be loaded to build target configuration. The name of the file is /tangosol-coherence-override-{mode}.xml, but may be overridden by Java environment property tangosol.coherence.override. In above situation, after loading initial tangosol-coherence.xml, Coherence loads tangosol-coherence-override-dev.xml, to finalize configuration with tangosol-coherence-override.xml. Oracle SOA suite uses exactly this default scheme:

Inside of configuration file, for each configurable, Coherence specifies system property name used to override given parameter. Using this Java environment variable one can configure system via a command line arguments. Let's take a look inside of tangosol-coherence-override.xml shipped with Oracle SOA Suite:

fabric-runtime.jar/tangosol-coherence-override.xml:
<!DOCTYPE coherence PUBLIC "-//Tangosol, Inc.//DTD Tangosol Coherence 3.0//EN" "http://www.tangosol.com/dtd/coherence_3_0.dtd">
<coherence>
  <configurable-cache-factory-config>
  <class-name>com.tangosol.net.DefaultConfigurableCacheFactory</class-name>
  <init-params>
  <init-param>
  <param-type>java.lang.String</param-type>
  <param-value>soa-coherence-cache-config.xml</param-value>
  </init-param>
  </init-params>
  </configurable-cache-factory-config>
  <cluster-config>
  <unicast-listener>
  <well-known-addresses>
  <socket-address id="1">
  <address system-property="tangosol.coherence.wka1"></address>
  <port system-property="tangosol.coherence.wka1.port">8088</port>
  </socket-address>
  <socket-address id="2">
  <address system-property="tangosol.coherence.wka2"></address>
  <port system-property="tangosol.coherence.wka2.port">8088</port>
  </socket-address>
  <socket-address id="3">
  <address system-property="tangosol.coherence.wka3"></address>
  <port system-property="tangosol.coherence.wka3.port">8088</port>
  </socket-address>
  <socket-address id="4">
  <address system-property="tangosol.coherence.wka4"></address>
  <port system-property="tangosol.coherence.wka4.port">8088</port>
  </socket-address>
  <socket-address id="5">
  <address system-property="tangosol.coherence.wka5"></address>
  <port system-property="tangosol.coherence.wka5.port">8088</port>
  </socket-address>
  <socket-address id="6">
  <address system-property="tangosol.coherence.wka6"></address>
  <port system-property="tangosol.coherence.wka6.port">8088</port>
  </socket-address>
  <socket-address id="7">
  <address system-property="tangosol.coherence.wka7"></address>
  <port system-property="tangosol.coherence.wka7.port">8088</port>
  </socket-address>
  <socket-address id="8">
  <address system-property="tangosol.coherence.wka8"></address>
  <port system-property="tangosol.coherence.wka8.port">8088</port>
  </socket-address>
  <socket-address id="9">
  <address system-property="tangosol.coherence.wka9"></address>
  <port system-property="tangosol.coherence.wka9.port">8088</port>
  </socket-address>
  </well-known-addresses>
  </unicast-listener>
  </cluster-config>
</coherence>

That's the full story.  On one hand easy, on the other hand harmful. Files create a chain of overrides that must be overridden by Java environment property. The use of properties makes a command line to become very long.

Theoretically long command line is not a problem, as there is almost no technical limit for the program arguments length.  To discover actual limit on your system, use xargs utility.

xargs --show-limits
Your environment variables take up 918 bytes
POSIX lower and upper limits on argument length: 2048, 129024
Maximum length of command we could actually use: 128106
Size of command buffer we are actually using: 128106

It looks like a very big space for command arguments - in 128kB one can fit a lot of configurables.

According to the preferences you may decide to move configuration from command line into configuration files. Such configuration is not officially recommended by Oracle, but is technically possible with use of core configuration system property: tangosol.coherence.override.

As you see on below diagram by setting tangosol.coherence.override, one can bypass all configuration chinned to the root tangosol-coherence.xml file. By using this technique one, must ensure that custom configuration files contains information from all bypassed files. It's recommended to split your custom file into two: first with your setting, and the second with core SOA configuration options. Note that after upgrade of SOA environment you should discover changes in shipped Coherence configurables.

When moving configuration to files, it's recommended to remove all "system-properties" from xml configuration. Such step will guarantee that no configuration will be further overwritten by potential settings in startup arguments.

To simplify configuration to as few steps as possible, ideally one, xml-override specified in custom-coherence-configuration.xml should point to soa-coherence-configuration.xml using its full path pointing to a file system. With such setting the only thing needed on command line arguments is link to first custom-coherence-configuration.xml configuration file.

Exemplary blueprint configuration is presented below. Note that variables that need to be fixed during file preparation are marked with dollar char and uppercase e.g. $CLUSTER_NAME. In appendix I'm showing how to deal with variable substitution in an automated way. Both files custom-coherence-override.xml, and soa-coherence-override.xml mud the placed in $DOMAIN_CONFIG.

###

Appendix. WLS configuration and Coherence config files

WebLogic startup script
export JAVA_OPTIONS
JAVA_OPTIONS="${JAVA_OPTIONS} -Dtangosol.coherence.override=$DOMAIN_CONFIG/custom-coherence-override.xml"

custom-coherence-override.xml
<?xml version='1.0'?>
<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"
   xml-override="$DOMAIN_CONFIG/soa-coherence-override.xml"
   >

  

  <cluster-config>
  <member-identity>
  
  <cluster-name>$CLUSTER_NAME</cluster-name>
  <site-name>$CLUSTER_SITE</site-name>
  </member-identity>

  <unicast-listener>

  
  <well-known-addresses>
  <socket-address id="1">
  <address>$WKA1_ADDRESS</address>
  <port>$WKA1_PORT</port>
  </socket-address>
  <socket-address id="2">
  <address>$WKA2_ADDRESS</address>
  <port>$WKA2_PORT</port>
  </socket-address>
  <socket-address id="3">
  <address>$WKA3_ADDRESS</address>
  <port>$WKA3_PORT</port>
  </socket-address>
  <socket-address id="4">
  <address>$WKA4_ADDRESS</address>
  <port>$WKA4_PORT</port>
  </socket-address>
  </well-known-addresses>

  
  <address>$NODE_TCP_ADDRESS</address>
  
  <port>$NODE_TCP_PORT</port>
  
  <port-auto-adjust>false</port-auto-adjust>

  </unicast-listener>

  
  <multicast-listener>
  <address>$CLUSTER_ADDRESS</address>
  <port>$CLUSTER_PORT</port>
  <time-to-live>4</time-to-live>
  </multicast-listener>

  </cluster-config>

  <logging-config>
  <destination>stdout</destination>
  <severity-level>9</severity-level>
  <character-limit>8192</character-limit>
  </logging-config>

</coherence>

soa-coherence-override.xml
<?xml version='1.0'?>
<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"
   >
  

  <configurable-cache-factory-config>
  <class-name>com.tangosol.net.DefaultConfigurableCacheFactory</class-name>
  <init-params>
  <init-param>
  <param-type>java.lang.String</param-type>
  <param-value>soa-coherence-cache-config.xml</param-value>
  </init-param>
  </init-params>
  </configurable-cache-factory-config>
</coherence>

Appendix. Variable substitution code

Below variable substation code is used in SOACONFIG tool.

function execOrDie {
  stepMsg=$1; shift
  exitCode=$1; shift
  runCMD=$1; shift

  echoTab "$stepMsg" 50
  $runCMD $@ >>$LOG_FILE
  if [ $? -ne 0 ]; then
  log Error.
  logDate "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
  logDate "Error running: $stepMsg. Exiting."
  logDate "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
  exit $exitCode
  else
removeTmpFiles
  log Done.
  return 0
  fi
}

function reverseExitCode {
  $@
  if [ $? -eq 0 ]; then
  return 255
  else
  return 0
  fi
}

function removeTmpFiles {
rm -rf CFG_BUILD
}

function substituteVariablesInFiles {
  cfgFiles=$1
  outDir=$2

log "Processing variable substitution"
  for file in $cfgFiles; do
   log "|---Processing file: $file"
   variables=$(cat $file | perl -ne 'if (/\${(\w+)}/) { print "$1\n";}')
   if [ ! -z "$variables" ]; then
for variable in $variables; do
   log "|   |--- $variable"
done
   else
  log "|   |--- (no variables)"
   fi
   cat $file | perl -p -e 's/\$\{(\w+)\}/(exists $ENV{$1}?$ENV{$1}:"\${$1}")/eg' > $outDir/$(basename $file)
  done
execOrDie "\---Verifying that all variables were replaced" 20 reverseExitCode egrep '\$\{\w+\}' $outDir/*
}

References

Oracle Best Practices For High Availability
http://www.oracle.com/technetwork/database/availability/fmw11gsoamultidc-aa-1998491.pdf
Extending the Domain for SOA Components
http://docs.oracle.com/cd/E28280_01/core.1111/e12036/extend_soa.htm#SOEDG543
Coherence Developer's Guide - Understanding Configuration
http://docs.oracle.com/cd/E18686_01/coh.37/e18677/gs_config.htm#COHDG5002
SOACONFIG tool
https://github.com/rstyczynski/tools/tree/master/SOACONFIG

Snails In Noble Software

Tuesday, March 10, 2015

File based Coherence configuration for Oracle SOA Suite

Appendix. WLS configuration and Coherence config files

Appendix. Variable substitution code

References

1 comment: