Tomcat-集群方案

多个tomcat要一起协同工作有多种办法,这里总结一下几种方案的优缺点,以及各自的实现方法。

集群方案

方案 实现方式 适用场景 优点 缺点
session sticky 基于七层的源地址hash 小型规模 配置简单 后端任何节点掉线,session丢失
session cluster tomcat自带集群方案 中小型规模 配置简单 占用内存大,每个节点都要持有整个集群的会话信息
session server 利用memcached或redis将session集中管理 任何规模 性能较好 配置复杂

session stickey

主机名 IP地址 用途
nginx 192.168.4.207 前端负载 ip_hash
tomcat1 192.168.4.208 测试节点1
tomcat2 192.168.4.207 测试节点2

利用nginx/haproxy/httpd基于访问ip的hash路由策略,保证访问的ip始终被路由到同一个tomcat上。

nginx配置

这里使用的是nginx1.8,具体安装步骤暂且不表。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
vim /etc/nginx/nginx.conf
http {
...
upstream tomcat_cluster {
server 192.168.4.208:8080 weight=1;
server 192.168.4.209:8080 weight=1;
#ip_hash;
#这里先禁用ip hash,看看有什么效果
}
...
server {
location / {
proxy_pass http://tomcat_cluster;
}
}
......
}

tomcat配置

同样安装好JDK和tomcat,分别在tomcat1和tomcat2的/webapps/ROOT目录下创建index.jsp文件。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
vim index.jsp
<%@page language="java" %>
<html>
<head><title>Tomcat111</title></head>
<body>
<h1><font color="red">TomcatA.chen-hao.com.cn</font></h1>
<table align="centre" border="1">
<tr>
<td>Session ID</td>
<% session.setAttribute("chen-hao.com.cn","chen-hao.com.cn"); %>
<td><%= session.getId() %></td>
</tr>
<tr>
<td>Created on</td>
<td><%= session.getCreationTime() %></td>
</tr>
</table>
</body>
</html>

tomcat2 需修改<head><title>Tomcat222</title></head>,其他保持一致即可。

验证1

  • 注释掉ip hash,验证如下

  • 开启ip hash,验证如下

session cluster

主机名 IP地址 用途
nginx 192.168.4.207 前端负载 轮序调度
tomcat1 192.168.4.208 测试节点1
tomcat2 192.168.4.207 测试节点2

使用tomcat自带的cluster方式,多个tomcat间自动实时复制session信息。
参考官方文档:http://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html

tomcat的配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
vim conf/server.xml
<Engine name="Catalina" defaultHost="localhost" jvmRoute="Tomcat222">
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564"
frequency="500"
dropTime="3000"/>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4000"
autoBind="100"
selectorTimeout="5000"
maxThreads="6"/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=""/>
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>

tomcat1和tomcat2配置一样,只需分别修改

1
<Engine name="Catalina" defaultHost="localhost" jvmRoute="Tomcat222">

其中:
Cluster:定义集群;可放在Engine或Host配置段但生效范围不用;
Channel: 是做成员关系判断和发送、接收session信息;
Manager:会话管理器,tomcat支持多种管理器;
Membership:定义集群成员;
Receiver:会话接收器;
Sender:会话发送器;
注意几点:

  1. 确保在web.xml中添加
  2. 确保上面有jvmRoute=”Tomcat222”属性;
  3. 确保各节点间时间是同步的;

验证2

这里再访问前端nginx可以看出:

无论怎么刷新SessionID都不会变

session server


利用memcached/redis把多个tomcat的session集中管理,前端在利用nginx负载均衡和动静态资源分离,在兼顾系统水平扩展的同时又能保证较高的性能。

gradle配置

由于源码构建基于 gradle,所以需要先配置 gradle 环境。
官网下载地址:
https://gradle.org/gradle-download/

1
2
3
4
curl -s https://get.sdkman.io | bash
chmod +x /root/.sdkman/bin/sdkman-init.sh
. /root/.bash_profile
sdk install gradle 3.3

确认gradle安装成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
gradle
Starting a Gradle Daemon (subsequent builds will be faster)
:help
Welcome to Gradle 3.3.
To run a build, run gradle <task> ...
To see a list of available tasks, run gradle tasks
To see a list of command-line options, run gradle --help
To see more detail about a task, run gradle help --task <task>
BUILD SUCCESSFUL
Total time: 3.283 secs

编译tomcat-redis-session-manager

让Tomcat把会话Session保存到Redis数据库中,需要使用tomcat-redis-session-manager这样的第三方库。
主页见: https://github.com/jcoleman/tomcat-redis-session-manager
按照自己的配置,编译多次未成功,这里直接贴出一个编译成功的配置。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
vim build.gradle
apply plugin: 'java'
apply plugin: 'maven'
apply plugin: 'signing'
group = 'com.orangefunction'
version = '2.0.0'
repositories {
mavenCentral()
}
compileJava {
sourceCompatibility = 1.7
targetCompatibility = 1.7
}
dependencies {
compile group: 'org.apache.tomcat', name: 'tomcat-catalina', version: '7.0.27'
compile group: 'redis.clients', name: 'jedis', version: '2.5.2'
compile group: 'org.apache.commons', name: 'commons-pool2', version: '2.2'
//compile group: 'commons-codec', name: 'commons-codec', version: '1.9'
testCompile group: 'junit', name: 'junit', version: '4.+'
testCompile 'org.hamcrest:hamcrest-core:1.3'
testCompile 'org.hamcrest:hamcrest-library:1.3'
testCompile 'org.mockito:mockito-all:1.9.5'
testCompile group: 'org.apache.tomcat', name: 'tomcat-coyote', version: '7.0.27'
}
task javadocJar(type: Jar, dependsOn: javadoc) {
classifier = 'javadoc'
from 'build/docs/javadoc'
}
task sourcesJar(type: Jar) {
from sourceSets.main.allSource
classifier = 'sources'
}
artifacts {
archives jar
archives javadocJar
archives sourcesJar
}
//signing
// sign configurations.archives
//required = false
//}
uploadArchives {
repositories {
mavenDeployer {
beforeDeployment { MavenDeployment deployment -> signing.signPom(deployment) }
//repository(url: "https://oss.sonatype.org/service/local/staging/deploy/maven2/") {
//authentication(userName: sonatypeUsername, password: sonatypePassword)
//}
//repository(url: "https://oss.sonatype.org/content/repositories/snapshots") {
// authentication(userName: sonatypeUsername, password: sonatypePassword)
//}
pom.project {
name 'tomcat-redis-session-manager'
packaging 'jar'
description 'Tomcat Redis Session Manager is a Tomcat extension to store sessions in Redis'
url 'https://github.com/jcoleman/tomcat-redis-session-manager'
issueManagement {
url 'https://github.com:jcoleman/tomcat-redis-session-manager/issues'
system 'GitHub Issues'
}
scm {
url 'https://github.com:jcoleman/tomcat-redis-session-manager'
connection 'scm:git:git://github.com/jcoleman/tomcat-redis-session-manager.git'
developerConnection 'scm:git:git@github.com:jcoleman/tomcat-redis-session-manager.git'
}
licenses {
license {
name 'MIT'
url 'http://opensource.org/licenses/MIT'
distribution 'repo'
}
}
developers {
developer {
id 'jcoleman'
name 'James Coleman'
email 'jtc331@gmail.com'
url 'https://github.com/jcoleman'
}
}
}
}
}
}

执行编译命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
gradle build --stacktrace
:compileJava
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-catalina/7.0.27/tomcat-catalina-7.0.27.pom
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-servlet-api/7.0.27/tomcat-servlet-api-7.0.27.pom
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-juli/7.0.27/tomcat-juli-7.0.27.pom
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/7.0.27/tomcat-annotations-api-7.0.27.pom
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-api/7.0.27/tomcat-api-7.0.27.pom
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-util/7.0.27/tomcat-util-7.0.27.pom
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-catalina/7.0.27/tomcat-catalina-7.0.27.jar
Download https://repo1.maven.org/maven2/redis/clients/jedis/2.5.2/jedis-2.5.2.jar
Download https://repo1.maven.org/maven2/org/apache/commons/commons-pool2/2.2/commons-pool2-2.2.jar
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-servlet-api/7.0.27/tomcat-servlet-api-7.0.27.jar
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-juli/7.0.27/tomcat-juli-7.0.27.jar
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/7.0.27/tomcat-annotations-api-7.0.27.jar
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-api/7.0.27/tomcat-api-7.0.27.jar
Download https://repo1.maven.org/maven2/org/apache/tomcat/tomcat-util/7.0.27/tomcat-util-7.0.27.jar
warning: [options] bootstrap class path not set in conjunction with -source 1.7
1 warning
:processResources UP-TO-DATE
:classes
:jar
:javadoc
:javadocJar
:sourcesJar
:assemble
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:test UP-TO-DATE
:check UP-TO-DATE
:build
BUILD SUCCESSFUL

在tomcat-redis-session-manager/build/libs/目录下查看是否生成tomcat-redis-session-manager的jar包

1
2
3
cd /work/tomcat-redis-session-manager/build/libs/
ls
tomcat-redis-session-manager-2.0.0.jar tomcat-redis-session-manager-2.0.0-javadoc.jar tomcat-redis-session-manager-2.0.0-sources.jar

此时只需要将tomcat-redis-session-manager-2.0.0.jar以及另外两个依赖包:jedis- 2.5.2.jar,commons-pool2-2.2.jar一并拷入tomcat的lib目录下即可(依赖包可以在gradle的用户目录下找到)

1
2
3
4
find / -name 'jedis-2.5.2.jar'
/root/.gradle/caches/modules-2/files-2.1/redis.clients/jedis/2.5.2/83d833f48052456f197ba55ef2a82fc4909880b/jedis-2.5.2.jar
find -name 'commons-pool2-2.2.jar'
./.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-pool2/2.2/bd1a6e384f3cf0f9b9a60e1e6c1c1ecbbee7e0b7/commons-pool2-2.2.jar

redis配置

修改配置使用 tomcat-redis-session-manager-master 作为 tomcat session 管理器

  1. 分别将生成的 tomcat-redis-session-manager-master 及依赖jar包覆盖到 tomcat 安装目录的 lib 文件夹
  2. 分别修改2台 tomcat 的 context.xml 文件,使 tomcat-redis-session-manager-master 作为session管理器,同时指定redis地址和端口。
    编辑context.xml配置文件
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    vim conf/context.xml
    <Context>
    <Valve className="com.orangefunction.tomcat.redissessions.RedisSessionHandlerValve" />
    <Manager className="com.orangefunction.tomcat.redissessions.RedisSessionManager"
    host="127.0.0.1" <!-- 可选,默认是"localhost" -->
    password="abc" <!-- 可选 -->
    port="6379" <!-- 可选,默认是 "6379" -->
    database="0" <!-- 可选,默认是 "0" -->
    maxInactiveInterval="60"/> <!-- 可选,默认是 "60" (单位:秒)--> />
    </Context>

这里提醒下,一定要注意tomcat的版本,jdk的版本,要和编译的配置填的版本一致,否则tomcat可能会报错
redis安装启动后,tomcat配置文件编辑完成之后,则重新启动tomcat即可

验证3

访问网页查看 session id

可以看出我们的 session id 为 764A35AC567302EDBA2AC05C4EACC7E7.Tomcat111

直接到后端redis 看看此session是否保存

1
2
3
/usr/local/redis-3.0.0/src/redis-cli
127.0.0.1:6379> get 764A35AC567302EDBA2AC05C4EACC7E7.Tomcat111
"\xac\xed\x00\x05sr\x00Dcom.orangefunction.tomcat.redissessions.SessionSerializationMetadataB\xd9\xd9\xf7v\xa2\xdbL\x03\x00\x01[\x00\x15sessionAttributesHasht\x00\x02[Bxpw\x14\x00\x00\x00\x10rG\x034\xbe\x9a>F\xd2KN\xf2}\xdb\xde\x7fxsr\x00\x0ejava.lang.Long;\x8b\xe4\x90\xcc\x8f#\xdf\x02\x00\x01J\x00\x05valuexr\x00\x10java.lang.Number\x86\xac\x95\x1d\x0b\x94\xe0\x8b\x02\x00\x00xp\x00\x00\x01Y\xa5)\x96\xccsq\x00~\x00\x03\x00\x00\x01Y\xa5)\x96\xccsr\x00\x11java.lang.Integer\x12\xe2\xa0\xa4\xf7\x81\x878\x02\x00\x01I\x00\x05valuexq\x00~\x00\x04\x00\x00\a\bsr\x00\x11java.lang.Boolean\xcd r\x80\xd5\x9c\xfa\xee\x02\x00\x01Z\x00\x05valuexp\x01q\x00~\x00\nsq\x00~\x00\x03\x00\x00\x01Y\xa5)\x96\xd8t\x00*764A35AC567302EDBA2AC05C4EACC7E7.Tomcat111sq\x00~\x00\a\x00\x00\x00\x01t\x00\x0fchen-hao.com.cnq\x00~\x00\x0ew\b\x00\x00\x01Y\xa5)\x96\xcc"

由此可见,session已经保存在redis当中,这里暂且演示一台,至于多台,同样配置即可,再加上前端nginx,tomcat的集群就可以了。

本文标题:Tomcat-集群方案

文章作者:火柴

发布时间:2017年01月13日 - 13:01

最后更新:2017年08月10日 - 19:08

原始链接:http://www.chen-hao.com.cn/posts/41890/

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。

火柴 wechat
扫描上方二维码关注我的博客!
0%