fix(ci): 修 release/next 误部署到 PROD 的严重 bug + 容器名 -release 物理隔离
Some checks failed
Java CI with Maven / build (11) (push) Has been cancelled
Java CI with Maven / build (17) (push) Has been cancelled
Java CI with Maven / build (8) (push) Has been cancelled

事故复盘:build #5 触发 release/next 部署,但 Initialize 阶段
  env.DEPLOY_HOST = env.RELEASE_DEPLOY_HOST
没有生效,DEPLOY_HOST 保持 environment 块默认值 172.17.16.14(PROD),导致
release.yml 被部署到 PROD 服务器;同时容器名与 prod 同名(aiot-gateway 等),
docker compose up -d 直接 force-recreate prod 容器,配置切到 release 库 / Nacos
namespace / Redis db1 — prod 业务断了。

根因:Jenkins declarative pipeline 的 environment 块声明的变量是 build-scope
constant,在 script 块里 env.X = ... 的赋值在某些场景不生效。

修复:
1. environment 块只声明常量 PROD_DEPLOY_HOST/PROD_DEPLOY_PATH/RELEASE_DEPLOY_HOST/
   RELEASE_DEPLOY_PATH,DEPLOY_HOST/DEPLOY_PATH/COMPOSE_FILE/CONTAINER_NAME_SUFFIX
   全部在 Initialize 阶段动态创建(不在 environment 声明则 env.X = 赋值生效)
2. 增加防呆:未知分支(既不是 master 也不是 release/next)DEPLOY_HOST 设空,
   后续 ssh 命令会因目标空直接报错,不会误伤任何机器
3. release 容器名加 -release 后缀(aiot-gateway-release 等),物理隔离:
   即便部署目标 host 错了,容器名不与 prod 重叠,docker compose 不会 recreate
   prod 同名容器
4. getContainerNameForService 改读 env.CONTAINER_NAME_SUFFIX(Initialize 阶段写入),
   不再依赖 @NonCPS 函数里访问 env.BRANCH_NAME

prod 影响:master 分支行为完全不变(DEPLOY_HOST→PROD_DEPLOY_HOST 同值、容器名
suffix='')。
This commit is contained in:
lzh
2026-04-28 17:38:17 +08:00
parent 7950d87a73
commit 8148bf7471
2 changed files with 26 additions and 14 deletions

26
Jenkinsfile vendored
View File

@@ -29,12 +29,12 @@ pipeline {
// 服务配置
CORE_SERVICES = 'viewsh-gateway,viewsh-module-system-server,viewsh-module-infra-server,viewsh-module-iot-server,viewsh-module-iot-gateway,viewsh-module-ops-server'
// 部署配置(默认 Prodrelease/next 分支会在 Initialize 阶段覆盖为 Release
DEPLOY_HOST = '172.17.16.14'
DEPLOY_PATH = '/opt/aiot-platform-cloud'
// 部署配置常量Initialize 阶段根据分支选择具体目标DEPLOY_HOST/PATH 不在 environment 块声明,
// 避免 Jenkins declarative environment 的 readonly 行为导致 script 块里的赋值不生效,
// 从而使 release/next 误部署到 PROD
SSH_KEY = '/var/jenkins_home/.ssh/id_rsa'
// Release 服务器配置
PROD_DEPLOY_HOST = '172.17.16.14'
PROD_DEPLOY_PATH = '/opt/aiot-platform-cloud'
RELEASE_DEPLOY_HOST = '172.17.16.7'
RELEASE_DEPLOY_PATH = '/opt/aiot-platform-cloud'
@@ -81,10 +81,20 @@ pipeline {
env.DEPLOY_HOST = env.RELEASE_DEPLOY_HOST
env.DEPLOY_PATH = env.RELEASE_DEPLOY_PATH
env.COMPOSE_FILE = 'docker-compose.release.yml'
env.CONTAINER_NAME_SUFFIX = '-release'
echo "📦 Deploy target: RELEASE (${env.DEPLOY_HOST}) using ${env.COMPOSE_FILE}"
} else {
} else if (env.BRANCH_NAME == 'master') {
env.DEPLOY_HOST = env.PROD_DEPLOY_HOST
env.DEPLOY_PATH = env.PROD_DEPLOY_PATH
env.COMPOSE_FILE = 'docker-compose.core.yml'
env.CONTAINER_NAME_SUFFIX = ''
echo "📦 Deploy target: PRODUCTION (${env.DEPLOY_HOST}) using ${env.COMPOSE_FILE}"
} else {
// 防呆:未知分支不允许部署,避免 BRANCH_NAME 缺失时误伤
env.DEPLOY_HOST = ''
env.COMPOSE_FILE = ''
env.CONTAINER_NAME_SUFFIX = ''
echo "⚠️ Unknown branch '${env.BRANCH_NAME}' — build only, no deploy"
}
// 【优化2】动态检测系统资源
@@ -1110,6 +1120,8 @@ def sortServicesByDependency(def services) {
// 获取服务对应的容器名称
@NonCPS
def getContainerNameForService(String service) {
// 容器名 suffix 在 Initialize 阶段按分支决定release/next='-release'、master=''
// 物理隔离防止 release 部署误伤 prod 同名容器
def map = [
'viewsh-gateway': 'aiot-gateway',
'viewsh-module-system-server': 'aiot-system-server',
@@ -1118,7 +1130,7 @@ def getContainerNameForService(String service) {
'viewsh-module-iot-gateway': 'aiot-iot-gateway',
'viewsh-module-ops-server': 'aiot-ops-server'
]
return map.get(service, "aiot-${service}")
return map.get(service, "aiot-${service}") + (env.CONTAINER_NAME_SUFFIX ?: '')
}
// 获取服务对应的模块路径

View File

@@ -59,7 +59,7 @@ x-common-env: &common-env
services:
viewsh-gateway:
image: ${REGISTRY_HOST:-172.17.16.7:5000}/viewsh-gateway:${IMAGE_TAG:-latest}
container_name: aiot-gateway
container_name: aiot-gateway-release
restart: on-failure:5
ports:
- "48080:48080"
@@ -82,7 +82,7 @@ services:
viewsh-module-system-server:
image: ${REGISTRY_HOST:-172.17.16.7:5000}/viewsh-module-system-server:${IMAGE_TAG:-latest}
container_name: aiot-system-server
container_name: aiot-system-server-release
restart: on-failure:5
ports:
- "48081:48081"
@@ -117,7 +117,7 @@ services:
viewsh-module-infra-server:
image: ${REGISTRY_HOST:-172.17.16.7:5000}/viewsh-module-infra-server:${IMAGE_TAG:-latest}
container_name: aiot-infra-server
container_name: aiot-infra-server-release
restart: on-failure:5
ports:
- "48082:48082"
@@ -140,7 +140,7 @@ services:
viewsh-module-iot-server:
image: ${REGISTRY_HOST:-172.17.16.7:5000}/viewsh-module-iot-server:${IMAGE_TAG:-latest}
container_name: aiot-iot-server
container_name: aiot-iot-server-release
restart: on-failure:5
ports:
- "48091:48091"
@@ -182,7 +182,7 @@ services:
viewsh-module-iot-gateway:
image: ${REGISTRY_HOST:-172.17.16.7:5000}/viewsh-module-iot-gateway:${IMAGE_TAG:-latest}
container_name: aiot-iot-gateway
container_name: aiot-iot-gateway-release
restart: on-failure:5
ports:
- "1883:1883"
@@ -203,7 +203,7 @@ services:
ROCKETMQ_ACCESS_KEY: ""
ROCKETMQ_SECRET_KEY: ""
VIEWSH_IOT_GATEWAY_RPC_URL: "http://aiot-iot-server:48091"
VIEWSH_IOT_GATEWAY_RPC_URL: "http://aiot-iot-server-release:48091"
volumes:
- app-logs:/app/logs
deploy:
@@ -216,7 +216,7 @@ services:
viewsh-module-ops-server:
image: ${REGISTRY_HOST:-172.17.16.7:5000}/viewsh-module-ops-server:${IMAGE_TAG:-latest}
container_name: aiot-ops-server
container_name: aiot-ops-server-release
restart: on-failure:5
ports:
- "48092:48092"