Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

모니터링 시스템 도입 완료 #378

Merged
merged 17 commits into from
Jun 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
228f718
feat(Monitoring): Prometheus 의존성 추가
limehee Jun 6, 2024
99ae0d2
feat(Gradle): 모니터링을 위한 Prometheus 및 Logback 추가
limehee Jun 16, 2024
5e77fc7
feat(Monitoring): logback-spring.xml 추가
limehee Jun 16, 2024
4866a4c
feat(Monitoring): Jenkinsfile 모니터링을 위한 컨테이너 설정 추가 및 가독성 향상
limehee Jun 16, 2024
32cd101
refactor(Jenkinsfile): buildApplication 환경 변수 설정 수정
limehee Jun 16, 2024
94222ce
fix(Jenkinsfile): 문법 오류 수정
limehee Jun 16, 2024
4c81bb7
fix(Jenkinsfile): 문법 오류 수정
limehee Jun 16, 2024
e14cc4d
fix(Jenkinsfile): 문법 오류 수정
limehee Jun 16, 2024
f22abef
fix(Jenkinsfile): 변수 오타 수정
limehee Jun 16, 2024
6963c46
fix(Jenkinsfile): 변수 오타 수정
limehee Jun 16, 2024
5d78b65
fix(Jenkinsfile): 스프링 컨테이너 네트워크 설정 오류 수정
limehee Jun 16, 2024
480380c
refactor(Jenkinsfile): 스프링 컨테이너 네트워크 설정 예외 추가
limehee Jun 16, 2024
e086dfb
refactor(Whitelist): 화이트리스트 관련 설정 리팩토링
limehee Jun 16, 2024
55fa438
refactor(Jenkins): 화이트리스트 설정에 따른 헬스 체크 Basic Auth 추가
limehee Jun 16, 2024
3415961
fix(Monitoring): Log 파일명에 시간이 표시되지 않는 문제 수정
limehee Jun 16, 2024
776094b
refactor(Monitoring): 로그를 하루 단위로 기록하도록 설정
limehee Jun 16, 2024
7cb4bc7
Merge branch 'develop' into feat/#348
limehee Jun 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ gradle/
/src/main/resources/*.yml
!/src/main/resources/application.yml
src/main/java/page/clab/api/global/auth/application/DataLoader.java
src/main/java/page/clab/api/auth/service/DataLoader.java
src/main/java/page/clab/api/global/auth/service/DataLoader.java
src/main/java/page/clab/api/global/config/SecurityProperties.java
/config/whitelist.json

### STS ###
Expand Down
50 changes: 27 additions & 23 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,39 @@ repositories {

dependencies {
// Spring Project
implementation 'org.springframework.boot:spring-boot-starter-actuator' // 모니터링
implementation 'org.springframework.boot:spring-boot-starter-web' // 웹 MVC
implementation 'org.springframework.boot:spring-boot-starter-validation' // 유효성 검사
implementation 'org.springframework.boot:spring-boot-starter-thymeleaf' // 템플릿 엔진
implementation 'org.springframework.boot:spring-boot-starter-webflux' // WebFlux
developmentOnly 'org.springframework.boot:spring-boot-devtools' // 개발 도구

// Security
implementation 'org.springframework.boot:spring-boot-starter-security' // Spring Security
implementation 'com.warrenstrange:googleauth:1.5.0' // Google Authenticator
implementation 'io.jsonwebtoken:jjwt-api:0.11.5' // JWT 라이브러리
runtimeOnly 'io.jsonwebtoken:jjwt-impl:0.11.5' // JWT 구현체
runtimeOnly 'io.jsonwebtoken:jjwt-jackson:0.11.5' // JWT Jackson 모듈

// Monitoring
implementation 'org.springframework.boot:spring-boot-starter-actuator' // Spring Boot Actuator
implementation 'io.micrometer:micrometer-registry-prometheus' // Prometheus
implementation 'ch.qos.logback:logback-classic:1.5.6' // Logback
implementation 'ch.qos.logback:logback-core:1.5.6' // Logback

// DB
implementation 'org.postgresql:postgresql:42.7.1' // PostgreSQL JDBC Driver
implementation 'org.springframework.boot:spring-boot-starter-data-redis' // Redis
implementation 'org.springframework.boot:spring-boot-starter-data-jpa' // Spring Data JPA
implementation 'org.springframework.boot:spring-boot-starter-validation' // Hibernate Validator
implementation 'org.hibernate.validator:hibernate-validator:8.0.1.Final' // Bean Validation
implementation 'jakarta.validation:jakarta.validation-api:3.0.2' // Jakarta Bean Validation

// QueryDSL
implementation 'com.querydsl:querydsl-jpa:5.1.0:jakarta'
annotationProcessor "com.querydsl:querydsl-apt:5.1.0:jakarta"
annotationProcessor "jakarta.annotation:jakarta.annotation-api"
annotationProcessor "jakarta.persistence:jakarta.persistence-api"

// Util
compileOnly 'org.projectlombok:lombok' // 롬복
annotationProcessor 'org.projectlombok:lombok' // 롬복
Expand All @@ -68,28 +94,6 @@ dependencies {
implementation 'com.slack.api:slack-api-client:1.39.0'
implementation 'com.slack.api:slack-app-backend:1.39.0'


// DB
implementation 'org.postgresql:postgresql:42.7.1' // PostgreSQL JDBC Driver
implementation 'org.springframework.boot:spring-boot-starter-data-redis' // Redis
implementation 'org.springframework.boot:spring-boot-starter-data-jpa' // Spring Data JPA
implementation 'org.springframework.boot:spring-boot-starter-validation' // Hibernate Validator
implementation 'org.hibernate.validator:hibernate-validator:8.0.1.Final' // Bean Validation
implementation 'jakarta.validation:jakarta.validation-api:3.0.2' // Jakarta Bean Validation

// QueryDSL
implementation 'com.querydsl:querydsl-jpa:5.1.0:jakarta'
annotationProcessor "com.querydsl:querydsl-apt:5.1.0:jakarta"
annotationProcessor "jakarta.annotation:jakarta.annotation-api"
annotationProcessor "jakarta.persistence:jakarta.persistence-api"

// Security
implementation 'org.springframework.boot:spring-boot-starter-security' // Spring Security
implementation 'com.warrenstrange:googleauth:1.5.0' // Google Authenticator
implementation 'io.jsonwebtoken:jjwt-api:0.11.5' // JWT 라이브러리
runtimeOnly 'io.jsonwebtoken:jjwt-impl:0.11.5' // JWT 구현체
runtimeOnly 'io.jsonwebtoken:jjwt-jackson:0.11.5' // JWT Jackson 모듈

// XSS Filter
implementation 'com.navercorp.lucy:lucy-xss-servlet:2.0.1' // Lucy XSS Servlet Filter
implementation 'com.navercorp.lucy:lucy-xss:1.6.3' // Lucy XSS Filter
Expand Down
148 changes: 116 additions & 32 deletions jenkins/Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,30 @@ pipeline {
DOCKER_HUB_USER = credentials('dockerhub_user')
DOCKER_HUB_PASSWORD = credentials('dockerhub_password')

SERVER_CONFIG = credentials('server_config')
SERVER_CLOUD = credentials('server_cloud')
EXTERNAL_SERVER_CONFIG_PATH = credentials('external_server_config_path')
EXTERNAL_SERVER_CLOUD_PATH = credentials('external_server_cloud_path')
EXTERNAL_SERVER_LOGS_PATH = credentials('external_server_logs_path')

INTERNAL_SERVER_CONFIG_PATH = credentials('internal_server_config_path')
INTERNAL_SERVER_CLOUD_PATH = credentials('internal_server_cloud_path')
INTERNAL_SERVER_LOGS_PATH = credentials('internal_server_logs_path')

BLUE_CONTAINER = credentials('blue_container')
GREEN_CONTAINER = credentials('green_container')
BLUE_URL = credentials('blue_url')
GREEN_URL = credentials('green_url')
IMAGE_NAME = credentials('image_name')
NETWORK_NAME = credentials('network_name')

APPLICATION_NETWORK = credentials('application_network')
MONITORING_NETWORK = credentials('monitoring_network')

PROFILE = credentials('profile')
PORT_A = credentials('port_a')
PORT_B = credentials('port_b')

WHITELIST_ADMIN_USERNAME = credentials('whitelist_admin_username')
WHITELIST_ADMIN_PASSWORD = credentials('whitelist_admin_password')

DOCKERFILE_PATH = "${env.WORKSPACE}/jenkins/Dockerfile"
NGINX_CONTAINER_NAME = 'nginx'
POSTGRESQL_CONTAINER_NAME = 'postgresql'
Expand Down Expand Up @@ -81,7 +92,9 @@ pipeline {

stage('애플리케이션 빌드') {
steps {
sh './gradlew clean build -Penv=stage --stacktrace --info'
script {
buildApplication()
}
}
}

Expand Down Expand Up @@ -134,7 +147,9 @@ pipeline {
}

def sendSlackNotification(message, color) {
withEnv(["SLACK_WEBHOOK_URL=${env.SLACK_WEBHOOK_URL}"]) {
withEnv([
"SLACK_WEBHOOK_URL=${env.SLACK_WEBHOOK_URL}"
]) {
def payload = """{
"attachments": [
{
Expand Down Expand Up @@ -169,7 +184,12 @@ def getChangeLog() {

def backupPostgres() {
def BACKUP_FILE = "postgres_backup_${new Date().format('yyyy-MM-dd_HH-mm-ss')}.sql"
withEnv(["BACKUP_DIR=${env.BACKUP_DIR}", "POSTGRESQL_CONTAINER_NAME=${env.POSTGRESQL_CONTAINER_NAME}", "PG_PASSWORD=${env.PG_PASSWORD}", "PG_USER=${env.PG_USER}"]) {
withEnv([
"BACKUP_DIR=${env.BACKUP_DIR}",
"POSTGRESQL_CONTAINER_NAME=${env.POSTGRESQL_CONTAINER_NAME}",
"PG_PASSWORD=${env.PG_PASSWORD}",
"PG_USER=${env.PG_USER}"
]) {
sh """
echo "Backing up PostgreSQL database to ${BACKUP_DIR}/${BACKUP_FILE}..."
docker exec -e PGPASSWORD=${PG_PASSWORD} ${POSTGRESQL_CONTAINER_NAME} sh -c 'pg_dumpall -c -U ${PG_USER} > ${BACKUP_DIR}/${BACKUP_FILE}'
Expand All @@ -189,7 +209,14 @@ def dockerLogin() {

def determineContainers() {
script {
withEnv(["BLUE_CONTAINER=${env.BLUE_CONTAINER}", "GREEN_CONTAINER=${env.GREEN_CONTAINER}", "BLUE_URL=${env.BLUE_URL}", "GREEN_URL=${env.GREEN_URL}", "PORT_A=${env.PORT_A}", "PORT_B=${env.PORT_B}"]) {
withEnv([
"BLUE_CONTAINER=${env.BLUE_CONTAINER}",
"GREEN_CONTAINER=${env.GREEN_CONTAINER}",
"BLUE_URL=${env.BLUE_URL}",
"GREEN_URL=${env.GREEN_URL}",
"PORT_A=${env.PORT_A}",
"PORT_B=${env.PORT_B}"
]) {
def blueRunning = sh(script: "docker ps --filter 'name=${BLUE_CONTAINER}' --format '{{.Names}}' | grep -q '${BLUE_CONTAINER}'", returnStatus: true) == 0
if (blueRunning) {
env.CURRENT_CONTAINER = BLUE_CONTAINER
Expand All @@ -209,8 +236,24 @@ def determineContainers() {
}
}

def buildApplication() {
withEnv([
"PROFILE=${env.PROFILE}"
]) {
sh """
echo "Building application with profile ${PROFILE}..."
./gradlew clean build -Penv=${PROFILE} --stacktrace --info
"""
}
}

def buildAndPushDockerImage() {
withEnv(["DOCKER_HUB_REPO=${env.DOCKER_HUB_REPO}", "DEPLOY_CONTAINER=${env.DEPLOY_CONTAINER}", "DOCKERFILE_PATH=${env.DOCKERFILE_PATH}", "IMAGE_NAME=${env.IMAGE_NAME}"]) {
withEnv([
"DOCKER_HUB_REPO=${env.DOCKER_HUB_REPO}",
"DEPLOY_CONTAINER=${env.DEPLOY_CONTAINER}",
"DOCKERFILE_PATH=${env.DOCKERFILE_PATH}",
"IMAGE_NAME=${env.IMAGE_NAME}"
]) {
sh """
docker build -f ${DOCKERFILE_PATH} -t ${IMAGE_NAME}:${DEPLOY_CONTAINER} .
docker tag ${IMAGE_NAME}:${DEPLOY_CONTAINER} ${DOCKER_HUB_REPO}:${DEPLOY_CONTAINER}
Expand All @@ -220,7 +263,20 @@ def buildAndPushDockerImage() {
}

def deployNewInstance() {
withEnv(["NEW_PORT=${env.NEW_PORT}", "NETWORK_NAME=${env.NETWORK_NAME}", "SERVER_CONFIG=${env.SERVER_CONFIG}", "SERVER_CLOUD=${env.SERVER_CLOUD}", "DEPLOY_CONTAINER=${env.DEPLOY_CONTAINER}", "IMAGE_NAME=${env.IMAGE_NAME}"]) {
withEnv([
"PROFILE=${env.PROFILE}",
"NEW_PORT=${env.NEW_PORT}",
"APPLICATION_NETWORK=${env.APPLICATION_NETWORK}",
"MONITORING_NETWORK=${env.MONITORING_NETWORK}",
"EXTERNAL_SERVER_CONFIG_PATH=${env.EXTERNAL_SERVER_CONFIG_PATH}",
"EXTERNAL_SERVER_CLOUD_PATH=${env.EXTERNAL_SERVER_CLOUD_PATH}",
"EXTERNAL_SERVER_LOGS_PATH=${env.EXTERNAL_SERVER_LOGS_PATH}",
"INTERNAL_SERVER_CONFIG_PATH=${env.INTERNAL_SERVER_CONFIG_PATH}",
"INTERNAL_SERVER_CLOUD_PATH=${env.INTERNAL_SERVER_CLOUD_PATH}",
"INTERNAL_SERVER_LOGS_PATH=${env.INTERNAL_SERVER_LOGS_PATH}",
"DEPLOY_CONTAINER=${env.DEPLOY_CONTAINER}",
"IMAGE_NAME=${env.IMAGE_NAME}"
]) {
sh """
echo "Stopping and removing existing container if it exists"
if docker ps | grep -q ${DEPLOY_CONTAINER}; then
Expand All @@ -231,11 +287,22 @@ def deployNewInstance() {
echo "Running new container ${DEPLOY_CONTAINER} with image ${IMAGE_NAME}:${DEPLOY_CONTAINER}"
docker run -d --name ${DEPLOY_CONTAINER} \\
-p ${NEW_PORT}:8080 \\
--network ${NETWORK_NAME} \\
-v ${SERVER_CONFIG}:/config \\
-v ${SERVER_CLOUD}:/cloud \\
--network ${APPLICATION_NETWORK} \\
-v ${EXTERNAL_SERVER_CONFIG_PATH}:${INTERNAL_SERVER_CONFIG_PATH} \\
-v ${EXTERNAL_SERVER_CLOUD_PATH}:${INTERNAL_SERVER_CLOUD_PATH} \\
-v ${EXTERNAL_SERVER_LOGS_PATH}:${INTERNAL_SERVER_LOGS_PATH} \\
-e LOG_PATH=${INTERNAL_SERVER_LOGS_PATH} \\
-e SPRING_PROFILES_ACTIVE=${PROFILE} \\
${IMAGE_NAME}:${DEPLOY_CONTAINER}

echo "Checking if monitoring network ${MONITORING_NETWORK} exists"
if docker network ls --format '{{.Name}}' | grep -q '^${MONITORING_NETWORK}\$'; then
echo "Connecting to monitoring network ${MONITORING_NETWORK}"
docker network connect ${MONITORING_NETWORK} ${DEPLOY_CONTAINER}
else
echo "Monitoring network ${MONITORING_NETWORK} does not exist. Skipping connection."
fi

echo "Listing all containers"
docker ps -a
"""
Expand All @@ -244,32 +311,49 @@ def deployNewInstance() {
}

def performHealthCheck() {
def PUBLIC_IP = sh(script: "curl -s ifconfig.me", returnStdout: true).trim()
echo "Public IP address: ${PUBLIC_IP}"

def start_time = System.currentTimeMillis()
def timeout = start_time + 240000 // 4 minutes

while (System.currentTimeMillis() < timeout) {
def elapsed = (System.currentTimeMillis() - start_time) / 1000
echo "Checking health... ${elapsed} seconds elapsed."
if (sh(script: "curl -s http://${PUBLIC_IP}:${env.NEW_PORT}/actuator/health | grep 'UP'", returnStatus: true) == 0) {
echo "New application started successfully after ${elapsed} seconds."
break
withEnv([
"WHITELIST_ADMIN_USERNAME=${env.WHITELIST_ADMIN_USERNAME}",
"WHITELIST_ADMIN_PASSWORD=${env.WHITELIST_ADMIN_PASSWORD}"
]) {
def PUBLIC_IP = sh(script: "curl -s ifconfig.me", returnStdout: true).trim()
echo "Public IP address: ${PUBLIC_IP}"

def start_time = System.currentTimeMillis()
def timeout = start_time + 240000 // 4 minutes

while (System.currentTimeMillis() < timeout) {
def elapsed = (System.currentTimeMillis() - start_time) / 1000
echo "Checking health... ${elapsed} seconds elapsed."
def status = sh(
script: """curl -s -u ${WHITELIST_ADMIN_USERNAME}:${WHITELIST_ADMIN_PASSWORD} \
http://${PUBLIC_IP}:${env.NEW_PORT}/actuator/health | grep 'UP'""",
returnStatus: true
)
if (status == 0) {
echo "New application started successfully after ${elapsed} seconds."
return
}
sleep 5
}
sleep 1
}

if (System.currentTimeMillis() >= timeout) {
sendSlackNotification(":scream_cat: New Staging application did not start successfully within 4 minutes.", env.SLACK_COLOR_FAILURE)
sh "docker stop ${env.DEPLOY_CONTAINER}"
sh "docker rm ${env.DEPLOY_CONTAINER}"
error "Health check failed"
if (System.currentTimeMillis() >= timeout) {
sendSlackNotification(":scream_cat: New Staging application did not start successfully within 4 minutes.", env.SLACK_COLOR_FAILURE)
sh "docker stop ${env.DEPLOY_CONTAINER}"
sh "docker rm ${env.DEPLOY_CONTAINER}"
error "Health check failed"
}
}
}

def switchTrafficAndCleanup() {
withEnv(["NEW_PORT=${env.NEW_PORT}", "OLD_PORT=${env.OLD_PORT}", "NEW_TARGET=${env.NEW_TARGET}", "CURRENT_CONTAINER=${env.CURRENT_CONTAINER}", "DEPLOY_CONTAINER=${env.DEPLOY_CONTAINER}", "NGINX_CONTAINER_NAME=${env.NGINX_CONTAINER_NAME}"]) {
withEnv([
"NEW_PORT=${env.NEW_PORT}",
"OLD_PORT=${env.OLD_PORT}",
"NEW_TARGET=${env.NEW_TARGET}",
"CURRENT_CONTAINER=${env.CURRENT_CONTAINER}",
"DEPLOY_CONTAINER=${env.DEPLOY_CONTAINER}",
"NGINX_CONTAINER_NAME=${env.NGINX_CONTAINER_NAME}"
]) {
sh """
echo "Switching traffic to ${DEPLOY_CONTAINER} on port ${NEW_PORT}."
docker exec ${NGINX_CONTAINER_NAME} bash -c '
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@
@Slf4j
public class WhitelistService {

@Value("${security.swagger.whitelist.enabled}")
@Value("${security.whitelist.enabled}")
private boolean whitelistEnabled;

@Value("${security.swagger.whitelist.path}")
@Value("${security.whitelist.path}")
private String whitelistPath;

public List<String> loadWhitelistIps() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ public class AuthenticationConfig {

private final CustomUserDetailsService customUserDetailsService;

private final OpenApiAccountProperties openApiAccountProperties;
private final WhitelistAccountProperties whitelistAccountProperties;

@Bean
public AuthenticationManager authenticationManager() throws Exception {
Expand All @@ -32,9 +32,9 @@ public AuthenticationManager authenticationManager() throws Exception {

@Bean
public InMemoryUserDetailsManager userDetailsService() {
UserDetails user = User.withUsername(openApiAccountProperties.getUsername())
.password(passwordEncoder().encode(openApiAccountProperties.getPassword()))
.roles(openApiAccountProperties.getRole())
UserDetails user = User.withUsername(whitelistAccountProperties.getUsername())
.password(passwordEncoder().encode(whitelistAccountProperties.getPassword()))
.roles(whitelistAccountProperties.getRole())
.build();
return new InMemoryUserDetailsManager(user);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,9 @@ public class SecurityConfig {

private final AuthenticationConfig authenticationConfig;

private final OpenApiAccountProperties openApiAccountProperties;
private final WhitelistAccountProperties whitelistAccountProperties;

private final OpenApiPatternsProperties OpenApiPatternsProperties;
private final WhitelistPatternsProperties WhitelistPatternsProperties;

private final IPInfoConfig ipInfoConfig;

Expand Down Expand Up @@ -115,7 +115,7 @@ public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {

private ExpressionUrlAuthorizationConfigurer<HttpSecurity>.ExpressionInterceptUrlRegistry configureRequests(ExpressionUrlAuthorizationConfigurer<HttpSecurity>.ExpressionInterceptUrlRegistry authorizeRequests) {
return authorizeRequests
.requestMatchers(OpenApiPatternsProperties.getPatterns()).hasRole(openApiAccountProperties.getRole())
.requestMatchers(WhitelistPatternsProperties.getPatterns()).hasRole(whitelistAccountProperties.getRole())
.requestMatchers(SecurityConstants.PERMIT_ALL).permitAll()
.requestMatchers(HttpMethod.GET, SecurityConstants.PERMIT_ALL_API_ENDPOINTS_GET).permitAll()
.requestMatchers(HttpMethod.POST, SecurityConstants.PERMIT_ALL_API_ENDPOINTS_POST).permitAll()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
@Setter
@Getter
@Configuration
@ConfigurationProperties(prefix = "security.account.swagger")
public class OpenApiAccountProperties {
@ConfigurationProperties(prefix = "security.account.whitelist-admin")
public class WhitelistAccountProperties {

private String username;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
@Setter
@Getter
@Configuration
@ConfigurationProperties(prefix = "security.swagger")
public class OpenApiPatternsProperties {
@ConfigurationProperties(prefix = "security.whitelist")
public class WhitelistPatternsProperties {

private String[] patterns;

Expand Down
Loading