Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: 뉴스 관련 코드 최적화 #102

Merged
merged 131 commits into from
Jan 2, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
131 commits
Select commit Hold shift + click to select a range
7a9f21d
[#2] feat : 패키지 구조 생성
junseok708 Dec 6, 2024
7c5fedd
[#2] feat : common schedule 추가
junseok708 Dec 6, 2024
4f3aede
[#2] feat : schedule yml 추가
junseok708 Dec 6, 2024
6ca25ba
[#2] feat : schedule build 추가
junseok708 Dec 6, 2024
5b93c1b
[#2] feat : schedule exception 추가
junseok708 Dec 6, 2024
a635930
[#2] feat : schedule 크롤링 설계
junseok708 Dec 6, 2024
e397af2
[#2] feat : schedule 자동실행 설계
junseok708 Dec 6, 2024
ceafe5d
Merge branch 'main' into issue/2
junseok708 Dec 6, 2024
fa7abd8
[#2] feat : schedule 삭제 news 패키지 생성
junseok708 Dec 6, 2024
1ce7d8a
[#2] fix : 위치 수정
junseok708 Dec 6, 2024
9c1ef4d
[#2] feat : 기본 설정 설계
junseok708 Dec 9, 2024
f79c963
Merge branch 'dev' into issue/2
junseok708 Dec 9, 2024
3ec410d
[#2] feat : 필요 없는 문자 삭제 및 출판일 수정 메소드 작성
junseok708 Dec 9, 2024
4cc8c94
[#2] fix : 그레이들 수정
junseok708 Dec 9, 2024
944c47c
[#2] feat : requestDto 생성
junseok708 Dec 9, 2024
a8f0027
[#2] feat : repository 생성
junseok708 Dec 9, 2024
19d10ce
[#2] feat : 빌더 및 Enum Default 설정
junseok708 Dec 9, 2024
709056f
[#2] fix : entity 속성 수정
junseok708 Dec 9, 2024
3296706
[#2] fix : Enum 추가
junseok708 Dec 9, 2024
e0abd37
[#2] feat : DB 저장 로직 작성
junseok708 Dec 9, 2024
572a71d
Merge branch 'dev' into issue/2
junseok708 Dec 9, 2024
ed00e89
[#26] feat : issue26 자동화 생성
junseok708 Dec 10, 2024
08beff5
[#26] feat : 배치 자동화 구현
junseok708 Dec 10, 2024
19ff1f9
[#26] feat : 배치 어노테이션 위치 변경
junseok708 Dec 10, 2024
7a9a57f
[#26] feat : 배치 의존성 생성
junseok708 Dec 10, 2024
91db040
[#26] feat : 3시간마다 크롤링 자동화 생성
junseok708 Dec 10, 2024
eee10cd
[#26] feat :배치 yml 추가
junseok708 Dec 10, 2024
9bc61ef
Merge branch 'issue/26' into issue/28
junseok708 Dec 10, 2024
7d480eb
[#26] fix : enum클래스 default 변경
junseok708 Dec 10, 2024
701aa77
[#26] fix : 히스토리 삭제
junseok708 Dec 10, 2024
7ce4f92
[#26] fix : 패치 삭제
junseok708 Dec 10, 2024
002a0a3
[#26] fix : 스케쥴러로 사용
junseok708 Dec 10, 2024
4e8bee2
[#26] fix : 리뷰 수정
junseok708 Dec 10, 2024
73a63cb
[#2] fix : 리뷰 수정
junseok708 Dec 10, 2024
85d9852
[#2] fix : 배치 삭제
junseok708 Dec 11, 2024
42c1a4c
[#2] fix : entity 수정
junseok708 Dec 11, 2024
c04b968
[#2] fix : 상수 처리, 오타 수정
junseok708 Dec 11, 2024
be2ea45
[#2] fix : repository 수정
junseok708 Dec 11, 2024
b9c7d2c
[#2] fix : EOL 수정
junseok708 Dec 11, 2024
af1cb8a
[#2] feat : 중복 처리 기능 생성
junseok708 Dec 11, 2024
7bc208b
[#2] feat : 중복 처리 기능 생성
junseok708 Dec 11, 2024
9c97b7c
Merge branch 'refs/heads/issue/2' into issue/28
junseok708 Dec 11, 2024
0372575
[#28] fix : 배치 삭제 필요 없는 entity 삭제
junseok708 Dec 11, 2024
081d9f3
[#28] fix : 그레이들 common 모듈 추가
junseok708 Dec 11, 2024
422289b
Merge branch 'refs/heads/dev' into issue/26
junseok708 Dec 11, 2024
87c5b29
Merge branch 'refs/heads/issue/2' into issue/26
junseok708 Dec 11, 2024
9c1b86e
[#28] fix : 임포트 수정
junseok708 Dec 11, 2024
10f9c75
[#26] fix : 임포트 수정, cron명 수정
junseok708 Dec 11, 2024
aa85be9
Merge remote-tracking branch 'origin/issue/26' into issue/26
junseok708 Dec 11, 2024
b0426e1
[#26] fix : 크론 명 수정, 상수 처리
junseok708 Dec 11, 2024
6035032
Merge branch 'refs/heads/issue/26' into issue/28
junseok708 Dec 11, 2024
6311467
[#28] feat : 필요없는 entity 삭제
junseok708 Dec 11, 2024
b32c4f2
[#28] feat : 잘못된 삭제 수정
junseok708 Dec 11, 2024
08ceb1d
Merge branch 'refs/heads/dev' into issue/26
junseok708 Dec 11, 2024
ec3fb89
Merge branch 'dev' into issue/26
junseok708 Dec 11, 2024
2a0ddc5
Merge branch 'dev' into issue/26
junseok708 Dec 11, 2024
a025d19
[#26] fix : dev내용 수정
junseok708 Dec 11, 2024
7df2eac
[#26] fix : yml 수정
junseok708 Dec 11, 2024
1bcec06
Merge branch 'refs/heads/issue/26' into issue/28
junseok708 Dec 11, 2024
61e08ad
[#28] fix : 테스트 코드 환경 설정
junseok708 Dec 12, 2024
d239658
Merge branch 'dev' into issue/28
junseok708 Dec 12, 2024
7d7d99f
[#37] fix : repository 위치 수정
junseok708 Dec 12, 2024
3675ca4
[#37] feat : 스케줄러 생성
junseok708 Dec 12, 2024
23bfaca
[#37] feat : 알림 메소드 생성
junseok708 Dec 12, 2024
12ed3b7
[#37] feat : 시간변경 entity 넣기
junseok708 Dec 12, 2024
2e72099
[#37] feat : 어제 크롤링한 내용 가져오기
junseok708 Dec 12, 2024
78334b3
[#37] fix : 스케줄러 정리
junseok708 Dec 12, 2024
70c7d71
[#37] fix : 알람일 수정
junseok708 Dec 12, 2024
28c7e1b
[#37] fix : 잘못된 트렌젝션 수정, 잘못된 int타입 수정
junseok708 Dec 12, 2024
60ab06b
[#28] fix : 로직 변경
junseok708 Dec 12, 2024
332bbfe
[#26] feat : 삭제 로직 및 스케쥴러 생성
junseok708 Dec 12, 2024
6b4dcfc
[#26] fix : 스케줄러 시작/끝 알림 수정
junseok708 Dec 12, 2024
70c29c2
[#28] feat : 테스트 코드 생성
junseok708 Dec 12, 2024
4e08a30
Delete schedule/src/test/java/itcast/news/application/NewsServiceTest…
junseok708 Dec 12, 2024
1ce2ee5
[#26] fix : integer 수정
junseok708 Dec 12, 2024
8682621
Merge remote-tracking branch 'origin/issue/26' into issue/26
junseok708 Dec 12, 2024
cd70d20
[#26] fix : static 추가 및 수정
junseok708 Dec 12, 2024
c9f5bf7
[#26] feat : 크롤링 기능에 Ai요약 기능 추가
junseok708 Dec 13, 2024
556ebd3
[#28] feat : 테스트 생성 및 로직 수정
junseok708 Dec 13, 2024
5b2514a
[#28] feat : 테스트 생성
junseok708 Dec 13, 2024
af3c55e
Merge branch 'refs/heads/dev' into issue/28
junseok708 Dec 13, 2024
e4783b7
[#29] feat : 예외 처리 추가 및 그레이들 추가
junseok708 Dec 15, 2024
c669f1a
[#29] feat : 그레이들 수정
junseok708 Dec 15, 2024
3648c1c
[#29] fix : comment 수정
junseok708 Dec 16, 2024
40f969e
[#29] fix : comment 수정
junseok708 Dec 16, 2024
f03718f
Merge branch 'dev' into issue/29
junseok708 Dec 16, 2024
c3950ee
[#29] fix : 추가 수정 부분
junseok708 Dec 16, 2024
6fcc572
[#29] fix : 추가 수정 부분
junseok708 Dec 16, 2024
efdc9f1
[#29] fix : 그레이들 통일
junseok708 Dec 16, 2024
4ec3beb
Merge branch 'refs/heads/dev' into issue/29
junseok708 Dec 16, 2024
0df68be
Merge branch 'refs/heads/dev' into issue/29
junseok708 Dec 17, 2024
aa454ea
[#29] fix: 클론 수정
junseok708 Dec 17, 2024
fa85a29
Merge branch 'refs/heads/dev' into issue/29
junseok708 Dec 17, 2024
5b71e70
[#69] feat: 서비스단 생성
junseok708 Dec 18, 2024
cbe14e0
[#69] feat: 크론 표현식 추가
junseok708 Dec 18, 2024
3dabd1a
[#69] feat: 선택 및 발송 스케줄러 생성
junseok708 Dec 18, 2024
4f67028
[#69] feat: repository 생성
junseok708 Dec 18, 2024
350b869
[#69] feat: 선택 로직 구현
junseok708 Dec 18, 2024
5fd3df9
[#69] feat: 선택 로직에 필요한 익셉션 생성
junseok708 Dec 18, 2024
17d35f8
[#69] fix: 크론표현식 수정 및 스케줄러 로그 수정
junseok708 Dec 18, 2024
cb37b72
[#69] fix: 뉴스 선택 로직 수정
junseok708 Dec 18, 2024
865b767
[#69] feat: 테스트 클레스 생성
junseok708 Dec 18, 2024
ffda79d
[#69] fix: 알림 기능 삭제
junseok708 Dec 18, 2024
c976ed8
[#69] fix: 알림 기능 테스트 삭제
junseok708 Dec 18, 2024
11c1aa8
[#69] fix: 뉴스 선택 sql 수정
junseok708 Dec 18, 2024
4cfa95b
[#69] feat: 메일 발송 이메일 찾는 repository 생성
junseok708 Dec 18, 2024
7baf169
[#69] feat: 메일에 넣는 내용 빌더 생성
junseok708 Dec 18, 2024
780b726
[#69] feat: 발송 로직 및 메일 발송 로직 생성, 예외 처리 적용
junseok708 Dec 18, 2024
060a871
[#69] feat: 예외 처리에 필요한 코드 작성
junseok708 Dec 18, 2024
9ab0b8c
[#69] docs: 필요 없는 스케줄러 삭제
junseok708 Dec 18, 2024
f9b643b
[#69] test: retrieveUserEmails 메소드 테스트 생성
junseok708 Dec 18, 2024
c400286
[#69] test: selectNews 메소드 테스트
junseok708 Dec 18, 2024
cf4dbe4
Merge branch 'refs/heads/dev' into issue/69
junseok708 Dec 19, 2024
b6f5a08
[#69] fix: PR 수정
junseok708 Dec 19, 2024
5f24269
[#79] fix: 테스트 코드 및 엔티티 수정
junseok708 Dec 19, 2024
32e29db
Merge branch 'dev' into issue/69
junseok708 Dec 19, 2024
5fd43a2
[#69] fix: 체크 아웃 커밋
junseok708 Dec 19, 2024
2e2689e
Merge branch 'refs/heads/dev' into issue/69
junseok708 Dec 19, 2024
7c0dd8b
Merge branch 'dev' into issue/69
junseok708 Dec 20, 2024
2b723ed
[#79] fix: 뉴스 히스토리 리포지토리 생성
junseok708 Dec 20, 2024
e18f343
[#79] feat: 뉴스 히스토리 빌더 생성
junseok708 Dec 20, 2024
8475bca
[#79] feat: 뉴스 히스토리 메소드 생성
junseok708 Dec 20, 2024
af38317
[#79] fix: 중복된 JPA 삭제
junseok708 Dec 20, 2024
0746f3b
[#79] fix: 테스트 코드 수정
junseok708 Dec 20, 2024
8496911
[#79] fix: yml파일 추가
junseok708 Dec 20, 2024
7144a59
Merge branch 'dev' into issue/69
junseok708 Dec 20, 2024
a180476
[#69] fix: 코멘트 수정
junseok708 Dec 24, 2024
e98808d
[#69] fix: 코멘트 수정
junseok708 Dec 24, 2024
964308a
[#98] refactor: 뉴스 크롤링 부분 코드 최적화
junseok708 Dec 24, 2024
7c7d3f2
[#98] test: 나눠진 로직 테스트
junseok708 Dec 26, 2024
d17b551
Merge branch 'dev' into issue/98
junseok708 Dec 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion admin/src/test/java/itcast/AdminNewsServiceTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ public void successNewsRetrieve() {
Long userId = 1L;
Long newsId = 1L;
LocalDateTime fixedTime = LocalDateTime.of(2024, 12, 1, 12, 0);
LocalDate sendAt = LocalDate.of(2024, 12, 1);
LocalDate fixedDate2 = LocalDate.of(2024, 12, 1);

User user = User.builder()
.id(userId)
Expand Down
1 change: 1 addition & 0 deletions common/src/main/java/itcast/exception/ErrorCodes.java
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ public enum ErrorCodes {
NEWS_CRAWLING_ERROR("뉴스 크롤링에 실패했습니다", 2004L, HttpStatus.BAD_REQUEST),
BLOG_SELECT_ERROR("블로그 선택에 실패하였습니다.", 2004L, HttpStatus.BAD_REQUEST),
NEWS_SELECT_ERROR("뉴스 선택에 실패하였습니다.", 2004L, HttpStatus.BAD_REQUEST),
GPT_SERVICE_ERROR("GPT요약 중 오류가 발생했습니다 ",2009L,HttpStatus.BAD_REQUEST),

TODAY_NEWS_NOT_FOUND("뉴스 선택에 실패했습니다", 2005L, HttpStatus.NOT_FOUND),
NOT_FOUND_EMAIL("이메일을 찾을 수 없습니다", 2006L, HttpStatus.NOT_FOUND),
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,17 @@
package itcast.ai.dto.request;

import lombok.Builder;


public record GPTSummaryRequest(
String model,
Message message,
float temperature
) {
@Builder
public GPTSummaryRequest(String model, Message message, float temperature) {
this.model = model;
this.message = message;
this.temperature = temperature;
}
}
81 changes: 49 additions & 32 deletions schedule/src/main/java/itcast/news/application/NewsService.java
Original file line number Diff line number Diff line change
Expand Up @@ -39,49 +39,67 @@ public void newsCrawling() throws IOException {
List<String> links = findLinks(URL);
links = isValidLinks(links);

List<News> newsList = new ArrayList<>();

links.forEach(link -> {
try {
Document url = Jsoup.connect(link).get();
String titles = url.select("#title_area").text();
String content = url.select("#dic_area").text();
String date =
url.select(".media_end_head_info_datestamp_bunch").text();
String thumbnail =
url.selectFirst("meta[property=og:image]").attr("content");

titles = cleanContent(titles);
content = cleanContent(content);
LocalDateTime publishedAt = convertDateTime(date);
News news = processNews(link);
if (news != null) {
newsList.add(news);
}
});

if (!newsList.isEmpty()) {
newsRepository.saveAll(newsList);
}
}

public News processNews(String link) {
try {
Document url = Jsoup.connect(link).get();
String titles = url.select("#title_area").text();
String content = url.select("#dic_area").text();
String date = url.select(".media_end_head_info_datestamp_bunch").text();
String thumbnail = url.selectFirst("meta[property=og:image]").attr("content");

titles = cleanContent(titles);
content = cleanContent(content);
LocalDateTime publishedAt = convertDateTime(date);

if (thumbnail.isEmpty()) {
throw new ItCastApplicationException(INVALID_NEWS_CONTENT);
}
if (thumbnail.isEmpty()) {
log.error("썸네일이 존재하지 않습니다. {}", link);
throw new ItCastApplicationException(INVALID_NEWS_CONTENT);
}

CreateNewsRequest newsRequest = new CreateNewsRequest(titles, content, link, thumbnail, publishedAt);
News news = newsRepository.save(newsRequest.toEntity(titles, content, link, thumbnail, publishedAt));
Message message = new Message("user", content);
GPTSummaryRequest request = new GPTSummaryRequest("gpt-4o-mini", message, 0.7f);
gptService.updateNewsBySummaryContent(request, news.getId());
} catch (IOException e) {
throw new ItCastApplicationException(NEWS_CRAWLING_ERROR);
}
});
CreateNewsRequest newsRequest = new CreateNewsRequest(titles, content, link, thumbnail, publishedAt);
News news = newsRequest.toEntity(titles, content, link, thumbnail, publishedAt);
updateNewsSummary(news, content);
return news;
} catch (IOException e) {
throw new ItCastApplicationException(CRAWLING_PARSE_ERROR);
}
}

public void updateNewsSummary(News news, String content) {
try {
Message message = new Message("user", content);
GPTSummaryRequest request = new GPTSummaryRequest("gpt-4o-mini", message, 0.7f);
gptService.updateNewsBySummaryContent(request, news.getId());
} catch (Exception e) {
throw new ItCastApplicationException(GPT_SERVICE_ERROR);
}
}

public List<String> findLinks(String url) throws IOException {
Document document = Jsoup.connect(url).get();
Elements articles = document.select(".sa_thumb_inner");

List<String> links = new ArrayList<>();
articles.forEach(article -> {
if (links.size() >= LINK_SIZE) {
return;
}
String link = article.select("a").attr("href");
links.add(link);
});
return links;
return articles.stream()
.map(article -> article.select("a").attr("href"))
.limit(LINK_SIZE)
.toList();
}

public List<String> isValidLinks(List<String> links) {
Expand Down Expand Up @@ -127,9 +145,8 @@ public String cleanContent(String info) {
throw new ItCastApplicationException(INVALID_NEWS_CONTENT);
}

info = info.replaceAll("\\[.*?\\]", "")
return info.replaceAll("\\[.*?\\]", "")
.replaceAll("\\(.*?\\)", "")
.trim();
return info;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ public void selectNews(LocalDate yesterday) {
}

LocalDate sendAt = LocalDate.now().plusDays(ALARM_DAY);

newsList.forEach(news -> {
news.newsUpdate(sendAt);
});
Expand All @@ -60,7 +59,6 @@ public void sendNews() {
news.getLink(),
news.getThumbnail()))
.toList();

List<String> emails = retrieveUserEmails(Interest.NEWS);

if (emails == null || emails.isEmpty()) {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package itcast.news.application;

import itcast.ai.application.GPTService;
import itcast.ai.dto.request.GPTSummaryRequest;
import itcast.domain.news.News;
import itcast.news.repository.NewsRepository;
import org.jsoup.Connection;
Expand All @@ -11,16 +12,15 @@
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.ArgumentCaptor;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;

import java.io.IOException;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

import static org.junit.jupiter.api.Assertions.*;
Expand Down Expand Up @@ -168,4 +168,31 @@ void findLinksTest() throws IOException {
}
}

@Test
@DisplayName("updateNewsSummary 메소드 테스트")
void updateNewsSummaryTest() {
// give
News news = News.builder()
.id(1L)
.title("Sample News")
.content("Original Content")
.build();
String content = "Updated Summary Content";

doNothing().when(gptService).updateNewsBySummaryContent(any(GPTSummaryRequest.class), eq(news.getId()));

// when
assertDoesNotThrow(() -> newsService.updateNewsSummary(news, content));

ArgumentCaptor<GPTSummaryRequest> captor = ArgumentCaptor.forClass(GPTSummaryRequest.class);
verify(gptService, times(1)).updateNewsBySummaryContent(captor.capture(), eq(news.getId()));

// then
GPTSummaryRequest capturedRequest = captor.getValue();
assertEquals("gpt-4o-mini", capturedRequest.model());
assertEquals("user", capturedRequest.message().getRole());
assertEquals(content, capturedRequest.message().getContent());
assertEquals(0.7f, capturedRequest.temperature());
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,6 @@
@ExtendWith(MockitoExtension.class)
public class SelectNewsServiceTest {
private static final int YESTERDAY = 1;
private static final int ALARM_HOUR = 2;
private static final int ALARM_DAY = 2;

@Mock
private UserRepository userRepository;
Expand Down