修复规则

去掉小说内容前面的多余空格

README.md

@@ -50,17 +50,24 @@ novel-plus -- 父工程
 | Spring Security      | 安全框架
 | Apache Shiro               | 安全框架
 | Redis               | 缓存方案
-| Aliyun OSS          | 阿里云对象存储服务(图片存储方式之一，一行配置即可切换)
-| FastDFS             | 开源轻量级分布式文件系统(图片存储方式之一，一行配置即可切换)
+| Aliyun OSS          | 阿里云对象存储服务（图片存储备选方案）
 | Lombok              | 简化对象封装工具
 | Docker              | 应用容器引擎
 | MySQL               | 数据库服务
 | Thymeleaf           | 模板引擎
 | Layui               | 前端 UI 框架
 
-## 项目演示
+## 项目截图
 
-https://www.bilibili.com/video/BV1Zo4y187Mi
+### 绿色主题模版
+
+[![点击查看大图](https://www.xxyopen.com/images/green_novel.png)](https://www.xxyopen.com/images/green_novel.png)
+[![点击查看大图](https://www.xxyopen.com/images/resource/os/novel-plus/green3.png)](https://www.xxyopen.com/images/resource/os/novel-plus/green3.png)
+[![点击查看大图](https://www.xxyopen.com/images/resource/os/novel-plus/green2.png)](https://www.xxyopen.com/images/resource/os/novel-plus/green2.png)
+
+## 演示视频
+
+https://www.bilibili.com/video/BV18e41197xs
 
 ## 增值服务
 

doc/api/api.md

@@ -1,116 +0,0 @@
-
- <h1 class="curproject-name"> 小说精品屋-plus </h1> 
- 小说精品屋-plus接口
-
-
-# 作家模块
-
-## 小说章节分页列表查询接口
-<a id=小说章节分页列表查询接口> </a>
-### 基本信息
-
-**Path：** /book/queryIndexList
-
-**Method：** GET
-
-**接口描述：**
-<p>作家后台章节管理页面需要请求该接口获取小说章节分页列表信息</p>
-
-
-### 请求参数
-**Query**
-
-| 参数名称  |  是否必须 | 示例  | 备注  |
-| ------------ | ------------ | ------------ | ------------ |
-| bookId | 是  |  1334337530296893441 |  小说ID |
-| curr | 否  |  1 |  查询页码，默认1 |
-| limit | 否  |  5 |  分页大小，默认5 |
-
-### 返回数据
-
-<table>
-  <thead class="ant-table-thead">
-    <tr>
-      <th key=name>名称</th><th key=type>类型</th><th key=required>是否必须</th><th key=default>默认值</th><th key=desc>备注</th><th key=sub>其他信息</th>
-    </tr>
-  </thead><tbody className="ant-table-tbody"><tr key=0-0><td key=0><span style="padding-left: 0px"><span style="color: #8c8a8a"></span> code</span></td><td key=1><span>number</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">响应状态吗，200表示成功</span></td><td key=5></td></tr><tr key=0-1><td key=0><span style="padding-left: 0px"><span style="color: #8c8a8a"></span> msg</span></td><td key=1><span>string</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">响应信息</span></td><td key=5></td></tr><tr key=0-2><td key=0><span style="padding-left: 0px"><span style="color: #8c8a8a"></span> data</span></td><td key=1><span>object</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">响应数据</span></td><td key=5></td></tr><tr key=0-2-0><td key=0><span style="padding-left: 20px"><span style="color: #8c8a8a">├─</span> total</span></td><td key=1><span>number</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">总数量</span></td><td key=5></td></tr><tr key=0-2-1><td key=0><span style="padding-left: 20px"><span style="color: #8c8a8a">├─</span> list</span></td><td key=1><span>object []</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">章节数据集合</span></td><td key=5><p key=3><span style="font-weight: '700'">item 类型: </span><span>object</span></p></td></tr><tr key=0-2-1-0><td key=0><span style="padding-left: 40px"><span style="color: #8c8a8a">├─</span> id</span></td><td key=1><span>string</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">章节ID</span></td><td key=5></td></tr><tr key=0-2-1-1><td key=0><span style="padding-left: 40px"><span style="color: #8c8a8a">├─</span> bookId</span></td><td key=1><span>string</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">小说ID</span></td><td key=5></td></tr><tr key=0-2-1-2><td key=0><span style="padding-left: 40px"><span style="color: #8c8a8a">├─</span> indexName</span></td><td key=1><span>string</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">章节名</span></td><td key=5></td></tr><tr key=0-2-1-3><td key=0><span style="padding-left: 40px"><span style="color: #8c8a8a">├─</span> isVip</span></td><td key=1><span>number</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">是否收费，1:收费，0:免费</span></td><td key=5></td></tr><tr key=0-2-1-4><td key=0><span style="padding-left: 40px"><span style="color: #8c8a8a">├─</span> updateTime</span></td><td key=1><span>string</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">更新时间</span></td><td key=5></td></tr><tr key=0-2-2><td key=0><span style="padding-left: 20px"><span style="color: #8c8a8a">├─</span> pageNum</span></td><td key=1><span>number</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">页码</span></td><td key=5></td></tr><tr key=0-2-3><td key=0><span style="padding-left: 20px"><span style="color: #8c8a8a">├─</span> pageSize</span></td><td key=1><span>number</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">分页大小</span></td><td key=5></td></tr><tr key=0-2-4><td key=0><span style="padding-left: 20px"><span style="color: #8c8a8a">├─</span> size</span></td><td key=1><span>number</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">当前页数量</span></td><td key=5></td></tr>
-               </tbody>
-              </table>
-            
-## 小说章节删除接口
-<a id=小说章节删除接口> </a>
-### 基本信息
-
-**Path：** /author/deleteIndex/{indexId}
-
-**Method：** DELETE
-
-**接口描述：**
-<p>作家后台章节管理页面点击删除按钮请求该接口删除小说章节内容</p>
-
-
-### 请求参数
-**Headers**
-
-| 参数名称  | 参数值  |  是否必须 | 示例  | 备注  |
-| ------------ | ------------ | ------------ | ------------ | ------------ |
-| Content-Type  |  application/x-www-form-urlencoded | 是  |   |   |
-| Authorization  |   | 是  |  eyJhbGciOiJIUzUxMiJ9.eyJleHAiOjE2MDgzNDg0NzksInN1YiI6IntcImlkXCI6MTI1NTA2MDMyODMyMjAyNzUyMCxcInVzZXJuYW1lXCI6XCIxMzU2MDQyMTMyNFwiLFwibmlja05hbWVcIjpcIjEzNTYwNDIxMzI0XCJ9IiwiY3JlYXRlZCI6MTYwNzc0MzY3OTkxM30.0qhwis_zPb6t8wGNejMhDZ2iHCL9Tgh2UHd1gcQBCp8t6RW3ggSwtfo4l_RgMT_v8jOkLW91GzTVWlNnTE6LCA |  认证JWT，请求登录接口成功后返回 |
-**路径参数**
-
-| 参数名称 | 示例  | 备注  |
-| ------------ | ------------ | ------------ |
-| indexId |  1337603246936645632 |  章节ID |
-
-### 返回数据
-
-<table>
-  <thead class="ant-table-thead">
-    <tr>
-      <th key=name>名称</th><th key=type>类型</th><th key=required>是否必须</th><th key=default>默认值</th><th key=desc>备注</th><th key=sub>其他信息</th>
-    </tr>
-  </thead><tbody className="ant-table-tbody"><tr key=0-0><td key=0><span style="padding-left: 0px"><span style="color: #8c8a8a"></span> code</span></td><td key=1><span>number</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">响应状态吗，200表示成功</span></td><td key=5></td></tr><tr key=0-1><td key=0><span style="padding-left: 0px"><span style="color: #8c8a8a"></span> msg</span></td><td key=1><span>string</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">响应信息</span></td><td key=5></td></tr>
-               </tbody>
-              </table>
-            
-## 小说章节发布接口
-<a id=小说章节发布接口> </a>
-### 基本信息
-
-**Path：** /author/addBookContent
-
-**Method：** POST
-
-**接口描述：**
-<p>作家后台章节发布页面点击提交按钮请求该接口新增小说章节内容</p>
-
-
-### 请求参数
-**Headers**
-
-| 参数名称  | 参数值  |  是否必须 | 示例  | 备注  |
-| ------------ | ------------ | ------------ | ------------ | ------------ |
-| Content-Type  |  application/x-www-form-urlencoded | 是  |   |   |
-| Authorization  |   | 是  |  eyJhbGciOiJIUzUxMiJ9.eyJleHAiOjE2MDgzNDg0NzksInN1YiI6IntcImlkXCI6MTI1NTA2MDMyODMyMjAyNzUyMCxcInVzZXJuYW1lXCI6XCIxMzU2MDQyMTMyNFwiLFwibmlja05hbWVcIjpcIjEzNTYwNDIxMzI0XCJ9IiwiY3JlYXRlZCI6MTYwNzc0MzY3OTkxM30.0qhwis_zPb6t8wGNejMhDZ2iHCL9Tgh2UHd1gcQBCp8t6RW3ggSwtfo4l_RgMT_v8jOkLW91GzTVWlNnTE6LCA |  认证JWT，请求登录接口成功后返回 |
-**Body**
-
-| 参数名称  | 参数类型  |  是否必须 | 示例  | 备注  |
-| ------------ | ------------ | ------------ | ------------ | ------------ |
-| bookId | text  |  是 |  1334337530296893441  |  小说ID |
-| indexName | text  |  是 |  第六章未婚妻（下）  |  章节名 |
-| content | text  |  是 |  开始之时，李七夜还是生疏无比，那怕他对于刀术的所有奥义了然于胸，但是，他出刀之时依然会颤抖，无法达到妙及巅毫的要求。  |  章节内容 |
-| isVip | text  |  是 |  1  |  是否收费，1:收费，0:免费 |
-
-
-
-### 返回数据
-
-<table>
-  <thead class="ant-table-thead">
-    <tr>
-      <th key=name>名称</th><th key=type>类型</th><th key=required>是否必须</th><th key=default>默认值</th><th key=desc>备注</th><th key=sub>其他信息</th>
-    </tr>
-  </thead><tbody className="ant-table-tbody"><tr key=0-0><td key=0><span style="padding-left: 0px"><span style="color: #8c8a8a"></span> code</span></td><td key=1><span>number</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">响应状态吗，200表示成功</span></td><td key=5></td></tr><tr key=0-1><td key=0><span style="padding-left: 0px"><span style="color: #8c8a8a"></span> msg</span></td><td key=1><span>string</span></td><td key=2>必须</td><td key=3></td><td key=4><span style="white-space: pre-wrap">响应信息</span></td><td key=5></td></tr>
-               </tbody>
-              </table>
-            

doc/sql/20230508.sql

@@ -0,0 +1,4 @@
+INSERT INTO `sys_menu` (`menu_id`, `parent_id`, `name`, `url`, `perms`, `type`, `icon`, `order_num`)
+VALUES (305, '301', '修改', null, 'novel:websiteInfo:edit', '2', null, '6');
+INSERT INTO sys_role_menu (role_id, menu_id)
+VALUES (1, 305);

doc/sql/20240512.sql

@@ -0,0 +1,3 @@
+update crawl_source
+set crawl_rule = replace(crawl_rule, 'ibiquge.net', 'ibiquzw.org')
+where id = 16;

doc/sql/Dockerfile

@@ -0,0 +1,2 @@
+FROM mysql:8.0
+COPY novel_plus.sql /docker-entrypoint-initdb.d/init.sql

novel-admin/Dockerfile

@@ -1,6 +0,0 @@
-FROM java:8
-ADD novel-admin-1.0.0.jar /root
-ENV dburl=""
-ENV username=""
-ENV	password=""
-ENTRYPOINT ["sh","-c","java -Dspring.datasource.url=${dburl} -Dspring.datasource.username=${username} -Dspring.datasource.password=${password} -jar /root/novel-admin-1.0.0.jar"]

novel-admin/pom.xml

@@ -5,7 +5,7 @@
 
     <groupId>com.java2nb</groupId>
     <artifactId>novel-admin</artifactId>
-    <version>4.1.0</version>
+    <version>4.4.0</version>
     <packaging>jar</packaging>
 
     <name>novel-admin</name>
@@ -14,7 +14,7 @@
     <parent>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-starter-parent</artifactId>
-        <version>2.1.8.RELEASE</version>
+        <version>2.1.18.RELEASE</version>
         <relativePath/>
     </parent>
 
@@ -25,6 +25,7 @@
         <velocity.version>1.7</velocity.version>
         <activiti.version>5.22.0</activiti.version>
         <sharding.jdbc.version>3.0.0</sharding.jdbc.version>
+        <jackson.version>2.15.1</jackson.version>
     </properties>
 
     <dependencies>
@@ -63,12 +64,12 @@
         <dependency>
             <groupId>mysql</groupId>
             <artifactId>mysql-connector-java</artifactId>
-            <version>8.0.11</version>
+            <version>8.0.29</version>
         </dependency>
         <dependency>
             <groupId>org.mybatis</groupId>
             <artifactId>mybatis</artifactId>
-            <version>3.4.4</version>
+            <version>3.5.6</version>
         </dependency>
 
         <dependency>
@@ -80,7 +81,7 @@
         <dependency>
             <groupId>com.alibaba</groupId>
             <artifactId>druid</artifactId>
-            <version>1.0.28</version>
+            <version>1.2.9</version>
         </dependency>
         <!--commons -->
         <dependency>
@@ -96,18 +97,18 @@
         <dependency>
             <groupId>commons-io</groupId>
             <artifactId>commons-io</artifactId>
-            <version>2.5</version>
+            <version>2.9.0</version>
         </dependency>
         <!--shiro -->
         <dependency>
             <groupId>org.apache.shiro</groupId>
             <artifactId>shiro-core</artifactId>
-            <version>1.3.2</version>
+            <version>1.11.0</version>
         </dependency>
         <dependency>
             <groupId>org.apache.shiro</groupId>
             <artifactId>shiro-spring</artifactId>
-            <version>1.7.0</version>
+            <version>1.11.0</version>
         </dependency>
         <!-- shiro ehcache -->
         <dependency>
@@ -119,7 +120,7 @@
                     <artifactId>ehcache-core</artifactId>
                 </exclusion>
             </exclusions>
-            <version>1.4.0</version>
+            <version>1.11.0</version>
         </dependency>
         <dependency>
             <groupId>com.github.theborakompanioni</groupId>
@@ -130,7 +131,7 @@
         <dependency>
             <groupId>com.alibaba</groupId>
             <artifactId>fastjson</artifactId>
-            <version>1.2.31</version>
+            <version>1.2.83</version>
         </dependency>
         <!--velocity代码生成使用模板 -->
         <dependency>
@@ -208,6 +209,12 @@
             <version>${sharding.jdbc.version}</version>
         </dependency>
 
+        <dependency>
+            <groupId>com.fasterxml.jackson.core</groupId>
+            <artifactId>jackson-databind</artifactId>
+            <version>${jackson.version}</version>
+        </dependency>
+
         <!--war包部署需要-->
         <!--<dependency>-->
         <!--<groupId>org.springframework.boot</groupId>-->
@@ -289,6 +296,9 @@
                                 <zip destfile='${project.build.directory}/build/${project.artifactId}.zip'>
                                     <zipfileset filemode="755" dir='${project.build.directory}/build/'/>
                                 </zip>
+
+                                <copy file="${basedir}/src/main/build/docker/Dockerfile"
+                                      tofile="${project.build.directory}/build/Dockerfile"/>
                             </tasks>
                         </configuration>
                     </execution>

novel-admin/src/main/build/docker/Dockerfile

@@ -0,0 +1,9 @@
+FROM openjdk:8
+ADD novel-admin.jar /root
+ENV dburl=""
+ENV username=""
+ENV	password=""
+ENV redishost = ""
+ENV redisport = ""
+ENV redispwd = ""
+ENTRYPOINT ["sh","-c","java -Dspring.datasource.url=${dburl} -Dspring.datasource.username=${username} -Dspring.datasource.password=${password} -Dspring.redis.host=${redishost} -Dspring.redis.port=${redisport} -Dspring.redis.password=${redispwd} -jar /root/novel-admin.jar"]

novel-admin/src/main/java/com/java2nb/common/config/ShiroConfig.java

@@ -68,6 +68,7 @@ public class ShiroConfig {
         filterChainDefinitionMap.put("/js/**", "anon");
         filterChainDefinitionMap.put("/fonts/**", "anon");
         filterChainDefinitionMap.put("/img/**", "anon");
+        filterChainDefinitionMap.put("/favicon.ico", "anon");
         filterChainDefinitionMap.put("/docs/**", "anon");
         filterChainDefinitionMap.put("/layuimini/**", "anon");
         filterChainDefinitionMap.put("/upload/**", "anon");

novel-admin/src/main/java/com/java2nb/novel/controller/FriendLinkController.java

@@ -12,6 +12,7 @@ import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.data.redis.core.RedisTemplate;
 import org.springframework.stereotype.Controller;
 import org.springframework.ui.Model;
+import org.springframework.validation.annotation.Validated;
 import org.springframework.web.bind.annotation.*;
 
 import java.util.List;
@@ -83,7 +84,7 @@ public class FriendLinkController {
     @ResponseBody
     @PostMapping("/save")
     @RequiresPermissions("novel:friendLink:add")
-    public R save(FriendLinkDO friendLink) {
+    public R save(@Validated FriendLinkDO friendLink) {
         if (friendLinkService.save(friendLink) > 0) {
             redisTemplate.delete(CacheKey.INDEX_LINK_KEY);
             return R.ok();
@@ -98,7 +99,7 @@ public class FriendLinkController {
     @ResponseBody
     @RequestMapping("/update")
     @RequiresPermissions("novel:friendLink:edit")
-    public R update(FriendLinkDO friendLink) {
+    public R update(@Validated FriendLinkDO friendLink) {
         friendLinkService.update(friendLink);
         redisTemplate.delete(CacheKey.INDEX_LINK_KEY);
         return R.ok();

novel-admin/src/main/java/com/java2nb/novel/domain/FriendLinkDO.java

@@ -1,163 +1,175 @@
 package com.java2nb.novel.domain;
 
-import java.io.Serializable;
-
-
-import java.math.BigDecimal;
-
 import com.fasterxml.jackson.databind.annotation.JsonSerialize;
 import com.java2nb.common.jsonserializer.LongToStringSerializer;
-
-
+import org.hibernate.validator.constraints.URL;
 import org.springframework.format.annotation.DateTimeFormat;
+
+import java.io.Serializable;
 import java.util.Date;
 
 
-
 /**
- * 
- * 
  * @author xiongxy
  * @email 1179705413@qq.com
  * @date 2023-04-14 15:12:25
  */
 public class FriendLinkDO implements Serializable {
-	private static final long serialVersionUID = 1L;
 
-	
-	//主键
-			private Integer id;
-	//链接名
-			private String linkName;
-	//链接url
-			private String linkUrl;
-	//排序号
-			private Integer sort;
-	//是否开启，0：不开启，1：开启
-			private Integer isOpen;
-	//创建人id
-		//java中的long能表示的范围比js中number大,也就意味着部分数值在js中存不下(变成不准确的值)
-	//所以通过序列化成字符串来解决
-	@JsonSerialize(using = LongToStringSerializer.class)
-			private Long createUserId;
-	//创建时间
-			@DateTimeFormat(pattern = "yyyy-MM-dd HH:mm:ss")
-		private Date createTime;
-	//更新者用户id
-		//java中的long能表示的范围比js中number大,也就意味着部分数值在js中存不下(变成不准确的值)
-	//所以通过序列化成字符串来解决
-	@JsonSerialize(using = LongToStringSerializer.class)
-			private Long updateUserId;
-	//更新时间
-			@DateTimeFormat(pattern = "yyyy-MM-dd HH:mm:ss")
-		private Date updateTime;
+    private static final long serialVersionUID = 1L;
 
-	/**
-	 * 设置：主键
-	 */
-	public void setId(Integer id) {
-		this.id = id;
-	}
-	/**
-	 * 获取：主键
-	 */
-	public Integer getId() {
-		return id;
-	}
-	/**
-	 * 设置：链接名
-	 */
-	public void setLinkName(String linkName) {
-		this.linkName = linkName;
-	}
-	/**
-	 * 获取：链接名
-	 */
-	public String getLinkName() {
-		return linkName;
-	}
-	/**
-	 * 设置：链接url
-	 */
-	public void setLinkUrl(String linkUrl) {
-		this.linkUrl = linkUrl;
-	}
-	/**
-	 * 获取：链接url
-	 */
-	public String getLinkUrl() {
-		return linkUrl;
-	}
-	/**
-	 * 设置：排序号
-	 */
-	public void setSort(Integer sort) {
-		this.sort = sort;
-	}
-	/**
-	 * 获取：排序号
-	 */
-	public Integer getSort() {
-		return sort;
-	}
-	/**
-	 * 设置：是否开启，0：不开启，1：开启
-	 */
-	public void setIsOpen(Integer isOpen) {
-		this.isOpen = isOpen;
-	}
-	/**
-	 * 获取：是否开启，0：不开启，1：开启
-	 */
-	public Integer getIsOpen() {
-		return isOpen;
-	}
-	/**
-	 * 设置：创建人id
-	 */
-	public void setCreateUserId(Long createUserId) {
-		this.createUserId = createUserId;
-	}
-	/**
-	 * 获取：创建人id
-	 */
-	public Long getCreateUserId() {
-		return createUserId;
-	}
-	/**
-	 * 设置：创建时间
-	 */
-	public void setCreateTime(Date createTime) {
-		this.createTime = createTime;
-	}
-	/**
-	 * 获取：创建时间
-	 */
-	public Date getCreateTime() {
-		return createTime;
-	}
-	/**
-	 * 设置：更新者用户id
-	 */
-	public void setUpdateUserId(Long updateUserId) {
-		this.updateUserId = updateUserId;
-	}
-	/**
-	 * 获取：更新者用户id
-	 */
-	public Long getUpdateUserId() {
-		return updateUserId;
-	}
-	/**
-	 * 设置：更新时间
-	 */
-	public void setUpdateTime(Date updateTime) {
-		this.updateTime = updateTime;
-	}
-	/**
-	 * 获取：更新时间
-	 */
-	public Date getUpdateTime() {
-		return updateTime;
-	}
+
+    //主键
+    private Integer id;
+    //链接名
+    private String linkName;
+    //链接url
+    @URL
+    private String linkUrl;
+    //排序号
+    private Integer sort;
+    //是否开启，0：不开启，1：开启
+    private Integer isOpen;
+    //创建人id
+    //java中的long能表示的范围比js中number大,也就意味着部分数值在js中存不下(变成不准确的值)
+    //所以通过序列化成字符串来解决
+    @JsonSerialize(using = LongToStringSerializer.class)
+    private Long createUserId;
+    //创建时间
+    @DateTimeFormat(pattern = "yyyy-MM-dd HH:mm:ss")
+    private Date createTime;
+    //更新者用户id
+    //java中的long能表示的范围比js中number大,也就意味着部分数值在js中存不下(变成不准确的值)
+    //所以通过序列化成字符串来解决
+    @JsonSerialize(using = LongToStringSerializer.class)
+    private Long updateUserId;
+    //更新时间
+    @DateTimeFormat(pattern = "yyyy-MM-dd HH:mm:ss")
+    private Date updateTime;
+
+    /**
+     * 设置：主键
+     */
+    public void setId(Integer id) {
+        this.id = id;
+    }
+
+    /**
+     * 获取：主键
+     */
+    public Integer getId() {
+        return id;
+    }
+
+    /**
+     * 设置：链接名
+     */
+    public void setLinkName(String linkName) {
+        this.linkName = linkName;
+    }
+
+    /**
+     * 获取：链接名
+     */
+    public String getLinkName() {
+        return linkName;
+    }
+
+    /**
+     * 设置：链接url
+     */
+    public void setLinkUrl(String linkUrl) {
+        this.linkUrl = linkUrl;
+    }
+
+    /**
+     * 获取：链接url
+     */
+    public String getLinkUrl() {
+        return linkUrl;
+    }
+
+    /**
+     * 设置：排序号
+     */
+    public void setSort(Integer sort) {
+        this.sort = sort;
+    }
+
+    /**
+     * 获取：排序号
+     */
+    public Integer getSort() {
+        return sort;
+    }
+
+    /**
+     * 设置：是否开启，0：不开启，1：开启
+     */
+    public void setIsOpen(Integer isOpen) {
+        this.isOpen = isOpen;
+    }
+
+    /**
+     * 获取：是否开启，0：不开启，1：开启
+     */
+    public Integer getIsOpen() {
+        return isOpen;
+    }
+
+    /**
+     * 设置：创建人id
+     */
+    public void setCreateUserId(Long createUserId) {
+        this.createUserId = createUserId;
+    }
+
+    /**
+     * 获取：创建人id
+     */
+    public Long getCreateUserId() {
+        return createUserId;
+    }
+
+    /**
+     * 设置：创建时间
+     */
+    public void setCreateTime(Date createTime) {
+        this.createTime = createTime;
+    }
+
+    /**
+     * 获取：创建时间
+     */
+    public Date getCreateTime() {
+        return createTime;
+    }
+
+    /**
+     * 设置：更新者用户id
+     */
+    public void setUpdateUserId(Long updateUserId) {
+        this.updateUserId = updateUserId;
+    }
+
+    /**
+     * 获取：更新者用户id
+     */
+    public Long getUpdateUserId() {
+        return updateUserId;
+    }
+
+    /**
+     * 设置：更新时间
+     */
+    public void setUpdateTime(Date updateTime) {
+        this.updateTime = updateTime;
+    }
+
+    /**
+     * 获取：更新时间
+     */
+    public Date getUpdateTime() {
+        return updateTime;
+    }
 }

novel-admin/src/main/java/com/java2nb/novel/service/impl/BookSettingServiceImpl.java

@@ -34,7 +34,6 @@ public class BookSettingServiceImpl implements BookSettingService {
             List<Long> bookIds = list.stream().map(BookSettingDO::getBookId).collect(Collectors.toList());
             Map<Long, String> bookNameMap = bookDao.batchGet(bookIds).stream()
                 .collect(Collectors.toMap(BookDO::getId, BookDO::getBookName));
-            list = list.stream().filter(v -> bookNameMap.containsKey(v.getBookId())).collect(Collectors.toList());
             list.forEach(v -> v.setBookName(bookNameMap.get(v.getBookId())));
         }
         return list;

novel-admin/src/main/resources/application.yml

@@ -18,11 +18,12 @@ spring:
   profiles:
     active: dev
   
-  #上传文件的最大值（10M）
+  #上传文件的最大值（100M）
   servlet:
     multipart:
-      max-file-size: 10485760
-        
+      max-file-size: 100MB
+      max-request-size: 100MB
+          
   devtools:
     restart:
       enabled: true

novel-admin/src/main/resources/templates/novel/friendLink/add.html

@@ -23,7 +23,7 @@
                             <div class="col-sm-8">
                                 <input id="linkUrl" name="linkUrl"
                                        class="form-control"
-                                       type="text" required>
+                                       type="url" required>
 
                             </div>
                         </div>

novel-admin/src/main/resources/templates/novel/friendLink/edit.html

@@ -26,7 +26,7 @@
                                 <input id="linkUrl" name="linkUrl"
                                        th:value="${friendLink.linkUrl}"
                                        class="form-control"
-                                       type="text" required>
+                                       type="url" required>
                             </div>
                         </div>
                         <div class="form-group">

novel-common/pom.xml

@@ -5,7 +5,7 @@
     <parent>
         <artifactId>novel</artifactId>
         <groupId>com.java2nb</groupId>
-        <version>4.1.0</version>
+        <version>4.4.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 
@@ -88,7 +88,7 @@
         <dependency>
             <groupId>org.apache.httpcomponents</groupId>
             <artifactId>httpclient</artifactId>
-            <version>4.5.2</version>
+            <version>4.5.14</version>
             <exclusions>
                 <exclusion>
                     <groupId>commons-logging</groupId>

novel-common/src/main/java/com/java2nb/novel/core/advice/PageExceptionHandler.java

@@ -10,7 +10,7 @@ import org.springframework.web.bind.annotation.ExceptionHandler;
  * @author 11797
  */
 @Slf4j
-@ControllerAdvice(basePackages = "com.java2nb.novel.page")
+@ControllerAdvice(basePackages = "com.java2nb.novel.controller.page")
 public class PageExceptionHandler {
 
 

novel-common/src/main/java/com/java2nb/novel/core/utils/FileUtil.java

@@ -14,7 +14,10 @@ import org.springframework.http.ResponseEntity;
 
 import javax.imageio.ImageIO;
 import java.awt.image.BufferedImage;
-import java.io.*;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
 import java.util.Date;
 import java.util.Objects;
 
@@ -37,10 +40,13 @@ public class FileUtil {
             //本地图片保存
             HttpHeaders headers = new HttpHeaders();
             HttpEntity<String> requestEntity = new HttpEntity<>(null, headers);
-            ResponseEntity<Resource> resEntity = RestTemplateUtil.getInstance(Charsets.ISO_8859_1.name()).exchange(picSrc, HttpMethod.GET, requestEntity, Resource.class);
+            ResponseEntity<Resource> resEntity = RestTemplates.newInstance(Charsets.ISO_8859_1.name())
+                .exchange(picSrc, HttpMethod.GET, requestEntity, Resource.class);
             input = Objects.requireNonNull(resEntity.getBody()).getInputStream();
             Date currentDate = new Date();
-            picSrc = visitPrefix + DateUtils.formatDate(currentDate, "yyyy") + "/" + DateUtils.formatDate(currentDate, "MM") + "/" + DateUtils.formatDate(currentDate, "dd") + "/"
+            picSrc =
+                visitPrefix + DateUtils.formatDate(currentDate, "yyyy") + "/" + DateUtils.formatDate(currentDate, "MM")
+                    + "/" + DateUtils.formatDate(currentDate, "dd") + "/"
                     + UUIDUtil.getUUID32()
                     + picSrc.substring(picSrc.lastIndexOf("."));
             File picFile = new File(picSavePath + picSrc);
@@ -67,7 +73,6 @@ public class FileUtil {
             closeStream(input, out);
         }
 
-
         return picSrc;
     }
 

novel-common/src/main/java/com/java2nb/novel/core/utils/HttpUtil.java

@@ -1,38 +1,24 @@
 package com.java2nb.novel.core.utils;
 
+import lombok.extern.slf4j.Slf4j;
 import org.springframework.http.*;
 import org.springframework.web.client.RestTemplate;
 
 /**
  * @author Administrator
  */
+@Slf4j
 public class HttpUtil {
 
-    private static RestTemplate restTemplate = RestTemplateUtil.getInstance("utf-8");
-
-
-    public static String getByHttpClient(String url) {
-        try {
-
-            ResponseEntity<String> forEntity = restTemplate.getForEntity(url, String.class);
-            if (forEntity.getStatusCode() == HttpStatus.OK) {
-                return forEntity.getBody();
-            } else {
-                return null;
-            }
-        } catch (Exception e) {
-            e.printStackTrace();
-            return null;
-        }
-    }
+    private static final RestTemplate REST_TEMPLATE = RestTemplates.newInstance("utf-8");
 
     public static String getByHttpClientWithChrome(String url) {
         try {
-
             HttpHeaders headers = new HttpHeaders();
-            headers.add("user-agent","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Safari/537.36");
+            headers.add("user-agent",
+                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Safari/537.36");
             HttpEntity<String> requestEntity = new HttpEntity<>(null, headers);
-            ResponseEntity<String> forEntity = restTemplate.exchange(url.toString(), HttpMethod.GET, requestEntity, String.class);
+            ResponseEntity<String> forEntity = REST_TEMPLATE.exchange(url, HttpMethod.GET, requestEntity, String.class);
 
             if (forEntity.getStatusCode() == HttpStatus.OK) {
                 return forEntity.getBody();
@@ -40,8 +26,9 @@ public class HttpUtil {
                 return null;
             }
         } catch (Exception e) {
-            e.printStackTrace();
+            log.error(e.getMessage(), e);
             return null;
         }
     }
+
 }

novel-common/src/main/java/com/java2nb/novel/core/utils/RestTemplates.java

@@ -26,16 +26,16 @@ import java.util.List;
 import java.util.Objects;
 
 @Component
-public class RestTemplateUtil {
+public class RestTemplates {
 
     private static HttpProxyProperties httpProxyProperties;
 
-    RestTemplateUtil(HttpProxyProperties properties) {
+    RestTemplates(HttpProxyProperties properties) {
         httpProxyProperties = properties;
     }
 
     @SneakyThrows
-    public static RestTemplate getInstance(String charset) {
+    public static RestTemplate newInstance(String charset) {
 
         TrustStrategy acceptingTrustStrategy = (X509Certificate[] chain, String authType) -> true;
 

novel-common/src/main/java/com/java2nb/novel/entity/User.java

@@ -14,43 +14,48 @@ public class User {
     private Long id;
 
 
-    @NotBlank(groups = {AddGroup.class},message="手机号不能为空！")
-    @Pattern(groups = {AddGroup.class},regexp="^1[3|4|5|6|7|8|9][0-9]{9}$",message="手机号格式不正确！")
+    @NotBlank(groups = {AddGroup.class}, message = "手机号不能为空！")
+    @Pattern(groups = {AddGroup.class}, regexp = "^1[3|4|5|6|7|8|9][0-9]{9}$", message = "手机号格式不正确！")
+    @Null(groups = {UpdateGroup.class})
     @Generated("org.mybatis.generator.api.MyBatisGenerator")
     private String username;
 
-    @NotBlank(groups = {AddGroup.class},message="密码不能为空！")
+    @NotBlank(groups = {AddGroup.class}, message = "密码不能为空！")
     @Null(groups = {UpdateGroup.class})
     @Generated("org.mybatis.generator.api.MyBatisGenerator")
     private String password;
 
     @Null(groups = {AddGroup.class})
+    @Pattern(groups = {
+        UpdateGroup.class}, regexp = "[\u4E00-\u9FA5A-Za-z0-9_]{1,11}", message = "昵称格式不正确！")
     @Generated("org.mybatis.generator.api.MyBatisGenerator")
     private String nickName;
 
     @Null(groups = {AddGroup.class})
+    @Pattern(groups = {
+        UpdateGroup.class}, regexp = "^/localPic/\\d{4}/\\d{2}/\\d{2}/[A-Za-z0-9]+\\.(jpg|jpeg|swf|gif|png|JPG|JPEG|SWF|GIF|PNG)$", message = "只能上传图片格式的文件！")
     @Generated("org.mybatis.generator.api.MyBatisGenerator")
     private String userPhoto;
 
     @Null(groups = {AddGroup.class})
-    @Min(value = 0,groups = {UpdateGroup.class})
-    @Max(value = 1,groups = {UpdateGroup.class})
+    @Min(value = 0, groups = {UpdateGroup.class})
+    @Max(value = 1, groups = {UpdateGroup.class})
     @Generated("org.mybatis.generator.api.MyBatisGenerator")
     private Byte userSex;
 
-    @Null(groups = {AddGroup.class,UpdateGroup.class})
+    @Null(groups = {AddGroup.class, UpdateGroup.class})
     @Generated("org.mybatis.generator.api.MyBatisGenerator")
     private Long accountBalance;
 
-    @Null(groups = {AddGroup.class,UpdateGroup.class})
+    @Null(groups = {AddGroup.class, UpdateGroup.class})
     @Generated("org.mybatis.generator.api.MyBatisGenerator")
     private Byte status;
 
-    @Null(groups = {AddGroup.class,UpdateGroup.class})
+    @Null(groups = {AddGroup.class, UpdateGroup.class})
     @Generated("org.mybatis.generator.api.MyBatisGenerator")
     private Date createTime;
 
-    @Null(groups = {AddGroup.class,UpdateGroup.class})
+    @Null(groups = {AddGroup.class, UpdateGroup.class})
     @Generated("org.mybatis.generator.api.MyBatisGenerator")
     private Date updateTime;
 

novel-common/src/main/resources/application-common-dev.yml

@@ -69,8 +69,8 @@ sharding:
 
 content:
   save:
-    storage: db #存储介质，db：数据库，file：txt文本
-    path: /Users/xiongxiaoyang/books  #txt小说文本保存路径
+    storage: db # 小说内容存储方式：db-数据库，txt-TXT文本
+    path: /Users/xiongxiaoyang/books  # 小说TXT文本保存路径
 
 # HTTP 代理配置
 http:

novel-common/src/main/resources/application-common.yml

@@ -11,10 +11,11 @@ spring:
     generator:
       write-numbers-as-strings: true
 
-  #上传文件的最大值（1M）
+  #上传文件的最大值（100M）
   servlet:
     multipart:
-      max-file-size: 1048576
+      max-file-size: 100MB
+      max-request-size: 100MB
 
 mybatis:
   configuration:

novel-crawl/Dockerfile

@@ -1,6 +0,0 @@
-FROM java:8
-ADD novel-crawl-1.1.0.jar /root
-ENV dburl=""
-ENV username=""
-ENV	password=""
-ENTRYPOINT ["sh","-c","java -Dspring.datasource.url=${dburl} -Dspring.datasource.username=${username} -Dspring.datasource.password=${password} -jar /root/novel-crawl-1.1.0.jar"]

novel-crawl/pom.xml

@@ -5,7 +5,7 @@
     <parent>
         <artifactId>novel</artifactId>
         <groupId>com.java2nb</groupId>
-        <version>4.1.0</version>
+        <version>4.4.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 
@@ -26,6 +26,12 @@
             <artifactId>spring-boot-starter-security</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>com.fasterxml.jackson.core</groupId>
+            <artifactId>jackson-databind</artifactId>
+            <version>${jackson.version}</version>
+        </dependency>
+
 
     </dependencies>
 
@@ -66,6 +72,9 @@
                                 <zip destfile='${project.build.directory}/build/${project.artifactId}.zip'>
                                     <zipfileset filemode="755" dir='${project.build.directory}/build/'/>
                                 </zip>
+
+                                <copy file="${basedir}/src/main/build/docker/Dockerfile"
+                                      tofile="${project.build.directory}/build/Dockerfile"/>
                             </tasks>
                         </configuration>
                     </execution>

novel-crawl/src/main/build/config/application-common-prod.yml

@@ -39,8 +39,8 @@ crawl:
 #小说内容保存配置
 content:
   save:
-    storage: db #存储介质，db：数据库，file：txt文本
-    path: /Users/xiongxiaoyang/books  #txt小说文本保存路径
+    storage: db # 小说内容存储方式：db-数据库，txt-TXT文本
+    path: /Users/xiongxiaoyang/books  # 小说TXT文本保存路径
 
 # HTTP 代理配置
 http:

novel-crawl/src/main/build/docker/Dockerfile

@@ -0,0 +1,9 @@
+FROM openjdk:8
+ADD novel-crawl.jar /root
+ENV dburl=""
+ENV username=""
+ENV	password=""
+ENV redishost = ""
+ENV redisport = ""
+ENV redispwd = ""
+ENTRYPOINT ["sh","-c","java -Dspring.datasource.url=${dburl} -Dspring.datasource.username=${username} -Dspring.datasource.password=${password} -Dspring.redis.host=${redishost} -Dspring.redis.port=${redisport} -Dspring.redis.password=${redispwd} -jar /root/novel-crawl.jar"]

novel-crawl/src/main/java/com/java2nb/novel/controller/page/PageController.java

@@ -1,4 +1,4 @@
-package com.java2nb.novel.controller;
+package com.java2nb.novel.controller.page;
 
 import lombok.RequiredArgsConstructor;
 import lombok.extern.slf4j.Slf4j;
@@ -15,7 +15,6 @@ import org.springframework.web.bind.annotation.RequestMapping;
 public class PageController {
 
 
-
     @RequestMapping("{url}.html")
     public String module(@PathVariable("url") String url) {
         return url;
@@ -27,13 +26,14 @@ public class PageController {
     }
 
     @RequestMapping("{module}/{classify}/{url}.html")
-    public String module3(@PathVariable("module") String module, @PathVariable("classify") String classify, @PathVariable("url") String url) {
+    public String module3(@PathVariable("module") String module, @PathVariable("classify") String classify,
+        @PathVariable("url") String url) {
         return module + "/" + classify + "/" + url;
     }
 
     /**
      * 首页
-     * */
+     */
     @RequestMapping(path = {"/", "/index", "/index.html"})
     public String index() {
         return "crawl/crawlSource_list";

novel-crawl/src/main/java/com/java2nb/novel/core/config/SecurityConfiguration.java

@@ -1,4 +1,5 @@
 package com.java2nb.novel.core.config;
+
 import lombok.RequiredArgsConstructor;
 import org.springframework.beans.factory.annotation.Value;
 import org.springframework.context.annotation.Bean;
@@ -14,6 +15,7 @@ import org.springframework.security.crypto.password.PasswordEncoder;
 
 /**
  * SpringSecurity配置
+ *
  * @author Administrator
  */
 @Configuration
@@ -48,17 +50,18 @@ public class SecurityConfiguration extends WebSecurityConfigurerAdapter {
 
     @Override
     protected void configure(HttpSecurity http) throws Exception {
-        http.csrf().disable()//禁用了 csrf 功能
-                .authorizeRequests()//限定签名成功的请求
-                .antMatchers("/**").hasRole("ADMIN")
-                .anyRequest().permitAll()//其他没有限定的请求，允许访问
-                .and().anonymous()//对于没有配置权限的其他请求允许匿名访问
-                .and().formLogin()//使用 spring security 默认登录页面
-                .and().httpBasic();//启用http 基础验证
+        http.csrf().disable()
+            .authorizeRequests()
+            .antMatchers("/css/**").permitAll()
+            .antMatchers("/favicon.ico").permitAll()
+            .antMatchers("/**").hasRole("ADMIN")
+            .and().formLogin().loginPage("/login.html").loginProcessingUrl("/login").permitAll()
+            .and().logout()
+            .logoutUrl("/logout")
+            .logoutSuccessUrl("/")
+            .and().httpBasic();
 
     }
 
 
-
-
 }

novel-crawl/src/main/java/com/java2nb/novel/core/crawl/CrawlParser.java

@@ -1,23 +1,23 @@
 package com.java2nb.novel.core.crawl;
 
-import com.java2nb.novel.core.utils.HttpUtil;
 import com.java2nb.novel.core.utils.RandomBookInfoUtil;
-import com.java2nb.novel.core.utils.RestTemplateUtil;
 import com.java2nb.novel.core.utils.StringUtil;
 import com.java2nb.novel.entity.Book;
 import com.java2nb.novel.entity.BookContent;
 import com.java2nb.novel.entity.BookIndex;
 import com.java2nb.novel.utils.Constants;
+import com.java2nb.novel.utils.CrawlHttpClient;
 import io.github.xxyopen.util.IdWorker;
+import lombok.RequiredArgsConstructor;
 import lombok.SneakyThrows;
-import lombok.extern.slf4j.Slf4j;
 import org.apache.commons.lang3.StringUtils;
-import org.springframework.http.HttpStatus;
-import org.springframework.http.ResponseEntity;
-import org.springframework.web.client.RestTemplate;
+import org.springframework.stereotype.Component;
 
 import java.text.SimpleDateFormat;
-import java.util.*;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
@@ -26,20 +26,19 @@ import java.util.regex.Pattern;
  *
  * @author Administrator
  */
-@Slf4j
+@Component
+@RequiredArgsConstructor
 public class CrawlParser {
 
-    private static final IdWorker idWorker = IdWorker.INSTANCE;
+    private final IdWorker ID_WORKER = IdWorker.INSTANCE;
 
-    private static final RestTemplate restTemplate = RestTemplateUtil.getInstance("utf-8");
-
-    private static final ThreadLocal<Integer> retryCount = new ThreadLocal<>();
+    private final CrawlHttpClient crawlHttpClient;
 
     @SneakyThrows
-    public static void parseBook(RuleBean ruleBean, String bookId, CrawlBookHandler handler) {
+    public void parseBook(RuleBean ruleBean, String bookId, CrawlBookHandler handler) {
         Book book = new Book();
         String bookDetailUrl = ruleBean.getBookDetailUrl().replace("{bookId}", bookId);
-        String bookDetailHtml = getByHttpClientWithChrome(bookDetailUrl);
+        String bookDetailHtml = crawlHttpClient.get(bookDetailUrl);
         if (bookDetailHtml != null) {
             Pattern bookNamePatten = PatternFactory.getPattern(ruleBean.getBookNamePatten());
             Matcher bookNameMatch = bookNamePatten.matcher(bookDetailHtml);
@@ -89,14 +88,15 @@ public class CrawlParser {
                         }
                     }
 
-                    String desc = bookDetailHtml.substring(bookDetailHtml.indexOf(ruleBean.getDescStart()) + ruleBean.getDescStart().length());
+                    String desc = bookDetailHtml.substring(
+                        bookDetailHtml.indexOf(ruleBean.getDescStart()) + ruleBean.getDescStart().length());
                     desc = desc.substring(0, desc.indexOf(ruleBean.getDescEnd()));
                     //过滤掉简介中的特殊标签
                     desc = desc.replaceAll("<a[^<]+</a>", "")
-                            .replaceAll("<font[^<]+</font>", "")
-                            .replaceAll("<p>\\s*</p>", "")
-                            .replaceAll("<p>", "")
-                            .replaceAll("</p>", "<br/>");
+                        .replaceAll("<font[^<]+</font>", "")
+                        .replaceAll("<p>\\s*</p>", "")
+                        .replaceAll("<p>", "")
+                        .replaceAll("</p>", "<br/>");
                     //设置书籍简介
                     book.setBookDesc(desc);
                     if (StringUtils.isNotBlank(ruleBean.getStatusPatten())) {
@@ -112,14 +112,16 @@ public class CrawlParser {
                         }
                     }
 
-                    if (StringUtils.isNotBlank(ruleBean.getUpadateTimePatten()) && StringUtils.isNotBlank(ruleBean.getUpadateTimeFormatPatten())) {
+                    if (StringUtils.isNotBlank(ruleBean.getUpadateTimePatten()) && StringUtils.isNotBlank(
+                        ruleBean.getUpadateTimeFormatPatten())) {
                         Pattern updateTimePatten = PatternFactory.getPattern(ruleBean.getUpadateTimePatten());
                         Matcher updateTimeMatch = updateTimePatten.matcher(bookDetailHtml);
                         boolean isFindUpdateTime = updateTimeMatch.find();
                         if (isFindUpdateTime) {
                             String updateTime = updateTimeMatch.group(1);
                             //设置更新时间
-                            book.setLastIndexUpdateTime(new SimpleDateFormat(ruleBean.getUpadateTimeFormatPatten()).parse(updateTime));
+                            book.setLastIndexUpdateTime(
+                                new SimpleDateFormat(ruleBean.getUpadateTimeFormatPatten()).parse(updateTime));
 
                         }
                     }
@@ -141,7 +143,8 @@ public class CrawlParser {
         handler.handle(book);
     }
 
-    public static boolean parseBookIndexAndContent(String sourceBookId, Book book, RuleBean ruleBean, Map<Integer, BookIndex> existBookIndexMap, CrawlBookChapterHandler handler) {
+    public boolean parseBookIndexAndContent(String sourceBookId, Book book, RuleBean ruleBean,
+        Map<Integer, BookIndex> existBookIndexMap, CrawlBookChapterHandler handler) {
 
         Date currentDate = new Date();
 
@@ -149,11 +152,12 @@ public class CrawlParser {
         List<BookContent> contentList = new ArrayList<>();
         //读取目录
         String indexListUrl = ruleBean.getBookIndexUrl().replace("{bookId}", sourceBookId);
-        String indexListHtml = getByHttpClientWithChrome(indexListUrl);
+        String indexListHtml = crawlHttpClient.get(indexListUrl);
 
         if (indexListHtml != null) {
             if (StringUtils.isNotBlank(ruleBean.getBookIndexStart())) {
-                indexListHtml = indexListHtml.substring(indexListHtml.indexOf(ruleBean.getBookIndexStart()) + ruleBean.getBookIndexStart().length());
+                indexListHtml = indexListHtml.substring(
+                    indexListHtml.indexOf(ruleBean.getBookIndexStart()) + ruleBean.getBookIndexStart().length());
             }
 
             Pattern indexIdPatten = PatternFactory.getPattern(ruleBean.getIndexIdPatten());
@@ -174,14 +178,16 @@ public class CrawlParser {
                 BookIndex hasIndex = existBookIndexMap.get(indexNum);
                 String indexName = indexNameMatch.group(1);
 
-                if (hasIndex == null || !StringUtils.deleteWhitespace(hasIndex.getIndexName()).equals(StringUtils.deleteWhitespace(indexName))) {
+                if (hasIndex == null || !StringUtils.deleteWhitespace(hasIndex.getIndexName())
+                    .equals(StringUtils.deleteWhitespace(indexName))) {
 
                     String sourceIndexId = indexIdMatch.group(1);
                     String bookContentUrl = ruleBean.getBookContentUrl();
                     int calStart = bookContentUrl.indexOf("{cal_");
                     if (calStart != -1) {
                         //内容页URL需要进行计算才能得到
-                        String calStr = bookContentUrl.substring(calStart, calStart + bookContentUrl.substring(calStart).indexOf("}"));
+                        String calStr = bookContentUrl.substring(calStart,
+                            calStart + bookContentUrl.substring(calStart).indexOf("}"));
                         String[] calArr = calStr.split("_");
                         int calType = Integer.parseInt(calArr[1]);
                         if (calType == 1) {
@@ -206,13 +212,25 @@ public class CrawlParser {
 
                     }
 
-                    String contentUrl = bookContentUrl.replace("{bookId}", sourceBookId).replace("{indexId}", sourceIndexId);
+                    String contentUrl = bookContentUrl.replace("{bookId}", sourceBookId)
+                        .replace("{indexId}", sourceIndexId);
 
                     //查询章节内容
-                    String contentHtml = getByHttpClientWithChrome(contentUrl);
+                    String contentHtml = crawlHttpClient.get(contentUrl);
                     if (contentHtml != null && !contentHtml.contains("正在手打中")) {
-                        String content = contentHtml.substring(contentHtml.indexOf(ruleBean.getContentStart()) + ruleBean.getContentStart().length());
+                        String content = contentHtml.substring(
+                            contentHtml.indexOf(ruleBean.getContentStart()) + ruleBean.getContentStart().length());
                         content = content.substring(0, content.indexOf(ruleBean.getContentEnd()));
+                        // 小说内容过滤
+                        String filterContent = ruleBean.getFilterContent();
+                        if (StringUtils.isNotBlank(filterContent)) {
+                            String[] filterRules = filterContent.replace("\r\n", "\n").split("\n");
+                            for (String filterRule : filterRules) {
+                                if (StringUtils.isNotBlank(filterRule)) {
+                                    content = content.replaceAll(filterRule, "");
+                                }
+                            }
+                        }
                         //插入章节目录和章节内容
                         BookIndex bookIndex = new BookIndex();
                         bookIndex.setIndexName(indexName);
@@ -235,7 +253,7 @@ public class CrawlParser {
                         } else {
                             //章节插入
                             //设置目录和章节内容
-                            Long indexId = idWorker.nextId();
+                            Long indexId = ID_WORKER.nextId();
                             bookIndex.setId(indexId);
                             bookIndex.setBookId(book.getId());
 
@@ -257,7 +275,6 @@ public class CrawlParser {
                 isFindIndex = indexIdMatch.find() & indexNameMatch.find();
             }
 
-
             if (indexList.size() > 0) {
                 //如果有爬到最新章节，则设置小说主表的最新章节信息
                 //获取爬取到的最新章节
@@ -290,56 +307,4 @@ public class CrawlParser {
         return false;
 
     }
-
-
-    private static String getByHttpClient(String url) {
-        try {
-            ResponseEntity<String> forEntity = restTemplate.getForEntity(url, String.class);
-            if (forEntity.getStatusCode() == HttpStatus.OK) {
-                String body = forEntity.getBody();
-                assert body != null;
-                if (body.length() < Constants.INVALID_HTML_LENGTH) {
-                    return processErrorHttpResult(url);
-                }
-                //成功获得html内容
-                return body;
-            }
-        } catch (Exception e) {
-            e.printStackTrace();
-        }
-        return processErrorHttpResult(url);
-
-    }
-
-    private static String getByHttpClientWithChrome(String url) {
-        try {
-
-            String body = HttpUtil.getByHttpClientWithChrome(url);
-            if (body != null && body.length() < Constants.INVALID_HTML_LENGTH) {
-                return processErrorHttpResult(url);
-            }
-            //成功获得html内容
-            return body;
-        } catch (Exception e) {
-            e.printStackTrace();
-        }
-        return processErrorHttpResult(url);
-
-    }
-
-    @SneakyThrows
-    private static String processErrorHttpResult(String url) {
-        Integer count = retryCount.get();
-        if (count == null) {
-            count = 0;
-        }
-        if (count < Constants.HTTP_FAIL_RETRY_COUNT) {
-            Thread.sleep(new Random().nextInt(10 * 1000));
-            retryCount.set(++count);
-            return getByHttpClient(url);
-        }
-        return null;
-    }
-
-
 }

novel-crawl/src/main/java/com/java2nb/novel/core/crawl/RuleBean.java

@@ -6,6 +6,7 @@ import java.util.Map;
 
 /**
  * 爬虫解析规则bean
+ *
  * @author Administrator
  */
 @Data
@@ -13,17 +14,17 @@ public class RuleBean {
 
     /**
      * 小说更新列表url
-     * */
+     */
     private String updateBookListUrl;
 
     /**
      * 分类列表页URL规则
-     * */
+     */
     private String bookListUrl;
 
-    private Map<String,String> catIdRule;
+    private Map<String, String> catIdRule;
 
-    private Map<String,Byte> bookStatusRule;
+    private Map<String, Byte> bookStatusRule;
 
     private String bookIdPatten;
     private String pagePatten;
@@ -51,5 +52,7 @@ public class RuleBean {
 
     private String bookIndexStart;
 
+    private String filterContent;
+
 
 }

novel-crawl/src/main/java/com/java2nb/novel/core/listener/StarterListener.java

@@ -1,10 +1,12 @@
 package com.java2nb.novel.core.listener;
 
 import com.fasterxml.jackson.databind.ObjectMapper;
-import com.java2nb.novel.core.crawl.ChapterBean;
 import com.java2nb.novel.core.crawl.CrawlParser;
 import com.java2nb.novel.core.crawl.RuleBean;
-import com.java2nb.novel.entity.*;
+import com.java2nb.novel.entity.Book;
+import com.java2nb.novel.entity.BookIndex;
+import com.java2nb.novel.entity.CrawlSingleTask;
+import com.java2nb.novel.entity.CrawlSource;
 import com.java2nb.novel.service.BookService;
 import com.java2nb.novel.service.CrawlService;
 import com.java2nb.novel.utils.Constants;
@@ -33,6 +35,8 @@ public class StarterListener implements ServletContextListener {
 
     private final CrawlService crawlService;
 
+    private final CrawlParser crawlParser;
+
     @Value("${crawl.update.thread}")
     private int updateThreadCount;
 
@@ -56,20 +60,24 @@ public class StarterListener implements ServletContextListener {
                                 CrawlSource source = crawlService.queryCrawlSource(needUpdateBook.getCrawlSourceId());
                                 RuleBean ruleBean = new ObjectMapper().readValue(source.getCrawlRule(), RuleBean.class);
                                 //解析小说基本信息
-                                CrawlParser.parseBook(ruleBean, needUpdateBook.getCrawlBookId(),book -> {
+                                crawlParser.parseBook(ruleBean, needUpdateBook.getCrawlBookId(), book -> {
                                     //这里只做老书更新
                                     book.setId(needUpdateBook.getId());
                                     book.setWordCount(needUpdateBook.getWordCount());
-                                    if (needUpdateBook.getPicUrl() != null && needUpdateBook.getPicUrl().contains(Constants.LOCAL_PIC_PREFIX)) {
+                                    if (needUpdateBook.getPicUrl() != null && needUpdateBook.getPicUrl()
+                                        .contains(Constants.LOCAL_PIC_PREFIX)) {
                                         //本地图片则不更新
                                         book.setPicUrl(null);
                                     }
                                     //查询已存在的章节
-                                    Map<Integer, BookIndex> existBookIndexMap = bookService.queryExistBookIndexMap(needUpdateBook.getId());
+                                    Map<Integer, BookIndex> existBookIndexMap = bookService.queryExistBookIndexMap(
+                                        needUpdateBook.getId());
                                     //解析章节目录
-                                    CrawlParser.parseBookIndexAndContent(needUpdateBook.getCrawlBookId(), book, ruleBean, existBookIndexMap,chapter -> {
-                                        bookService.updateBookAndIndexAndContent(book, chapter.getBookIndexList(), chapter.getBookContentList(), existBookIndexMap);
-                                    });
+                                    crawlParser.parseBookIndexAndContent(needUpdateBook.getCrawlBookId(), book,
+                                        ruleBean, existBookIndexMap, chapter -> {
+                                            bookService.updateBookAndIndexAndContent(book, chapter.getBookIndexList(),
+                                                chapter.getBookContentList(), existBookIndexMap);
+                                        });
                                 });
                             } catch (Exception e) {
                                 log.error(e.getMessage(), e);
@@ -88,7 +96,6 @@ public class StarterListener implements ServletContextListener {
 
         }
 
-
         new Thread(() -> {
             log.info("程序启动,开始执行单本采集任务线程。。。");
             while (true) {
@@ -103,7 +110,8 @@ public class StarterListener implements ServletContextListener {
                         CrawlSource source = crawlService.queryCrawlSource(task.getSourceId());
                         RuleBean ruleBean = new ObjectMapper().readValue(source.getCrawlRule(), RuleBean.class);
 
-                        if (crawlService.parseBookAndSave(task.getCatId(), ruleBean, task.getSourceId(), task.getSourceBookId())) {
+                        if (crawlService.parseBookAndSave(task.getCatId(), ruleBean, task.getSourceId(),
+                            task.getSourceBookId())) {
                             //采集成功
                             crawlStatus = 1;
                         }

novel-crawl/src/main/java/com/java2nb/novel/service/impl/CrawlServiceImpl.java

@@ -2,17 +2,11 @@ package com.java2nb.novel.service.impl;
 
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.github.pagehelper.PageHelper;
-import io.github.xxyopen.model.page.PageBean;
 import com.java2nb.novel.core.cache.CacheKey;
 import com.java2nb.novel.core.cache.CacheService;
 import com.java2nb.novel.core.crawl.CrawlParser;
 import com.java2nb.novel.core.crawl.RuleBean;
 import com.java2nb.novel.core.enums.ResponseStatus;
-import io.github.xxyopen.model.page.builder.pagehelper.PageBuilder;
-import io.github.xxyopen.util.IdWorker;
-import io.github.xxyopen.util.ThreadUtil;
-import io.github.xxyopen.web.exception.BusinessException;
-import io.github.xxyopen.web.util.BeanUtil;
 import com.java2nb.novel.entity.Book;
 import com.java2nb.novel.entity.CrawlSingleTask;
 import com.java2nb.novel.entity.CrawlSource;
@@ -22,8 +16,15 @@ import com.java2nb.novel.mapper.CrawlSourceDynamicSqlSupport;
 import com.java2nb.novel.mapper.CrawlSourceMapper;
 import com.java2nb.novel.service.BookService;
 import com.java2nb.novel.service.CrawlService;
+import com.java2nb.novel.utils.CrawlHttpClient;
 import com.java2nb.novel.vo.CrawlSingleTaskVO;
 import com.java2nb.novel.vo.CrawlSourceVO;
+import io.github.xxyopen.model.page.PageBean;
+import io.github.xxyopen.model.page.builder.pagehelper.PageBuilder;
+import io.github.xxyopen.util.IdWorker;
+import io.github.xxyopen.util.ThreadUtil;
+import io.github.xxyopen.web.exception.BusinessException;
+import io.github.xxyopen.web.util.BeanUtil;
 import io.github.xxyopen.web.util.SpringUtil;
 import lombok.RequiredArgsConstructor;
 import lombok.SneakyThrows;
@@ -38,7 +39,6 @@ import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
-import static com.java2nb.novel.core.utils.HttpUtil.getByHttpClientWithChrome;
 import static com.java2nb.novel.mapper.CrawlSourceDynamicSqlSupport.*;
 import static org.mybatis.dynamic.sql.SqlBuilder.isEqualTo;
 import static org.mybatis.dynamic.sql.select.SelectDSL.select;
@@ -51,6 +51,7 @@ import static org.mybatis.dynamic.sql.select.SelectDSL.select;
 @Slf4j
 public class CrawlServiceImpl implements CrawlService {
 
+    private final CrawlParser crawlParser;
 
     private final CrawlSourceMapper crawlSourceMapper;
 
@@ -62,6 +63,8 @@ public class CrawlServiceImpl implements CrawlService {
 
     private final IdWorker idWorker = IdWorker.INSTANCE;
 
+    private final CrawlHttpClient crawlHttpClient;
+
 
     @Override
     public void addCrawlSource(CrawlSource source) {
@@ -71,15 +74,16 @@ public class CrawlServiceImpl implements CrawlService {
         crawlSourceMapper.insertSelective(source);
 
     }
+
     @Override
     public void updateCrawlSource(CrawlSource source) {
-        if(source.getId()!=null){
-            Optional<CrawlSource> opt=crawlSourceMapper.selectByPrimaryKey(source.getId());
-            if(opt.isPresent()) {
-                CrawlSource crawlSource =opt.get();
+        if (source.getId() != null) {
+            Optional<CrawlSource> opt = crawlSourceMapper.selectByPrimaryKey(source.getId());
+            if (opt.isPresent()) {
+                CrawlSource crawlSource = opt.get();
                 if (crawlSource.getSourceStatus() == (byte) 1) {
                     //关闭
-                    openOrCloseCrawl(crawlSource.getId(),(byte)0);
+                    openOrCloseCrawl(crawlSource.getId(), (byte) 0);
                 }
                 Date currentDate = new Date();
                 crawlSource.setUpdateTime(currentDate);
@@ -89,14 +93,15 @@ public class CrawlServiceImpl implements CrawlService {
             }
         }
     }
+
     @Override
     public PageBean<CrawlSource> listCrawlByPage(int page, int pageSize) {
         PageHelper.startPage(page, pageSize);
         SelectStatementProvider render = select(id, sourceName, sourceStatus, createTime, updateTime)
-                .from(crawlSource)
-                .orderBy(updateTime)
-                .build()
-                .render(RenderingStrategies.MYBATIS3);
+            .from(crawlSource)
+            .orderBy(updateTime)
+            .build()
+            .render(RenderingStrategies.MYBATIS3);
         List<CrawlSource> crawlSources = crawlSourceMapper.selectMany(render);
         PageBean<CrawlSource> pageBean = PageBuilder.build(crawlSources);
         pageBean.setList(BeanUtil.copyList(crawlSources, CrawlSourceVO.class));
@@ -113,7 +118,8 @@ public class CrawlServiceImpl implements CrawlService {
         if (sourceStatus == (byte) 0) {
             //关闭,直接修改数据库状态，并直接修改数据库状态后获取该爬虫正在运行的线程集合全部停止
             SpringUtil.getBean(CrawlService.class).updateCrawlSourceStatus(sourceId, sourceStatus);
-            Set<Long> runningCrawlThreadId = (Set<Long>) cacheService.getObject(CacheKey.RUNNING_CRAWL_THREAD_KEY_PREFIX + sourceId);
+            Set<Long> runningCrawlThreadId = (Set<Long>) cacheService.getObject(
+                CacheKey.RUNNING_CRAWL_THREAD_KEY_PREFIX + sourceId);
             if (runningCrawlThreadId != null) {
                 for (Long ThreadId : runningCrawlThreadId) {
                     Thread thread = ThreadUtil.findThread(ThreadId);
@@ -157,11 +163,12 @@ public class CrawlServiceImpl implements CrawlService {
 
     @Override
     public CrawlSource queryCrawlSource(Integer sourceId) {
-        SelectStatementProvider render = select(CrawlSourceDynamicSqlSupport.sourceStatus, CrawlSourceDynamicSqlSupport.crawlRule)
-                .from(crawlSource)
-                .where(id, isEqualTo(sourceId))
-                .build()
-                .render(RenderingStrategies.MYBATIS3);
+        SelectStatementProvider render = select(CrawlSourceDynamicSqlSupport.sourceStatus,
+            CrawlSourceDynamicSqlSupport.crawlRule)
+            .from(crawlSource)
+            .where(id, isEqualTo(sourceId))
+            .build()
+            .render(RenderingStrategies.MYBATIS3);
         return crawlSourceMapper.selectMany(render).get(0);
     }
 
@@ -182,10 +189,10 @@ public class CrawlServiceImpl implements CrawlService {
     public PageBean<CrawlSingleTask> listCrawlSingleTaskByPage(int page, int pageSize) {
         PageHelper.startPage(page, pageSize);
         SelectStatementProvider render = select(CrawlSingleTaskDynamicSqlSupport.crawlSingleTask.allColumns())
-                .from(CrawlSingleTaskDynamicSqlSupport.crawlSingleTask)
-                .orderBy(CrawlSingleTaskDynamicSqlSupport.createTime.descending())
-                .build()
-                .render(RenderingStrategies.MYBATIS3);
+            .from(CrawlSingleTaskDynamicSqlSupport.crawlSingleTask)
+            .orderBy(CrawlSingleTaskDynamicSqlSupport.createTime.descending())
+            .build()
+            .render(RenderingStrategies.MYBATIS3);
         List<CrawlSingleTask> crawlSingleTasks = crawlSingleTaskMapper.selectMany(render);
         PageBean<CrawlSingleTask> pageBean = PageBuilder.build(crawlSingleTasks);
         pageBean.setList(BeanUtil.copyList(crawlSingleTasks, CrawlSingleTaskVO.class));
@@ -200,7 +207,8 @@ public class CrawlServiceImpl implements CrawlService {
     @Override
     public CrawlSingleTask getCrawlSingleTask() {
 
-        List<CrawlSingleTask> list = crawlSingleTaskMapper.selectMany(select(CrawlSingleTaskDynamicSqlSupport.crawlSingleTask.allColumns())
+        List<CrawlSingleTask> list = crawlSingleTaskMapper.selectMany(
+            select(CrawlSingleTaskDynamicSqlSupport.crawlSingleTask.allColumns())
                 .from(CrawlSingleTaskDynamicSqlSupport.crawlSingleTask)
                 .where(CrawlSingleTaskDynamicSqlSupport.taskStatus, isEqualTo((byte) 2))
                 .orderBy(CrawlSingleTaskDynamicSqlSupport.createTime)
@@ -226,12 +234,12 @@ public class CrawlServiceImpl implements CrawlService {
 
     @Override
     public CrawlSource getCrawlSource(Integer id) {
-            Optional<CrawlSource> opt=crawlSourceMapper.selectByPrimaryKey(id);
-            if(opt.isPresent()) {
-                CrawlSource crawlSource =opt.get();
-                return crawlSource;
-            }
-            return null;
+        Optional<CrawlSource> opt = crawlSourceMapper.selectByPrimaryKey(id);
+        if (opt.isPresent()) {
+            CrawlSource crawlSource = opt.get();
+            return crawlSource;
+        }
+        return null;
     }
 
     /**
@@ -251,10 +259,10 @@ public class CrawlServiceImpl implements CrawlService {
                 if (StringUtils.isNotBlank(ruleBean.getCatIdRule().get("catId" + catId))) {
                     //拼接分类URL
                     String catBookListUrl = ruleBean.getBookListUrl()
-                            .replace("{catId}", ruleBean.getCatIdRule().get("catId" + catId))
-                            .replace("{page}", page + "");
+                        .replace("{catId}", ruleBean.getCatIdRule().get("catId" + catId))
+                        .replace("{page}", page + "");
 
-                    String bookListHtml = getByHttpClientWithChrome(catBookListUrl);
+                    String bookListHtml = crawlHttpClient.get(catBookListUrl);
                     if (bookListHtml != null) {
                         Pattern bookIdPatten = Pattern.compile(ruleBean.getBookIdPatten());
                         Matcher bookIdMatcher = bookIdPatten.matcher(bookListHtml);
@@ -268,14 +276,12 @@ public class CrawlServiceImpl implements CrawlService {
                                     return;
                                 }
 
-
                                 String bookId = bookIdMatcher.group(1);
                                 parseBookAndSave(catId, ruleBean, sourceId, bookId);
                             } catch (Exception e) {
                                 log.error(e.getMessage(), e);
                             }
 
-
                             isFindBookId = bookIdMatcher.find();
                         }
 
@@ -306,7 +312,7 @@ public class CrawlServiceImpl implements CrawlService {
 
         final AtomicBoolean parseResult = new AtomicBoolean(false);
 
-        CrawlParser.parseBook(ruleBean, bookId, book -> {
+        crawlParser.parseBook(ruleBean, bookId, book -> {
             if (book.getBookName() == null || book.getAuthorName() == null) {
                 return;
             }
@@ -330,9 +336,11 @@ public class CrawlServiceImpl implements CrawlService {
                 book.setCrawlLastTime(new Date());
                 book.setId(idWorker.nextId());
                 //解析章节目录
-                boolean parseIndexContentResult = CrawlParser.parseBookIndexAndContent(bookId, book, ruleBean, new HashMap<>(0), chapter -> {
-                    bookService.saveBookAndIndexAndContent(book, chapter.getBookIndexList(), chapter.getBookContentList());
-                });
+                boolean parseIndexContentResult = crawlParser.parseBookIndexAndContent(bookId, book, ruleBean,
+                    new HashMap<>(0), chapter -> {
+                        bookService.saveBookAndIndexAndContent(book, chapter.getBookIndexList(),
+                            chapter.getBookContentList());
+                    });
                 parseResult.set(parseIndexContentResult);
 
             } else {
@@ -356,11 +364,12 @@ public class CrawlServiceImpl implements CrawlService {
 
     @Override
     public List<CrawlSource> queryCrawlSourceByStatus(Byte sourceStatus) {
-        SelectStatementProvider render = select(CrawlSourceDynamicSqlSupport.id, CrawlSourceDynamicSqlSupport.sourceStatus, CrawlSourceDynamicSqlSupport.crawlRule)
-                .from(crawlSource)
-                .where(CrawlSourceDynamicSqlSupport.sourceStatus, isEqualTo(sourceStatus))
-                .build()
-                .render(RenderingStrategies.MYBATIS3);
+        SelectStatementProvider render = select(CrawlSourceDynamicSqlSupport.id,
+            CrawlSourceDynamicSqlSupport.sourceStatus, CrawlSourceDynamicSqlSupport.crawlRule)
+            .from(crawlSource)
+            .where(CrawlSourceDynamicSqlSupport.sourceStatus, isEqualTo(sourceStatus))
+            .build()
+            .render(RenderingStrategies.MYBATIS3);
         return crawlSourceMapper.selectMany(render);
     }
 }

novel-crawl/src/main/java/com/java2nb/novel/utils/Constants.java

@@ -7,7 +7,7 @@ public class Constants {
 
     /**
      * 本地图片保存前缀
-     * */
+     */
     public static final String LOCAL_PIC_PREFIX = "/localPic/";
 
     /**
@@ -23,5 +23,5 @@ public class Constants {
     /**
      * 爬取小说http请求失败重试次数
      */
-    public static final Integer HTTP_FAIL_RETRY_COUNT = 5;
+    public static final Integer HTTP_FAIL_RETRY_COUNT = 3;
 }

novel-crawl/src/main/java/com/java2nb/novel/utils/CrawlHttpClient.java

@@ -0,0 +1,57 @@
+package com.java2nb.novel.utils;
+
+import com.java2nb.novel.core.utils.HttpUtil;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.stereotype.Component;
+
+import java.util.Objects;
+import java.util.Random;
+
+/**
+ * @author Administrator
+ */
+@Slf4j
+@Component
+public class CrawlHttpClient {
+
+    @Value("${crawl.interval.min}")
+    private Integer intervalMin;
+
+    @Value("${crawl.interval.max}")
+    private Integer intervalMax;
+
+    private final Random random = new Random();
+
+    private static final ThreadLocal<Integer> RETRY_COUNT = new ThreadLocal<>();
+
+    public String get(String url) {
+        if (Objects.nonNull(intervalMin) && Objects.nonNull(intervalMax) && intervalMax > intervalMin) {
+            try {
+                Thread.sleep(random.nextInt(intervalMax - intervalMin + 1) + intervalMin);
+            } catch (InterruptedException e) {
+                log.error(e.getMessage(), e);
+            }
+        }
+        String body = HttpUtil.getByHttpClientWithChrome(url);
+        if (Objects.isNull(body) || body.length() < Constants.INVALID_HTML_LENGTH) {
+            return processErrorHttpResult(url);
+        }
+        //成功获得html内容
+        return body;
+    }
+
+    private String processErrorHttpResult(String url) {
+        Integer count = RETRY_COUNT.get();
+        if (count == null) {
+            count = 0;
+        }
+        if (count < Constants.HTTP_FAIL_RETRY_COUNT) {
+            RETRY_COUNT.set(++count);
+            return get(url);
+        }
+        RETRY_COUNT.remove();
+        return null;
+    }
+
+}

novel-crawl/src/main/resources/application.yml

@@ -14,12 +14,18 @@ admin:
   username: admin
   password: admin
 
-#爬虫自动更新的线程数
-#建议小说数量不多或者正在运行新书入库爬虫的情况下设置为1即可
-#随着小说数量的增多可以逐渐增加，但建议不要超出CPU的线程数
+
+
 crawl:
   update:
+    #爬虫自动更新的线程数
+    #建议小说数量不多或者正在运行新书入库爬虫的情况下设置为1即可
+    #随着小说数量的增多可以逐渐增加，但建议不要超出CPU的线程数
     thread: 1
+  # 采集间隔时间，单位：毫秒
+  interval:
+    min: 300
+    max: 500
 
 
 

novel-crawl/src/main/resources/static/css/signin.css

@@ -0,0 +1,44 @@
+body {
+    padding-top: 40px;
+    padding-bottom: 40px;
+    background-color: #eee;
+}
+
+.form-signin {
+    max-width: 330px;
+    padding: 15px;
+    margin: 0 auto;
+}
+
+.form-signin .form-signin-heading,
+.form-signin .checkbox {
+    margin-bottom: 10px;
+}
+
+.form-signin .checkbox {
+    font-weight: 400;
+}
+
+.form-signin .form-control {
+    position: relative;
+    box-sizing: border-box;
+    height: auto;
+    padding: 10px;
+    font-size: 16px;
+}
+
+.form-signin .form-control:focus {
+    z-index: 2;
+}
+
+.form-signin input[type="email"] {
+    margin-bottom: -1px;
+    border-bottom-right-radius: 0;
+    border-bottom-left-radius: 0;
+}
+
+.form-signin input[type="password"] {
+    margin-bottom: 10px;
+    border-top-left-radius: 0;
+    border-top-right-radius: 0;
+}