1.场景

例如有一张成语字典表，数据在13000条左右，需要将数据经过处理存到另一个存储中（可以是数据库），如何才能加快处理速度？

2.推荐工具

ETL工具kettle

3.解决方案

3.1.多线程互斥加快处理速度

代码地址：https://gitee.com/mgang/scene-handle/tree/master/handle-fastjob

1). 创建线程池执行对象

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

@Component
public class MainExecutor {
    private Logger log = LoggerFactory.getLogger(this.getClass());
    private ThreadPoolExecutor executor;
    private BlockingQueue blockingQueue;
    private int initNum = 3;
    private int maxNum = 10;
    private int queueSize = 100;
    private int keepAliveTime = 60;//单位秒

    @PostConstruct
    protected void init(){
        blockingQueue = new ArrayBlockingQueue<>(queueSize);
        executor = new ThreadPoolExecutor(initNum, maxNum, keepAliveTime,
                TimeUnit.SECONDS, blockingQueue,
                new ThreadPoolExecutor.CallerRunsPolicy());
    }
    /**
     * 添加任务
     * @param runnable
     */
    public void addTask(Runnable runnable){
        log.info("线程池ThreadPoolExecutor-ArrayBlockingQueue队列当前大小："+blockingQueue.size());
        executor.execute(runnable);
    }
}

2). 定义好业务处理线程

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

public class WorkThread implements Runnable {

    private IYesoulChenYuService yesoulChenYuService =
            (IYesoulChenYuService) SpringUtil.getBean("yesoulChenYuServiceImpl");

    @Override
    public void run() {
        yesoulChenYuService.moveData();
    }
}

3). 实现具体的业务逻辑

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

@Service
public class YesoulChenYuServiceImpl extends ServiceImpl<YesoulChenYuMapper, YesoulChenYu> implements IYesoulChenYuService {

    @Autowired
    @Qualifier("yesoulChenYu2ServiceImpl")
    private IYesoulChenYu2Service yesoulChenYu2Service;

    @Autowired
    private YesoulChenYuMapper yesoulChenYuMapper;

    @Override
    public void moveData() {
        List<YesoulChenYu> list = null;
        synchronized (this) {
            //1.先查询数据 线程锁
            list = yesoulChenYuMapper.selectList();
            System.out.println(JSONObject.toJSONString(list));
            for (YesoulChenYu yesoulChenYu : list) {
                //1.1.将状态修改为正在同步中 1
                YesoulChenYu update = new YesoulChenYu();
                update.setId(yesoulChenYu.getId());
                update.setFlag("1");
                this.updateById(update);
            }
        }
        for (YesoulChenYu yesoulChenYu : list) {
            //2.保存到目标库
            TargetYesoulChenYu t = new TargetYesoulChenYu();
            BeanUtils.copyProperties(yesoulChenYu, t);
            yesoulChenYu2Service.saveData(t);
            //3.修改同步标识
            YesoulChenYu update = new YesoulChenYu();
            update.setId(yesoulChenYu.getId());
            update.setFlag("2");
            this.updateById(update);
        }
    }
}