Redis slowing down when rewriting AOF files.

redis 가 AOF 방식이고, aof파일에 대한 rewritting 이 일어나는 동안,

jedis(java) 클라이언트가 redis connection pool 반환을 받지 못하는 상태가 종종 발생 한다.

redis.log

[4554] 17 Jan 10:54:11 * Starting automatic rewriting of AOF on 100% growth

[4554] 17 Jan 10:54:12 * Background append only file rewriting started by pid 13226

[13226] 17 Jan 10:55:58 * SYNC append only file rewrite performed

[4554] 17 Jan 10:55:59 * Background AOF rewrite terminated with success

[4554] 17 Jan 10:55:59 * Parent diff successfully flushed to the rewritten AOF (20865463 bytes)

[4554] 17 Jan 10:55:59 * Background AOF rewrite successful

tomcat.log

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool at redis.clients.util.Pool.getResource(Pool.java:22)

관련해서 redis git 에 관련된 이슈가 등록되어 있다.

 antirez commented2 years ago

Unfortunately this is what happens when Redis is used with a very slow disk (EBS rocks at slowness), the only thing you can do is, in order to mitigate the problem, relax your AOF fsync policy from everysec to none (that in Linux this means that data will be flushed in 30 seconds at max). However even with this change if disk can't cope with I/O soon or later you'll see Redis blocking.

It's worth to spend a few words to show how this is not avoidable and is not Redis fault:

1) You ask Redis to fsync every second the AOF, to flush the kernel buffers to disk.
2) Redis will try, and usually disk will be able to receive the generated I/O without too much problems.
3) Then Redis has to rewrite the log because it is too big, so Redis creates another child to rewrite the log, however the log rewriting will use additional I/O.
4) At this point if the disk is slow, the additional I/O used to rewrite the log saturated it to the point that Redis will try to fsync data every second but the background thread is not able to accomplish the sync in a second. Redis will log a warning, and eventually if after two seconds the thread is still waiting to perform the fsync, it starts blocking, in your interest: you want data to be transmitted on disk!

So you can either relax the fsync policy, or use ephemeral storage for Redis (and move backups from time to time into EBS), and so forth.

Cheers,
Salvatore

p.s. closing the issue.

Closed

 antirez closed the issue 2 years ago

현상에 대한 원인과 마지막 부분에 답이 있는데.

So you can either relax the fsync policy, or use ephemeral storage for Redis (and move backups from time to time into EBS), and so forth.

디스크를 SSD로 바꾸는게 제일 좋을듯하다.

참조

https://github.com/antirez/redis

https://github.com/antirez/redis/issues/368

http://redis.io/topics/persistence

http://redis.io/commands/BGREWRITEAOF