Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OUT OF MEMORY crash? #4059

Open
steadyrun opened this issue Jun 15, 2017 · 4 comments
Open

OUT OF MEMORY crash? #4059

steadyrun opened this issue Jun 15, 2017 · 4 comments

Comments

@steadyrun
Copy link

Could you tell me what problem is in my server?

my server has 1 core 2GB memory(AWS t2.small) & I use just 1 custom config, max-active=512.

=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:M 14 Jun 11:19:39.860 # ------------------------------------------------
1:M 14 Jun 11:19:39.860 # !!! Software Failure. Press left mouse button to continue
1:M 14 Jun 11:19:39.860 # Guru Meditation: "Redis aborting for OUT OF MEMORY" #server.c:3828
1:M 14 Jun 11:19:39.860 # (forcing SIGSEGV in order to print the stack trace)
1:M 14 Jun 11:19:39.860 # ------------------------------------------------
1:M 14 Jun 11:19:39.860 # Redis 3.2.8 crashed by signal: 11
1:M 14 Jun 11:19:39.860 # Crashed running the instuction at: 0x45bf4a
1:M 14 Jun 11:19:39.860 # Accessing address: 0xffffffffffffffff
1:M 14 Jun 11:19:39.860 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
redis-server *:6379(_serverPanic+0x7a)[0x45bf4a]

Backtrace:
redis-server *:6379(logStackTrace+0x29)[0x45d599]
redis-server *:6379(sigsegvHandler+0xa6)[0x45dc36]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f8bf859f890]
redis-server *:6379(_serverPanic+0x7a)[0x45bf4a]
redis-server *:6379(redisOutOfMemoryHandler+0x2c)[0x423d0c]
redis-server *:6379(zrealloc+0xc9)[0x42d209]
redis-server *:6379(sdsMakeRoomFor+0x2bd)[0x42ab0d]
redis-server *:6379(sdsgrowzero+0x6e)[0x42b15e]
redis-server *:6379(lookupStringForBitCommand+0x65)[0x470d05]
redis-server *:6379(setbitCommand+0x7c)[0x470ecc]
redis-server *:6379(call+0x81)[0x4260e1]
redis-server *:6379(processCommand+0x387)[0x4291b7]
redis-server *:6379(processInputBuffer+0x105)[0x435bd5]
redis-server *:6379(aeProcessEvents+0x133)[0x420653]
redis-server *:6379(aeMain+0x2b)[0x42099b]
redis-server *:6379(main+0x40b)[0x41d99b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f8bf8206b45]
redis-server *:6379[0x41dbef]

------ INFO OUTPUT ------

Server

redis_version:3.2.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:151525c861545b87
redis_mode:standalone
os:Linux 3.10.0-514.6.1.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.9.2
process_id:1
run_id:9afef020b94f003a8fef89c61a5ec4b6e49621d6
tcp_port:6379
uptime_in_seconds:185200
uptime_in_days:2
hz:10
lru_clock:4266955
executable:/data/redis-server
config_file:

Clients

connected_clients:3
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

Memory

used_memory:1477301928
used_memory_human:1.38G
used_memory_rss:1373413376
used_memory_rss_human:1.28G
used_memory_peak:1477341216
used_memory_peak_human:1.38G
total_system_memory:1927479296
total_system_memory_human:1.80G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:0.93
mem_allocator:jemalloc-4.0.3

Persistence

loading:0
rdb_changes_since_last_save:25
rdb_bgsave_in_progress:0
rdb_last_save_time:1497257580
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

Stats

total_connections_received:4223
total_commands_processed:20956
instantaneous_ops_per_sec:2
total_net_input_bytes:439390
total_net_output_bytes:39671735
instantaneous_input_kbps:0.14
instantaneous_output_kbps:0.01
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:321
keyspace_misses:257
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:59
migrate_cached_sockets:0

Replication

role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:115.96
used_cpu_user:65.18
used_cpu_sys_children:0.01
used_cpu_user_children:0.33

Commandstats

cmdstat_setbit:calls=8,usec=324704,usec_per_call=40588.00
cmdstat_getbit:calls=226,usec=827,usec_per_call=3.66
cmdstat_hget:calls=264,usec=662,usec_per_call=2.51
cmdstat_hincrby:calls=21,usec=110,usec_per_call=5.24
cmdstat_ping:calls=5286,usec=7508,usec_per_call=1.42
cmdstat_info:calls=15061,usec=600260,usec_per_call=39.86
cmdstat_config:calls=1,usec=9,usec_per_call=9.00
cmdstat_bitcount:calls=88,usec=971622,usec_per_call=11041.16
cmdstat_command:calls=1,usec=399,usec_per_call=399.00

Cluster

cluster_enabled:0

Keyspace

db0:keys=10,expires=0,avg_ttl=0
hash_init_value: 1496994265

------ CLIENT LIST OUTPUT ------
id=4209 addr=10.10.1.217:44788 fd=6 name= age=6472 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=4223 addr=10.10.2.184:43474 fd=7 name= age=56 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setbit
id=4224 addr=10.10.2.184:43480 fd=8 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hget

------ CURRENT CLIENT INFO ------
id=4223 addr=10.10.2.184:43474 fd=7 name= age=56 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setbit
argv[0]: 'SETBIT'
argv[1]: '20170614@猷⑤굹뚰봽

                         argv[2]: '3955627448'

argv[3]: '1'
1:M 14 Jun 11:19:39.869 # key '20170614@猷⑤굹뚰봽 found in DB containing the following object:
1:M 14 Jun 11:19:39.869 # Object type: 0
1:M 14 Jun 11:19:39.869 # Object encoding: 0
1:M 14 Jun 11:19:39.869 # Object refcount: 1
1:M 14 Jun 11:19:39.869 # Object raw string len: 377027058

------ REGISTERS ------
1:M 14 Jun 11:19:39.869 #
RAX:0000000000000000 RBX:00000000004e67f0
RCX:ffffffffffffffff RDX:0000000000000000
RDI:0000000000000000 RSI:00007f8bf858c7a0
RBP:00000000004e8ed0 RSP:00007ffff0190690
R8 :00007f8bf8ec9780 R9 :00007f8bf823099a
R10:00007f8bf85896a0 R11:0000000000000000
R12:0000000000000ef4 R13:000000001678f9f2
R14:0000000000000003 R15:000000001d88c2b8
RIP:000000000045bf4a EFL:0000000000010202
CSGSFS:0000000000000033
1:M 14 Jun 11:19:39.869 # (00007ffff019069f) -> 0000000000000019
1:M 14 Jun 11:19:39.869 # (00007ffff019069e) -> 0000000006ffc8c6
1:M 14 Jun 11:19:39.869 # (00007ffff019069d) -> 000000001d78c2b8
1:M 14 Jun 11:19:39.869 # (00007ffff019069c) -> 00007f8bf7d26400
1:M 14 Jun 11:19:39.869 # (00007ffff019069b) -> 000000001678f9f2
1:M 14 Jun 11:19:39.869 # (00007ffff019069a) -> 0000000000000000
1:M 14 Jun 11:19:39.869 # (00007ffff0190699) -> 000000000042ab0d
1:M 14 Jun 11:19:39.869 # (00007ffff0190698) -> 00007f8b9e800000
1:M 14 Jun 11:19:39.869 # (00007ffff0190697) -> 00007f8b9e800009
1:M 14 Jun 11:19:39.869 # (00007ffff0190696) -> 0000000000000009
1:M 14 Jun 11:19:39.869 # (00007ffff0190695) -> 000000000042d209
1:M 14 Jun 11:19:39.869 # (00007ffff0190694) -> 000000001d88c2b8
1:M 14 Jun 11:19:39.869 # (00007ffff0190693) -> 0000000000423d0c
1:M 14 Jun 11:19:39.869 # (00007ffff0190692) -> 000000001d88c2c2
1:M 14 Jun 11:19:39.869 # (00007ffff0190691) -> 0000000018000000
1:M 14 Jun 11:19:39.869 # (00007ffff0190690) -> 0000000000000000

------ FAST MEMORY TEST ------
1:M 14 Jun 11:19:39.869 # Bio thread for job type #0 terminated
1:M 14 Jun 11:19:39.869 # Bio thread for job type #1 terminated
*** Preparing to test memory region 730000 (94208 bytes)
*** Preparing to test memory region 242e000 (135168 bytes)
*** Preparing to test memory region 7f8b9e800000 (1478492160 bytes)
*** Preparing to test memory region 7f8bf6bff000 (8388608 bytes)
*** Preparing to test memory region 7f8bf7400000 (12582912 bytes)
*** Preparing to test memory region 7f8bf858c000 (16384 bytes)
*** Preparing to test memory region 7f8bf87a9000 (16384 bytes)
*** Preparing to test memory region 7f8bf8ec9000 (16384 bytes)
*** Preparing to test memory region 7f8bf8ece000 (16384 bytes)
*** Preparing to test memory region 7f8bf8ed4000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: _serverPanic (base: 0x45bed0)
Module: redis-server *:6379 (base 0x400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x45bed0 -D -b binary -m i386:x86-64 /tmp/dump.bin

1:M 14 Jun 11:19:44.280 # dump of function (hexdump of 250 bytes):
8b05165e2d0041544189d4554889f5534889fb85c07505e894f9ffffbe58184f00bf0300000031c0e8037dfcffbe90184f00bf0300000031c0e8f27cfcff4589e04889e94889dabe50274f00bf0300000031c0e8d87cfcffbed0184f00bf0300000031c0e8c77cfcffbe58184f00bf0300000031c0e8b67cfcffc60425ffffffff785b5d415cc3660f1f84000000000048c7070000000048c7470800000000c74710000000008b0544592d0085c00f8eb7060000415741564155415455534881ec4801000048897c241048c744242800000000488d9c24c0000000c744242400000000eb1e0f1f0083442424014883442428408b4424243905f3
Function at 0x423c00 is serverLog

=== REDIS BUG REPORT END. Make sure to include from START to END. ===
@badboy
Copy link
Contributor

badboy commented Jun 15, 2017 

used_memory_human:1.38G

For some reason or another, your instance is using over 1 Gb of memory.
With only 2 Gb of main memory and your system requiring some of it as well, you simply run out of more to use for Redis.
Reduce memory usage, increase memory or set a proper memory limit for Redis.

@charsyam
Copy link
Contributor

@steadyrun Yes, It is because of "out of memory"
Your redis server wasted all memory.

@steadyrun
Copy link
Author

@charsyam @badboy Can you suggest proper setting?
I'm not expert of redis..

@scu2013
Copy link

scu2013 commented Jun 16, 2017

you can set maxmemory 1G in your conf file. BUT reducing your redis cost is most important.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@badboy @charsyam @steadyrun @scu2013