hashtable crash

[maczpc]

when i set 67108864 or more elements, the hashtalbe crash.

(gdb) p *ht
$5 = {nelems = 67108864, table_size = 134217728, elem_size = 16, mask = 134217727, kdef = 0x7ffff7ff6ea0, vdef = 0x7ffff7ff68e0,
table = 0x7ffe68e92020 ""}

my init:
reph = ph_ht_init(&(mth->ht), 10000,
&ph_ht_string_key_def, &ph_ht_uint32_val_def);

[wez]

Can you provide a bt full? Curious about the value of elem.

Could you reduce this problem to a test case? For example, can you modify tests/hashtable.c so that it reproduces this problem and then paste your modified version of that file?

[maczpc]

my modified in tests/hashtable.c :

// if (howmany > 1000) {
// need to fix the format to allow for this many
// abort();
// }

load_data(&ht, 100000000);

Core was generated by `/data1/hadoop/pengcheng/libphenom/tests/.libs/lt-hashtable.t'.
Program terminated with signal 11, Segmentation fault.
#0 find_new_slot (ht=0x7fffe1db7c30, size=268435456) at corelib/hash/table.c:187

warning: Source file is more recent than executable.
187 if (elem->status == PH_HT_ELEM_EMPTY) {
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.el6.x86_64 keyutils-libs-1.4-3.el6.x86_64 krb5-libs-1.9-22.el6.x86_64 libcom_err-1.41.12-11.el6.x86_64 libselinux-2.0.94-5.2.el6.x86_64 openssl-1.0.0-20.el6.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) bt
#0 find_new_slot (ht=0x7fffe1db7c30, size=268435456) at corelib/hash/table.c:187
#1 rebuild_table (ht=0x7fffe1db7c30, size=268435456) at corelib/hash/table.c:212
#2 0x00007f0b1a40fd3c in ph_ht_insert (ht=0x7fffe1db7c30, key=0x7f0b0a5b2010, value=0x7f0b0a5b2010, flags=0)
at corelib/hash/table.c:245
#3 0x0000000000403bbc in load_data (ht=0x7fffe1db7c30, howmany=100000000) at tests/hashtable.c:33
#4 0x0000000000404e73 in main (argc=1, argv=0x7fffe1db7d58) at tests/hashtable.c:147
(gdb) p elem
$1 = (struct ph_ht_elem *) 0x7f0b4a5b1034
(gdb) p *elem
Cannot access memory at address 0x7f0b4a5b1034
(gdb) p *ht
$2 = {nelems = 67108864, table_size = 134217728, elem_size = 20, mask = 134217727, kdef = 0x7f0b1a61e7a0, vdef = 0x7f0b1a61e7d0,
table = 0x7f09aa6e6020 ""}
(gdb) quit

Thanks very much!

[maczpc]

this bt full

Program terminated with signal 11, Segmentation fault.
#0 find_new_slot (ht=0x7fffe1db7c30, size=268435456) at corelib/hash/table.c:187

warning: Source file is more recent than executable.
187 if (elem->status == PH_HT_ELEM_EMPTY) {
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.el6.x86_64 keyutils-libs-1.4-3.el6.x86_64 krb5-libs-1.9-22.el6.x86_64 libcom_err-1.41.12-11.el6.x86_64 libselinux-2.0.94-5.2.el6.x86_64 openssl-1.0.0-20.el6.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) bt full
#0 find_new_slot (ht=0x7fffe1db7c30, size=268435456) at corelib/hash/table.c:187
pos = 134217729
elem = 0x7f0b4a5b1034
#1 rebuild_table (ht=0x7fffe1db7c30, size=268435456) at corelib/hash/table.c:212
src =
dest =
table = 0x7f0aaa5b1020 ""
mask = 268435455
i =
done =
#2 0x00007f0b1a40fd3c in ph_ht_insert (ht=0x7fffe1db7c30, key=0x7f0b0a5b2010, value=0x7f0b0a5b2010, flags=0)
at corelib/hash/table.c:245
elem = 0x7f0a1b9e5408
#3 0x0000000000403bbc in load_data (ht=0x7fffe1db7c30, howmany=100000000) at tests/hashtable.c:33
data = 0x7f0aea5b2010
i = 67108864
oiter = {slot = 3789257648, size = 32767, slots = 0x7f0b1a3f7f09 "H\205\300t\004\360\377@\004\311\303fff.\017\037\204"}
func = "load_data"
#4 0x0000000000404e73 in main (argc=1, argv=0x7fffe1db7d58) at tests/hashtable.c:147
ht = {nelems = 67108864, table_size = 134217728, elem_size = 20, mask = 134217727, kdef = 0x7f0b1a61e7a0,
vdef = 0x7f0b1a61e7d0, table = 0x7f09aa6e6020 ""}
a = 0x2208270
b = 0x2208180
oiter = {slot = 2, size = 2, slots = 0x0}
iter = {slot = 32768, size = 32768}
func = "main"
(gdb)

[wez]

Resolved by the attached commit; thanks for reporting this!

Also note: the hash table will work more efficiently for you if you call ph_ht_init and tell it that you have such a large number of items. You're hinting at 10k items but are really storing a massive amount; what happens at runtime is that each time the table starts to get full, it will be rebuilt at twice the size and rehashed. If there are many powers-of-2 between 10k and the end number of elements then it will rebuild and rehash that many times.

If you don't know the final size at the time that you call ph_ht_init(), but find out a bit later on, you can call ph_ht_grow (http://facebook.github.io/libphenom/#hashtable--ph_ht_grow) to provide a new hint and have the table resize just once and then your inserts will take less time compared to not providing this hint.

[maczpc]

You're welcome. Thank you for your advice.
Is libphenom used in production environment ?

[dhobsd]

Yes

[lvshaco]

来信收到，谢谢
来自新浪邮箱手机网页版