Page allocation failure #17
Comments
I'm curious how big that stream was and what network protocols were used?
This must be the wifi IPC driver trying to allocate skb for the packet coming from wifi. The choice at this point is either allocate skb dynamically as packets arrive or use a preallocated pool of skbs. I have implemented the former and this kind of error is the expected consequence. The latter wouldn't have this issue, but since there's no way to tell the firmware to pause reception it would lose packets in this situation instead. Also the skbs taken from the pool need to be returned back into it, and that means changes in the esp-hosted driver. |
jcmvbkbc
pushed a commit
that referenced
this issue
Apr 2, 2024
An errant disk backup on my desktop got into debugfs and triggered the following deadlock scenario in the amdgpu debugfs files. The machine also hard-resets immediately after those lines are printed (although I wasn't able to reproduce that part when reading by hand): [ 1318.016074][ T1082] ====================================================== [ 1318.016607][ T1082] WARNING: possible circular locking dependency detected [ 1318.017107][ T1082] 6.8.0-rc7-00015-ge0c8221b72c0 #17 Not tainted [ 1318.017598][ T1082] ------------------------------------------------------ [ 1318.018096][ T1082] tar/1082 is trying to acquire lock: [ 1318.018585][ T1082] ffff98c44175d6a0 (&mm->mmap_lock){++++}-{3:3}, at: __might_fault+0x40/0x80 [ 1318.019084][ T1082] [ 1318.019084][ T1082] but task is already holding lock: [ 1318.020052][ T1082] ffff98c4c13f55f8 (reservation_ww_class_mutex){+.+.}-{3:3}, at: amdgpu_debugfs_mqd_read+0x6a/0x250 [amdgpu] [ 1318.020607][ T1082] [ 1318.020607][ T1082] which lock already depends on the new lock. [ 1318.020607][ T1082] [ 1318.022081][ T1082] [ 1318.022081][ T1082] the existing dependency chain (in reverse order) is: [ 1318.023083][ T1082] [ 1318.023083][ T1082] -> #2 (reservation_ww_class_mutex){+.+.}-{3:3}: [ 1318.024114][ T1082] __ww_mutex_lock.constprop.0+0xe0/0x12f0 [ 1318.024639][ T1082] ww_mutex_lock+0x32/0x90 [ 1318.025161][ T1082] dma_resv_lockdep+0x18a/0x330 [ 1318.025683][ T1082] do_one_initcall+0x6a/0x350 [ 1318.026210][ T1082] kernel_init_freeable+0x1a3/0x310 [ 1318.026728][ T1082] kernel_init+0x15/0x1a0 [ 1318.027242][ T1082] ret_from_fork+0x2c/0x40 [ 1318.027759][ T1082] ret_from_fork_asm+0x11/0x20 [ 1318.028281][ T1082] [ 1318.028281][ T1082] -> #1 (reservation_ww_class_acquire){+.+.}-{0:0}: [ 1318.029297][ T1082] dma_resv_lockdep+0x16c/0x330 [ 1318.029790][ T1082] do_one_initcall+0x6a/0x350 [ 1318.030263][ T1082] kernel_init_freeable+0x1a3/0x310 [ 1318.030722][ T1082] kernel_init+0x15/0x1a0 [ 1318.031168][ T1082] ret_from_fork+0x2c/0x40 [ 1318.031598][ T1082] ret_from_fork_asm+0x11/0x20 [ 1318.032011][ T1082] [ 1318.032011][ T1082] -> #0 (&mm->mmap_lock){++++}-{3:3}: [ 1318.032778][ T1082] __lock_acquire+0x14bf/0x2680 [ 1318.033141][ T1082] lock_acquire+0xcd/0x2c0 [ 1318.033487][ T1082] __might_fault+0x58/0x80 [ 1318.033814][ T1082] amdgpu_debugfs_mqd_read+0x103/0x250 [amdgpu] [ 1318.034181][ T1082] full_proxy_read+0x55/0x80 [ 1318.034487][ T1082] vfs_read+0xa7/0x360 [ 1318.034788][ T1082] ksys_read+0x70/0xf0 [ 1318.035085][ T1082] do_syscall_64+0x94/0x180 [ 1318.035375][ T1082] entry_SYSCALL_64_after_hwframe+0x46/0x4e [ 1318.035664][ T1082] [ 1318.035664][ T1082] other info that might help us debug this: [ 1318.035664][ T1082] [ 1318.036487][ T1082] Chain exists of: [ 1318.036487][ T1082] &mm->mmap_lock --> reservation_ww_class_acquire --> reservation_ww_class_mutex [ 1318.036487][ T1082] [ 1318.037310][ T1082] Possible unsafe locking scenario: [ 1318.037310][ T1082] [ 1318.037838][ T1082] CPU0 CPU1 [ 1318.038101][ T1082] ---- ---- [ 1318.038350][ T1082] lock(reservation_ww_class_mutex); [ 1318.038590][ T1082] lock(reservation_ww_class_acquire); [ 1318.038839][ T1082] lock(reservation_ww_class_mutex); [ 1318.039083][ T1082] rlock(&mm->mmap_lock); [ 1318.039328][ T1082] [ 1318.039328][ T1082] *** DEADLOCK *** [ 1318.039328][ T1082] [ 1318.040029][ T1082] 1 lock held by tar/1082: [ 1318.040259][ T1082] #0: ffff98c4c13f55f8 (reservation_ww_class_mutex){+.+.}-{3:3}, at: amdgpu_debugfs_mqd_read+0x6a/0x250 [amdgpu] [ 1318.040560][ T1082] [ 1318.040560][ T1082] stack backtrace: [ 1318.041053][ T1082] CPU: 22 PID: 1082 Comm: tar Not tainted 6.8.0-rc7-00015-ge0c8221b72c0 #17 3316c85d50e282c5643b075d1f01a4f6365e39c2 [ 1318.041329][ T1082] Hardware name: Gigabyte Technology Co., Ltd. B650 AORUS PRO AX/B650 AORUS PRO AX, BIOS F20 12/14/2023 [ 1318.041614][ T1082] Call Trace: [ 1318.041895][ T1082] <TASK> [ 1318.042175][ T1082] dump_stack_lvl+0x4a/0x80 [ 1318.042460][ T1082] check_noncircular+0x145/0x160 [ 1318.042743][ T1082] __lock_acquire+0x14bf/0x2680 [ 1318.043022][ T1082] lock_acquire+0xcd/0x2c0 [ 1318.043301][ T1082] ? __might_fault+0x40/0x80 [ 1318.043580][ T1082] ? __might_fault+0x40/0x80 [ 1318.043856][ T1082] __might_fault+0x58/0x80 [ 1318.044131][ T1082] ? __might_fault+0x40/0x80 [ 1318.044408][ T1082] amdgpu_debugfs_mqd_read+0x103/0x250 [amdgpu 8fe2afaa910cbd7654c8cab23563a94d6caebaab] [ 1318.044749][ T1082] full_proxy_read+0x55/0x80 [ 1318.045042][ T1082] vfs_read+0xa7/0x360 [ 1318.045333][ T1082] ksys_read+0x70/0xf0 [ 1318.045623][ T1082] do_syscall_64+0x94/0x180 [ 1318.045913][ T1082] ? do_syscall_64+0xa0/0x180 [ 1318.046201][ T1082] ? lockdep_hardirqs_on+0x7d/0x100 [ 1318.046487][ T1082] ? do_syscall_64+0xa0/0x180 [ 1318.046773][ T1082] ? do_syscall_64+0xa0/0x180 [ 1318.047057][ T1082] ? do_syscall_64+0xa0/0x180 [ 1318.047337][ T1082] ? do_syscall_64+0xa0/0x180 [ 1318.047611][ T1082] entry_SYSCALL_64_after_hwframe+0x46/0x4e [ 1318.047887][ T1082] RIP: 0033:0x7f480b70a39d [ 1318.048162][ T1082] Code: 91 ba 0d 00 f7 d8 64 89 02 b8 ff ff ff ff eb b2 e8 18 a3 01 00 0f 1f 84 00 00 00 00 00 80 3d a9 3c 0e 00 00 74 17 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 5b c3 66 2e 0f 1f 84 00 00 00 00 00 53 48 83 [ 1318.048769][ T1082] RSP: 002b:00007ffde77f5c68 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 1318.049083][ T1082] RAX: ffffffffffffffda RBX: 0000000000000800 RCX: 00007f480b70a39d [ 1318.049392][ T1082] RDX: 0000000000000800 RSI: 000055c9f2120c00 RDI: 0000000000000008 [ 1318.049703][ T1082] RBP: 0000000000000800 R08: 000055c9f2120a94 R09: 0000000000000007 [ 1318.050011][ T1082] R10: 0000000000000000 R11: 0000000000000246 R12: 000055c9f2120c00 [ 1318.050324][ T1082] R13: 0000000000000008 R14: 0000000000000008 R15: 0000000000000800 [ 1318.050638][ T1082] </TASK> amdgpu_debugfs_mqd_read() holds a reservation when it calls put_user(), which may fault and acquire the mmap_sem. This violates the established locking order. Bounce the mqd data through a kernel buffer to get put_user() out of the illegal section. Fixes: 445d85e ("drm/amdgpu: add debugfs interface for reading MQDs") Cc: [email protected] # v6.5+ Reviewed-by: Shashank Sharma <[email protected]> Signed-off-by: Johannes Weiner <[email protected]> Signed-off-by: Alex Deucher <[email protected]>
|
I tested the commit 8678b10 reference this issue. But, the page error still exist. The data size for a day is around 430MB per day. Will new tag xtensa-20240502 solve this issue? |
It is entirely unrelated to this issue. github is linking it to this issue because the commit description has the text
I don't think so. |
jcmvbkbc
pushed a commit
that referenced
this issue
Jul 20, 2024
The code in ocfs2_dio_end_io_write() estimates number of necessary transaction credits using ocfs2_calc_extend_credits(). This however does not take into account that the IO could be arbitrarily large and can contain arbitrary number of extents. Extent tree manipulations do often extend the current transaction but not in all of the cases. For example if we have only single block extents in the tree, ocfs2_mark_extent_written() will end up calling ocfs2_replace_extent_rec() all the time and we will never extend the current transaction and eventually exhaust all the transaction credits if the IO contains many single block extents. Once that happens a WARN_ON(jbd2_handle_buffer_credits(handle) <= 0) is triggered in jbd2_journal_dirty_metadata() and subsequently OCFS2 aborts in response to this error. This was actually triggered by one of our customers on a heavily fragmented OCFS2 filesystem. To fix the issue make sure the transaction always has enough credits for one extent insert before each call of ocfs2_mark_extent_written(). Heming Zhao said: ------ PANIC: "Kernel panic - not syncing: OCFS2: (device dm-1): panic forced after error" PID: xxx TASK: xxxx CPU: 5 COMMAND: "SubmitThread-CA" #0 machine_kexec at ffffffff8c069932 #1 __crash_kexec at ffffffff8c1338fa #2 panic at ffffffff8c1d69b9 #3 ocfs2_handle_error at ffffffffc0c86c0c [ocfs2] #4 __ocfs2_abort at ffffffffc0c88387 [ocfs2] #5 ocfs2_journal_dirty at ffffffffc0c51e98 [ocfs2] #6 ocfs2_split_extent at ffffffffc0c27ea3 [ocfs2] #7 ocfs2_change_extent_flag at ffffffffc0c28053 [ocfs2] #8 ocfs2_mark_extent_written at ffffffffc0c28347 [ocfs2] #9 ocfs2_dio_end_io_write at ffffffffc0c2bef9 [ocfs2] #10 ocfs2_dio_end_io at ffffffffc0c2c0f5 [ocfs2] #11 dio_complete at ffffffff8c2b9fa7 #12 do_blockdev_direct_IO at ffffffff8c2bc09f #13 ocfs2_direct_IO at ffffffffc0c2b653 [ocfs2] #14 generic_file_direct_write at ffffffff8c1dcf14 #15 __generic_file_write_iter at ffffffff8c1dd07b #16 ocfs2_file_write_iter at ffffffffc0c49f1f [ocfs2] #17 aio_write at ffffffff8c2cc72e #18 kmem_cache_alloc at ffffffff8c248dde #19 do_io_submit at ffffffff8c2ccada #20 do_syscall_64 at ffffffff8c004984 #21 entry_SYSCALL_64_after_hwframe at ffffffff8c8000ba Link: https://lkml.kernel.org/r/[email protected] Link: https://lkml.kernel.org/r/[email protected] Fixes: c15471f ("ocfs2: fix sparse file & data ordering issue in direct io") Signed-off-by: Jan Kara <[email protected]> Reviewed-by: Joseph Qi <[email protected]> Reviewed-by: Heming Zhao <[email protected]> Cc: Mark Fasheh <[email protected]> Cc: Joel Becker <[email protected]> Cc: Junxiao Bi <[email protected]> Cc: Changwei Ge <[email protected]> Cc: Gang He <[email protected]> Cc: Jun Piao <[email protected]> Cc: <[email protected]> Signed-off-by: Andrew Morton <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
After running the Linux around a day, there is a page allocation failure as below. During this period, I am forwarding a stream from UART to a remote server using nc. Please help me to fix it and my application needs to run 24/7.
E (86428606) wifi:null wpa_sta_ap_set_rsnxe
E (86437956) wifi:null wpa_sta_ap_set_rsnxe
[86554.227453] irq/5-600c0004.: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null)
[86554.228646] CPU: 0 PID: 29 Comm: irq/5-600c0004. Not tainted 6.8.0 #1
[86554.233113] Stack:
[86554.235243] > 00000100 00000000 3dbfbc00 423b8f69
[86554.239881] 423b8f69 3d87d11b 3dbfc000 3d8850a4
[86554.244551] 8217cfcf 3dbfbc50 00000000 3dbfc350
[86554.249458] 3dbfbc30 00000004 00000830 00060f00
[86554.254156] 8217d4c9 3dbfbc70 00000820 00000000
[86554.258700] > 3dbfbc8c 00000820 3dbfbca0 00000000
[86554.263303] 8217d525 3dbfbcf0 00000000 00000000
[86554.267990] > 00000000 000007c8 3d9574a0 00000110
[86554.272683] 3dbfbcd0 3dbfbcb0 0000000c 423b88b0
[86554.277369] 3dbfbc80 3d87d698 00000820 3d87d448
[86554.282043] 00000820 3dbfbcd0 3d86c0fc 00000000
[86554.286736] 3dbfbe90 3dbfbe50 00000000 00000000
[86554.291421] 3d87d698 3dbfa000 00000000 3d8c2a60
[86554.296094] 00000001 00000000 00000820 00000000
[86554.300809] 8217d56d 3dbfbd70 00000000 00000820
[86554.305482] > 0000000f 3d8c0002 3dbfc000 3d870204
[86554.310195] Call Trace:
[86554.312552] [<42372119>] 0x42372119
[86554.316149] [<4217cfcf>] 0x4217cfcf
[86554.319680] [<4217d4c9>] 0x4217d4c9
[86554.323216] [<4217d525>] 0x4217d525
[86554.326772] [<4217d56d>] 0x4217d56d
[86554.330325] [<4227be0d>] 0x4227be0d
[86554.333877] [<42268c02>] 0x42268c02
[86554.337449] [<4226b0b1>] 0x4226b0b1
[86554.341152] [<4225d3cc>] 0x4225d3cc
[86554.344572] [<42144d48>] 0x42144d48
[86554.348140] [<4213504e>] 0x4213504e
[86554.351673] [<421209b4>] 0x421209b4
[86554.356342] Mem-Info:
[86554.357545] active_anon:0 inactive_anon:0 isolated_anon:0
[86554.357545] active_file:1 inactive_file:1 isolated_file:0
[86554.357545] unevictable:15 dirty:0 writeback:0
[86554.357545] slab_reclaimable:178 slab_unreclaimable:1105
[86554.357545] mapped:0 shmem:0 pagetables:0
[86554.357545] sec_pagetables:0 bounce:0
[86554.357545] kernel_misc_reclaimable:0
[86554.357545] free:33 free_pcp:0 free_cma:0
[86554.394913] Node 0 active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:60kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:336kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? yes
[86554.419561] Normal free:132kB boost:0kB min:340kB low:424kB high:508kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:60kB writepending:0kB present:8192kB managed:7328kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[86554.445055] lowmem_reserve[]: 0 0
[86554.447426] Normal: 334kB (U) 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 132kB
[86554.459128] 17 total pagecache pages
[86554.461577] 2048 pages RAM
[86554.464143] 0 pages HighMem/MovableOnly
[86554.468877] 216 pages reserved
[86554.490664] syslogd invoked oom-killer: gfp_mask=0x140cc0(GFP_USER|__GFP_COMP), order=0, oom_score_adj=0
[86554.491709] CPU: 0 PID: 52 Comm: syslogd Not tainted 6.8.0 #1
[86554.495628] Stack:
[86554.497776] > 00000100 00000000 3de8f9d0 423b8f69
[86554.502400] 423b8f69 3d87d11b 3dbff180 3d8850a4
[86554.507072] 8236fe74 3de8fa20 0000005c 3dbff4d0
[86554.511750] 3de8fa00 00000004 00000000 00060800
[86554.516445] 8216e6a7 3de8fa40 3de8fb10 4237b088
[86554.521121] > 00140cc0 3de8fb1c 00000000 00000000
[86554.525824] 8216ea7e 3de8fa90 3de8fb10 423b62c8
[86554.530504] > 3de8fa3c 00000001 3de8e000 3d88551c
[86554.535186] 3d87d70c 00000000 00000001 00000001
[86554.539870] 00000000 00000020 3de8e000 00000003
[86554.544565] 0000000a 00060200 3dee6400 3d87024c
[86554.549256] 8217d391 3de8fae0 3de8fb10 7fffffff
[86554.553925] > 00000020 00000000 00000000 00000000
[86554.558621] 00000000 ff001078 00000000 00140cc0
[86554.563313] 00000000 3d87d698 001408c0 3d87d448
[86554.568000] 3de8fae0 3d8816c0 3dee6400 00000000
[86554.572611] Call Trace:
[86554.575054] [<42372119>] 0x42372119
[86554.578612] [<4236fe74>] 0x4236fe74
[86554.582319] [<4216e6a7>] 0x4216e6a7
[86554.585733] [<4216ea7e>] 0x4216ea7e
[86554.589309] [<4217d391>] 0x4217d391
[86554.592839] [<4217d8cc>] 0x4217d8cc
[86554.596400] [<4216b88d>] 0x4216b88d
[86554.599966] [<4216d2e4>] 0x4216d2e4
[86554.603515] [<4216d3a9>] 0x4216d3a9
[86554.607104] [<421897f7>] 0x421897f7
[86554.610624] [<42189f65>] 0x42189f65
[86554.614194] [<4218a222>] 0x4218a222
[86554.617788] [<4218a447>] 0x4218a447
[86554.621300] [<4218ae38>] 0x4218ae38
[86554.624860] [<4218b419>] 0x4218b419
[86554.628445] [<421810e7>] 0x421810e7
[86554.631978] [<42181172>] 0x42181172
[86554.635535] [<42181188>] 0x42181188
[86554.639131] [<421208e0>] 0x421208e0
[86554.642671] [<4212066a>] 0x4212066a
[86554.678594] Mem-Info:
[86554.679287] active_anon:0 inactive_anon:0 isolated_anon:0
[86554.679287] active_file:0 inactive_file:1 isolated_file:0
[86554.679287] unevictable:15 dirty:0 writeback:0
[86554.679287] slab_reclaimable:178 slab_unreclaimable:1106
[86554.679287] mapped:0 shmem:0 pagetables:0
[86554.679287] sec_pagetables:0 bounce:0
[86554.679287] kernel_misc_reclaimable:0
[86554.679287] free:33 free_pcp:0 free_cma:0
[86554.710744] Node 0 active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:60kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:336kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no
[86554.743198] Normal free:136kB boost:0kB min:340kB low:424kB high:508kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:60kB writepending:0kB present:8192kB managed:7328kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
The text was updated successfully, but these errors were encountered: