From 6ac061db9c58ca5b9270b1b3940d2464fb3ff183 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:30 +0000 Subject: binder: use EPOLLERR from eventpoll.h Use EPOLLERR instead of POLLERR to make sure it is cast to the correct __poll_t type. This fixes the following sparse issue: drivers/android/binder.c:5030:24: warning: incorrect type in return expression (different base types) drivers/android/binder.c:5030:24: expected restricted __poll_t drivers/android/binder.c:5030:24: got int Fixes: f88982679f54 ("binder: check for binder_thread allocation failure in binder_poll()") Cc: stable@vger.kernel.org Cc: Eric Biggers Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-2-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'drivers/android') diff --git a/drivers/android/binder.c b/drivers/android/binder.c index 92128aae2d06..71a40a4c546f 100644 --- a/drivers/android/binder.c +++ b/drivers/android/binder.c @@ -5030,7 +5030,7 @@ static __poll_t binder_poll(struct file *filp, thread = binder_get_thread(proc); if (!thread) - return POLLERR; + return EPOLLERR; binder_inner_proc_lock(thread->proc); thread->looper |= BINDER_LOOPER_STATE_POLL; -- cgit From 3f489c2067c5824528212b0fc18b28d51332d906 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:31 +0000 Subject: binder: fix use-after-free in shinker's callback The mmap read lock is used during the shrinker's callback, which means that using alloc->vma pointer isn't safe as it can race with munmap(). As of commit dd2283f2605e ("mm: mmap: zap pages with read mmap_sem in munmap") the mmap lock is downgraded after the vma has been isolated. I was able to reproduce this issue by manually adding some delays and triggering page reclaiming through the shrinker's debug sysfs. The following KASAN report confirms the UAF: ================================================================== BUG: KASAN: slab-use-after-free in zap_page_range_single+0x470/0x4b8 Read of size 8 at addr ffff356ed50e50f0 by task bash/478 CPU: 1 PID: 478 Comm: bash Not tainted 6.6.0-rc5-00055-g1c8b86a3799f-dirty #70 Hardware name: linux,dummy-virt (DT) Call trace: zap_page_range_single+0x470/0x4b8 binder_alloc_free_page+0x608/0xadc __list_lru_walk_one+0x130/0x3b0 list_lru_walk_node+0xc4/0x22c binder_shrink_scan+0x108/0x1dc shrinker_debugfs_scan_write+0x2b4/0x500 full_proxy_write+0xd4/0x140 vfs_write+0x1ac/0x758 ksys_write+0xf0/0x1dc __arm64_sys_write+0x6c/0x9c Allocated by task 492: kmem_cache_alloc+0x130/0x368 vm_area_alloc+0x2c/0x190 mmap_region+0x258/0x18bc do_mmap+0x694/0xa60 vm_mmap_pgoff+0x170/0x29c ksys_mmap_pgoff+0x290/0x3a0 __arm64_sys_mmap+0xcc/0x144 Freed by task 491: kmem_cache_free+0x17c/0x3c8 vm_area_free_rcu_cb+0x74/0x98 rcu_core+0xa38/0x26d4 rcu_core_si+0x10/0x1c __do_softirq+0x2fc/0xd24 Last potentially related work creation: __call_rcu_common.constprop.0+0x6c/0xba0 call_rcu+0x10/0x1c vm_area_free+0x18/0x24 remove_vma+0xe4/0x118 do_vmi_align_munmap.isra.0+0x718/0xb5c do_vmi_munmap+0xdc/0x1fc __vm_munmap+0x10c/0x278 __arm64_sys_munmap+0x58/0x7c Fix this issue by performing instead a vma_lookup() which will fail to find the vma that was isolated before the mmap lock downgrade. Note that this option has better performance than upgrading to a mmap write lock which would increase contention. Plus, mmap_write_trylock() has been recently removed anyway. Fixes: dd2283f2605e ("mm: mmap: zap pages with read mmap_sem in munmap") Cc: stable@vger.kernel.org Cc: Liam Howlett Cc: Minchan Kim Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-3-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 138f6d43d13b..9d2eff70c3ba 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -1005,7 +1005,9 @@ enum lru_status binder_alloc_free_page(struct list_head *item, goto err_mmget; if (!mmap_read_trylock(mm)) goto err_mmap_read_lock_failed; - vma = binder_alloc_get_vma(alloc); + vma = vma_lookup(mm, page_addr); + if (vma && vma != binder_alloc_get_vma(alloc)) + goto err_invalid_vma; list_lru_isolate(lru, item); spin_unlock(lock); @@ -1031,6 +1033,8 @@ enum lru_status binder_alloc_free_page(struct list_head *item, mutex_unlock(&alloc->mutex); return LRU_REMOVED_RETRY; +err_invalid_vma: + mmap_read_unlock(mm); err_mmap_read_lock_failed: mmput_async(mm); err_mmget: -- cgit From 9a9ab0d963621d9d12199df9817e66982582d5a5 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:32 +0000 Subject: binder: fix race between mmput() and do_exit() Task A calls binder_update_page_range() to allocate and insert pages on a remote address space from Task B. For this, Task A pins the remote mm via mmget_not_zero() first. This can race with Task B do_exit() and the final mmput() refcount decrement will come from Task A. Task A | Task B ------------------+------------------ mmget_not_zero() | | do_exit() | exit_mm() | mmput() mmput() | exit_mmap() | remove_vma() | fput() | In this case, the work of ____fput() from Task B is queued up in Task A as TWA_RESUME. So in theory, Task A returns to userspace and the cleanup work gets executed. However, Task A instead sleep, waiting for a reply from Task B that never comes (it's dead). This means the binder_deferred_release() is blocked until an unrelated binder event forces Task A to go back to userspace. All the associated death notifications will also be delayed until then. In order to fix this use mmput_async() that will schedule the work in the corresponding mm->async_put_work WQ instead of Task A. Fixes: 457b9a6f09f0 ("Staging: android: add binder driver") Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-4-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 9d2eff70c3ba..adcec5ec0959 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -271,7 +271,7 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate, } if (mm) { mmap_write_unlock(mm); - mmput(mm); + mmput_async(mm); } return 0; @@ -304,7 +304,7 @@ err_page_ptr_cleared: err_no_vma: if (mm) { mmap_write_unlock(mm); - mmput(mm); + mmput_async(mm); } return vma ? -ENOMEM : -ESRCH; } -- cgit From 3091c21d3e9322428691ce0b7a0cfa9c0b239eeb Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:33 +0000 Subject: binder: fix async space check for 0-sized buffers Move the padding of 0-sized buffers to an earlier stage to account for this round up during the alloc->free_async_space check. Fixes: 74310e06be4d ("android: binder: Move buffer out of area shared with user space") Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-5-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index adcec5ec0959..abff1bafcc43 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -407,6 +407,10 @@ static struct binder_buffer *binder_alloc_new_buf_locked( alloc->pid, extra_buffers_size); return ERR_PTR(-EINVAL); } + + /* Pad 0-size buffers so they get assigned unique addresses */ + size = max(size, sizeof(void *)); + if (is_async && alloc->free_async_space < size + sizeof(struct binder_buffer)) { binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, @@ -415,9 +419,6 @@ static struct binder_buffer *binder_alloc_new_buf_locked( return ERR_PTR(-ENOSPC); } - /* Pad 0-size buffers so they get assigned unique addresses */ - size = max(size, sizeof(void *)); - while (n) { buffer = rb_entry(n, struct binder_buffer, rb_node); BUG_ON(!buffer->free); -- cgit From c6d05e0762ab276102246d24affd1e116a46aa0c Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:34 +0000 Subject: binder: fix unused alloc->free_async_space Each transaction is associated with a 'struct binder_buffer' that stores the metadata about its buffer area. Since commit 74310e06be4d ("android: binder: Move buffer out of area shared with user space") this struct is no longer embedded within the buffer itself but is instead allocated on the heap to prevent userspace access to this driver-exclusive info. Unfortunately, the space of this struct is still being accounted for in the total buffer size calculation, specifically for async transactions. This results in an additional 104 bytes added to every async buffer request, and this area is never used. This wasted space can be substantial. If we consider the maximum mmap buffer space of SZ_4M, the driver will reserve half of it for async transactions, or 0x200000. This area should, in theory, accommodate up to 262,144 buffers of the minimum 8-byte size. However, after adding the extra 'sizeof(struct binder_buffer)', the total number of buffers drops to only 18,724, which is a sad 7.14% of the actual capacity. This patch fixes the buffer size calculation to enable the utilization of the entire async buffer space. This is expected to reduce the number of -ENOSPC errors that are seen on the field. Fixes: 74310e06be4d ("android: binder: Move buffer out of area shared with user space") Signed-off-by: Carlos Llamas Reviewed-by: Alice Ryhl Link: https://lore.kernel.org/r/20231201172212.1813387-6-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index abff1bafcc43..9b5c4d446efa 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -344,8 +344,7 @@ static bool debug_low_async_space_locked(struct binder_alloc *alloc, int pid) continue; if (!buffer->async_transaction) continue; - total_alloc_size += binder_alloc_buffer_size(alloc, buffer) - + sizeof(struct binder_buffer); + total_alloc_size += binder_alloc_buffer_size(alloc, buffer); num_buffers++; } @@ -411,8 +410,7 @@ static struct binder_buffer *binder_alloc_new_buf_locked( /* Pad 0-size buffers so they get assigned unique addresses */ size = max(size, sizeof(void *)); - if (is_async && - alloc->free_async_space < size + sizeof(struct binder_buffer)) { + if (is_async && alloc->free_async_space < size) { binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, "%d: binder_alloc_buf size %zd failed, no async space left\n", alloc->pid, size); @@ -520,7 +518,7 @@ static struct binder_buffer *binder_alloc_new_buf_locked( buffer->pid = pid; buffer->oneway_spam_suspect = false; if (is_async) { - alloc->free_async_space -= size + sizeof(struct binder_buffer); + alloc->free_async_space -= size; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC_ASYNC, "%d: binder_alloc_buf size %zd async free %zd\n", alloc->pid, size, alloc->free_async_space); @@ -658,8 +656,7 @@ static void binder_free_buf_locked(struct binder_alloc *alloc, BUG_ON(buffer->user_data > alloc->buffer + alloc->buffer_size); if (buffer->async_transaction) { - alloc->free_async_space += buffer_size + sizeof(struct binder_buffer); - + alloc->free_async_space += buffer_size; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC_ASYNC, "%d: binder_free_buf size %zd async free %zd\n", alloc->pid, size, alloc->free_async_space); -- cgit From 122a3c1cb0ff304c2b8934584fcfea4edb2fe5e3 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:35 +0000 Subject: binder: fix trivial typo of binder_free_buf_locked() Fix minor misspelling of the function in the comment section. No functional changes in this patch. Cc: stable@vger.kernel.org Fixes: 0f966cba95c7 ("binder: add flag to clear buffer on txn complete") Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-7-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 9b5c4d446efa..a124d2743c69 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -704,7 +704,7 @@ void binder_alloc_free_buf(struct binder_alloc *alloc, /* * We could eliminate the call to binder_alloc_clear_buf() * from binder_alloc_deferred_release() by moving this to - * binder_alloc_free_buf_locked(). However, that could + * binder_free_buf_locked(). However, that could * increase contention for the alloc mutex if clear_on_free * is used frequently for large buffers. The mutex is not * needed for correctness here. -- cgit From e1090371e02b601cbfcea175c2a6cc7c955fa830 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:36 +0000 Subject: binder: fix comment on binder_alloc_new_buf() return value Update the comments of binder_alloc_new_buf() to reflect that the return value of the function is now ERR_PTR(-errno) on failure. No functional changes in this patch. Cc: stable@vger.kernel.org Fixes: 57ada2fb2250 ("binder: add log information for binder transaction failures") Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-8-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index a124d2743c69..a56cbfd9ba44 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -556,7 +556,7 @@ err_alloc_buf_struct_failed: * is the sum of the three given sizes (each rounded up to * pointer-sized boundary) * - * Return: The allocated buffer or %NULL if error + * Return: The allocated buffer or %ERR_PTR(-errno) if error */ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, size_t data_size, -- cgit From da483f8b390546fbe36abd72f58d612a8032e2a8 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:37 +0000 Subject: binder: remove extern from function prototypes The kernel coding style does not require 'extern' in function prototypes in .h files, so remove them from drivers/android/binder_alloc.h as they are not needed. No functional changes in this patch. Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-9-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.h | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.h b/drivers/android/binder_alloc.h index dc1e2b01dd64..82380febdd85 100644 --- a/drivers/android/binder_alloc.h +++ b/drivers/android/binder_alloc.h @@ -121,27 +121,27 @@ static inline void binder_selftest_alloc(struct binder_alloc *alloc) {} enum lru_status binder_alloc_free_page(struct list_head *item, struct list_lru_one *lru, spinlock_t *lock, void *cb_arg); -extern struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, - size_t data_size, - size_t offsets_size, - size_t extra_buffers_size, - int is_async, - int pid); -extern void binder_alloc_init(struct binder_alloc *alloc); -extern int binder_alloc_shrinker_init(void); -extern void binder_alloc_shrinker_exit(void); -extern void binder_alloc_vma_close(struct binder_alloc *alloc); -extern struct binder_buffer * +struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, + size_t data_size, + size_t offsets_size, + size_t extra_buffers_size, + int is_async, + int pid); +void binder_alloc_init(struct binder_alloc *alloc); +int binder_alloc_shrinker_init(void); +void binder_alloc_shrinker_exit(void); +void binder_alloc_vma_close(struct binder_alloc *alloc); +struct binder_buffer * binder_alloc_prepare_to_free(struct binder_alloc *alloc, uintptr_t user_ptr); -extern void binder_alloc_free_buf(struct binder_alloc *alloc, - struct binder_buffer *buffer); -extern int binder_alloc_mmap_handler(struct binder_alloc *alloc, - struct vm_area_struct *vma); -extern void binder_alloc_deferred_release(struct binder_alloc *alloc); -extern int binder_alloc_get_allocated_count(struct binder_alloc *alloc); -extern void binder_alloc_print_allocated(struct seq_file *m, - struct binder_alloc *alloc); +void binder_alloc_free_buf(struct binder_alloc *alloc, + struct binder_buffer *buffer); +int binder_alloc_mmap_handler(struct binder_alloc *alloc, + struct vm_area_struct *vma); +void binder_alloc_deferred_release(struct binder_alloc *alloc); +int binder_alloc_get_allocated_count(struct binder_alloc *alloc); +void binder_alloc_print_allocated(struct seq_file *m, + struct binder_alloc *alloc); void binder_alloc_print_pages(struct seq_file *m, struct binder_alloc *alloc); -- cgit From df9aabead791d7a3d59938abe288720f5c1367f7 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:38 +0000 Subject: binder: keep vma addresses type as unsigned long The vma addresses in binder are currently stored as void __user *. This requires casting back and forth between the mm/ api which uses unsigned long. Since we also do internal arithmetic on these addresses we end up having to cast them _again_ to an integer type. Lets stop all the unnecessary casting which kills code readability and store the virtual addresses as the native unsigned long from mm/. Note that this approach is preferred over uintptr_t as Linus explains in [1]. Opportunistically add a few cosmetic touchups. Link: https://lore.kernel.org/all/CAHk-=wj2OHy-5e+srG1fy+ZU00TmZ1NFp6kFLbVLMXHe7A1d-g@mail.gmail.com/ [1] Signed-off-by: Carlos Llamas Reviewed-by: Alice Ryhl Link: https://lore.kernel.org/r/20231201172212.1813387-10-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder.c | 21 ++++---- drivers/android/binder_alloc.c | 91 +++++++++++++++------------------ drivers/android/binder_alloc.h | 6 +-- drivers/android/binder_alloc_selftest.c | 6 +-- drivers/android/binder_trace.h | 2 +- 5 files changed, 57 insertions(+), 69 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder.c b/drivers/android/binder.c index 71a40a4c546f..437d1097b118 100644 --- a/drivers/android/binder.c +++ b/drivers/android/binder.c @@ -2077,9 +2077,8 @@ static void binder_transaction_buffer_release(struct binder_proc *proc, * Convert the address to an offset relative to * the base of the transaction buffer. */ - fda_offset = - (parent->buffer - (uintptr_t)buffer->user_data) + - fda->parent_offset; + fda_offset = parent->buffer - buffer->user_data + + fda->parent_offset; for (fd_index = 0; fd_index < fda->num_fds; fd_index++) { u32 fd; @@ -2597,7 +2596,7 @@ static int binder_translate_fd_array(struct list_head *pf_head, * Convert the address to an offset relative to * the base of the transaction buffer. */ - fda_offset = (parent->buffer - (uintptr_t)t->buffer->user_data) + + fda_offset = parent->buffer - t->buffer->user_data + fda->parent_offset; sender_ufda_base = (void __user *)(uintptr_t)sender_uparent->buffer + fda->parent_offset; @@ -2672,8 +2671,9 @@ static int binder_fixup_parent(struct list_head *pf_head, proc->pid, thread->pid); return -EINVAL; } - buffer_offset = bp->parent_offset + - (uintptr_t)parent->buffer - (uintptr_t)b->user_data; + + buffer_offset = bp->parent_offset + parent->buffer - b->user_data; + return binder_add_fixup(pf_head, buffer_offset, bp->buffer, 0); } @@ -3250,7 +3250,7 @@ static void binder_transaction(struct binder_proc *proc, ALIGN(extra_buffers_size, sizeof(void *)) - ALIGN(secctx_sz, sizeof(u64)); - t->security_ctx = (uintptr_t)t->buffer->user_data + buf_offset; + t->security_ctx = t->buffer->user_data + buf_offset; err = binder_alloc_copy_to_buffer(&target_proc->alloc, t->buffer, buf_offset, secctx, secctx_sz); @@ -3527,8 +3527,7 @@ static void binder_transaction(struct binder_proc *proc, goto err_translate_failed; } /* Fixup buffer pointer to target proc address space */ - bp->buffer = (uintptr_t) - t->buffer->user_data + sg_buf_offset; + bp->buffer = t->buffer->user_data + sg_buf_offset; sg_buf_offset += ALIGN(bp->length, sizeof(u64)); num_valid = (buffer_offset - off_start_offset) / @@ -4698,7 +4697,7 @@ retry: } trd->data_size = t->buffer->data_size; trd->offsets_size = t->buffer->offsets_size; - trd->data.ptr.buffer = (uintptr_t)t->buffer->user_data; + trd->data.ptr.buffer = t->buffer->user_data; trd->data.ptr.offsets = trd->data.ptr.buffer + ALIGN(t->buffer->data_size, sizeof(void *)); @@ -5981,7 +5980,7 @@ static void print_binder_transaction_ilocked(struct seq_file *m, } if (buffer->target_node) seq_printf(m, " node %d", buffer->target_node->debug_id); - seq_printf(m, " size %zd:%zd data %pK\n", + seq_printf(m, " size %zd:%zd data %lx\n", buffer->data_size, buffer->offsets_size, buffer->user_data); } diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index a56cbfd9ba44..179b67a3ef70 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -125,23 +125,20 @@ static void binder_insert_allocated_buffer_locked( static struct binder_buffer *binder_alloc_prepare_to_free_locked( struct binder_alloc *alloc, - uintptr_t user_ptr) + unsigned long user_ptr) { struct rb_node *n = alloc->allocated_buffers.rb_node; struct binder_buffer *buffer; - void __user *uptr; - - uptr = (void __user *)user_ptr; while (n) { buffer = rb_entry(n, struct binder_buffer, rb_node); BUG_ON(buffer->free); - if (uptr < buffer->user_data) + if (user_ptr < buffer->user_data) { n = n->rb_left; - else if (uptr > buffer->user_data) + } else if (user_ptr > buffer->user_data) { n = n->rb_right; - else { + } else { /* * Guard against user threads attempting to * free the buffer when in use by kernel or @@ -168,7 +165,7 @@ static struct binder_buffer *binder_alloc_prepare_to_free_locked( * Return: Pointer to buffer or NULL */ struct binder_buffer *binder_alloc_prepare_to_free(struct binder_alloc *alloc, - uintptr_t user_ptr) + unsigned long user_ptr) { struct binder_buffer *buffer; @@ -179,18 +176,17 @@ struct binder_buffer *binder_alloc_prepare_to_free(struct binder_alloc *alloc, } static int binder_update_page_range(struct binder_alloc *alloc, int allocate, - void __user *start, void __user *end) + unsigned long start, unsigned long end) { - void __user *page_addr; - unsigned long user_page_addr; - struct binder_lru_page *page; struct vm_area_struct *vma = NULL; + struct binder_lru_page *page; struct mm_struct *mm = NULL; + unsigned long page_addr; bool need_mm = false; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: %s pages %pK-%pK\n", alloc->pid, - allocate ? "allocate" : "free", start, end); + "%d: %s allocate pages %lx-%lx\n", alloc->pid, + allocate ? "allocate" : "free", start, end); if (end <= start) return 0; @@ -249,18 +245,17 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate, __GFP_HIGHMEM | __GFP_ZERO); if (!page->page_ptr) { - pr_err("%d: binder_alloc_buf failed for page at %pK\n", - alloc->pid, page_addr); + pr_err("%d: binder_alloc_buf failed for page at %lx\n", + alloc->pid, page_addr); goto err_alloc_page_failed; } page->alloc = alloc; INIT_LIST_HEAD(&page->lru); - user_page_addr = (uintptr_t)page_addr; - ret = vm_insert_page(vma, user_page_addr, page[0].page_ptr); + ret = vm_insert_page(vma, page_addr, page->page_ptr); if (ret) { pr_err("%d: binder_alloc_buf failed to map page at %lx in userspace\n", - alloc->pid, user_page_addr); + alloc->pid, page_addr); goto err_vm_insert_page_failed; } @@ -377,9 +372,9 @@ static struct binder_buffer *binder_alloc_new_buf_locked( struct binder_buffer *buffer; size_t buffer_size; struct rb_node *best_fit = NULL; - void __user *has_page_addr; - void __user *end_page_addr; size_t size, data_offsets_size; + unsigned long has_page_addr; + unsigned long end_page_addr; int ret; /* Check binder_alloc is fully initialized */ @@ -477,15 +472,13 @@ static struct binder_buffer *binder_alloc_new_buf_locked( "%d: binder_alloc_buf size %zd got buffer %pK size %zd\n", alloc->pid, size, buffer, buffer_size); - has_page_addr = (void __user *) - (((uintptr_t)buffer->user_data + buffer_size) & PAGE_MASK); + has_page_addr = (buffer->user_data + buffer_size) & PAGE_MASK; WARN_ON(n && buffer_size != size); - end_page_addr = - (void __user *)PAGE_ALIGN((uintptr_t)buffer->user_data + size); + end_page_addr = PAGE_ALIGN(buffer->user_data + size); if (end_page_addr > has_page_addr) end_page_addr = has_page_addr; - ret = binder_update_page_range(alloc, 1, (void __user *) - PAGE_ALIGN((uintptr_t)buffer->user_data), end_page_addr); + ret = binder_update_page_range(alloc, 1, PAGE_ALIGN(buffer->user_data), + end_page_addr); if (ret) return ERR_PTR(ret); @@ -498,7 +491,7 @@ static struct binder_buffer *binder_alloc_new_buf_locked( __func__, alloc->pid); goto err_alloc_buf_struct_failed; } - new_buffer->user_data = (u8 __user *)buffer->user_data + size; + new_buffer->user_data = buffer->user_data + size; list_add(&new_buffer->entry, &buffer->entry); new_buffer->free = 1; binder_insert_free_buffer(alloc, new_buffer); @@ -536,8 +529,7 @@ static struct binder_buffer *binder_alloc_new_buf_locked( return buffer; err_alloc_buf_struct_failed: - binder_update_page_range(alloc, 0, (void __user *) - PAGE_ALIGN((uintptr_t)buffer->user_data), + binder_update_page_range(alloc, 0, PAGE_ALIGN(buffer->user_data), end_page_addr); return ERR_PTR(-ENOMEM); } @@ -574,15 +566,14 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, return buffer; } -static void __user *buffer_start_page(struct binder_buffer *buffer) +static unsigned long buffer_start_page(struct binder_buffer *buffer) { - return (void __user *)((uintptr_t)buffer->user_data & PAGE_MASK); + return buffer->user_data & PAGE_MASK; } -static void __user *prev_buffer_end_page(struct binder_buffer *buffer) +static unsigned long prev_buffer_end_page(struct binder_buffer *buffer) { - return (void __user *) - (((uintptr_t)(buffer->user_data) - 1) & PAGE_MASK); + return (buffer->user_data - 1) & PAGE_MASK; } static void binder_delete_free_buffer(struct binder_alloc *alloc, @@ -597,7 +588,7 @@ static void binder_delete_free_buffer(struct binder_alloc *alloc, if (prev_buffer_end_page(prev) == buffer_start_page(buffer)) { to_free = false; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: merge free, buffer %pK share page with %pK\n", + "%d: merge free, buffer %lx share page with %lx\n", alloc->pid, buffer->user_data, prev->user_data); } @@ -607,7 +598,7 @@ static void binder_delete_free_buffer(struct binder_alloc *alloc, if (buffer_start_page(next) == buffer_start_page(buffer)) { to_free = false; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: merge free, buffer %pK share page with %pK\n", + "%d: merge free, buffer %lx share page with %lx\n", alloc->pid, buffer->user_data, next->user_data); @@ -616,17 +607,17 @@ static void binder_delete_free_buffer(struct binder_alloc *alloc, if (PAGE_ALIGNED(buffer->user_data)) { binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: merge free, buffer start %pK is page aligned\n", + "%d: merge free, buffer start %lx is page aligned\n", alloc->pid, buffer->user_data); to_free = false; } if (to_free) { binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: merge free, buffer %pK do not share page with %pK or %pK\n", + "%d: merge free, buffer %lx do not share page with %lx or %lx\n", alloc->pid, buffer->user_data, prev->user_data, - next ? next->user_data : NULL); + next ? next->user_data : 0); binder_update_page_range(alloc, 0, buffer_start_page(buffer), buffer_start_page(buffer) + PAGE_SIZE); } @@ -662,10 +653,8 @@ static void binder_free_buf_locked(struct binder_alloc *alloc, alloc->pid, size, alloc->free_async_space); } - binder_update_page_range(alloc, 0, - (void __user *)PAGE_ALIGN((uintptr_t)buffer->user_data), - (void __user *)(((uintptr_t) - buffer->user_data + buffer_size) & PAGE_MASK)); + binder_update_page_range(alloc, 0, PAGE_ALIGN(buffer->user_data), + (buffer->user_data + buffer_size) & PAGE_MASK); rb_erase(&buffer->rb_node, &alloc->allocated_buffers); buffer->free = 1; @@ -754,7 +743,7 @@ int binder_alloc_mmap_handler(struct binder_alloc *alloc, SZ_4M); mutex_unlock(&binder_alloc_mmap_lock); - alloc->buffer = (void __user *)vma->vm_start; + alloc->buffer = vma->vm_start; alloc->pages = kcalloc(alloc->buffer_size / PAGE_SIZE, sizeof(alloc->pages[0]), @@ -787,7 +776,7 @@ err_alloc_buf_struct_failed: kfree(alloc->pages); alloc->pages = NULL; err_alloc_pages_failed: - alloc->buffer = NULL; + alloc->buffer = 0; mutex_lock(&binder_alloc_mmap_lock); alloc->buffer_size = 0; err_already_mapped: @@ -840,7 +829,7 @@ void binder_alloc_deferred_release(struct binder_alloc *alloc) int i; for (i = 0; i < alloc->buffer_size / PAGE_SIZE; i++) { - void __user *page_addr; + unsigned long page_addr; bool on_lru; if (!alloc->pages[i].page_ptr) @@ -850,7 +839,7 @@ void binder_alloc_deferred_release(struct binder_alloc *alloc) &alloc->pages[i].lru); page_addr = alloc->buffer + i * PAGE_SIZE; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%s: %d: page %d at %pK %s\n", + "%s: %d: page %d at %lx %s\n", __func__, alloc->pid, i, page_addr, on_lru ? "on lru" : "active"); __free_page(alloc->pages[i].page_ptr); @@ -870,7 +859,7 @@ void binder_alloc_deferred_release(struct binder_alloc *alloc) static void print_binder_buffer(struct seq_file *m, const char *prefix, struct binder_buffer *buffer) { - seq_printf(m, "%s %d: %pK size %zd:%zd:%zd %s\n", + seq_printf(m, "%s %d: %lx size %zd:%zd:%zd %s\n", prefix, buffer->debug_id, buffer->user_data, buffer->data_size, buffer->offsets_size, buffer->extra_buffers_size, @@ -984,7 +973,7 @@ enum lru_status binder_alloc_free_page(struct list_head *item, struct binder_lru_page, lru); struct binder_alloc *alloc; - uintptr_t page_addr; + unsigned long page_addr; size_t index; struct vm_area_struct *vma; @@ -996,7 +985,7 @@ enum lru_status binder_alloc_free_page(struct list_head *item, goto err_page_already_freed; index = page - alloc->pages; - page_addr = (uintptr_t)alloc->buffer + index * PAGE_SIZE; + page_addr = alloc->buffer + index * PAGE_SIZE; mm = alloc->mm; if (!mmget_not_zero(mm)) diff --git a/drivers/android/binder_alloc.h b/drivers/android/binder_alloc.h index 82380febdd85..cb19677a5c15 100644 --- a/drivers/android/binder_alloc.h +++ b/drivers/android/binder_alloc.h @@ -56,7 +56,7 @@ struct binder_buffer { size_t data_size; size_t offsets_size; size_t extra_buffers_size; - void __user *user_data; + unsigned long user_data; int pid; }; @@ -101,7 +101,7 @@ struct binder_alloc { struct mutex mutex; struct vm_area_struct *vma; struct mm_struct *mm; - void __user *buffer; + unsigned long buffer; struct list_head buffers; struct rb_root free_buffers; struct rb_root allocated_buffers; @@ -133,7 +133,7 @@ void binder_alloc_shrinker_exit(void); void binder_alloc_vma_close(struct binder_alloc *alloc); struct binder_buffer * binder_alloc_prepare_to_free(struct binder_alloc *alloc, - uintptr_t user_ptr); + unsigned long user_ptr); void binder_alloc_free_buf(struct binder_alloc *alloc, struct binder_buffer *buffer); int binder_alloc_mmap_handler(struct binder_alloc *alloc, diff --git a/drivers/android/binder_alloc_selftest.c b/drivers/android/binder_alloc_selftest.c index c2b323bc3b3a..341c73b4a807 100644 --- a/drivers/android/binder_alloc_selftest.c +++ b/drivers/android/binder_alloc_selftest.c @@ -93,11 +93,11 @@ static bool check_buffer_pages_allocated(struct binder_alloc *alloc, struct binder_buffer *buffer, size_t size) { - void __user *page_addr; - void __user *end; + unsigned long page_addr; + unsigned long end; int page_index; - end = (void __user *)PAGE_ALIGN((uintptr_t)buffer->user_data + size); + end = PAGE_ALIGN(buffer->user_data + size); page_addr = buffer->user_data; for (; page_addr < end; page_addr += PAGE_SIZE) { page_index = (page_addr - alloc->buffer) / PAGE_SIZE; diff --git a/drivers/android/binder_trace.h b/drivers/android/binder_trace.h index 8cc07e6a4273..fe38c6fc65d0 100644 --- a/drivers/android/binder_trace.h +++ b/drivers/android/binder_trace.h @@ -317,7 +317,7 @@ DEFINE_EVENT(binder_buffer_class, binder_transaction_update_buffer_release, TRACE_EVENT(binder_update_page_range, TP_PROTO(struct binder_alloc *alloc, bool allocate, - void __user *start, void __user *end), + unsigned long start, unsigned long end), TP_ARGS(alloc, allocate, start, end), TP_STRUCT__entry( __field(int, proc) -- cgit From 0d35bf3bf2da8d43fd12fea7699dc936999bf96e Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:39 +0000 Subject: binder: split up binder_update_page_range() The binder_update_page_range() function performs both allocation and freeing of binder pages. However, these two operations are unrelated and have no common logic. In fact, when a free operation is requested, the allocation logic is skipped entirely. This behavior makes the error path unnecessarily complex. To improve readability of the code, this patch splits the allocation and freeing operations into separate functions. No functional changes are introduced by this patch. Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-11-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 79 +++++++++++++++++++++--------------------- 1 file changed, 40 insertions(+), 39 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 179b67a3ef70..3051ea7ca44f 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -175,8 +175,32 @@ struct binder_buffer *binder_alloc_prepare_to_free(struct binder_alloc *alloc, return buffer; } -static int binder_update_page_range(struct binder_alloc *alloc, int allocate, - unsigned long start, unsigned long end) +static void binder_free_page_range(struct binder_alloc *alloc, + unsigned long start, unsigned long end) +{ + struct binder_lru_page *page; + unsigned long page_addr; + + trace_binder_update_page_range(alloc, false, start, end); + + for (page_addr = start; page_addr < end; page_addr += PAGE_SIZE) { + size_t index; + int ret; + + index = (page_addr - alloc->buffer) / PAGE_SIZE; + page = &alloc->pages[index]; + + trace_binder_free_lru_start(alloc, index); + + ret = list_lru_add(&binder_alloc_lru, &page->lru); + WARN_ON(!ret); + + trace_binder_free_lru_end(alloc, index); + } +} + +static int binder_allocate_page_range(struct binder_alloc *alloc, + unsigned long start, unsigned long end) { struct vm_area_struct *vma = NULL; struct binder_lru_page *page; @@ -185,16 +209,13 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate, bool need_mm = false; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: %s allocate pages %lx-%lx\n", alloc->pid, - allocate ? "allocate" : "free", start, end); + "%d: allocate pages %lx-%lx\n", + alloc->pid, start, end); if (end <= start) return 0; - trace_binder_update_page_range(alloc, allocate, start, end); - - if (allocate == 0) - goto free_range; + trace_binder_update_page_range(alloc, true, start, end); for (page_addr = start; page_addr < end; page_addr += PAGE_SIZE) { page = &alloc->pages[(page_addr - alloc->buffer) / PAGE_SIZE]; @@ -270,32 +291,12 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate, } return 0; -free_range: - for (page_addr = end - PAGE_SIZE; 1; page_addr -= PAGE_SIZE) { - bool ret; - size_t index; - - index = (page_addr - alloc->buffer) / PAGE_SIZE; - page = &alloc->pages[index]; - - trace_binder_free_lru_start(alloc, index); - - ret = list_lru_add(&binder_alloc_lru, &page->lru); - WARN_ON(!ret); - - trace_binder_free_lru_end(alloc, index); - if (page_addr == start) - break; - continue; - err_vm_insert_page_failed: - __free_page(page->page_ptr); - page->page_ptr = NULL; + __free_page(page->page_ptr); + page->page_ptr = NULL; err_alloc_page_failed: err_page_ptr_cleared: - if (page_addr == start) - break; - } + binder_free_page_range(alloc, start, page_addr); err_no_vma: if (mm) { mmap_write_unlock(mm); @@ -477,8 +478,8 @@ static struct binder_buffer *binder_alloc_new_buf_locked( end_page_addr = PAGE_ALIGN(buffer->user_data + size); if (end_page_addr > has_page_addr) end_page_addr = has_page_addr; - ret = binder_update_page_range(alloc, 1, PAGE_ALIGN(buffer->user_data), - end_page_addr); + ret = binder_allocate_page_range(alloc, PAGE_ALIGN(buffer->user_data), + end_page_addr); if (ret) return ERR_PTR(ret); @@ -529,8 +530,8 @@ static struct binder_buffer *binder_alloc_new_buf_locked( return buffer; err_alloc_buf_struct_failed: - binder_update_page_range(alloc, 0, PAGE_ALIGN(buffer->user_data), - end_page_addr); + binder_free_page_range(alloc, PAGE_ALIGN(buffer->user_data), + end_page_addr); return ERR_PTR(-ENOMEM); } @@ -618,8 +619,8 @@ static void binder_delete_free_buffer(struct binder_alloc *alloc, alloc->pid, buffer->user_data, prev->user_data, next ? next->user_data : 0); - binder_update_page_range(alloc, 0, buffer_start_page(buffer), - buffer_start_page(buffer) + PAGE_SIZE); + binder_free_page_range(alloc, buffer_start_page(buffer), + buffer_start_page(buffer) + PAGE_SIZE); } list_del(&buffer->entry); kfree(buffer); @@ -653,8 +654,8 @@ static void binder_free_buf_locked(struct binder_alloc *alloc, alloc->pid, size, alloc->free_async_space); } - binder_update_page_range(alloc, 0, PAGE_ALIGN(buffer->user_data), - (buffer->user_data + buffer_size) & PAGE_MASK); + binder_free_page_range(alloc, PAGE_ALIGN(buffer->user_data), + (buffer->user_data + buffer_size) & PAGE_MASK); rb_erase(&buffer->rb_node, &alloc->allocated_buffers); buffer->free = 1; -- cgit From 377e1684db7a1e23261f3c3ebf76523c0554d512 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:40 +0000 Subject: binder: do unlocked work in binder_alloc_new_buf() Extract non-critical sections from binder_alloc_new_buf_locked() that don't require holding the alloc->mutex. While we are here, consolidate the checks for size overflow and zero-sized padding into a separate sanitized_size() helper function. Also add a few touchups to follow the coding guidelines. Signed-off-by: Carlos Llamas Reviewed-by: Alice Ryhl Link: https://lore.kernel.org/r/20231201172212.1813387-12-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 95 +++++++++++++++++++++++++----------------- 1 file changed, 56 insertions(+), 39 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 3051ea7ca44f..40a2ca0c0dea 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -363,9 +363,7 @@ static bool debug_low_async_space_locked(struct binder_alloc *alloc, int pid) static struct binder_buffer *binder_alloc_new_buf_locked( struct binder_alloc *alloc, - size_t data_size, - size_t offsets_size, - size_t extra_buffers_size, + size_t size, int is_async, int pid) { @@ -373,39 +371,10 @@ static struct binder_buffer *binder_alloc_new_buf_locked( struct binder_buffer *buffer; size_t buffer_size; struct rb_node *best_fit = NULL; - size_t size, data_offsets_size; unsigned long has_page_addr; unsigned long end_page_addr; int ret; - /* Check binder_alloc is fully initialized */ - if (!binder_alloc_get_vma(alloc)) { - binder_alloc_debug(BINDER_DEBUG_USER_ERROR, - "%d: binder_alloc_buf, no vma\n", - alloc->pid); - return ERR_PTR(-ESRCH); - } - - data_offsets_size = ALIGN(data_size, sizeof(void *)) + - ALIGN(offsets_size, sizeof(void *)); - - if (data_offsets_size < data_size || data_offsets_size < offsets_size) { - binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: got transaction with invalid size %zd-%zd\n", - alloc->pid, data_size, offsets_size); - return ERR_PTR(-EINVAL); - } - size = data_offsets_size + ALIGN(extra_buffers_size, sizeof(void *)); - if (size < data_offsets_size || size < extra_buffers_size) { - binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: got transaction with invalid extra_buffers_size %zd\n", - alloc->pid, extra_buffers_size); - return ERR_PTR(-EINVAL); - } - - /* Pad 0-size buffers so they get assigned unique addresses */ - size = max(size, sizeof(void *)); - if (is_async && alloc->free_async_space < size) { binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, "%d: binder_alloc_buf size %zd failed, no async space left\n", @@ -421,13 +390,14 @@ static struct binder_buffer *binder_alloc_new_buf_locked( if (size < buffer_size) { best_fit = n; n = n->rb_left; - } else if (size > buffer_size) + } else if (size > buffer_size) { n = n->rb_right; - else { + } else { best_fit = n; break; } } + if (best_fit == NULL) { size_t allocated_buffers = 0; size_t largest_alloc_size = 0; @@ -505,10 +475,7 @@ static struct binder_buffer *binder_alloc_new_buf_locked( binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, "%d: binder_alloc_buf size %zd got %pK\n", alloc->pid, size, buffer); - buffer->data_size = data_size; - buffer->offsets_size = offsets_size; buffer->async_transaction = is_async; - buffer->extra_buffers_size = extra_buffers_size; buffer->pid = pid; buffer->oneway_spam_suspect = false; if (is_async) { @@ -527,6 +494,7 @@ static struct binder_buffer *binder_alloc_new_buf_locked( alloc->oneway_spam_detected = false; } } + return buffer; err_alloc_buf_struct_failed: @@ -535,6 +503,28 @@ err_alloc_buf_struct_failed: return ERR_PTR(-ENOMEM); } +/* Calculate the sanitized total size, returns 0 for invalid request */ +static inline size_t sanitized_size(size_t data_size, + size_t offsets_size, + size_t extra_buffers_size) +{ + size_t total, tmp; + + /* Align to pointer size and check for overflows */ + tmp = ALIGN(data_size, sizeof(void *)) + + ALIGN(offsets_size, sizeof(void *)); + if (tmp < data_size || tmp < offsets_size) + return 0; + total = tmp + ALIGN(extra_buffers_size, sizeof(void *)); + if (total < tmp || total < extra_buffers_size) + return 0; + + /* Pad 0-sized buffers so they get a unique address */ + total = max(total, sizeof(void *)); + + return total; +} + /** * binder_alloc_new_buf() - Allocate a new binder buffer * @alloc: binder_alloc for this proc @@ -559,11 +549,38 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, int pid) { struct binder_buffer *buffer; + size_t size; + + /* Check binder_alloc is fully initialized */ + if (!binder_alloc_get_vma(alloc)) { + binder_alloc_debug(BINDER_DEBUG_USER_ERROR, + "%d: binder_alloc_buf, no vma\n", + alloc->pid); + return ERR_PTR(-ESRCH); + } + + size = sanitized_size(data_size, offsets_size, extra_buffers_size); + if (unlikely(!size)) { + binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, + "%d: got transaction with invalid size %zd-%zd-%zd\n", + alloc->pid, data_size, offsets_size, + extra_buffers_size); + return ERR_PTR(-EINVAL); + } mutex_lock(&alloc->mutex); - buffer = binder_alloc_new_buf_locked(alloc, data_size, offsets_size, - extra_buffers_size, is_async, pid); + buffer = binder_alloc_new_buf_locked(alloc, size, is_async, pid); + if (IS_ERR(buffer)) { + mutex_unlock(&alloc->mutex); + goto out; + } + + buffer->data_size = data_size; + buffer->offsets_size = offsets_size; + buffer->extra_buffers_size = extra_buffers_size; mutex_unlock(&alloc->mutex); + +out: return buffer; } -- cgit From 89f71743bf42217dd4092fda703a8e4f6f4e55ac Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:41 +0000 Subject: binder: remove pid param in binder_alloc_new_buf() Binder attributes the buffer allocation to the current->tgid everytime. There is no need to pass this as a parameter so drop it. Also add a few touchups to follow the coding guidelines. No functional changes are introduced in this patch. Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-13-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder.c | 2 +- drivers/android/binder_alloc.c | 18 ++++++++---------- drivers/android/binder_alloc.h | 7 ++----- drivers/android/binder_alloc_selftest.c | 2 +- 4 files changed, 12 insertions(+), 17 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder.c b/drivers/android/binder.c index 437d1097b118..45674af6310f 100644 --- a/drivers/android/binder.c +++ b/drivers/android/binder.c @@ -3225,7 +3225,7 @@ static void binder_transaction(struct binder_proc *proc, t->buffer = binder_alloc_new_buf(&target_proc->alloc, tr->data_size, tr->offsets_size, extra_buffers_size, - !reply && (t->flags & TF_ONE_WAY), current->tgid); + !reply && (t->flags & TF_ONE_WAY)); if (IS_ERR(t->buffer)) { char *s; diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 40a2ca0c0dea..b5c3e56318e1 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -319,7 +319,7 @@ static inline struct vm_area_struct *binder_alloc_get_vma( return smp_load_acquire(&alloc->vma); } -static bool debug_low_async_space_locked(struct binder_alloc *alloc, int pid) +static bool debug_low_async_space_locked(struct binder_alloc *alloc) { /* * Find the amount and size of buffers allocated by the current caller; @@ -328,10 +328,11 @@ static bool debug_low_async_space_locked(struct binder_alloc *alloc, int pid) * and at some point we'll catch them in the act. This is more efficient * than keeping a map per pid. */ - struct rb_node *n; struct binder_buffer *buffer; size_t total_alloc_size = 0; + int pid = current->tgid; size_t num_buffers = 0; + struct rb_node *n; for (n = rb_first(&alloc->allocated_buffers); n != NULL; n = rb_next(n)) { @@ -364,8 +365,7 @@ static bool debug_low_async_space_locked(struct binder_alloc *alloc, int pid) static struct binder_buffer *binder_alloc_new_buf_locked( struct binder_alloc *alloc, size_t size, - int is_async, - int pid) + int is_async) { struct rb_node *n = alloc->free_buffers.rb_node; struct binder_buffer *buffer; @@ -476,7 +476,6 @@ static struct binder_buffer *binder_alloc_new_buf_locked( "%d: binder_alloc_buf size %zd got %pK\n", alloc->pid, size, buffer); buffer->async_transaction = is_async; - buffer->pid = pid; buffer->oneway_spam_suspect = false; if (is_async) { alloc->free_async_space -= size; @@ -489,7 +488,7 @@ static struct binder_buffer *binder_alloc_new_buf_locked( * of async space left (which is less than 10% of total * buffer size). */ - buffer->oneway_spam_suspect = debug_low_async_space_locked(alloc, pid); + buffer->oneway_spam_suspect = debug_low_async_space_locked(alloc); } else { alloc->oneway_spam_detected = false; } @@ -532,7 +531,6 @@ static inline size_t sanitized_size(size_t data_size, * @offsets_size: user specified buffer offset * @extra_buffers_size: size of extra space for meta-data (eg, security context) * @is_async: buffer for async transaction - * @pid: pid to attribute allocation to (used for debugging) * * Allocate a new buffer given the requested sizes. Returns * the kernel version of the buffer pointer. The size allocated @@ -545,8 +543,7 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, size_t data_size, size_t offsets_size, size_t extra_buffers_size, - int is_async, - int pid) + int is_async) { struct binder_buffer *buffer; size_t size; @@ -569,7 +566,7 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, } mutex_lock(&alloc->mutex); - buffer = binder_alloc_new_buf_locked(alloc, size, is_async, pid); + buffer = binder_alloc_new_buf_locked(alloc, size, is_async); if (IS_ERR(buffer)) { mutex_unlock(&alloc->mutex); goto out; @@ -578,6 +575,7 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, buffer->data_size = data_size; buffer->offsets_size = offsets_size; buffer->extra_buffers_size = extra_buffers_size; + buffer->pid = current->tgid; mutex_unlock(&alloc->mutex); out: diff --git a/drivers/android/binder_alloc.h b/drivers/android/binder_alloc.h index cb19677a5c15..bbc16bc6d5ac 100644 --- a/drivers/android/binder_alloc.h +++ b/drivers/android/binder_alloc.h @@ -49,15 +49,13 @@ struct binder_buffer { unsigned async_transaction:1; unsigned oneway_spam_suspect:1; unsigned debug_id:27; - struct binder_transaction *transaction; - struct binder_node *target_node; size_t data_size; size_t offsets_size; size_t extra_buffers_size; unsigned long user_data; - int pid; + int pid; }; /** @@ -125,8 +123,7 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, size_t data_size, size_t offsets_size, size_t extra_buffers_size, - int is_async, - int pid); + int is_async); void binder_alloc_init(struct binder_alloc *alloc); int binder_alloc_shrinker_init(void); void binder_alloc_shrinker_exit(void); diff --git a/drivers/android/binder_alloc_selftest.c b/drivers/android/binder_alloc_selftest.c index 341c73b4a807..ed753747e54c 100644 --- a/drivers/android/binder_alloc_selftest.c +++ b/drivers/android/binder_alloc_selftest.c @@ -119,7 +119,7 @@ static void binder_selftest_alloc_buf(struct binder_alloc *alloc, int i; for (i = 0; i < BUFFER_NUM; i++) { - buffers[i] = binder_alloc_new_buf(alloc, sizes[i], 0, 0, 0, 0); + buffers[i] = binder_alloc_new_buf(alloc, sizes[i], 0, 0, 0); if (IS_ERR(buffers[i]) || !check_buffer_pages_allocated(alloc, buffers[i], sizes[i])) { -- cgit From 9409af24e4503d14093b27db9425f7c99e64fef4 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:42 +0000 Subject: binder: separate the no-space debugging logic Move the no-space debugging logic into a separate function. Lets also mark this branch as unlikely in binder_alloc_new_buf_locked() as most requests will fit without issue. Also add a few cosmetic changes and suggestions from checkpatch. Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-14-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 71 ++++++++++++++++++++++++------------------ 1 file changed, 40 insertions(+), 31 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index b5c3e56318e1..3dca7b199246 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -319,6 +319,43 @@ static inline struct vm_area_struct *binder_alloc_get_vma( return smp_load_acquire(&alloc->vma); } +static void debug_no_space_locked(struct binder_alloc *alloc) +{ + size_t largest_alloc_size = 0; + struct binder_buffer *buffer; + size_t allocated_buffers = 0; + size_t largest_free_size = 0; + size_t total_alloc_size = 0; + size_t total_free_size = 0; + size_t free_buffers = 0; + size_t buffer_size; + struct rb_node *n; + + for (n = rb_first(&alloc->allocated_buffers); n; n = rb_next(n)) { + buffer = rb_entry(n, struct binder_buffer, rb_node); + buffer_size = binder_alloc_buffer_size(alloc, buffer); + allocated_buffers++; + total_alloc_size += buffer_size; + if (buffer_size > largest_alloc_size) + largest_alloc_size = buffer_size; + } + + for (n = rb_first(&alloc->free_buffers); n; n = rb_next(n)) { + buffer = rb_entry(n, struct binder_buffer, rb_node); + buffer_size = binder_alloc_buffer_size(alloc, buffer); + free_buffers++; + total_free_size += buffer_size; + if (buffer_size > largest_free_size) + largest_free_size = buffer_size; + } + + binder_alloc_debug(BINDER_DEBUG_USER_ERROR, + "allocated: %zd (num: %zd largest: %zd), free: %zd (num: %zd largest: %zd)\n", + total_alloc_size, allocated_buffers, + largest_alloc_size, total_free_size, + free_buffers, largest_free_size); +} + static bool debug_low_async_space_locked(struct binder_alloc *alloc) { /* @@ -398,42 +435,14 @@ static struct binder_buffer *binder_alloc_new_buf_locked( } } - if (best_fit == NULL) { - size_t allocated_buffers = 0; - size_t largest_alloc_size = 0; - size_t total_alloc_size = 0; - size_t free_buffers = 0; - size_t largest_free_size = 0; - size_t total_free_size = 0; - - for (n = rb_first(&alloc->allocated_buffers); n != NULL; - n = rb_next(n)) { - buffer = rb_entry(n, struct binder_buffer, rb_node); - buffer_size = binder_alloc_buffer_size(alloc, buffer); - allocated_buffers++; - total_alloc_size += buffer_size; - if (buffer_size > largest_alloc_size) - largest_alloc_size = buffer_size; - } - for (n = rb_first(&alloc->free_buffers); n != NULL; - n = rb_next(n)) { - buffer = rb_entry(n, struct binder_buffer, rb_node); - buffer_size = binder_alloc_buffer_size(alloc, buffer); - free_buffers++; - total_free_size += buffer_size; - if (buffer_size > largest_free_size) - largest_free_size = buffer_size; - } + if (unlikely(!best_fit)) { binder_alloc_debug(BINDER_DEBUG_USER_ERROR, "%d: binder_alloc_buf size %zd failed, no address space\n", alloc->pid, size); - binder_alloc_debug(BINDER_DEBUG_USER_ERROR, - "allocated: %zd (num: %zd largest: %zd), free: %zd (num: %zd largest: %zd)\n", - total_alloc_size, allocated_buffers, - largest_alloc_size, total_free_size, - free_buffers, largest_free_size); + debug_no_space_locked(alloc); return ERR_PTR(-ENOSPC); } + if (n == NULL) { buffer = rb_entry(best_fit, struct binder_buffer, rb_node); buffer_size = binder_alloc_buffer_size(alloc, buffer); -- cgit From c13500eaabd2343aa4cbb76b54ec624cb0c0ef8d Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:43 +0000 Subject: binder: relocate low space calculation Move the low async space calculation to debug_low_async_space_locked(). This logic not only fits better here but also offloads some of the many tasks currently done in binder_alloc_new_buf_locked(). No functional change in this patch. Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-15-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 3dca7b199246..167ee6f871dc 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -371,6 +371,15 @@ static bool debug_low_async_space_locked(struct binder_alloc *alloc) size_t num_buffers = 0; struct rb_node *n; + /* + * Only start detecting spammers once we have less than 20% of async + * space left (which is less than 10% of total buffer size). + */ + if (alloc->free_async_space >= alloc->buffer_size / 10) { + alloc->oneway_spam_detected = false; + return false; + } + for (n = rb_first(&alloc->allocated_buffers); n != NULL; n = rb_next(n)) { buffer = rb_entry(n, struct binder_buffer, rb_node); @@ -491,16 +500,8 @@ static struct binder_buffer *binder_alloc_new_buf_locked( binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC_ASYNC, "%d: binder_alloc_buf size %zd async free %zd\n", alloc->pid, size, alloc->free_async_space); - if (alloc->free_async_space < alloc->buffer_size / 10) { - /* - * Start detecting spammers once we have less than 20% - * of async space left (which is less than 10% of total - * buffer size). - */ - buffer->oneway_spam_suspect = debug_low_async_space_locked(alloc); - } else { - alloc->oneway_spam_detected = false; - } + if (debug_low_async_space_locked(alloc)) + buffer->oneway_spam_suspect = true; } return buffer; -- cgit From cbc174a64b8d0ab542752c167dc1334b52b88624 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:44 +0000 Subject: binder: relocate binder_alloc_clear_buf() Move this function up along with binder_alloc_get_page() so that their prototypes aren't necessary. No functional change in this patch. Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-16-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 124 ++++++++++++++++++++--------------------- 1 file changed, 61 insertions(+), 63 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 167ee6f871dc..99eacd8782b8 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -704,8 +704,68 @@ static void binder_free_buf_locked(struct binder_alloc *alloc, binder_insert_free_buffer(alloc, buffer); } +/** + * binder_alloc_get_page() - get kernel pointer for given buffer offset + * @alloc: binder_alloc for this proc + * @buffer: binder buffer to be accessed + * @buffer_offset: offset into @buffer data + * @pgoffp: address to copy final page offset to + * + * Lookup the struct page corresponding to the address + * at @buffer_offset into @buffer->user_data. If @pgoffp is not + * NULL, the byte-offset into the page is written there. + * + * The caller is responsible to ensure that the offset points + * to a valid address within the @buffer and that @buffer is + * not freeable by the user. Since it can't be freed, we are + * guaranteed that the corresponding elements of @alloc->pages[] + * cannot change. + * + * Return: struct page + */ +static struct page *binder_alloc_get_page(struct binder_alloc *alloc, + struct binder_buffer *buffer, + binder_size_t buffer_offset, + pgoff_t *pgoffp) +{ + binder_size_t buffer_space_offset = buffer_offset + + (buffer->user_data - alloc->buffer); + pgoff_t pgoff = buffer_space_offset & ~PAGE_MASK; + size_t index = buffer_space_offset >> PAGE_SHIFT; + struct binder_lru_page *lru_page; + + lru_page = &alloc->pages[index]; + *pgoffp = pgoff; + return lru_page->page_ptr; +} + +/** + * binder_alloc_clear_buf() - zero out buffer + * @alloc: binder_alloc for this proc + * @buffer: binder buffer to be cleared + * + * memset the given buffer to 0 + */ static void binder_alloc_clear_buf(struct binder_alloc *alloc, - struct binder_buffer *buffer); + struct binder_buffer *buffer) +{ + size_t bytes = binder_alloc_buffer_size(alloc, buffer); + binder_size_t buffer_offset = 0; + + while (bytes) { + unsigned long size; + struct page *page; + pgoff_t pgoff; + + page = binder_alloc_get_page(alloc, buffer, + buffer_offset, &pgoff); + size = min_t(size_t, bytes, PAGE_SIZE - pgoff); + memset_page(page, pgoff, 0, size); + bytes -= size; + buffer_offset += size; + } +} + /** * binder_alloc_free_buf() - free a binder buffer * @alloc: binder_alloc for this proc @@ -1148,68 +1208,6 @@ static inline bool check_buffer(struct binder_alloc *alloc, (!buffer->allow_user_free || !buffer->transaction); } -/** - * binder_alloc_get_page() - get kernel pointer for given buffer offset - * @alloc: binder_alloc for this proc - * @buffer: binder buffer to be accessed - * @buffer_offset: offset into @buffer data - * @pgoffp: address to copy final page offset to - * - * Lookup the struct page corresponding to the address - * at @buffer_offset into @buffer->user_data. If @pgoffp is not - * NULL, the byte-offset into the page is written there. - * - * The caller is responsible to ensure that the offset points - * to a valid address within the @buffer and that @buffer is - * not freeable by the user. Since it can't be freed, we are - * guaranteed that the corresponding elements of @alloc->pages[] - * cannot change. - * - * Return: struct page - */ -static struct page *binder_alloc_get_page(struct binder_alloc *alloc, - struct binder_buffer *buffer, - binder_size_t buffer_offset, - pgoff_t *pgoffp) -{ - binder_size_t buffer_space_offset = buffer_offset + - (buffer->user_data - alloc->buffer); - pgoff_t pgoff = buffer_space_offset & ~PAGE_MASK; - size_t index = buffer_space_offset >> PAGE_SHIFT; - struct binder_lru_page *lru_page; - - lru_page = &alloc->pages[index]; - *pgoffp = pgoff; - return lru_page->page_ptr; -} - -/** - * binder_alloc_clear_buf() - zero out buffer - * @alloc: binder_alloc for this proc - * @buffer: binder buffer to be cleared - * - * memset the given buffer to 0 - */ -static void binder_alloc_clear_buf(struct binder_alloc *alloc, - struct binder_buffer *buffer) -{ - size_t bytes = binder_alloc_buffer_size(alloc, buffer); - binder_size_t buffer_offset = 0; - - while (bytes) { - unsigned long size; - struct page *page; - pgoff_t pgoff; - - page = binder_alloc_get_page(alloc, buffer, - buffer_offset, &pgoff); - size = min_t(size_t, bytes, PAGE_SIZE - pgoff); - memset_page(page, pgoff, 0, size); - bytes -= size; - buffer_offset += size; - } -} - /** * binder_alloc_copy_user_to_buffer() - copy src user to tgt user * @alloc: binder_alloc for this proc -- cgit From ea2735ce19c1c6ce0f6011f813a1eea0272c231d Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:45 +0000 Subject: binder: refactor page range allocation Instead of looping through the page range twice to first determine if the mmap lock is required, simply do it per-page as needed. Split out all this logic into a separate binder_install_single_page() function. Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-17-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 107 ++++++++++++++++++----------------------- 1 file changed, 47 insertions(+), 60 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 99eacd8782b8..1caf0e3d3451 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -199,14 +199,51 @@ static void binder_free_page_range(struct binder_alloc *alloc, } } +static int binder_install_single_page(struct binder_alloc *alloc, + struct binder_lru_page *lru_page, + unsigned long addr) +{ + struct page *page; + int ret = 0; + + if (!mmget_not_zero(alloc->mm)) + return -ESRCH; + + mmap_write_lock(alloc->mm); + if (!alloc->vma) { + pr_err("%d: %s failed, no vma\n", alloc->pid, __func__); + ret = -ESRCH; + goto out; + } + + page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO); + if (!page) { + pr_err("%d: failed to allocate page\n", alloc->pid); + ret = -ENOMEM; + goto out; + } + + ret = vm_insert_page(alloc->vma, addr, page); + if (ret) { + pr_err("%d: %s failed to insert page at %lx with %d\n", + alloc->pid, __func__, addr, ret); + __free_page(page); + ret = -ENOMEM; + goto out; + } + + lru_page->page_ptr = page; +out: + mmap_write_unlock(alloc->mm); + mmput_async(alloc->mm); + return ret; +} + static int binder_allocate_page_range(struct binder_alloc *alloc, unsigned long start, unsigned long end) { - struct vm_area_struct *vma = NULL; struct binder_lru_page *page; - struct mm_struct *mm = NULL; unsigned long page_addr; - bool need_mm = false; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, "%d: allocate pages %lx-%lx\n", @@ -218,32 +255,9 @@ static int binder_allocate_page_range(struct binder_alloc *alloc, trace_binder_update_page_range(alloc, true, start, end); for (page_addr = start; page_addr < end; page_addr += PAGE_SIZE) { - page = &alloc->pages[(page_addr - alloc->buffer) / PAGE_SIZE]; - if (!page->page_ptr) { - need_mm = true; - break; - } - } - - if (need_mm && mmget_not_zero(alloc->mm)) - mm = alloc->mm; - - if (mm) { - mmap_write_lock(mm); - vma = alloc->vma; - } - - if (!vma && need_mm) { - binder_alloc_debug(BINDER_DEBUG_USER_ERROR, - "%d: binder_alloc_buf failed to map pages in userspace, no vma\n", - alloc->pid); - goto err_no_vma; - } - - for (page_addr = start; page_addr < end; page_addr += PAGE_SIZE) { - int ret; + unsigned long index; bool on_lru; - size_t index; + int ret; index = (page_addr - alloc->buffer) / PAGE_SIZE; page = &alloc->pages[index]; @@ -258,26 +272,15 @@ static int binder_allocate_page_range(struct binder_alloc *alloc, continue; } - if (WARN_ON(!vma)) - goto err_page_ptr_cleared; - trace_binder_alloc_page_start(alloc, index); - page->page_ptr = alloc_page(GFP_KERNEL | - __GFP_HIGHMEM | - __GFP_ZERO); - if (!page->page_ptr) { - pr_err("%d: binder_alloc_buf failed for page at %lx\n", - alloc->pid, page_addr); - goto err_alloc_page_failed; - } + page->alloc = alloc; INIT_LIST_HEAD(&page->lru); - ret = vm_insert_page(vma, page_addr, page->page_ptr); + ret = binder_install_single_page(alloc, page, page_addr); if (ret) { - pr_err("%d: binder_alloc_buf failed to map page at %lx in userspace\n", - alloc->pid, page_addr); - goto err_vm_insert_page_failed; + binder_free_page_range(alloc, start, page_addr); + return ret; } if (index + 1 > alloc->pages_high) @@ -285,24 +288,8 @@ static int binder_allocate_page_range(struct binder_alloc *alloc, trace_binder_alloc_page_end(alloc, index); } - if (mm) { - mmap_write_unlock(mm); - mmput_async(mm); - } - return 0; -err_vm_insert_page_failed: - __free_page(page->page_ptr); - page->page_ptr = NULL; -err_alloc_page_failed: -err_page_ptr_cleared: - binder_free_page_range(alloc, start, page_addr); -err_no_vma: - if (mm) { - mmap_write_unlock(mm); - mmput_async(mm); - } - return vma ? -ENOMEM : -ESRCH; + return 0; } static inline void binder_alloc_set_vma(struct binder_alloc *alloc, -- cgit From c7ac30fad18231a1637d38aa8a97d6b4788ed8ad Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:46 +0000 Subject: binder: malloc new_buffer outside of locks Preallocate new_buffer before acquiring the alloc->mutex and hand it down to binder_alloc_new_buf_locked(). The new buffer will be used in the vast majority of requests (measured at 98.2% in field data). The buffer is discarded otherwise. This change is required in preparation for transitioning alloc->mutex into a spinlock in subsequent commits. Signed-off-by: Carlos Llamas Reviewed-by: Alice Ryhl Link: https://lore.kernel.org/r/20231201172212.1813387-18-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 44 ++++++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 21 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 1caf0e3d3451..86f4929a55d5 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -395,24 +395,27 @@ static bool debug_low_async_space_locked(struct binder_alloc *alloc) return false; } +/* Callers preallocate @new_buffer, it is freed by this function if unused */ static struct binder_buffer *binder_alloc_new_buf_locked( struct binder_alloc *alloc, + struct binder_buffer *new_buffer, size_t size, int is_async) { struct rb_node *n = alloc->free_buffers.rb_node; - struct binder_buffer *buffer; - size_t buffer_size; struct rb_node *best_fit = NULL; + struct binder_buffer *buffer; unsigned long has_page_addr; unsigned long end_page_addr; + size_t buffer_size; int ret; if (is_async && alloc->free_async_space < size) { binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, "%d: binder_alloc_buf size %zd failed, no async space left\n", alloc->pid, size); - return ERR_PTR(-ENOSPC); + buffer = ERR_PTR(-ENOSPC); + goto out; } while (n) { @@ -436,7 +439,8 @@ static struct binder_buffer *binder_alloc_new_buf_locked( "%d: binder_alloc_buf size %zd failed, no address space\n", alloc->pid, size); debug_no_space_locked(alloc); - return ERR_PTR(-ENOSPC); + buffer = ERR_PTR(-ENOSPC); + goto out; } if (n == NULL) { @@ -455,22 +459,17 @@ static struct binder_buffer *binder_alloc_new_buf_locked( end_page_addr = has_page_addr; ret = binder_allocate_page_range(alloc, PAGE_ALIGN(buffer->user_data), end_page_addr); - if (ret) - return ERR_PTR(ret); + if (ret) { + buffer = ERR_PTR(ret); + goto out; + } if (buffer_size != size) { - struct binder_buffer *new_buffer; - - new_buffer = kzalloc(sizeof(*buffer), GFP_KERNEL); - if (!new_buffer) { - pr_err("%s: %d failed to alloc new buffer struct\n", - __func__, alloc->pid); - goto err_alloc_buf_struct_failed; - } new_buffer->user_data = buffer->user_data + size; list_add(&new_buffer->entry, &buffer->entry); new_buffer->free = 1; binder_insert_free_buffer(alloc, new_buffer); + new_buffer = NULL; } rb_erase(best_fit, &alloc->free_buffers); @@ -491,12 +490,10 @@ static struct binder_buffer *binder_alloc_new_buf_locked( buffer->oneway_spam_suspect = true; } +out: + /* Discard possibly unused new_buffer */ + kfree(new_buffer); return buffer; - -err_alloc_buf_struct_failed: - binder_free_page_range(alloc, PAGE_ALIGN(buffer->user_data), - end_page_addr); - return ERR_PTR(-ENOMEM); } /* Calculate the sanitized total size, returns 0 for invalid request */ @@ -542,7 +539,7 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, size_t extra_buffers_size, int is_async) { - struct binder_buffer *buffer; + struct binder_buffer *buffer, *next; size_t size; /* Check binder_alloc is fully initialized */ @@ -562,8 +559,13 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, return ERR_PTR(-EINVAL); } + /* Preallocate the next buffer */ + next = kzalloc(sizeof(*next), GFP_KERNEL); + if (!next) + return ERR_PTR(-ENOMEM); + mutex_lock(&alloc->mutex); - buffer = binder_alloc_new_buf_locked(alloc, size, is_async); + buffer = binder_alloc_new_buf_locked(alloc, next, size, is_async); if (IS_ERR(buffer)) { mutex_unlock(&alloc->mutex); goto out; -- cgit From 68aef12d094e4c96d972790f1620415460a4f3cf Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:47 +0000 Subject: binder: initialize lru pages in mmap callback Rather than repeatedly initializing some of the binder_lru_page members during binder_alloc_new_buf(), perform this initialization just once in binder_alloc_mmap_handler(), after the pages have been created. Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-19-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 86f4929a55d5..25efdbb2ad5d 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -274,9 +274,6 @@ static int binder_allocate_page_range(struct binder_alloc *alloc, trace_binder_alloc_page_start(alloc, index); - page->alloc = alloc; - INIT_LIST_HEAD(&page->lru); - ret = binder_install_single_page(alloc, page, page_addr); if (ret) { binder_free_page_range(alloc, start, page_addr); @@ -798,9 +795,9 @@ void binder_alloc_free_buf(struct binder_alloc *alloc, int binder_alloc_mmap_handler(struct binder_alloc *alloc, struct vm_area_struct *vma) { - int ret; - const char *failure_string; struct binder_buffer *buffer; + const char *failure_string; + int ret, i; if (unlikely(vma->vm_mm != alloc->mm)) { ret = -EINVAL; @@ -829,6 +826,11 @@ int binder_alloc_mmap_handler(struct binder_alloc *alloc, goto err_alloc_pages_failed; } + for (i = 0; i < alloc->buffer_size / PAGE_SIZE; i++) { + alloc->pages[i].alloc = alloc; + INIT_LIST_HEAD(&alloc->pages[i].lru); + } + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL); if (!buffer) { ret = -ENOMEM; -- cgit From 37ebbb4f73a0d299fa0c7dd043932a2f5fbbb779 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:48 +0000 Subject: binder: perform page installation outside of locks Split out the insertion of pages to be outside of the alloc->mutex in a separate binder_install_buffer_pages() routine. Since this is no longer serialized, we must look at the full range of pages used by the buffers. The installation is protected with mmap_sem in write mode since multiple tasks might race to install the same page. Besides avoiding unnecessary nested locking this helps in preparation of switching the alloc->mutex into a spinlock_t in subsequent patches. Signed-off-by: Carlos Llamas Reviewed-by: Alice Ryhl Link: https://lore.kernel.org/r/20231201172212.1813387-20-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 101 +++++++++++++++++++++++++++++------------ 1 file changed, 73 insertions(+), 28 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 25efdbb2ad5d..551f08e84408 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -175,6 +175,21 @@ struct binder_buffer *binder_alloc_prepare_to_free(struct binder_alloc *alloc, return buffer; } +static inline void +binder_set_installed_page(struct binder_lru_page *lru_page, + struct page *page) +{ + /* Pairs with acquire in binder_get_installed_page() */ + smp_store_release(&lru_page->page_ptr, page); +} + +static inline struct page * +binder_get_installed_page(struct binder_lru_page *lru_page) +{ + /* Pairs with release in binder_set_installed_page() */ + return smp_load_acquire(&lru_page->page_ptr); +} + static void binder_free_page_range(struct binder_alloc *alloc, unsigned long start, unsigned long end) { @@ -190,6 +205,9 @@ static void binder_free_page_range(struct binder_alloc *alloc, index = (page_addr - alloc->buffer) / PAGE_SIZE; page = &alloc->pages[index]; + if (!binder_get_installed_page(page)) + continue; + trace_binder_free_lru_start(alloc, index); ret = list_lru_add(&binder_alloc_lru, &page->lru); @@ -209,7 +227,14 @@ static int binder_install_single_page(struct binder_alloc *alloc, if (!mmget_not_zero(alloc->mm)) return -ESRCH; + /* + * Protected with mmap_sem in write mode as multiple tasks + * might race to install the same page. + */ mmap_write_lock(alloc->mm); + if (binder_get_installed_page(lru_page)) + goto out; + if (!alloc->vma) { pr_err("%d: %s failed, no vma\n", alloc->pid, __func__); ret = -ESRCH; @@ -232,15 +257,50 @@ static int binder_install_single_page(struct binder_alloc *alloc, goto out; } - lru_page->page_ptr = page; + /* Mark page installation complete and safe to use */ + binder_set_installed_page(lru_page, page); out: mmap_write_unlock(alloc->mm); mmput_async(alloc->mm); return ret; } -static int binder_allocate_page_range(struct binder_alloc *alloc, - unsigned long start, unsigned long end) +static int binder_install_buffer_pages(struct binder_alloc *alloc, + struct binder_buffer *buffer, + size_t size) +{ + struct binder_lru_page *page; + unsigned long start, final; + unsigned long page_addr; + + start = buffer->user_data & PAGE_MASK; + final = PAGE_ALIGN(buffer->user_data + size); + + for (page_addr = start; page_addr < final; page_addr += PAGE_SIZE) { + unsigned long index; + int ret; + + index = (page_addr - alloc->buffer) / PAGE_SIZE; + page = &alloc->pages[index]; + + if (binder_get_installed_page(page)) + continue; + + trace_binder_alloc_page_start(alloc, index); + + ret = binder_install_single_page(alloc, page, page_addr); + if (ret) + return ret; + + trace_binder_alloc_page_end(alloc, index); + } + + return 0; +} + +/* The range of pages should exclude those shared with other buffers */ +static void binder_allocate_page_range(struct binder_alloc *alloc, + unsigned long start, unsigned long end) { struct binder_lru_page *page; unsigned long page_addr; @@ -249,15 +309,11 @@ static int binder_allocate_page_range(struct binder_alloc *alloc, "%d: allocate pages %lx-%lx\n", alloc->pid, start, end); - if (end <= start) - return 0; - trace_binder_update_page_range(alloc, true, start, end); for (page_addr = start; page_addr < end; page_addr += PAGE_SIZE) { unsigned long index; bool on_lru; - int ret; index = (page_addr - alloc->buffer) / PAGE_SIZE; page = &alloc->pages[index]; @@ -272,21 +328,9 @@ static int binder_allocate_page_range(struct binder_alloc *alloc, continue; } - trace_binder_alloc_page_start(alloc, index); - - ret = binder_install_single_page(alloc, page, page_addr); - if (ret) { - binder_free_page_range(alloc, start, page_addr); - return ret; - } - if (index + 1 > alloc->pages_high) alloc->pages_high = index + 1; - - trace_binder_alloc_page_end(alloc, index); } - - return 0; } static inline void binder_alloc_set_vma(struct binder_alloc *alloc, @@ -405,7 +449,6 @@ static struct binder_buffer *binder_alloc_new_buf_locked( unsigned long has_page_addr; unsigned long end_page_addr; size_t buffer_size; - int ret; if (is_async && alloc->free_async_space < size) { binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, @@ -449,18 +492,14 @@ static struct binder_buffer *binder_alloc_new_buf_locked( "%d: binder_alloc_buf size %zd got buffer %pK size %zd\n", alloc->pid, size, buffer, buffer_size); - has_page_addr = (buffer->user_data + buffer_size) & PAGE_MASK; WARN_ON(n && buffer_size != size); + + has_page_addr = (buffer->user_data + buffer_size) & PAGE_MASK; end_page_addr = PAGE_ALIGN(buffer->user_data + size); if (end_page_addr > has_page_addr) end_page_addr = has_page_addr; - ret = binder_allocate_page_range(alloc, PAGE_ALIGN(buffer->user_data), - end_page_addr); - if (ret) { - buffer = ERR_PTR(ret); - goto out; - } - + binder_allocate_page_range(alloc, PAGE_ALIGN(buffer->user_data), + end_page_addr); if (buffer_size != size) { new_buffer->user_data = buffer->user_data + size; list_add(&new_buffer->entry, &buffer->entry); @@ -538,6 +577,7 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, { struct binder_buffer *buffer, *next; size_t size; + int ret; /* Check binder_alloc is fully initialized */ if (!binder_alloc_get_vma(alloc)) { @@ -574,6 +614,11 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, buffer->pid = current->tgid; mutex_unlock(&alloc->mutex); + ret = binder_install_buffer_pages(alloc, buffer, size); + if (ret) { + binder_alloc_free_buf(alloc, buffer); + buffer = ERR_PTR(ret); + } out: return buffer; } -- cgit From 258ce20ede33c551002705fa1488864fb287752c Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:49 +0000 Subject: binder: remove redundant debug log The debug information in this statement is already logged earlier in the same function. We can get rid of this duplicate log. Signed-off-by: Carlos Llamas Reviewed-by: Alice Ryhl Link: https://lore.kernel.org/r/20231201172212.1813387-21-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 3 --- 1 file changed, 3 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 551f08e84408..c9292eee8fee 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -512,9 +512,6 @@ static struct binder_buffer *binder_alloc_new_buf_locked( buffer->free = 0; buffer->allow_user_free = 0; binder_insert_allocated_buffer_locked(alloc, buffer); - binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: binder_alloc_buf size %zd got %pK\n", - alloc->pid, size, buffer); buffer->async_transaction = is_async; buffer->oneway_spam_suspect = false; if (is_async) { -- cgit From de0e6573125f8ea7a01a9b05a45b0c73116c73b2 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:50 +0000 Subject: binder: make oversized buffer code more readable The sections in binder_alloc_new_buf_locked() dealing with oversized buffers are scattered which makes them difficult to read. Instead, consolidate this code into a single block to improve readability. No functional change here. Signed-off-by: Carlos Llamas Reviewed-by: Alice Ryhl Link: https://lore.kernel.org/r/20231201172212.1813387-22-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index c9292eee8fee..ad9b73c6ddb7 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -483,32 +483,31 @@ static struct binder_buffer *binder_alloc_new_buf_locked( goto out; } - if (n == NULL) { + if (buffer_size != size) { + /* Found an oversized buffer and needs to be split */ buffer = rb_entry(best_fit, struct binder_buffer, rb_node); buffer_size = binder_alloc_buffer_size(alloc, buffer); + + WARN_ON(n || buffer_size == size); + new_buffer->user_data = buffer->user_data + size; + list_add(&new_buffer->entry, &buffer->entry); + new_buffer->free = 1; + binder_insert_free_buffer(alloc, new_buffer); + new_buffer = NULL; } binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, "%d: binder_alloc_buf size %zd got buffer %pK size %zd\n", alloc->pid, size, buffer, buffer_size); - WARN_ON(n && buffer_size != size); - has_page_addr = (buffer->user_data + buffer_size) & PAGE_MASK; end_page_addr = PAGE_ALIGN(buffer->user_data + size); if (end_page_addr > has_page_addr) end_page_addr = has_page_addr; binder_allocate_page_range(alloc, PAGE_ALIGN(buffer->user_data), end_page_addr); - if (buffer_size != size) { - new_buffer->user_data = buffer->user_data + size; - list_add(&new_buffer->entry, &buffer->entry); - new_buffer->free = 1; - binder_insert_free_buffer(alloc, new_buffer); - new_buffer = NULL; - } - rb_erase(best_fit, &alloc->free_buffers); + rb_erase(&buffer->rb_node, &alloc->free_buffers); buffer->free = 0; buffer->allow_user_free = 0; binder_insert_allocated_buffer_locked(alloc, buffer); -- cgit From ea9cdbf0c7273b55e251b2ed8f85794cfadab5d5 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:51 +0000 Subject: binder: rename lru shrinker utilities Now that the page allocation step is done separately we should rename the binder_free_page_range() and binder_allocate_page_range() functions to provide a more accurate description of what they do. Lets borrow the freelist concept used in other parts of the kernel for this. No functional change here. Signed-off-by: Carlos Llamas Reviewed-by: Alice Ryhl Link: https://lore.kernel.org/r/20231201172212.1813387-23-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 40 ++++++++++++++++----------------- drivers/android/binder_alloc.h | 4 ++-- drivers/android/binder_alloc_selftest.c | 6 ++--- 3 files changed, 25 insertions(+), 25 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index ad9b73c6ddb7..052a8c3b0ce1 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -26,7 +26,7 @@ #include "binder_alloc.h" #include "binder_trace.h" -struct list_lru binder_alloc_lru; +struct list_lru binder_freelist; static DEFINE_MUTEX(binder_alloc_mmap_lock); @@ -190,8 +190,8 @@ binder_get_installed_page(struct binder_lru_page *lru_page) return smp_load_acquire(&lru_page->page_ptr); } -static void binder_free_page_range(struct binder_alloc *alloc, - unsigned long start, unsigned long end) +static void binder_lru_freelist_add(struct binder_alloc *alloc, + unsigned long start, unsigned long end) { struct binder_lru_page *page; unsigned long page_addr; @@ -210,7 +210,7 @@ static void binder_free_page_range(struct binder_alloc *alloc, trace_binder_free_lru_start(alloc, index); - ret = list_lru_add(&binder_alloc_lru, &page->lru); + ret = list_lru_add(&binder_freelist, &page->lru); WARN_ON(!ret); trace_binder_free_lru_end(alloc, index); @@ -299,14 +299,14 @@ static int binder_install_buffer_pages(struct binder_alloc *alloc, } /* The range of pages should exclude those shared with other buffers */ -static void binder_allocate_page_range(struct binder_alloc *alloc, - unsigned long start, unsigned long end) +static void binder_lru_freelist_del(struct binder_alloc *alloc, + unsigned long start, unsigned long end) { struct binder_lru_page *page; unsigned long page_addr; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: allocate pages %lx-%lx\n", + "%d: pages %lx-%lx\n", alloc->pid, start, end); trace_binder_update_page_range(alloc, true, start, end); @@ -321,7 +321,7 @@ static void binder_allocate_page_range(struct binder_alloc *alloc, if (page->page_ptr) { trace_binder_alloc_lru_start(alloc, index); - on_lru = list_lru_del(&binder_alloc_lru, &page->lru); + on_lru = list_lru_del(&binder_freelist, &page->lru); WARN_ON(!on_lru); trace_binder_alloc_lru_end(alloc, index); @@ -504,8 +504,8 @@ static struct binder_buffer *binder_alloc_new_buf_locked( end_page_addr = PAGE_ALIGN(buffer->user_data + size); if (end_page_addr > has_page_addr) end_page_addr = has_page_addr; - binder_allocate_page_range(alloc, PAGE_ALIGN(buffer->user_data), - end_page_addr); + binder_lru_freelist_del(alloc, PAGE_ALIGN(buffer->user_data), + end_page_addr); rb_erase(&buffer->rb_node, &alloc->free_buffers); buffer->free = 0; @@ -671,8 +671,8 @@ static void binder_delete_free_buffer(struct binder_alloc *alloc, alloc->pid, buffer->user_data, prev->user_data, next ? next->user_data : 0); - binder_free_page_range(alloc, buffer_start_page(buffer), - buffer_start_page(buffer) + PAGE_SIZE); + binder_lru_freelist_add(alloc, buffer_start_page(buffer), + buffer_start_page(buffer) + PAGE_SIZE); } list_del(&buffer->entry); kfree(buffer); @@ -706,8 +706,8 @@ static void binder_free_buf_locked(struct binder_alloc *alloc, alloc->pid, size, alloc->free_async_space); } - binder_free_page_range(alloc, PAGE_ALIGN(buffer->user_data), - (buffer->user_data + buffer_size) & PAGE_MASK); + binder_lru_freelist_add(alloc, PAGE_ALIGN(buffer->user_data), + (buffer->user_data + buffer_size) & PAGE_MASK); rb_erase(&buffer->rb_node, &alloc->allocated_buffers); buffer->free = 1; @@ -953,7 +953,7 @@ void binder_alloc_deferred_release(struct binder_alloc *alloc) if (!alloc->pages[i].page_ptr) continue; - on_lru = list_lru_del(&binder_alloc_lru, + on_lru = list_lru_del(&binder_freelist, &alloc->pages[i].lru); page_addr = alloc->buffer + i * PAGE_SIZE; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, @@ -1152,13 +1152,13 @@ err_get_alloc_mutex_failed: static unsigned long binder_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { - return list_lru_count(&binder_alloc_lru); + return list_lru_count(&binder_freelist); } static unsigned long binder_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) { - return list_lru_walk(&binder_alloc_lru, binder_alloc_free_page, + return list_lru_walk(&binder_freelist, binder_alloc_free_page, NULL, sc->nr_to_scan); } @@ -1184,13 +1184,13 @@ int binder_alloc_shrinker_init(void) { int ret; - ret = list_lru_init(&binder_alloc_lru); + ret = list_lru_init(&binder_freelist); if (ret) return ret; binder_shrinker = shrinker_alloc(0, "android-binder"); if (!binder_shrinker) { - list_lru_destroy(&binder_alloc_lru); + list_lru_destroy(&binder_freelist); return -ENOMEM; } @@ -1205,7 +1205,7 @@ int binder_alloc_shrinker_init(void) void binder_alloc_shrinker_exit(void) { shrinker_free(binder_shrinker); - list_lru_destroy(&binder_alloc_lru); + list_lru_destroy(&binder_freelist); } /** diff --git a/drivers/android/binder_alloc.h b/drivers/android/binder_alloc.h index bbc16bc6d5ac..a5181916942e 100644 --- a/drivers/android/binder_alloc.h +++ b/drivers/android/binder_alloc.h @@ -15,7 +15,7 @@ #include #include -extern struct list_lru binder_alloc_lru; +extern struct list_lru binder_freelist; struct binder_transaction; /** @@ -61,7 +61,7 @@ struct binder_buffer { /** * struct binder_lru_page - page object used for binder shrinker * @page_ptr: pointer to physical page in mmap'd space - * @lru: entry in binder_alloc_lru + * @lru: entry in binder_freelist * @alloc: binder_alloc for a proc */ struct binder_lru_page { diff --git a/drivers/android/binder_alloc_selftest.c b/drivers/android/binder_alloc_selftest.c index ed753747e54c..fba7ab6ca451 100644 --- a/drivers/android/binder_alloc_selftest.c +++ b/drivers/android/binder_alloc_selftest.c @@ -158,8 +158,8 @@ static void binder_selftest_free_page(struct binder_alloc *alloc) int i; unsigned long count; - while ((count = list_lru_count(&binder_alloc_lru))) { - list_lru_walk(&binder_alloc_lru, binder_alloc_free_page, + while ((count = list_lru_count(&binder_freelist))) { + list_lru_walk(&binder_freelist, binder_alloc_free_page, NULL, count); } @@ -183,7 +183,7 @@ static void binder_selftest_alloc_free(struct binder_alloc *alloc, /* Allocate from lru. */ binder_selftest_alloc_buf(alloc, buffers, sizes, seq); - if (list_lru_count(&binder_alloc_lru)) + if (list_lru_count(&binder_freelist)) pr_err("lru list should be empty but is not\n"); binder_selftest_free_buf(alloc, buffers, sizes, seq, end); -- cgit From 67dcc880780569ec40391cae4d8299adc1e7a44e Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:52 +0000 Subject: binder: document the final page calculation The code to determine the page range for binder_lru_freelist_del() is quite obscure. It leverages the buffer_size calculated before doing an oversized buffer split. This is used to figure out if the last page is being shared with another active buffer. If so, the page gets trimmed out of the range as it has been previously removed from the freelist. This would be equivalent to getting the start page of the next in-use buffer explicitly. However, the code for this is much larger as we can see in binder_free_buf_locked() routine. Instead, lets settle on documenting the tricky step and using better names for now. I believe an ideal solution would be to count the binder_page->users to determine when a page should be added or removed from the freelist. However, this is a much bigger change than what I'm willing to risk at this time. Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-24-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 052a8c3b0ce1..edd9714ec9f5 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -446,8 +446,8 @@ static struct binder_buffer *binder_alloc_new_buf_locked( struct rb_node *n = alloc->free_buffers.rb_node; struct rb_node *best_fit = NULL; struct binder_buffer *buffer; - unsigned long has_page_addr; - unsigned long end_page_addr; + unsigned long next_used_page; + unsigned long curr_last_page; size_t buffer_size; if (is_async && alloc->free_async_space < size) { @@ -500,12 +500,16 @@ static struct binder_buffer *binder_alloc_new_buf_locked( "%d: binder_alloc_buf size %zd got buffer %pK size %zd\n", alloc->pid, size, buffer, buffer_size); - has_page_addr = (buffer->user_data + buffer_size) & PAGE_MASK; - end_page_addr = PAGE_ALIGN(buffer->user_data + size); - if (end_page_addr > has_page_addr) - end_page_addr = has_page_addr; + /* + * Now we remove the pages from the freelist. A clever calculation + * with buffer_size determines if the last page is shared with an + * adjacent in-use buffer. In such case, the page has been already + * removed from the freelist so we trim our range short. + */ + next_used_page = (buffer->user_data + buffer_size) & PAGE_MASK; + curr_last_page = PAGE_ALIGN(buffer->user_data + size); binder_lru_freelist_del(alloc, PAGE_ALIGN(buffer->user_data), - end_page_addr); + min(next_used_page, curr_last_page)); rb_erase(&buffer->rb_node, &alloc->free_buffers); buffer->free = 0; -- cgit From 8e905217c4543af9cf1754809846157a7dbbb261 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:53 +0000 Subject: binder: collapse print_binder_buffer() into caller The code in print_binder_buffer() is quite small so it can be collapsed into its single caller binder_alloc_print_allocated(). No functional change in this patch. Signed-off-by: Carlos Llamas Reviewed-by: Alice Ryhl Link: https://lore.kernel.org/r/20231201172212.1813387-25-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index edd9714ec9f5..60c829506d31 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -978,16 +978,6 @@ void binder_alloc_deferred_release(struct binder_alloc *alloc) __func__, alloc->pid, buffers, page_count); } -static void print_binder_buffer(struct seq_file *m, const char *prefix, - struct binder_buffer *buffer) -{ - seq_printf(m, "%s %d: %lx size %zd:%zd:%zd %s\n", - prefix, buffer->debug_id, buffer->user_data, - buffer->data_size, buffer->offsets_size, - buffer->extra_buffers_size, - buffer->transaction ? "active" : "delivered"); -} - /** * binder_alloc_print_allocated() - print buffer info * @m: seq_file for output via seq_printf() @@ -999,12 +989,18 @@ static void print_binder_buffer(struct seq_file *m, const char *prefix, void binder_alloc_print_allocated(struct seq_file *m, struct binder_alloc *alloc) { + struct binder_buffer *buffer; struct rb_node *n; mutex_lock(&alloc->mutex); - for (n = rb_first(&alloc->allocated_buffers); n != NULL; n = rb_next(n)) - print_binder_buffer(m, " buffer", - rb_entry(n, struct binder_buffer, rb_node)); + for (n = rb_first(&alloc->allocated_buffers); n; n = rb_next(n)) { + buffer = rb_entry(n, struct binder_buffer, rb_node); + seq_printf(m, " buffer %d: %lx size %zd:%zd:%zd %s\n", + buffer->debug_id, buffer->user_data, + buffer->data_size, buffer->offsets_size, + buffer->extra_buffers_size, + buffer->transaction ? "active" : "delivered"); + } mutex_unlock(&alloc->mutex); } -- cgit From f07b83a48e944c8a1cc1e9f6703fae5e34df2ba4 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:54 +0000 Subject: binder: refactor binder_delete_free_buffer() Skip the freelist call immediately as needed, instead of continuing the pointless checks. Also, drop the debug logs that we don't really need. Signed-off-by: Carlos Llamas Reviewed-by: Alice Ryhl Link: https://lore.kernel.org/r/20231201172212.1813387-26-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 44 +++++++++++------------------------------- 1 file changed, 11 insertions(+), 33 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 60c829506d31..c3fc90966867 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -636,48 +636,26 @@ static unsigned long prev_buffer_end_page(struct binder_buffer *buffer) static void binder_delete_free_buffer(struct binder_alloc *alloc, struct binder_buffer *buffer) { - struct binder_buffer *prev, *next = NULL; - bool to_free = true; + struct binder_buffer *prev, *next; + + if (PAGE_ALIGNED(buffer->user_data)) + goto skip_freelist; BUG_ON(alloc->buffers.next == &buffer->entry); prev = binder_buffer_prev(buffer); BUG_ON(!prev->free); - if (prev_buffer_end_page(prev) == buffer_start_page(buffer)) { - to_free = false; - binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: merge free, buffer %lx share page with %lx\n", - alloc->pid, buffer->user_data, - prev->user_data); - } + if (prev_buffer_end_page(prev) == buffer_start_page(buffer)) + goto skip_freelist; if (!list_is_last(&buffer->entry, &alloc->buffers)) { next = binder_buffer_next(buffer); - if (buffer_start_page(next) == buffer_start_page(buffer)) { - to_free = false; - binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: merge free, buffer %lx share page with %lx\n", - alloc->pid, - buffer->user_data, - next->user_data); - } + if (buffer_start_page(next) == buffer_start_page(buffer)) + goto skip_freelist; } - if (PAGE_ALIGNED(buffer->user_data)) { - binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: merge free, buffer start %lx is page aligned\n", - alloc->pid, buffer->user_data); - to_free = false; - } - - if (to_free) { - binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: merge free, buffer %lx do not share page with %lx or %lx\n", - alloc->pid, buffer->user_data, - prev->user_data, - next ? next->user_data : 0); - binder_lru_freelist_add(alloc, buffer_start_page(buffer), - buffer_start_page(buffer) + PAGE_SIZE); - } + binder_lru_freelist_add(alloc, buffer_start_page(buffer), + buffer_start_page(buffer) + PAGE_SIZE); +skip_freelist: list_del(&buffer->entry); kfree(buffer); } -- cgit From 162c79731448a5a052e93af7753df579dfe0bf7a Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:55 +0000 Subject: binder: avoid user addresses in debug logs Prefer logging vma offsets instead of addresses or simply drop the debug log altogether if not useful. Note this covers the instances affected by the switch to store addresses as unsigned long. However, there are other sections in the driver that could do the same. Signed-off-by: Carlos Llamas Reviewed-by: Alice Ryhl Link: https://lore.kernel.org/r/20231201172212.1813387-27-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder.c | 4 ++-- drivers/android/binder_alloc.c | 15 ++++++--------- 2 files changed, 8 insertions(+), 11 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder.c b/drivers/android/binder.c index 45674af6310f..c4bb18305e77 100644 --- a/drivers/android/binder.c +++ b/drivers/android/binder.c @@ -5980,9 +5980,9 @@ static void print_binder_transaction_ilocked(struct seq_file *m, } if (buffer->target_node) seq_printf(m, " node %d", buffer->target_node->debug_id); - seq_printf(m, " size %zd:%zd data %lx\n", + seq_printf(m, " size %zd:%zd offset %lx\n", buffer->data_size, buffer->offsets_size, - buffer->user_data); + proc->alloc.buffer - buffer->user_data); } static void print_binder_work_ilocked(struct seq_file *m, diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index c3fc90966867..5783675f2970 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -250,8 +250,8 @@ static int binder_install_single_page(struct binder_alloc *alloc, ret = vm_insert_page(alloc->vma, addr, page); if (ret) { - pr_err("%d: %s failed to insert page at %lx with %d\n", - alloc->pid, __func__, addr, ret); + pr_err("%d: %s failed to insert page at offset %lx with %d\n", + alloc->pid, __func__, addr - alloc->buffer, ret); __free_page(page); ret = -ENOMEM; goto out; @@ -305,10 +305,6 @@ static void binder_lru_freelist_del(struct binder_alloc *alloc, struct binder_lru_page *page; unsigned long page_addr; - binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%d: pages %lx-%lx\n", - alloc->pid, start, end); - trace_binder_update_page_range(alloc, true, start, end); for (page_addr = start; page_addr < end; page_addr += PAGE_SIZE) { @@ -939,8 +935,8 @@ void binder_alloc_deferred_release(struct binder_alloc *alloc) &alloc->pages[i].lru); page_addr = alloc->buffer + i * PAGE_SIZE; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, - "%s: %d: page %d at %lx %s\n", - __func__, alloc->pid, i, page_addr, + "%s: %d: page %d %s\n", + __func__, alloc->pid, i, on_lru ? "on lru" : "active"); __free_page(alloc->pages[i].page_ptr); page_count++; @@ -974,7 +970,8 @@ void binder_alloc_print_allocated(struct seq_file *m, for (n = rb_first(&alloc->allocated_buffers); n; n = rb_next(n)) { buffer = rb_entry(n, struct binder_buffer, rb_node); seq_printf(m, " buffer %d: %lx size %zd:%zd:%zd %s\n", - buffer->debug_id, buffer->user_data, + buffer->debug_id, + buffer->user_data - alloc->buffer, buffer->data_size, buffer->offsets_size, buffer->extra_buffers_size, buffer->transaction ? "active" : "delivered"); -- cgit From e50f4e6cc9bfaca655d3b6a3506d27cf2caa1d40 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:56 +0000 Subject: binder: reverse locking order in shrinker callback The locking order currently requires the alloc->mutex to be acquired first followed by the mmap lock. However, the alloc->mutex is converted into a spinlock in subsequent commits so the order needs to be reversed to avoid nesting the sleeping mmap lock under the spinlock. The shrinker's callback binder_alloc_free_page() is the only place that needs to be reordered since other functions have been refactored and no longer nest these locks. Some minor cosmetic changes are also included in this patch. Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-28-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 46 ++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 24 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 5783675f2970..a3e56637db4f 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -1061,35 +1061,39 @@ enum lru_status binder_alloc_free_page(struct list_head *item, void *cb_arg) __must_hold(lock) { - struct mm_struct *mm = NULL; - struct binder_lru_page *page = container_of(item, - struct binder_lru_page, - lru); - struct binder_alloc *alloc; + struct binder_lru_page *page = container_of(item, typeof(*page), lru); + struct binder_alloc *alloc = page->alloc; + struct mm_struct *mm = alloc->mm; + struct vm_area_struct *vma; + struct page *page_to_free; unsigned long page_addr; size_t index; - struct vm_area_struct *vma; - alloc = page->alloc; + if (!mmget_not_zero(mm)) + goto err_mmget; + if (!mmap_read_trylock(mm)) + goto err_mmap_read_lock_failed; if (!mutex_trylock(&alloc->mutex)) goto err_get_alloc_mutex_failed; - if (!page->page_ptr) goto err_page_already_freed; index = page - alloc->pages; page_addr = alloc->buffer + index * PAGE_SIZE; - mm = alloc->mm; - if (!mmget_not_zero(mm)) - goto err_mmget; - if (!mmap_read_trylock(mm)) - goto err_mmap_read_lock_failed; vma = vma_lookup(mm, page_addr); if (vma && vma != binder_alloc_get_vma(alloc)) goto err_invalid_vma; + trace_binder_unmap_kernel_start(alloc, index); + + page_to_free = page->page_ptr; + page->page_ptr = NULL; + + trace_binder_unmap_kernel_end(alloc, index); + list_lru_isolate(lru, item); + mutex_unlock(&alloc->mutex); spin_unlock(lock); if (vma) { @@ -1099,28 +1103,22 @@ enum lru_status binder_alloc_free_page(struct list_head *item, trace_binder_unmap_user_end(alloc, index); } + mmap_read_unlock(mm); mmput_async(mm); - - trace_binder_unmap_kernel_start(alloc, index); - - __free_page(page->page_ptr); - page->page_ptr = NULL; - - trace_binder_unmap_kernel_end(alloc, index); + __free_page(page_to_free); spin_lock(lock); - mutex_unlock(&alloc->mutex); return LRU_REMOVED_RETRY; err_invalid_vma: +err_page_already_freed: + mutex_unlock(&alloc->mutex); +err_get_alloc_mutex_failed: mmap_read_unlock(mm); err_mmap_read_lock_failed: mmput_async(mm); err_mmget: -err_page_already_freed: - mutex_unlock(&alloc->mutex); -err_get_alloc_mutex_failed: return LRU_SKIP; } -- cgit From 7710e2cca32e7f3958480e8bd44f50e29d0c2509 Mon Sep 17 00:00:00 2001 From: Carlos Llamas Date: Fri, 1 Dec 2023 17:21:57 +0000 Subject: binder: switch alloc->mutex to spinlock_t The alloc->mutex is a highly contended lock that causes performance issues on Android devices. When a low-priority task is given this lock and it sleeps, it becomes difficult for the task to wake up and complete its work. This delays other tasks that are also waiting on the mutex. The problem gets worse when there is memory pressure in the system, because this increases the contention on the alloc->mutex while the shrinker reclaims binder pages. Switching to a spinlock helps to keep the waiters running and avoids the overhead of waking up tasks. This significantly improves the transaction latency when the problematic scenario occurs. The performance impact of this patchset was measured by stress-testing the binder alloc contention. In this test, several clients of different priorities send thousands of transactions of different sizes to a single server. In parallel, pages get reclaimed using the shinker's debugfs. The test was run on a Pixel 8, Pixel 6 and qemu machine. The results were similar on all three devices: after: | sched | prio | average | max | min | |--------+------+---------+-----------+---------| | fifo | 99 | 0.135ms | 1.197ms | 0.022ms | | fifo | 01 | 0.136ms | 5.232ms | 0.018ms | | other | -20 | 0.180ms | 7.403ms | 0.019ms | | other | 19 | 0.241ms | 58.094ms | 0.018ms | before: | sched | prio | average | max | min | |--------+------+---------+-----------+---------| | fifo | 99 | 0.350ms | 248.730ms | 0.020ms | | fifo | 01 | 0.357ms | 248.817ms | 0.024ms | | other | -20 | 0.399ms | 249.906ms | 0.020ms | | other | 19 | 0.477ms | 297.756ms | 0.022ms | The key metrics above are the average and max latencies (wall time). These improvements should roughly translate to p95-p99 latencies on real workloads. The response time is up to 200x faster in these scenarios and there is no penalty in the regular path. Note that it is only possible to convert this lock after a series of changes made by previous patches. These mainly include refactoring the sections that might_sleep() and changing the locking order with the mmap_lock amongst others. Reviewed-by: Alice Ryhl Signed-off-by: Carlos Llamas Link: https://lore.kernel.org/r/20231201172212.1813387-29-cmllamas@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc.c | 46 +++++++++++++++++++++--------------------- drivers/android/binder_alloc.h | 10 ++++----- 2 files changed, 28 insertions(+), 28 deletions(-) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index a3e56637db4f..a4a4dc87ba53 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -169,9 +169,9 @@ struct binder_buffer *binder_alloc_prepare_to_free(struct binder_alloc *alloc, { struct binder_buffer *buffer; - mutex_lock(&alloc->mutex); + spin_lock(&alloc->lock); buffer = binder_alloc_prepare_to_free_locked(alloc, user_ptr); - mutex_unlock(&alloc->mutex); + spin_unlock(&alloc->lock); return buffer; } @@ -597,10 +597,10 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, if (!next) return ERR_PTR(-ENOMEM); - mutex_lock(&alloc->mutex); + spin_lock(&alloc->lock); buffer = binder_alloc_new_buf_locked(alloc, next, size, is_async); if (IS_ERR(buffer)) { - mutex_unlock(&alloc->mutex); + spin_unlock(&alloc->lock); goto out; } @@ -608,7 +608,7 @@ struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, buffer->offsets_size = offsets_size; buffer->extra_buffers_size = extra_buffers_size; buffer->pid = current->tgid; - mutex_unlock(&alloc->mutex); + spin_unlock(&alloc->lock); ret = binder_install_buffer_pages(alloc, buffer, size); if (ret) { @@ -785,17 +785,17 @@ void binder_alloc_free_buf(struct binder_alloc *alloc, * We could eliminate the call to binder_alloc_clear_buf() * from binder_alloc_deferred_release() by moving this to * binder_free_buf_locked(). However, that could - * increase contention for the alloc mutex if clear_on_free - * is used frequently for large buffers. The mutex is not + * increase contention for the alloc->lock if clear_on_free + * is used frequently for large buffers. This lock is not * needed for correctness here. */ if (buffer->clear_on_free) { binder_alloc_clear_buf(alloc, buffer); buffer->clear_on_free = false; } - mutex_lock(&alloc->mutex); + spin_lock(&alloc->lock); binder_free_buf_locked(alloc, buffer); - mutex_unlock(&alloc->mutex); + spin_unlock(&alloc->lock); } /** @@ -893,7 +893,7 @@ void binder_alloc_deferred_release(struct binder_alloc *alloc) struct binder_buffer *buffer; buffers = 0; - mutex_lock(&alloc->mutex); + spin_lock(&alloc->lock); BUG_ON(alloc->vma); while ((n = rb_first(&alloc->allocated_buffers))) { @@ -943,7 +943,7 @@ void binder_alloc_deferred_release(struct binder_alloc *alloc) } kfree(alloc->pages); } - mutex_unlock(&alloc->mutex); + spin_unlock(&alloc->lock); if (alloc->mm) mmdrop(alloc->mm); @@ -966,7 +966,7 @@ void binder_alloc_print_allocated(struct seq_file *m, struct binder_buffer *buffer; struct rb_node *n; - mutex_lock(&alloc->mutex); + spin_lock(&alloc->lock); for (n = rb_first(&alloc->allocated_buffers); n; n = rb_next(n)) { buffer = rb_entry(n, struct binder_buffer, rb_node); seq_printf(m, " buffer %d: %lx size %zd:%zd:%zd %s\n", @@ -976,7 +976,7 @@ void binder_alloc_print_allocated(struct seq_file *m, buffer->extra_buffers_size, buffer->transaction ? "active" : "delivered"); } - mutex_unlock(&alloc->mutex); + spin_unlock(&alloc->lock); } /** @@ -993,7 +993,7 @@ void binder_alloc_print_pages(struct seq_file *m, int lru = 0; int free = 0; - mutex_lock(&alloc->mutex); + spin_lock(&alloc->lock); /* * Make sure the binder_alloc is fully initialized, otherwise we might * read inconsistent state. @@ -1009,7 +1009,7 @@ void binder_alloc_print_pages(struct seq_file *m, lru++; } } - mutex_unlock(&alloc->mutex); + spin_unlock(&alloc->lock); seq_printf(m, " pages: %d:%d:%d\n", active, lru, free); seq_printf(m, " pages high watermark: %zu\n", alloc->pages_high); } @@ -1025,10 +1025,10 @@ int binder_alloc_get_allocated_count(struct binder_alloc *alloc) struct rb_node *n; int count = 0; - mutex_lock(&alloc->mutex); + spin_lock(&alloc->lock); for (n = rb_first(&alloc->allocated_buffers); n != NULL; n = rb_next(n)) count++; - mutex_unlock(&alloc->mutex); + spin_unlock(&alloc->lock); return count; } @@ -1073,8 +1073,8 @@ enum lru_status binder_alloc_free_page(struct list_head *item, goto err_mmget; if (!mmap_read_trylock(mm)) goto err_mmap_read_lock_failed; - if (!mutex_trylock(&alloc->mutex)) - goto err_get_alloc_mutex_failed; + if (!spin_trylock(&alloc->lock)) + goto err_get_alloc_lock_failed; if (!page->page_ptr) goto err_page_already_freed; @@ -1093,7 +1093,7 @@ enum lru_status binder_alloc_free_page(struct list_head *item, trace_binder_unmap_kernel_end(alloc, index); list_lru_isolate(lru, item); - mutex_unlock(&alloc->mutex); + spin_unlock(&alloc->lock); spin_unlock(lock); if (vma) { @@ -1113,8 +1113,8 @@ enum lru_status binder_alloc_free_page(struct list_head *item, err_invalid_vma: err_page_already_freed: - mutex_unlock(&alloc->mutex); -err_get_alloc_mutex_failed: + spin_unlock(&alloc->lock); +err_get_alloc_lock_failed: mmap_read_unlock(mm); err_mmap_read_lock_failed: mmput_async(mm); @@ -1149,7 +1149,7 @@ void binder_alloc_init(struct binder_alloc *alloc) alloc->pid = current->group_leader->pid; alloc->mm = current->mm; mmgrab(alloc->mm); - mutex_init(&alloc->mutex); + spin_lock_init(&alloc->lock); INIT_LIST_HEAD(&alloc->buffers); } diff --git a/drivers/android/binder_alloc.h b/drivers/android/binder_alloc.h index a5181916942e..70387234477e 100644 --- a/drivers/android/binder_alloc.h +++ b/drivers/android/binder_alloc.h @@ -9,7 +9,7 @@ #include #include #include -#include +#include #include #include #include @@ -72,7 +72,7 @@ struct binder_lru_page { /** * struct binder_alloc - per-binder proc state for binder allocator - * @mutex: protects binder_alloc fields + * @lock: protects binder_alloc fields * @vma: vm_area_struct passed to mmap_handler * (invariant after mmap) * @mm: copy of task->mm (invariant after open) @@ -96,7 +96,7 @@ struct binder_lru_page { * struct binder_buffer objects used to track the user buffers */ struct binder_alloc { - struct mutex mutex; + spinlock_t lock; struct vm_area_struct *vma; struct mm_struct *mm; unsigned long buffer; @@ -153,9 +153,9 @@ binder_alloc_get_free_async_space(struct binder_alloc *alloc) { size_t free_async_space; - mutex_lock(&alloc->mutex); + spin_lock(&alloc->lock); free_async_space = alloc->free_async_space; - mutex_unlock(&alloc->mutex); + spin_unlock(&alloc->lock); return free_async_space; } -- cgit From 96d1d578dec1e69ed086dd67f7e5974875d981b3 Mon Sep 17 00:00:00 2001 From: Randy Dunlap Date: Tue, 5 Dec 2023 14:53:18 -0800 Subject: android: binder: fix a kernel-doc enum warning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add kernel-doc notation for @LOOP_END to prevent a kernel-doc warning. binder_alloc_selftest.c:76: warning: Enum value 'LOOP_END' not described in enum 'buf_end_align_type' Signed-off-by: Randy Dunlap Cc: Greg Kroah-Hartman Cc: Arve Hjønnevåg Cc: Todd Kjos Cc: Martijn Coenen Cc: Joel Fernandes Cc: Christian Brauner Cc: Carlos Llamas Cc: Suren Baghdasaryan Acked-by: Carlos Llamas Link: https://lore.kernel.org/r/20231205225324.32362-1-rdunlap@infradead.org Signed-off-by: Greg Kroah-Hartman --- drivers/android/binder_alloc_selftest.c | 4 ++++ 1 file changed, 4 insertions(+) (limited to 'drivers/android') diff --git a/drivers/android/binder_alloc_selftest.c b/drivers/android/binder_alloc_selftest.c index fba7ab6ca451..81442fe20a69 100644 --- a/drivers/android/binder_alloc_selftest.c +++ b/drivers/android/binder_alloc_selftest.c @@ -72,6 +72,10 @@ enum buf_end_align_type { * buf1 ]|[ buf2 | buf2 | buf2 ][ ... */ NEXT_NEXT_UNALIGNED, + /** + * @LOOP_END: The number of enum values in &buf_end_align_type. + * It is used for controlling loop termination. + */ LOOP_END, }; -- cgit From 54ffdab820801372f3632bb86613407df34b921b Mon Sep 17 00:00:00 2001 From: Tanzir Hasan Date: Tue, 26 Dec 2023 17:09:23 +0000 Subject: android: binder: binderfs.c: removed asm-generic/errno-base.h asm-generic/errno-base.h can be replaced by linux/errno.h and the file will still build correctly. It is an asm-generic file which should be avoided if possible. Suggested-by: Al Viro Signed-off-by: Tanzir Hasan Acked-by: Carlos Llamas Link: https://lore.kernel.org/r/20231226-binderfs-v1-1-66829e92b523@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binderfs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'drivers/android') diff --git a/drivers/android/binderfs.c b/drivers/android/binderfs.c index 1224ab7aa070..d04ff6029480 100644 --- a/drivers/android/binderfs.c +++ b/drivers/android/binderfs.c @@ -29,7 +29,7 @@ #include #include #include -#include +#include #include #include -- cgit From 5850edccec30325707f953bc088497b3b9041231 Mon Sep 17 00:00:00 2001 From: Tanzir Hasan Date: Thu, 4 Jan 2024 19:31:36 +0000 Subject: android: removed duplicate linux/errno There are two linux/errno.h inclusions in this file. The second one has been removed and the file builds correctly. Fixes: 54ffdab82080 ("android: binder: binderfs.c: removed asm-generic/errno-base.h") Reviewed-by: Nick Desaulniers Tested-by: Nick Desaulniers Signed-off-by: Tanzir Hasan Acked-by: Carlos Llamas Link: https://lore.kernel.org/r/20240104-removeduperror-v1-1-d170d4b3675a@google.com Signed-off-by: Greg Kroah-Hartman --- drivers/android/binderfs.c | 1 - 1 file changed, 1 deletion(-) (limited to 'drivers/android') diff --git a/drivers/android/binderfs.c b/drivers/android/binderfs.c index d04ff6029480..3001d754ac36 100644 --- a/drivers/android/binderfs.c +++ b/drivers/android/binderfs.c @@ -29,7 +29,6 @@ #include #include #include -#include #include #include -- cgit