sched/core: Free dead mm structs asynchronously in finish_task_switch()

Although mm structs are not often freed from finish_task_switch() during
a context switch, they can still slow things down and waste CPU time on
high priority CPUs when freed. Since unbound workqueues are now affined
to the little CPU cluster, we can offload the mm struct frees away from
the current CPU entirely if it's a high-performance CPU, and defer them
onto a little CPU. This reduces the amount of time spent in context
switches and reclaims CPU time from more-important CPUs. This is
achieved without increasing the size of the mm struct by reusing the
mmput async work, which is guaranteed to not be in use by the time
mm_count reaches zero.

Signed-off-by: Sultan Alsawaf <[email protected]>
Signed-off-by: Carlos Ayrton Lopez Arroyo <[email protected]>

kernel/sched/core.c

@@ -3505,6 +3505,13 @@ static void finish_task_switch_dead(struct task_struct *prev)
 	queue_work(system_unbound_wq, &prev->async_free.work);
 }
 
+static void mmdrop_async_free(struct work_struct *work)
+{
+	struct mm_struct *mm = container_of(work, typeof(*mm), async_put_work);
+
+	__mmdrop(mm);
+}
+
 /**
  * finish_task_switch - clean up after a task-switch
  * @prev: the thread we just switched away from.
@@ -3581,9 +3588,12 @@ static struct rq *finish_task_switch(struct task_struct *prev)
 	 *   provided by mmdrop(),
 	 * - a sync_core for SYNC_CORE.
 	 */
-	if (mm) {
+	if (mm)
 		membarrier_mm_sync_core_before_usermode(mm);
-		mmdrop(mm);
+
+	if (mm && atomic_dec_and_test(&mm->mm_count)) {
+		INIT_WORK(&mm->async_put_work, mmdrop_async_free);
+		queue_work(system_unbound_wq, &mm->async_put_work);
 	}
 	if (unlikely(prev_state == TASK_DEAD)) {
 		if (prev->sched_class->task_dead)