The following commit has been merged in the linux branch: commit 41e20983fe553b39bc2b00e07c7a379f0c86a4bc Author: Wu Fengguang fengguang.wu@intel.com Date: Mon Oct 26 16:49:53 2009 -0700

vmscan: limit VM_EXEC protection to file pages

It is possible to have !Anon but SwapBacked pages, and some apps could create huge number of such pages with MAP_SHARED|MAP_ANONYMOUS. These pages go into the ANON lru list, and hence shall not be protected: we only care mapped executable files. Failing to do so may trigger OOM.

Tested-by: Christian Borntraeger borntraeger@de.ibm.com Reviewed-by: Rik van Riel riel@redhat.com Signed-off-by: Wu Fengguang fengguang.wu@intel.com Cc: Hugh Dickins hugh.dickins@tiscali.co.uk Signed-off-by: Andrew Morton akpm@linux-foundation.org Signed-off-by: Linus Torvalds torvalds@linux-foundation.org

diff --git a/mm/vmscan.c b/mm/vmscan.c index fbb9f6b..fbcac3b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1356,7 +1356,7 @@ static void shrink_active_list(unsigned long nr_pages, struct zone *zone, * IO, plus JVM can create lots of anon VM_EXEC pages, * so we ignore them here. */ - if ((vm_flags & VM_EXEC) && !PageAnon(page)) { + if ((vm_flags & VM_EXEC) && page_is_file_cache(page)) { list_add(&page->lru, &l_active); continue; }