Thank you Google for writing such a shitty Android library that makes my head ache once again. The problem has been reported to you ages ago, please fix your shit.
Luckily someone had made a workaround already, I just needed to copy-paste that solution to the project and do some minor adjustments. Now the adapter can remove pages from a pager just fine.