I'm gonna do a test with my alt and see if the vip I ignored will randomly come off my ignore list too. If they do, then ignore button probably has time limits. If not, then not sure why you're having the issue and I'm not.
I'll come back to this thread once I get results.