Does a Vision Model Know Where the Chair Went?