fix: preserve object when device is same #3858

Job	Run time
downgrade	21m 26s
test (1.10, ubuntu-latest, core_layers)	18m 2s
test (1.10, ubuntu-latest, normalize_layers)	16m 59s
test (1.10, ubuntu-latest, recurrent_layers)	1s
test (1.10, ubuntu-latest, autodiff)	21m 22s
test (1.10, ubuntu-latest, misc)	9m 29s
test (1.10, ubuntu-latest, reactant)	1m 29s
test (1, ubuntu-latest, core_layers)	1s
test (1, ubuntu-latest, normalize_layers)	1s
test (1, ubuntu-latest, recurrent_layers)	1s
test (1, ubuntu-latest, autodiff)	1s
test (1, ubuntu-latest, misc)	1s
test (1, ubuntu-latest, reactant)	1s
test (1, macos-latest, all)	1s
test (1, windows-latest, all)	1s
	1h 28m 56s

Provide feedback