I've seen this bug happen a lot too.
The trick to fix it is to build parks on top of it. For some reason, parks can replace those left over bits when other ploppables can't.
So just use parks to get rid of those left over models and bulldoze the parks then you can put whatever new buildings you want there.