I think so as well! The recycling of stone blocks was so common that it easily reached across the ages. I quite like it how in Witcher 3 (especially the Blood and Wine DLC) the player had an opportunity to restore various hamlets, but I feel like it's just one side of the coin.
Especially since once we establish that building materials are worth collecting and reusing, it makes the buildings which are left to itself even more interesting. It makes sense, for example, that a "haunted", burnt house becomes taboo.