When you manually remove a .VHD file from the library, is it possible that you cannot remove the template from the SCVMM library. The template has some Temporary Template dependencies on it. On the dependencies tab in your template configuration file you see the Object that is linked.
Double click on the linked object and copy the name.
Open a Virtual Machine Manager Powershell
remove-scvmtemplate -vmtemplate "Temporary Template1c2fe 00e-4d16-4975-b85b-8c4fad34fff5"
Once runned the linked temporary template has been removed and you are able te remove te template.