Why do you need to break covalent bonds in giant covalent structures to melt them?

Simple covalent molecules have low melting point (mp) and boiling point(bp) because you only need to break the weak Van Der Waals forces of attraction between the molecules right? my Question is... then why do giant covalent molecules have high mp and bp? i know that it is because in order to boil or melt them, their strong COVALENT bonds need to be broken. But why cant they boil or melt with just their Van Der waals broken? I also know that some giant covalent structures like diamond does not have any molecules and thus is purely made up of strong covalent bonds. But what about graphite that has Van Der Waals forces of attraction? Would it not be enough to break the Vander Waals force of attraction just like in Simple covalent molecules instead of breaking the strong covalent bonds among the carbon atoms of graphite?