Solution #1:
Right-click on the Recycle Bin on the Desktop, and select Properties from the menu.
Under the Global Tab, select Use one setting for all drives.
Turn on the option labeled Do not move files to the recycle bin. (For some unknown reason, decreasing the Maximum size of Recycle Bin to 0% doesn't do the trick.)
Notice that the option labeled Display delete confirmation dialog is grayed out at this point, meaning that there is no way to truly delete files in Windows without a confirmation dialog.
Also, if you have files waiting in the Recycle Bin, they will not be deleted until you do so manually, even after changing this setting.
Solution #2:
Create a shortcut to the Recycle Bin in your Send To folder. To delete files, right-click, and choose Recycle Bin from the Send To menu.
Note: Holding the Shift key while pressing on Delete Key will remove the file permanently without sending it yo recycle bin.
Cheers
Courtesy of:
http://www.annoyances.org