Most of the icons that show up in Control Panel are placed there by a corresponding .CPL file in your \Windows\System folder. The trick is to find the right one and rename it. You can delete it, but only if you're sure you never want to see it again.

Run FIND (Start, Find, File or Folders) and search your \Windows\System folder for *.CPL. Generally the application icon will be named for the application it belongs to. In this example we sought out the .CPL files for TweakUI, CyberMedia's Uninstaller and Office's FindFast. Identify the corresponding .CPL file and rename it, then open Control Panel to make sure you removed the right one(s). Again, it's best to rename a .CPL file (ex: .CPX) instead of deleting it right away since it's a lot easier to rename it again than to reinstall something later if you make a mistake.