I'm going to start things off with a simple app for browsing the web. Safely, securely, and requires no set up out of the box. Firefox portable. Its pre-configured to not store any cache or cookie data and works with most extensions like Sage or Chatzilla. I can still store passwords if you wish, and because its Firefox, you can set a master password to keep anyone from gaining access to those passwords should you misplace your USB drive.
All you need to do to "install" this app is download and run the executable. You can then have tell the executable where you're planning to extract the files to. I'd exctract the files to: "[Drive]:\Apps\Browsers\" because it not only keeps things organized this way, you know where to look. And if you're wondering why I choose to label the folder "browsers", I'm about to get to that. Firefox doesn't run ActiveX without the aid of an additional Add-on. But we don't want to waste the time with putting an addon on our browser that will only sow us down when we aren't using it. A browser called Maxthon is what we're going to use for ActiveX. Its portable just like Firefox, but you have to set it up after installing it to your USB drive. All you need to do is disable cache and cookie storing and set the bookmarks folder to somewhere on your USB drive( ex. "$Maxdir\favorites\"). I should mention that Maxthon is based of the IE engine, and I only use it for those few sites that run ActiveX.