When the PSP first launched, it was supposed to have all kinds of exciting possibilities for interaction with the PS3, but most of that never really came to pass. Now the PS Vita is on the way, and people are getting excited about its possibilities for interaction with the PS3. We won't know for sure what is actually going to happen until it happens, but here are the interaction possibilities that have been mentioned.
Basically, Remote Play is a way of connecting a PS Vita (or PSP) and PS3 via the internet to allow you to access the content on your PS3 remotely via your handheld. You can play music, watch videos, look at photos and play games (some games, anyway) that are stored on your PS3 via the internet on your handheld.
Remote Play on the PS Vita will probably be much like Remote Play on the PSP, except the PS Vita's controls better match the PS3's (mainly, it has two analog sticks), and the graphics will be much improved. It's also likely that many more games will have support for Remote Play on the PS Vita than there were on the PSP.
Supposing you have the PSP version of a game with a multiplayer mode and your friend has the PS3 version. You want to get into a game together, but you can't. The PSP doesn't support Cross-Platform play, probably largely because it's not powerful enough.
The PS Vita, though, will support Cross-Platform Play, and though it's probably something developers will have to build into each game, multiplayer gaming is popular enough that it's likely many (or even most) games with both PS Vita and PS3 versions and multiplayer options will also support Cross-Platform Play. Sure, the framerate will no doubt be slower on the PS Vita, but as long as the game itself can keep up, that's all you really need.
Title User Storage
Title User Storage is a system that allows 1 MB of remote storage on PlayStation Network servers (it's not specified if that's total per user, or per game) that is accessible by both the user's PS Vita and PS3. This is what will allow Continuation Play (see below) to work smoothly, but it will also be useable in unspecified ways by developers to shift data between the two platforms.
One of the most mentioned features of the PS Vita is the ability to play a game on the handheld and then switch to the PS3 and play the same game, right where you left off on the PS Vita (or vice versa). Of course, this would require there to be both PS3 and PS Vita versions of a game, and for the user to own them both (but I can imagine bundle deals on the PlayStation Store). The feature uses Title User Storage (see above) to keep game data remotely stored for near-seamless transition from one platform to the other.
Using a PS Vita as a PS3 controller could mean a couple of different things, and both of them are pretty exciting. First, it could simply mean using the PS Vita instead of a DualShock 3 controller, substituting the PS Vita's buttons and inputs for their equivalents on the DualShock, but also adding in the touch controls. This would add a whole new dimension to PS3 games, either by re-mapping some controls to touch controls, or by adding a completely new set of options to a PS3 game.
The other way a PS Vita could become a PS controller is by having the PS Vita handle additional gameplay elements that add to the PS3 experience. The PS3 would control what displays on the PS Vita screen, and give you access to more gameplay aspects or new abilities that you wouldn't have if you only had one version of the game. Of course, games making use of such features would favor players who own both devices over those who have only one, but imagine the cool ways developers could use this! (Though it does bring to mind the promise that the PSP would be able to function as a rearview mirror in the PS3 game Formula One 06, which never came to pass as far as I know, but I'll be optimistic here.)
I could be pessimistic and point out again how the PSP never fulfilled its interactive potential with the PS3, but I won't (OK, I did, but I won't dwell on it). The PS Vita is exciting and it looks like developers are as excited as gamers. So let's just think about what could be and hope it comes about in the best ways possible.