At least until hackers manage to crack the PS Vita (maybe it'll never happen, but I'm betting it will in some fashion, eventually), the only place you're going to find downloadable games for the PS Vita is on the PlayStation Store. But the PlayStation Store has several categories of downloadable games, some of which will be playable on the PS Vita, and some of which won't. Below are the different kinds of games you'll find at the PlayStation Store, with information on whether or not you can play them on a PS Vita.
Retail Games: PS Vita
It probably goes without saying, but any retail game marked as a PS Vita game, whether it's a boxed cartridge from a retail shop, a retail card with download code, or a download purchased directly from the PlayStation Store, will be playable on any PS Vita. And while it's not confirmed yet, early hands-on reports indicate that the PS Vita will most likely have region-free games, just like the PSP did, meaning you can import games from other regions (or download them, if you are able to set up a PlayStation network account in another region). It has also been reported that Sony's intention is to have all retail boxed games also available for download, so you won't ever have to go to the store to buy a new game if you don't want to (but you will need a lot of memory cards to keep them on).
Retail Games: PSP
All PSP retail games should also be playable on the PS Vita, but only if they are downloaded from the PlayStation Store. UMDs will not work in the PS Vita, so don't expect to be able to buy a packaged game at a game store and play in on your PS Vita. Only downloads will work. Download-only PSP games should also be playable on the PS Vita. Note, however, that this does not include PSOne Classics (yet). See below for more information.
Retail Games: PS3
One of the features that the PS Vita has is the ability to continue playing a game you've been playing on your PS3 at the same point you left off on the console. However, don't expect to be able to transfer a PS3 game onto your PS Vita. For most games, you'll need an actual PS Vita version of the game, which you'll have to buy separately. You will however, be able to play some PS3 games remotely on your PS Vita by having them run on the PS3 while use Remote Play to access them on the PS Vita. This also mean downloaded PS3 games will not work directly on a PS Vita.
Currently, PSP demos will not run on a PS Vita, though the actual games will. This may or may not be changed in an update--it remains to be seen. It's not unreasonable to expect that at least some PS Vita games will have demos, but again, that remains to be seen.
PS Minis will almost certainly run on a PS Vita, but I'm not aware of any specific information regarding them yet.
PSOne Classics is a line of games originally published for the PlayStation (aka PSOne). These games are ports of the original games, and not re-tooled in any way except to make them playable with PSP controls. Some older games, like the first two Final Fantasy games, have been released with graphics and gameplay updates, but are not part of the PSOne Classics line. As of this writing, PSOne Classics do not run on a PS Vita. This is expected to be addressed in a firmware update in the near future.
Neo Geo/PC Engine Games
These games are ports of classic Neo Geo and PC Engine games, similar to the PSOne Classics line. They are supported by the current PS Vita firmware and should run just fine.
The Japan Imports line features games exactly as they were released in Japan, and may or may not have English text. There is no information yet about whether or not they can be expected to run on a PS Vita, but if they are simply Japanese PSP or PS Vita games, they will run. If they're classic games originally published on the PlayStation or PS2, then they may not work.
The PS2 Classics line is a follow-up to the popular PSOne Classics, and presents PS2 games minimally re-tooled to run on a PS3. There is no information yet about whether or not they will run on a PS Vita, but I suspect that if they don't at launch, they soon will with a firmware update.
Homebrew games are small games made by indie developers and hackers, and are therefore unavailable from the PlayStation Store. While homebrew for the PS Vita is a future possibility, don't buy a PS Vita expecting to hack it out of the box. Even the PSP, frequently hacked and fitted with custom homebrew firmware, never had a very large library of original homebrew games.