Wow, that's tough. Hard to find (for good reason, honestly) banks that'll take a 3% down payment these days.
Without wanting to sound judgemental, how are you going to make your payments? Let's assume you CAN find someone to extend you a loan; figure a $275k loan (midpoint of your range less the down payment). Average 30-year fixed mortgage rates are 5.17%. That's a monthly payment of $1,504, or annual payments of $18,048; doesn't leave you a lot of money to live on with your wife unemployed (not even considering property taxes, homeowners insurance, the other direct costs of owning a house).
Even ARM rates aren't that much cheaper these days, at least nationally (and then you expose yourself to getting hosed in when rates adjust).
It sucks, it really does, but it's hard to see how the numbers make a lot of sense. That's a lot of $$ to put into a house relative to how much money you're bringing in. Four years ago, no doubt that you would have been able to find SOMEONE to put you into a mortgage, but frankly it's these kinds of situation (people buying more house than they could really afford) that ultimately underlies a lot of the economic issues we're still dealing with.