Keep your US bank accounts and don't worry
Keep your US bank accounts and don't worry so much about the IRS getting you anywhere in the world. If you want to come to Panama to retire, the chances that you make more money than $190,000 per year as a couple or $95,000 as an individual (the sums that are free of US taxes when you are a resident of Panama) is quite small.
As Panama uses the dollar, you could consider investing in Panama as well, especially in land, which in most cases will give you a better ROI than investments in the US. Nevertheless don't sell out your IRA or retirement account. And with the help of a Panamanian lawyer, you can get a bank account fast and easy in Panama, too. You need it to be able to get a visa, because you need to show you have at least $5,000 in the bank.
Transferring money from the US to Panama is easy, but it takes mostly a week before you can use it. It is also costly, because your US bank charges you and your Panama bank charges you, too. In general the costs in Panama are $30.00. The interest you can get in Panama is a little higher than most US banks as well, as money for the US banks is interest free, in Panama it isn't.