The minimum credit score for an FHA loan is 500, but most lenders require 580 or higher to get the lowest 3.5% down payment. Learn what score you really need and how to improve it fast as a first-time buyer.