fine is a revenue practice, normally if you pay them promptly, it is normally just a less than 100 quid fine.
in your case, give them a call and resolve it asap.
taking the wrong oyster card and get caught is a very common issue, and you will not be let go by TFL.