Yes, you should worry about fonts! It is recommended that you use standard fonts such as Arial, Times New Roman, or Verdana as these are universal fonts available to most email programs.
If you use other fonts, the recipient may not have that font installed on their computer and may not be able to view the message as intended.
Font sizes are also important. You would want to avoid font sizes larger than 14pt, because some bulk mail folders may increase the spam score for very large fonts in your email message.