Don't enable pinentry loopback mode for gpgsm

* lisp/epg.el (epg--start): Passphrase entry through the
minibuffer is currently not supported with gpgsm, therefore don't
pass "--pinentry-mode loopback" as an argument when the protocol
is CMS.  (Bug#67012)
* doc/misc/epa.texi (GnuPG Pinentry): Document it.

doc/misc/epa.texi

@@ -640,6 +640,9 @@ Customize variable @code{epg-pinentry-mode} to @code{loopback} in
 Emacs.
 @end enumerate
 
+Note that loopback Pinentry does not work with @command{gpgsm},
+therefore EasyPG will ignore this setting for it.
+
 There are other options available to use Emacs as Pinentry, you might
 come across a Pinentry called @command{pinentry-emacs} or
 @command{gpg-agent} option @code{allow-emacs-pinentry}.  However,

lisp/epg.el

@@ -595,7 +595,12 @@ callback data (if any)."
 		       (if (epg-context-textmode context) '("--textmode"))
 		       (if (epg-context-output-file context)
 			   (list "--output" (epg-context-output-file context)))
-		       (if (epg-context-pinentry-mode context)
+		       (if (and (epg-context-pinentry-mode context)
+				(not
+				 ;; loopback doesn't work with gpgsm
+				 (and (eq (epg-context-protocol context) 'CMS)
+				      (eq (epg-context-pinentry-mode context)
+					  'loopback))))
 			   (list "--pinentry-mode"
 				 (symbol-name (epg-context-pinentry-mode
 					       context))))