the Courier mail server can use both /etc/courier/esmtppercentrelay and /etc/courier/esmtppercentrelay.dat at the same time. Usually you would put a couple of your most frequent domains in /etc/courier/esmtppercentrelay, then put the rest in /etc/courier/esmtppercentrelay.dir, and use makepercentrelay to turn it into a database file.
"percent-hack" domains are a list of domains for which the Courier mail server accepts mail via ESMTP addressed as "email@example.com", where "percent.hack.domain" is a domain found in /etc/courier/esmtppercentrelay or /etc/courier/esmtppercentrelay.dat, and "local.domain" is any domain found in /etc/courier/locals. The Courier mail server removes the local domain, and rewrites the address as "firstname.lastname@example.org", then attempts to deliver it.
The percent hack applies only to mail received via ESMTP. The Courier mail server does not check this list of domains if the message is received via any other way (such as by running /usr/bin/sendmail directly from the command line). "percent.hack.domain" would likely to be a domain that the Courier mail server knows how to handle via some other means. It might be an entry in /etc/courier/aliases, or an entry in /etc/courier/esmtproutes.