I spent almost an entire day last week fighting with UTF-8 encoding and PayPal IPN integration with the Tentacle Software store.

From a development perspective, I really like IPN. It lets your order processing workflow accept payment notifications from PayPal when a transaction is completed (not just standard purchases, but recurring payments and even any refunds that you issue), and it’s been pretty much rock solid since we launched Disk Management a couple of months ago.

That was, of course, until the ordering process happened to send a non-ASCII character in a payment request to PayPal (an accented vowel, from a customer’s first name). My order processor started throwing nasty unhandled exceptions, and the license key wasn’t emailed out to the customer.

When you take someone’s money and they don’t get what they paid for, that’s a bad (bad) thing. Something you try to take care of immediately.

After much (much) debugging and hair-pulling, it turns out that you have to do a few things to get valid UTF-8 IPN responses from PayPal. There wasn’t much that Google had to say about the issue, and PayPal’s own documentation wasn’t very clear, so here it is for posterity.

Some of these steps were fixes to my code, but some of them should be things that PayPal does by default.

 

Tell the IPN gateway that you want to use UTF-8, please

None of the ASP.NET PayPal IPN integration examples I could find mentioned this one. You need to specify the character set you’re going to be sending to PayPal explicitly, when you build the initial purchase request URL.

public string GeneratePurchaseRequest(string merchantReference, decimal amount)
{
	string paypalReturnMethod = "2";
	string paypalCharset = "UTF-8";
	Encoding encoding = Encoding.UTF8;  

	StringBuilder url = new StringBuilder();

    url.Append(paypalAccessUrl + "cmd=_xclick&business=" + HttpUtility.UrlEncode(paypalEmailAddress, encoding));
    if (!string.IsNullOrEmpty(paypalCharset)) { url.AppendFormat("&charset={0}", paypalCharset); }
    if (amount != 0.00M) { url.AppendFormat("&amount={0:f2}", amount); }
    if (!string.IsNullOrEmpty(transactionCurrency)) { url.AppendFormat("&currency_code={0}", transactionCurrency); }
    if (!string.IsNullOrEmpty(siteTitle) url.AppendFormat("&item_name={0}", HttpUtility.UrlEncode(string.Format("{0} Invoice #{1}", siteTitle, merchantReference), encoding));
    if (!string.IsNullOrEmpty(merchantReference)) { url.AppendFormat("&invoice={0}", HttpUtility.UrlEncode(merchantReference, encoding)); }
    if (!string.IsNullOrEmpty(successPage)) { url.AppendFormat("&return={0}", HttpUtility.UrlEncode(successPage, encoding)); }
    if (!string.IsNullOrEmpty(successPage)) { url.AppendFormat("&rm={0}", HttpUtility.UrlEncode(paypalReturnMethod, encoding)); }
    if (!string.IsNullOrEmpty(paypalNotifyUrl)) { url.AppendFormat("&notify_url={0}", HttpUtility.UrlEncode(paypalNotifyUrl, encoding)); }

    return url.ToString();
}

Pay attention to line 9 above, that’s your winner. Plus all the UTF-8 UrlEncode() stuff.

 

Really tell the IPN gateway that you want to use UTF-8

This is the bit that got me. I was specifying the character set I wanted PayPal to use in my request, but when I went to verify the transaction with PayPal the non-ASCII character always got mangled, and the verification failed. Both request and response forms were set to UTF-8 encoding, and everything I looked at said I was doing the right thing.

Except, I wasn’t doing the right thing that mattered most. You have to explicitly set UTF-8 encoding in your PayPal profile, while you’re logged into the PayPal site.

No, as far as I can tell, you can’t set this programmatically – you have to actually log into their management interface and tell them no, I’m serious, I really really want to use UTF-8. Here’s how (because it’s buried in the user interface):

  1. Log into PayPal
  2. Click the ‘Profile’ link in the menu bar under ‘My Account’
  3. Click the ‘Language Encoding’ link under the ‘Selling Preferences’ column
  4. Click ‘More Options’
  5. Set ‘Encoding’ to ‘UTF-8’
  6. Click ‘Save’

Really.

 

Handle the UTF-8 payment notification verification

Now PayPal is actually sending through UTF-8 responses, and your verification won’t get mangled. Well, it won’t get mangled as long as you use the raw form response.

I’ve had mixed success doing this other ways (Form.ToString() or pulling out the individual query parameters and rebuilding the query string), but your mileage may vary. BinaryRead() seems to give me the most consistent results.

public bool VerifyIpn(HttpContext context)
{
    try
    {
		Encoding encoding = Encoding.UTF8;          

		HttpWebRequest req = (HttpWebRequest)WebRequest.Create(paypalAccessUrl);

        req.Method = "POST";
        req.ContentType = "application/x-www-form-urlencoded";
        byte[] param = context.Request.BinaryRead(HttpContext.Current.Request.ContentLength);
        string strRequest = encoding.GetString(param);
        strRequest += "&cmd=_notify-validate";
        req.ContentLength = strRequest.Length;

        //Send the request to PayPal and get the response
        StreamWriter streamOut = new StreamWriter(req.GetRequestStream());
        streamOut.Write(strRequest);
        streamOut.Close();

        StreamReader streamIn = new StreamReader(req.GetResponse().GetResponseStream());
        string strResponse = streamIn.ReadToEnd();
        streamIn.Close();

        if (strResponse == "VERIFIED")
        {
            //PayPal says we got paid, so let's post-process this order
        }
        else
        {
            //Do something else, like display "Pending payment" to the user
        }
    }
    catch (Exception error)
    {
        log.Fatal(error.Message, error);
        return null;
    }
}

Line 12 is where the magic UTF-8 decoding happens.

 

Basic IPN plumbing

I’ve skipped a bunch of the basic IPN handling and setup steps, because I’m assuming you’ve already got that working and you’ve made it to this page because someone from Germany tried to order from your store. There are plenty of examples out there about how to write your own IPN handler, so I won’t repeat those.

Hopefully this will save someone else hours of debugging. And swearing.

posted on Friday, April 09, 2010 10:06 PM | Filed Under [ Site Development ]

Comments

Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Random @ 8/12/2010 1:54 PM)

Thanks! You saved me with this post.
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Tom @ 12/22/2010 12:16 AM)

Thanks for this post, very helpful indeed :)
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (David @ 1/8/2011 1:37 AM)

Thanks for this post. Really helped
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Pierre Olivier Martel @ 1/28/2011 9:29 AM)

Thank you! This bug was driving me crazy, I couldn't understand why payments made from the US worked with the IPN while others made in France and Germany failed. Stupid character encodings!
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Sinan @ 3/25/2011 1:29 PM)

God Bless You!
You r da man!
Damn! U saved my life!

THANK U SO MUCH!!

I didn't know that I could set the encoding of paypal before I read here and got crazy because of the encoding problems.
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Yan @ 7/16/2011 8:05 AM)

Amazing post !! Thanks so much ! You saved my day.
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Mark @ 7/27/2011 1:39 AM)

Worked perfectly, thanks for sharing!
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Pablo Medina @ 1/18/2012 10:49 PM)

and more than one and a half year later, this post is still a life saver !!!
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (hg @ 2/9/2012 11:08 AM)

Just had exactly the same problem with a German order today. This fixes things nicely!
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Jon @ 2/15/2012 2:59 AM)

Completely hidden away within the depths of PayPal... the charset in the form request got me thinking something was amiss with the encoding... your post came up trumps!
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Paul @ 5/14/2012 5:48 PM)

Thanks! In the latest version of Paypal (using a business account), I managed to find the option in Profile > My selling tools > PayPal button language encoding.
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Fran @ 5/18/2012 7:58 PM)

The profile language encoding option saved my day. I was going crazy with this.
Thank you!
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (zakir saiyed @ 5/26/2012 5:48 AM)

Solved my problem with International characters

I have to specify the character set going to be sending to PayPal explicitly, when I build the initial purchase request URL.

AND

Followed below steps.

IPNs default to charset = windows-1252, however, you can change this by going to your Go to Profile > My Selling Tools
Down at the bottom click on "PayPal button language encoding" and then More Options. On this page you can change your character set and languages for various notifications.
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Raghu @ 7/26/2012 6:07 PM)

You saved a complete day (and maybe more) for me. This is the exact same issue I am facing. Thank you very much.
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Larry Smith @ 7/28/2012 9:41 AM)

Yet another thank you! I had the same problem today (IPN validation failed for a customer in Germany). Fortunately, the rep I spoke to at PayPal's Merchant Technical Support" line suspected an encoding problem though he wasn't aware of the problem you described. I then found your link and passed it onto them. I requested they send it their IT dep't and they're going to do it (they sounded sincere). Hopefully some fix will eventually be available. Thanks again (greatly appreciated).
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Henry @ 4/16/2013 7:03 PM)

Do you have any idea of using UTF-8 with Java?
I see that .NET problem solved with your solution. Any idea on the Java part?
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Sam Wood @ 4/16/2013 9:39 PM)

Hi Henry, I have no experience with Java, but I suspect this would be handled in a similar way; you'll need to explicitly request UTF-8 and then decode PayPal's response as UTF-8, as well as enabling UTF-8 in the PayPal Merchant UI.
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (ezra @ 11/8/2013 10:50 PM)

Thank you so MUCH! PayPal's documentation is a messy maze of erratic contents... You juste saved my life! :D
 
Gravatar
# re: PayPal Instant Payment Notification and UTF-8 encoding (Matthew Molloy @ 3/13/2014 5:41 AM)

Thanks for advice,

Paypal never ceases to astound me with its backwardness.

Post Comment

Title *
Name *
Email
Url
Comment *  
Remember me
Please add 8 and 8 and type the answer here:

Search

Site Sections

Recent Posts

Archives

Post Categories

WHS Add-In Tutorial

WHS Blogs

WHS Development