April 2010 Blog Posts

I spent almost an entire day last week fighting with UTF-8 encoding and PayPal IPN integration with the Tentacle Software store.

From a development perspective, I really like IPN. It lets your order processing workflow accept payment notifications from PayPal when a transaction is completed (not just standard purchases, but recurring payments and even any refunds that you issue), and it’s been pretty much rock solid since we launched Disk Management a couple of months ago.

That was, of course, until the ordering process happened to send a non-ASCII character in a payment request to PayPal (an accented vowel, from a customer’s first name). My order processor started throwing nasty unhandled exceptions, and the license key wasn’t emailed out to the customer.

When you take someone’s money and they don’t get what they paid for, that’s a bad (bad) thing. Something you try to take care of immediately.

After much (much) debugging and hair-pulling, it turns out that you have to do a few things to get valid UTF-8 IPN responses from PayPal. There wasn’t much that Google had to say about the issue, and PayPal’s own documentation wasn’t very clear, so here it is for posterity.

Some of these steps were fixes to my code, but some of them should be things that PayPal does by default.


Tell the IPN gateway that you want to use UTF-8, please

None of the ASP.NET PayPal IPN integration examples I could find mentioned this one. You need to specify the character set you’re going to be sending to PayPal explicitly, when you build the initial purchase request URL.

public string GeneratePurchaseRequest(string merchantReference, decimal amount)
	string paypalReturnMethod = "2";
	string paypalCharset = "UTF-8";
	Encoding encoding = Encoding.UTF8;  

	StringBuilder url = new StringBuilder();

    url.Append(paypalAccessUrl + "cmd=_xclick&business=" + HttpUtility.UrlEncode(paypalEmailAddress, encoding));
    if (!string.IsNullOrEmpty(paypalCharset)) { url.AppendFormat("&charset={0}", paypalCharset); }
    if (amount != 0.00M) { url.AppendFormat("&amount={0:f2}", amount); }
    if (!string.IsNullOrEmpty(transactionCurrency)) { url.AppendFormat("&currency_code={0}", transactionCurrency); }
    if (!string.IsNullOrEmpty(siteTitle) url.AppendFormat("&item_name={0}", HttpUtility.UrlEncode(string.Format("{0} Invoice #{1}", siteTitle, merchantReference), encoding));
    if (!string.IsNullOrEmpty(merchantReference)) { url.AppendFormat("&invoice={0}", HttpUtility.UrlEncode(merchantReference, encoding)); }
    if (!string.IsNullOrEmpty(successPage)) { url.AppendFormat("&return={0}", HttpUtility.UrlEncode(successPage, encoding)); }
    if (!string.IsNullOrEmpty(successPage)) { url.AppendFormat("&rm={0}", HttpUtility.UrlEncode(paypalReturnMethod, encoding)); }
    if (!string.IsNullOrEmpty(paypalNotifyUrl)) { url.AppendFormat("&notify_url={0}", HttpUtility.UrlEncode(paypalNotifyUrl, encoding)); }

    return url.ToString();

Pay attention to line 9 above, that’s your winner. Plus all the UTF-8 UrlEncode() stuff.


Really tell the IPN gateway that you want to use UTF-8

This is the bit that got me. I was specifying the character set I wanted PayPal to use in my request, but when I went to verify the transaction with PayPal the non-ASCII character always got mangled, and the verification failed. Both request and response forms were set to UTF-8 encoding, and everything I looked at said I was doing the right thing.

Except, I wasn’t doing the right thing that mattered most. You have to explicitly set UTF-8 encoding in your PayPal profile, while you’re logged into the PayPal site.

No, as far as I can tell, you can’t set this programmatically – you have to actually log into their management interface and tell them no, I’m serious, I really really want to use UTF-8. Here’s how (because it’s buried in the user interface):

  1. Log into PayPal
  2. Click the ‘Profile’ link in the menu bar under ‘My Account’
  3. Click the ‘Language Encoding’ link under the ‘Selling Preferences’ column
  4. Click ‘More Options’
  5. Set ‘Encoding’ to ‘UTF-8’
  6. Click ‘Save’



Handle the UTF-8 payment notification verification

Now PayPal is actually sending through UTF-8 responses, and your verification won’t get mangled. Well, it won’t get mangled as long as you use the raw form response.

I’ve had mixed success doing this other ways (Form.ToString() or pulling out the individual query parameters and rebuilding the query string), but your mileage may vary. BinaryRead() seems to give me the most consistent results.

public bool VerifyIpn(HttpContext context)
		Encoding encoding = Encoding.UTF8;          

		HttpWebRequest req = (HttpWebRequest)WebRequest.Create(paypalAccessUrl);

        req.Method = "POST";
        req.ContentType = "application/x-www-form-urlencoded";
        byte[] param = context.Request.BinaryRead(HttpContext.Current.Request.ContentLength);
        string strRequest = encoding.GetString(param);
        strRequest += "&cmd=_notify-validate";
        req.ContentLength = strRequest.Length;

        //Send the request to PayPal and get the response
        StreamWriter streamOut = new StreamWriter(req.GetRequestStream());

        StreamReader streamIn = new StreamReader(req.GetResponse().GetResponseStream());
        string strResponse = streamIn.ReadToEnd();

        if (strResponse == "VERIFIED")
            //PayPal says we got paid, so let's post-process this order
            //Do something else, like display "Pending payment" to the user
    catch (Exception error)
        log.Fatal(error.Message, error);
        return null;

Line 12 is where the magic UTF-8 decoding happens.


Basic IPN plumbing

I’ve skipped a bunch of the basic IPN handling and setup steps, because I’m assuming you’ve already got that working and you’ve made it to this page because someone from Germany tried to order from your store. There are plenty of examples out there about how to write your own IPN handler, so I won’t repeat those.

Hopefully this will save someone else hours of debugging. And swearing.

Andreas has posted another great WHS development tip for allowing your SettingsTab and ConsoleTab to notify each other of changes.

The easiest way to accomplish this is by using a singleton object which provides the necessary means of communication.

We can access a common instance (the singleton) from everywhere in our code by calling on ChangeNotifier.Instance. First we need to subscribe to the ChangeNotifier’s Changed event in our console tab and second we have to call the Notify() method (which will fire the Changed event) from our settings tab when setting changes are committed.

I’ve seen this come up quite a few times on various forums, and I’ve always recommended using some variation of the WHS Notification infrastructure (it’s what I use in Disk Management). I like Andreas’ idea much better; it’s a lot cleaner and less prone to errors than what I’ve been using.

Great stuff, Andreas.

Back in December, I wrote about HomeServerLand.com’s Add-in Central, a Windows Home Server Add-in that sounds remarkably like an App Store (minus the transaction fees).

I’m happy to see that they’ve released the public beta early, for everyone to play with.

We have been working tirelessly on AC during our exclusive private beta and we are highly confident in the latest stable release. We are constantly working on improving Add-In Central and with it being released to the public; we know that the community will help us make significant improvements.

The add-in software and service [is] designed to help you discover and track useful add-ins right from within the Windows Home Server Console.

Great work!


Site Sections

Recent Posts


Post Categories

WHS Add-In Tutorial

WHS Blogs

WHS Development