PCI at Groupon – the Tokenizer

at June 17th, 2014

Any successful e-commerce company invariably has to become PCI compliant. The Payment Card Industry (PCI) is a credit card industry consortium that sets standards and protocols for dealing with credit cards. One of these standards targeted at merchants is called the PCI-DSS, or PCI Data Security Standards. It is a set of rules for how credit card data must be handled by a merchant in order to prevent that data from getting into the wrong hands.

Groupon has designed a PCI solution called Tokenizer – a purpose-built system, built with performance, scalability, and fault-tolerance in mind as a way to streamline this process and we are seeing great results.

The PCI Security Standards Council certifies independent auditors to assess merchants’ compliance with the PCI-DSS. Companies that handle more than six million credit card transactions in a year are held to the highest standard, known as PCI Level 1.

When faced with becoming PCI-compliant, most e-commerce businesses that need access to the original credit card number usually follow one of two typical paths:

  • Put all of their code and systems under PCI scope. Development velocity grinds to a halt due to the order-of-magnitude increase in overhead compared with typical agile processes.
  • Carve up their app into in-scope (checkout page, credit card number storage, etc.) and out-of-scope (everything else) portions. The once-monolithic application becomes a spaghetti mess of separation and interdependency between very different systems. The “checkout” page stagnates and becomes visually inconsistent with the rest of the site due to the order-of-magnitude increase in overhead noted above.

Groupon faced this dilemma a few years back. Prior to designing a solution, we set out our design goals:

  1. Have as few devices/services in PCI scope as possible.
  2. Make as few changes to existing services as possible.
  3. Maximize protection of our customers’ credit card data.

After many hours meditating at the Deer Head Center for Excellence (see the end of this post) we arrived at our solution.

Groupon’s PCI Solution

For the core tenet of the solution we would focus on the credit card data that we had to protect. Rather than re-implement our checkout application, we would shield the application from the credit card data. How would we do this? Read on…

Prior to becoming PCI compliant, users would submit their credit card information directly to the checkout application. With the PCI-compliant solution in place, there is an HTTP proxy layer that transforms the customer’s web request, replacing the credit card number with an innocuous token. The modified request is forwarded to the checkout app, who processes the request and responds to the user via the proxy. Since the proxy’s job is to replace sensitive data with a token, we call it the Tokenizer.

Tokenizer High Level Diagram

Enter the Tokenizer

With this architecture, our PCI footprint is minimal, flexible, and rarely needs to change. The Tokenizer is a purpose-built system, built with performance, scalability, and fault-tolerance in mind. Its only job is to find credit card numbers in specific kinds of requests, replace the credit card numbers with something else, store the encrypted credit card number, and forward the request to another system.

The Tokenizer does not render any HTML or error messages — all of that comes from the application that it is proxying to. Therefore, the user interface components are under complete control of systems that are not in PCI scope, thereby preserving the normal high iteration velocity of a modern e-commerce company.

“Yes, But…” or All About Format-Preserving Tokens

You may be asking, “So the credit card number is replaced with something else right? What is it replaced with? How can you satisfy your design goal of having minimal changes to the rest of the system?”

A credit card is defined as a numeric sequence of 13 to 19 digits. Credit card companies are assigned a number space in which they may issue cards. You may have noticed that most VISA cards start with 4, MasterCard with 5, etc.

There is a special number space that begins with “99” that is by definition unassigned. The tokens that the Tokenizer generates are 19 digits long, and begin with “99”. Because the token follows the official definition of a credit card, it is called a Format-Preserving Token.

Within the 19 digit number, we encode a token version, the original card type, datacenter that created the token, a checksum digit, a sequence number, and the original last four digits of the customer-entered credit card number.

Groupon Token Breakdown:

9911211234567891234
 | || ||        lastfour
 | || |seqno(9)
 | || luhn
 | |card type(2)
 | version/colo
 cardspace

The only modification required of other systems at Groupon is that they need to recognize these tokens as valid credit cards, and look at the card type field to know the original card type, rather than running their normal pattern matching code for Visa, Mastercard, Discover, AmEx, etc. Usually this is a minor change within a CreditCard class.

Database schemas can stay the same, masked card display routines (only showing the card type and last four digits) may stay the same, even error cases can stay the same.

Error Handling

Since the Tokenizer has pattern matching rules for detecting card types, it can detect invalid card numbers as well. Rather than encrypting and storing invalid card numbers, the Tokenizer will generate a token that is invalid in the same way that the original input was invalid and replace the invalid card number with the generated invalid token.

For instance, if the credit card number is too short (e.g. a VISA with 15 digits instead of the typical 16) the Tokenizer will generate a token that is too short (e.g. 18 digits instead 19). If the card number the customer typed in does not pass the Luhn checksum test, then the token will not pass the Luhn checksum test. If the card is an unrecognizable type, then the token will be an unrecognizable type.

This allows all end-user messaging around card errors to be generated by the out-of-scope systems that the Tokenizer is proxying to. The checkout app will see an invalid card number, and return an appropriate error message via the Tokenizer back to the user. Less PCI-scoped code means that development teams may iterate on the product with less friction.

Security

Most similar tokenizing systems will use a cryptographic hashing algorithm like SHA-256 to transform the card number into something unrecognizable. The issue with hashed card numbers is that the plaintext space is relatively small (numbers with well known prefixes having 13-19 digits that pass the Luhn checksum test). Therefore, reverse-engineering of hashed credit card numbers is fairly simple with today’s computers. If the hashed tokens fall into the wrong hands, the assumption needs to be that the card numbers have been exposed.

This risk can be mitigated by using a salted hash or HMAC algorithm instead, with the salt treated as an encryption key and split between layers of the system, but the downside of having a token in a very different form than the original card number remains.

Aside from the card type and last four digits, the Groupon token is not derived from the card number at all. There is no way, given any number of tokens, to generate any number of credit card numbers. The information simply isn’t there for the bad guys to use. Less information is more secure.

Closing The Loop

At this point we have described how card numbers are replaced with tokens by the Tokenizer. Now we have many backend systems that have tokens stored instead of credit card numbers. You may be wondering how we collect money from tokens. Seems like squeezing water from a rock, right?

There is a complementary, internal-only system called the Detokenizer. The Detokenizer functions in a very similar way to the tokenizer, just in reverse.

We use our payment processor’s client library to construct a request back to the processor. Usually these requests are SOAP over HTTP. In the credit card number field in the SOAP payload, the out-of-scope system will insert the token. Naturally, we cannot send this request as-is to the payment processor, since the token is meaningless to them.

<?xml>
<transaction>
<card_number>990110000000121111</card_number>…

“Bzzzzt! That’s not a credit card number!”

Rather than sending the request directly to the payment processor, it is sent via the Detokenizer proxy. The Detokenizer recognizes the destination of the request and looks in a specific place in the payload (the card number field) for the token. If it’s found, the token is used to look up the original encrypted card number from the database, the card number is decrypted, and it replaces the token in the original request payload.

<?xml>
<transaction>
<card_number>4111111111111111</card_number>…

The Detokenizer then forwards the HTTP request to the payment processor and Groupon gets paid. Make it rain, Groupon.

Overall

Groupon’s current production footprint is on the order of thousands of servers. Our global PCI scope extends to one small source code repository, a dozen application servers, four database servers, and some network gear. Development, code deploys, and configuration changes follow the regimented rules of the PCI-DSS. But since the in-scope footprint is so small, the impact on the rest of the organization is minimal to nil.

There is a very small group of developers and operations staff who maintain these systems on a part-time basis. The rest of the time, these employees enjoy working on more high-velocity projects for the company. This design has meant that Groupon can still run at the speed of a startup, even though it is compliant with these strict industry rules.

And finally, The Deer Head Center For Excellence is Groupon’s in-office cocktail bar. Put up a monitor and serve up your own Deer Head.

Deer Head


12 thoughts on “PCI at Groupon – the Tokenizer

  1. When I add a credit card to my account, it is requiring the CVV. This is a clear violation of PCI-DSS requirements, whether you encrypt or tokenize. The system doesn't allow me to add a card for one-time only use for a purchase - it automatically saves it for future purchases. In fact, I've had transactions fail to process before because I placed an order, then removed the card from my "saved" credit cards immediately after. This prevented the transaction from processing because it didn't just process the order in real-time and discard the CVV - it saved the entire thing to my account, then when I removed it, the transaction failed. I seriously doubt that you have passed a PCI audit with this setup, but maybe you can enlighten me on how this is compliant.

    by Dave Shardell on June 25, 2014 at 6:28 am
  2. Hi Dave -- Thanks for your comment and questions. The CVV is not persisted in any Groupon systems. We pass it to a credit card transaction processor (a third party), who validates it and returns an opaque token for us to use in future transactions if the card information is correct. This is an asynchronous process, which explains the behavior you noticed when deleting a stored card immediately after initiating a transaction. This is in-line with the rules and guidance set forth in the PCI-DSS, and thus we maintain our PCI compliance. We take PCI compliance very seriously, and have always passed our audits.

    by Zack Steinkamp on June 25, 2014 at 10:46 am
  3. Thanks for the information. Do you have plans to allow people to make purchases without storing their CC data for future purchases to their account? I had hundreds of dollars in unauthorized purchases made last year because someone hacked my account which is why I delete my CC data after purchasing. I should be able to put in CC information for one-time only use. Also, does the 3rd-party save the CVV and use it for future purchases, or do they receive a token from the credit card issuer that allows them to handle future purchases without the CVV? Is the 3rd-party PCI compliant?

    by Dave on June 25, 2014 at 11:19 am
  4. Currently, Groupon always stores your card given a successful purchase in order to simplify future purchases. I'll make a suggestion to the Checkout team to offer the option to not store it. Thanks for that. As part of our PCI compliance, we must request, review, and submit Attestations of Compliance for all of our vendors who handle credit card data to our PCI auditors. We know the 3rd party payment processor is PCI compliant, so they must not store the CVV as per the rules in the PCI-DSS. It's quite likely that they handle CVV as you suggest -- they pass it up the chain, in this case to the credit card companies themselves, and receive an OK or NOT-OK response. There is no need to store CVV at any layer except at the card issuer for purposes of validation. The chain of compliance must be unbroken in order for Groupon to maintain our compliance.

    by Zack Steinkamp on June 25, 2014 at 11:39 am
  5. Thanks Zack for your responses, I feel more comfortable entering my CC information now that I know you do not store CVV and your partner is PCI compliant. And I understand why I need to leave it there until the transaction is successfully completed... though I do plan to continue removing it thereafter to avoid problems.

    by Dave on June 25, 2014 at 11:51 am
  6. Zach, Thanks for the write up. One thing that I would be curious to know: How did you comply with requirement 1.3.3 - Do not allow any direct connections inbound or outbound for traffic between the Internet and the cardholder data environment? If the tokenization proxy is in line with checkout applications, and it is storing CC nums, then the CDE would seem to be directly connected to the Internet. Thanks.

    by Tyler Rasmussen on April 22, 2015 at 4:53 pm
  7. @Tyler, Zack's diagram in this blog post is a simplified logical component diagram. There are other systems in place. For example, the tokenizer sits behind a DDOS protection layer, a web-application firewall (WAF), a traditional network firewall, a second WAF from a different vendor, and other network gear like routers and load balancers with their own network segmentation. Additionally, the credit card data is persisted in a database (not depicted in Zack's diagram) that is segmented from the tokenizer application in compliance with PCI-DSS 1.3.7 (segregate databases). As you pointed out, the tokenization application and CDE still need to comply with all the requirements of PCI. Tokenization allows us to focus our efforts on a smaller number of systems, fewer system administrators, and ultimately a smaller attack surface.

    by Kyle O on May 5, 2015 at 5:36 pm
  8. Curious - from this section 'The Detokenizer recognizes the destination of the request and looks in a specific place in the payload (the card number field) for the token. If it’s found, the token is used to look up the original encrypted card number from the database, the card number is decrypted, and it replaces the token in the original request payload,' how do you handle authentication to ensure that a user or process is authorized to retrieve the token? For example, if the non-in scope DB with the tokens were to be compromised, what is preventing those tokens to be used to extract the real CC number from the PCI DB?

    by Jonathan Smith on June 12, 2015 at 1:25 pm
  9. @Jonathan, great question. The user or process would never retrieve the PAN in a response. The tokenizer and detokenizer act more like proxies sitting in between the client making the request and the server ultimately serving the request. Continuing your example,

    For example, if the non-in scope DB with the tokens were to be compromised, what is preventing those tokens to be used to extract the real CC number from the PCI DB?
    The potential attacker would need to also compromise the application servers responsible for processing payments. The PCI DMZ firewalls only accept requests from known ingress networks. With the compromised tokens and the additionally compromised host, the attacker could replay the tokens through the detokenizer which would proxy the request to the payment gateway. In other words, an attacker could make fraudulent payments or refunds, but they would be unable to retrieve the personal account numbers because the information is only sent upstream to the payment vendor. Obviously, making fraudulent payments and refunds is a problem too. But, it's not a problem specifically addressed by PCI-DSS which was the scope of Zack's blog post.

    by Kyle O on July 13, 2015 at 5:52 pm
  10. Any thoughts on open-sourcing the proxies?

    by Randy on February 5, 2016 at 2:45 pm
  11. @Randy, we haven't discussed open sourcing the proxies. We didn't think there would be much interest since the proxies, frankly, are only a fraction of the work required to fully comply with PCI-DSS. If you have a use case, feel free to contact me at (my name + last initial)@groupon.com.

    by Kyle O on March 2, 2016 at 7:16 pm
  12. Most useful real-world implementation details I've read Thanks for taking the time to write this.

    by Martín on August 19, 2016 at 12:01 pm

Leave a Reply

Your email address will not be published. Required fields are marked *