HashiCorp Announces Vault, a Tool for Managing Secrets

at May 22nd, 2015

Sean Chittenden (Groupon Production Operations) invited Armon Dadgar from HashiCorp to give a Tech Talk about Vault, a new secrets management tool. Armon both unveiled Vault from our office and discussed its importance in securing and managing secrets in a modern datacenter. The implications of Vault’s new approach to this old problem are far reaching and could benefit engineering teams. We were delighted by the understanding HashiCorp brought to this problem space and are excited to see where this technology goes.

Thank you to Armon and HashiCorp for taking the time to share Vault with Groupon Engineering.

First Kill Bill hackathon

at April 17th, 2015

Groupon recently hosted a Payments Tech Talk where Stéphane and I went over our progress deploying Kill Bill internationally at Groupon, and what our roadmap for the rest of the year looks like. To our surprise, most attendees were engineers, and while we went over the architecture of Kill Bill, there was clearly an ask to dig deeper into the code. This is why we came up with the idea of a hackathon, which Groupon hosted in their San Francisco office last Saturday.

Because it was our first event of such kind, we decided to invite only engineers from a few select companies which we knew were already in the process of switching to Kill Bill and were familiar with the system. The event lasted from 11am to 7pm: Stéphane and I covered two brief technical sessions, but the rest of the afternoon was focused on coding and answering questions.

The day turned out to be quite productive: Greg wrote the first ever Kill Bill plugin in Scala, Sean gave Kaui a much needed facelift, and Ivan updated the Node.js client. We also had design discussions leading to API changes for 0.13.7 and talked about migration best practices. Even Stéphane and I managed to get some coding done: he wrote a performance benchmark testing specifically usage APIs, and I worked on automatic deployments of our Docker image to EC2 using Cloud 66.

The most exciting part of the day for me was when I realized the breadth of the ecosystem. When, at the beginning, we were all brainstorming about the type of projects to work on, quite a few ideas were generated, such as a Go client library, Chef recipes for deployment, and a new admin UI in AngularJS. While the very core of Kill Bill is a Java library, you don’t need to know Java to contribute to the project, and whether you are a back-end or front-end developer, whether you know Python or Rust, you can be part of the Kill Bill developer community (and if you want to do this as your full-time job, Groupon is hiring!).

We’re happy the event was a success, and we’ve already been asked to organize a follow-up one. This will happen at some point in the summer, and we will probably open it to a bigger audience. If you want to participate, get in touch!

Groupon Engineer Mike Burton Releases Android App Development for Dummies

at March 13th, 2015

indexAt Groupon, we do amazing things.

And none of them would be possible without our talented, passionate and curious employees who contribute to our distinct culture and make working at Groupon so great.

Our Head of Mobile Engineering, Mike Burton, is one example — and his passion extends well beyond the workplace. Driven by a penchant for all things Android, he’s just released the third edition of Android App Development for Dummies. The guide is the first to delve into the brand new Android Wear APIs and provide instruction on building apps for Android watches, TV and Kindle. Mike shares best practices and learnings from his experience building several successful apps both independently and on Groupon’s Android team.

I’m incredibly proud of Mike and have already purchased his book on Amazon. Be sure to message him at @roboguice if you’re a mobile developer and would like learn about opportunities on the team or to find out more about Mike’s extensive experience in the space.

Groupon in VentureBeat

at February 24th, 2015

Groupon’s excited to sponsor Venture Beat’s Mobile Summit for a second straight year. Great collection of mobile entrepreneurs and a ton of awesome ideas. I wrote an op-ed for the conference about the importance of mobile testing and tracking. Check it out here.

DMARC at Groupon

at December 17th, 2014

At Groupon we are a global company sending email in 47 countries worldwide. Our mission is to connect our customers with our merchant partners through price and discovery using email as one of the communication channels. Given the global reach and strength of our brand “bad actors” have attempted to misuse our brand and email domains through phishing activity to trick unsuspecting users into providing sensitive personal information. As such we began the work to implement Domain-based Message Authentication, Reporting & Conformance policies, or DMARC for short, globally to combat these “bad actors.”

DMARC is a policy-reporting layer built on top of standard email authentication protocols known as Sender Policy Framework (SPF) & Domain Keys Identified Mail (DKIM). At a high level SPF allows receiving email servers to check whether email from a domain is sent using approved infrastructure or IPs. DKIM applies similar concepts at the domain level but uses a private/public key pair to validate pre-defined portions of the email message from the domain in question. From an execution level SPF and DKIM both rely on DNS lookups to function correctly.

At Groupon SPF and DKIM are standard authentication protocols used in every country we operate. As such we took the next step to implement DMARC around the world in an effort to fight phishing and create a feedback loop for how our email domains are utilized in the wild. DMARC operates through a DNS record where we are able to tell participating email providers like Gmail, Hotmail, and Yahoo to take specific policy actions (none, quarantine, reject) for email failing SPF & DKIM.

When declaring a policy of “none”, defined as “p=none” in the below example, we are instructing the participating email providers to take no action with messages failing authentication. Even though no action is taken we still receive reports on how email is passing or failing authentication from those providers. The reports are sent to the email addresses defined below in the “rua=” and “ruf=” sections. The “rua” option refers to an aggregate report of failures. It can be thought of as a high level aggregate failure report. The “ruf” option is the more detailed reporting path, providing significantly more and detailed forensic reports for every failure. At Groupon we work with Agari, an email security company, to compile this data into human readable reports, which support our DMARC work globally. Overall, the “p=none” step is key in our DMARC rollout process as we use this data to create a baseline for authentication performance and ensure we are in a position to not block legitimate email when we choose to enforce a “quarantine” or “reject” policy.

v=DMARC1; p=none; fo=1; rua=mailto:example@example.com; ruf=mailto:example@example.com; rf=afrf; pct=100

After a complete and thorough audit at the “p=none” stage we move to publishing a “quarantine policy”, defined as “p=quarantine” in the below example. When declaring a quarantine policy we are instructing email providers to send any email failing SPF & DKIM to spam, which quarantines the email outside the users’ inboxes. It is at this stage that we take advantage of the “pct” feature. This gives us the ability to inform email providers about the percentage of email failing authentication to quarantine. At Groupon we found that anything less than 50% does not provide a significant enough sample size to analyze the data for when to move to publishing a “reject policy.”

v=DMARC1; p=quarantine; fo=1; rua=mailto:example@example.com; ruf=mailto:example@example.com; rf=afrf; pct=50

Once any remaining issues have been corrected at the quarantine stage we publish a “reject policy”, which is represented as “p=reject” in the below example. Publishing a “reject policy” instructs any participating email providers to block all email failing authentication from reaching the inbox or spam folder. As a practice at Groupon when we reach this stage we leave the “pct” option set to 100, which instructs participating email providers to block 100% of all email failing authentication. This is done to take full advantage of the anti-phishing benefits DMARC provides and is possible due to the work completed to ensure no legitimate email is blocked by accident.

Throughout the DMARC process we have alerts set to trigger if any failures on legitimate email exceed our internal thresholds. These alerts take center stage when we reach the “reject” phase. If our pre-defined thresholds are met, it initiates a rollback of DMARC policies from “quarantine” or “reject” to “none” in the effected region to ensure email is not inadvertently blocked.

v=DMARC1; p=reject; fo=1; rua=mailto:example@example.com; ruf=mailto:example@example.com; rf=afrf; pct=100

We follow the process of moving incrementally from a policy of “none” to “quarantine” and eventually “reject” to make changes in a controlled fashion. A staged rollout allows us to adjust the process as needed by responding to what the data highlights as our action items at each phase. This provides the opportunity to complete our due diligence while minimizing the overall risk of blocking legitimate email to our subscribers. I am happy to report that we are enforcing DMARC policies in 45 countries with 43 countries publishing a “reject policy.”

The implications of being able to globally reject phishing emails that are targeting our subscribers and brand are enormous. Recently in Brazil we tracked a phishing campaign offering discount iPhones in an attempt to steal credit card information. (screenshot below)


Due to our use of DMARC and the stellar implementation by my team in South America we were already publishing a “reject policy” for our mailing domain in Brazil, r.grouponmail.com.br. As a result we were able to proactively block around 50,000 phishing emails targeting Gmail, Hotmail, and Yahoo! addresses, which added another layer of protection for our subscribers. (data below)


We will continue to roll out DMARC through the remaining countries to ensure our subscribers are able to benefit from the anti-phishing protection they deserve. Once the process is completed all Groupon email operations will be covered by DMARC. For Game of Thrones fans, DMARC can be thought of as a member of the Night’s Watch, silently standing guard on The Wall. DMARC protects the Groupon realm from phishing attempts and keeps our subscribers and brand safe in the process.

Groupon Selected as One of the Best Apps of 2014

at December 8th, 2014

Screen Shot 2014-12-08 at 2.49.39 PM

We are all very excited that Google has named Groupon on of the Best Apps of 2014. We work very hard to make our app fun and delightful, and are happy that people love it and consistently give us great reviews. We’ve recently refreshed the UI, added your reviews and tips for many merchants, and made significant architectural changes to get us a 40% improvement in startup times. There’s a lot more to come so look forward to our releases in 2015!

Well done to all the teams that have contributed to this effort!

How do Groupon Customers Fare When it Comes to Gift Giving?

at November 24th, 2014

It’s that time of year!! And personally, it’s my favorite time of year! I love what the season represents: family, togetherness, generosity, and opportunities to show appreciation for one another.

This month I thought I would step back and take some time for something that’s always fun……PRESENTS! As the gift giving season is upon us, the Groupon Data Science team is here to tell you who are the best Groupon gift givers!

As a whole, the industry has been experiencing a shift toward online shopping and more recently a shift toward shopping on Mobile. Last year Mobile traffic accounted for 30+% site visits on Cyber Monday. At Groupon, mobile accounts for more than 50% of our transactions worldwide.

As more and more people decide to buy products on their phones, we thought it would be interesting to know who are the better gift givers: iPhone or Android users?

First off, Groupon users spend 45% more online than your average US consumer! So make sure you cozy up to your Groupon-loving friends this season!

Not only do Groupon customers spend more money online, they are more generous to others than to themselves! All customers spend more when buying a Groupon deal as a gift than when buying a Groupon deal for themselves. But as we see later, Groupon app users are the more generous gift givers.

Screen Shot 2014-11-24 at 4.26.17 PM

Q: Who gets more in the spirit in gift giving?

iPhone users. The data suggests that iPhone users tend to get a little more in the spirit spending upwards of 50% more on a gifted Groupon deal than on Groupon deal for themselves. If you have an iPhone user friend spending $50 on average on Groupon deal you can expect them to spend $75 on a gift! But Android friends aren’t too shabby either and compared to all Groupon users, are overall more generous when it comes to gift giving. When looking at a random Monday, the average Android user’s generosity surpasses that of an average iPhone user’s.

Screen Shot 2014-11-24 at 6.10.12 PM

Q: The holiday season can be hectic, taking care of oneself is important! Who takes care of themselves the best?

Android users. Android users spend 10-20% more on purchases during the holiday season. Cyber Monday seems to be the day when everyone goes for that upgrade and pays a few extra dollars to get something nice for themselves. It is the peak time for self-spend, especially for Android users. And with Groupon’s crazy Cyber Monday deals why wouldn’t you treat yo self, even Batman does.

Q: The most annoying giver gifts items better suited for himself then for the recipient, who is the biggest offender?

Neither. Neither Andriod users or iPhone users are guilty of having the same purchase profile for themselves as they do for gift giving. Interestingly, the different platforms’ gift giving patterns stay true to stereotypes: andoid = more techie, iphone = experience.

Screen Shot 2014-11-24 at 5.04.18 PM

So back to the original question: who gives the best gifts? It probably depends on what you’re looking for: cool gadgets or fun experiences! Either way, Groupon’s got it all this Holiday season and we’re kicking it off a little early with these killer Black Friday deals!

On The Subject of Girls, Technology, and Marshmallow Or: how the Evolution of Girl Scouts and STEM is evident at Groupon

at November 14th, 2014


Groupon recently opened its green doors to some of the Girl Scouts’ best and brightest for our Scout Out Engineering event. For the second and consecutive year, Groupon Engineering and the Groupon Employee Volunteer Program partnered with the Girl Scouts of Greater Chicago and Northwest Indiana to welcome 5th and 6th graders into the Chicago Groupon office for a morning of learning, fun, and tech engagement.

Scout Out Engineering introduces girls to engineering concepts through a combination of presentations and hands-on learning. Groupon’s goal is to excite these girls about technology and keep them interested in engineering and STEM education.

IMG_7898 (1)

IMG_7905 (1)

One tenet of the Girl Scouts that makes them great is their all inclusive, ‘every girl’ approach. For the Girl Scouts, every girl should be able to participate in any activity regardless of her background or skillset. Last year, Groupon was advised to plan for girls with no internet in their homes, no experience with computers, and no idea who – or what – Groupon was as a tech company. With those guidelines in mind, we planned the program as a hands-on engineering centered event that, for a tech company, was strangely void of computers.

If the focus of last year’s program was to introduce the idea of STEM education and emphasize its importance, then this year’s focus was to build on that foundation and actually do something about it.

In the six months leading up to our 2014 planning, ideas incubated and matured, technology advanced, and the profile of the ‘every girl’ evolved. In 2014 ‘every girl’ used a computer, a smartphone, and got exposed to some aspect of STEM education daily. The Girl Scouts encouraged Groupon to incorporate computers into the program–many of the girls may have already done some form of coding–and there were no limits on what technology the girls could be exposed to.

With these new guidelines we designed a program with a tech heavy core that better represented the work that happens here at Groupon. Hands-on computer learning took center stage and the focus on coding allowed participants the chance to code alongside top engineers and continue their learning outside Groupon’s green walls. A bridge building activity became an opportunity for girls to work cross functionally and employ a few of the key concepts that keep Groupon Engineering running. Girls learned about agile methodologies, iterated on their work, and closed the day with a real, live white boarding retrospective session (and, of course, pizza.)

IMG_7521 (1)

IMG_7907 (1)

Scout Out Engineering at Groupon exposed girls to technology in an immediate and accessible way. It became an event for Groupon employees to use their talents to spark interest in subject matter that they are passionate about, and it gave everyone the opportunity to realize how essential empowering young girls can be. When it comes to STEM education at Groupon, there has always been an abundance of employee support and our support for the Scout Out Engineering event was no different. From the planning team, to speakers, to volunteers, Groupon Engineering was ready and willing to donate time, energy, and resources to teach these girls a thing or two about tech.

IMG_7522 (1)