Geekfest Palo Alto: WhatsApp’s Secret Sauce: Erlang from behind the Trenches by Francesco Cesarini

at May 9th, 2016

Screen Shot 2016-01-15 at 4.02.33 PMWe are pleased to be hosting our 5th Geekfest talk in Palo Alto this Thursday (May 12th). This talk is open to Groupon employees as well as tech enthusiasts across the Bay Area. If interested, read details below and RSVP on our page.

5:45 PM – Networking and Food
6:15 PM – Talk Begins
7:00 PM – QA and Networking
8:00 PM – Events End

Erlang is a programming language designed for the Internet Age, although it pre-dates the Web. It is a language designed for multi-core computers, although it pre-dates them too. It is a “beacon language”, to quote Haskell guru Simon Peyton-Jones, in that it more clearly than any other language demonstrates the benefits of concurrency-oriented programming.

In this talk, Francesco will introduce Erlang from behind the trenches, looking at how its history influenced its constructs. He will be doing so from a personal prospective, with anecdotes from his time as an intern at the Ericsson computer science lab at a time when the language was being heavily influenced and later when working on the OTP R1 release.

About the Speaker:
Francesco Cesarini is the founder of Erlang Solutions Ltd. He has used Erlang on a daily basis since 1995, starting as an intern at Ericsson’s computer science laboratory, the birthplace of Erlang. He moved on to Ericsson’s Erlang training and consulting arm working on the first release of OTP, applying it to turnkey solutions and flagship telecom applications. In 1999, soon after Erlang was released as open source, he founded Erlang Solutions, who have become the world leaders in Erlang based consulting, contracting, training and systems development. Francesco has worked in major Erlang based projects both within and outside Ericsson, and as Technical Director, has led the development and consulting teams at Erlang Solutions.

He is also the co-author of ‘Erlang Programming’ and ‘Designing for Scalability with Erlang/OTP’ both published by O’Reilly and lectures at Oxford University.

Hope to see you there!

Geekfest Palo Alto: Intro to Geospatial in Redis 3.2 by Dave Nielsen of Redis Labs

at April 26th, 2016

Screen Shot 2016-01-15 at 4.02.33 PMWe are pleased to be hosting our 4th Geekfest talk in Palo Alto this Thursday (April 28th). This talk is open to Groupon employees as well as tech enthusiasts across the Bay Area. If interested, read details below and RSVP on our page:

6:30pm Networking
7:00pm Geospatial Talk
8:00pm Q&A
8:30pm End

Description: Join Dave in this talk where he demonstrates how to speed up mobile apps and web scale systems with Redis geospatial data structures and functions.

In his talk, Dave will demo an open source Geospatial app that depends solely on Redis. Data structures include Geospatial Indexes. Functions demoed are GEOADD, GEOHASH, GEOPOS, GEODIST, GEORADIUS, GEORADIUSBYMEMBER .

About the Speaker

Dave works for Redis Labs organizing workshops, hackathons, meetups and other events to help developers learn when and how to use Redis. Dave is also the co-founder and lead organizer of CloudCamp, a community of Cloud Computing enthusiasts in over 100 cities around the world. Dave graduated from Cal Poly: San Luis Obispo and has worked in developer relations for 12 years at companies like PayPal, Strikeiron and Platform D. Dave gained modest notoriety when he proposed to his girlfriend in the book “PayPal Hacks.”

Groupon’s Workflow Service, “Backbeat”, Now Open Source

at April 14th, 2016

Screen Shot 2016-04-14 at 3.24.13 PM
Groupon operates globally in 28 countries. In order to stay financially nimble at this scale, the company needs automated processes for just about everything. Groupon’s Financial Engineering Developers (FED) team developed and maintain our system for automating merchant payments. The system has paid millions of merchants billions of dollars. One of the tools that helped us reach this scale is a workflow service we created called Backbeat. We recently open-sourced this service and want to share how it helped us and how it can help you.

Let’s see how Backbeat helps out with a simplified example of Groupon’s merchant payment process. For this example we will say there are 3 distinct steps for making a payment-

1. Get Sold Vouchers – We query our inventory systems to discover Groupon vouchers that were sold for a particular contract.

2. Calculate Payment – We get the relevant payment term on the contract from the contract service. Payment term describes how we pay the merchant for that particular contract. We multiply the number of unpaid vouchers by the amount we owe the merchant per voucher. We take this total and create a payment object in our database.

3. Send to Bank– Here we take any payment objects that have not been sent to the bank and call the bank’s API telling them where we want to send the money and how much. If successful, we mark the payments as “sent_to_bank” so that we do not accidentally send them again.

These separate steps, or activities as we call them, need to happen in a specific order. It can be best organized into a simple workflow. A workflow is the series of activities that are necessary to complete a task. Our workflow for this example would look like this-

Screen Shot 2016-04-14 at 3.19.19 PM

We decided to break the payment process into separate steps because it is easier to retry at an individual activity level if they fail. For example: If the bank’s API cannot be reached for whatever reason, we will want to retry the “Send to Bank” activity, but not the whole “Make Payment” process. It also allows for resource level throttling. For example, it may be the case that as a client of the bank API you are only allowed X number of connections to their banking service. Now that “Send to Bank” is its own step, we can have it run asynchronous and limit the number of processes that can run that type of activity.

An alternative approach may be to maintain the workflow implicitly in our application code. Various activities would check the status of previous activities before executing, and we could use Zookeeper or database locks to ensure activities don’t race. This is a lot of bookkeeping for authoring new activities. Additional bookkeeping increases the risk of introducing a bug, and in payments, that might mean making a double payment or none at all! Splitting the business logic from the workflow state provides more visibility into the state of the system, simplifies authoring new workflows, and makes the system less brittle when changes are required.

Introducing Backbeat

Backbeat is a free to use, open-source, workflow service. It allows applications to process activities across N processes/machines while maintaining the order and time in which activities will run. Activities can spawn new activities as they run and will be automatically retried if they fail. Activities can be blocking or non-blocking so you can run things in a parallel or sequentially at any level in a workflow.

It is Backbeat’s responsibility to tell the application when it is time to run an activity and the application responds with the status of the activity (completed or errored) along with any child activities it may want to run. When Backbeat tells the client application to run an activity, the application can put it onto a queue and process it later. Here is sequence diagram of a client interacting with backbeat for our “Make Payment” example above-

Screen Shot 2016-04-14 at 3.20.28 PM

In our example we were running things sequentially, one activity after another. There may be cases where you want to run some activities in parallel and when all of those activities finish you want to perform some other business logic-

Screen Shot 2016-04-14 at 3.21.27 PM

Backbeat can do this by allowing activities to have different modes like blocking, non-blocking, or fire-and-forget. It has other features like scheduling activities to run in the future and linking activities across workflows.

Backbeat was written entirely in Ruby and uses PostgreSQL for its data-store. It uses Redis along with Sidekiq to do its async processing and scheduling. Several engineering teams at Groupon are using Backbeat in production to solve their asynchronous workflow problems. We invite you to start using it for your applications today. The software is free to use and customize with directions on how to set it up on your own server. We have a ruby client for your ruby applications that allows for easy interaction with Backbeat, and we have other language clients in development. We invite the open-source community to contribute and work with us to continually improve it.

Here is the Github Repo and Wiki Page for getting started.

Codeburner – security-focused static code analysis for everyone

at March 11th, 2016


Last year, the Application Security team set out to improve upon a challenging situation: with a single security team and such a large developer community, how do we keep on top of security analysis for the ever-increasing mountain of code?

The answer came about as the result of a GeekOn project to trigger automated static code analysis based on internal deployment notifications.

After some time in development adding features and getting things just right, we’re proud to announce the open source release of Codeburner!

What is Codeburner?

Codeburner uses the OWASP pipeline project to run multiple open source and commercial static analysis tools against your code, and provides a unified web interface to sort and act on the issues it finds.

Since the core backend and scanning engine is built on Rails, Codeburner also provides a full REST API for easy integration with other tools or an existing CI process.

Key Features:

  • Asynchronous scanning (via sidekiq) that scales
  • Advanced false positive filtering
  • Publish issues via GitHub or JIRA
  • Track statistics and graph security trends in your applications
  • Integrates with a variety of open source and commercial scanning tools
  • Full REST API for extension and integration with other tools, CI processes, etc.


You can find full documentation for Codeburner at

Get Involved!

If you’d like to contribute, fork us on GitHub and check out the Developer Guide.

Girl Scouts explore the world of STEM at Groupon’s 4th Scout Out Engineering

at February 15th, 2016


This President’s Day, we welcomed 100 3rd – 6th grade scouts to Groupon, 85 in our Chicago office and 15 in Palo Alto, with the support of almost 50 employee volunteers from the Engineering and Product teams. Throughout the day, the girls got to work with our technical employees to walk through activities on, practice the process of iteration in a hands-on bridge building activity, hear from a panel of female employees, and say hello to our friends across the country all while earning their very own “Scout Out Engineering” merit badge.

_1060225 copy

Chicago volunteer and Software Development Engineer, Shilpa, mentioned, “It’s exciting to see so many young girls here, because I don’t want to be the only girl on the Engineering team. Diversity on our Engineering teams is so important. People solve problems that they can see; and when we have diverse teams, we are more likely to solve a problem from multiple angles and build a better product.”

_1060334 copy

I could not be more proud of the work that employee volunteers have done to grow this program as we’ve traditionally hosted Scout Out Engineering in October with one event in Chicago – and this year, we have had so much excitement around supporting STEM Education and women in technology that we added a February event to the calendar and expanded programming to the west coast.

_1060386 copy

In addition to expanding the program, this year, even the grown-ups got in on the fun as we know it’s so important for students to have the opportunity to continue learning outside of the classroom. Girl Scout Moms, Diana and Carmen, hopped on, and they were hooked. “As Girl Scout parents and troop leaders, this type of program helps our girls to see themselves as the engineers of tomorrow. It gives the girls the opportunity to see different types of STEM careers and meet a number of grown up scouts in the jobs that they could fill in the future. It was fun to hear the girls saying ‘I want to do this when I grow up!’ as they worked with the Groupon volunteers.” both moms stated.

_1060417 copy

Palo Alto volunteer and fellow Software Development Engineer, Sarah, said, “It’s really cool to see girls have the opportunity to be immersed into this kind of environment at such a young age. Growing up, I didn’t have opportunities to meet people like me, so I’m excited to volunteer and help open the doors for young girls to stay interested in STEM.”

STEM Education is an area that Groupon’s Social Responsibility team has committed to as we work to help people and communities thrive and prosper. Our portfolio of community initiatives is designed to help us achieve greater impact and value for society, our employees and the business. Our hope for STEM Education is for more students in diverse and under-served communities to have access and be inspired to pursue education and careers in science, technology, engineering, and math. We are excited to continue this work with the Girl Scouts and show them how cool technology really is!

_1060431 (1) copy

Geekfest Palo Alto Meetup: Machine Learning from Lukas Biewald

at January 15th, 2016

Screen Shot 2016-01-15 at 4.02.33 PM

Groupon’s Palo Alto office is excited to be the newest chapter participating in Geekfest! Our first talk in this monthly series will be from Lukas Biewald of Crowdflower. Lukas will be speaking about machine learning, active learning and human in the loop computing; exploring how artificial intelligence systems interact with human intelligence systems.

The event is Tuesday evening, January 19th at the Groupon office in Palo Alto. See our Meetup page for details and to RSVP.

Geekfest is a technology-agnostic software developer meetup. We strive to have both technical presentations as well as presentations about intra- and inter-personal relationships. We have chapters in Chicago, Seattle and now Palo Alto. If you are interested in speaking at Geekfest please email us at

Hope to see you there on Tuesday!

Kill Bill Metrics

at December 3rd, 2015

Thank you to Kofi Jedamzik for contributing to this project and blog post.

CyberSource rate

Status quo

At Groupon, the majority of applications use a system called Grapher for monitoring. It can plot simple rrd graphs with one minute resolution. However, during the Kill Bill migration, we encountered multiple deficiencies with the existing solution:

  • Grapher offers limited support for templates, which makes it difficult to reuse and maintain the rrd graph definitions.
  • Timerange and Timezone are stored in a cookie, thus sharing the graph link will most likely lead to a different graph (so we ended up sharing screenshots).
  • It is time consuming to change and add new metrics because most of them are based on Splunk searches or cron jobs.
  • The biggest deficiency is the high cardinality of our metrics: we need a lot of context around our metrics. For example, we wanted to get notified when a specific payment method in a specific country for a specific client starts failing. But the number of combinations caused capacity problems in our Splunk cluster.

As a result, we started looking for a simple solution to improve the situation. The starting point was the dropwizard metrics library, which has been established as the de facto standard for metrics in Java based applications. The library makes it very easy to measure different metrics within your application. It supports five different metric-types: Gauges, Counters, Histograms, Meters and Timers. You can also use numerous modules to instrument common libraries like Jetty, Logback, Log4j, Apache HttpClient, Ehcache, JDBI and Jersey.

The collected metrics are kept in metric registries, on top of those you can use reporters to publish your metrics: we use the JmxReporter to expose metrics as JMX MBeans and Metrics Servlets to publish the metrics as JSON objects via HTTP.

Storing metrics in a time series database

Additionally, we just started experimenting with InfluxDB. InfluxDB is a new time series database with promising features like tags and fields.

Tags are indexed and allow fast querying by tag values, which should give us the abilities to breakdown numerous attributes. Another noteworthy feature is its mechanism for downsampling stored data, called Continuous Queries: it lets you aggregate and precompute expensive queries on the fly.

Unfortunately, InfluxDB reporting is not yet supported by the metrics library but as InfluxDB supports the graphite line protocol we use the metrics-graphite reporter to stream the metrics directly into InfluxDB. The drawback is that we have to parse the metric name to extract the metadata. This can be done with InfluxDB’s Graphite Plugin which lets you extract tags from metric names by using a template. For example, instead of duration, host=myhost, method=create-payment, country=de value=1234 we send 1234 and configure a template like this: to extract the tags. This feature is a bit limited as you can only use wildcards and the dot separator, regular expressions would be a nice feature for the future.

Visualizing time series

To visualize the time series, we use Grafana. It supports a variety of time series data sources including InfluxDB and has lots of visualization options, for example annotations, which allow you to mark events like restarts or deployments in your graphs.

Dashboard templating lets you create dynamic visualizations, it even supports variables to change query parameters.

Grafana annotations


This stack was easy to deploy and mitigated a few of our pain points. Compared to the rrd graph syntax, it’s very nice to have a full featured graph editor.

Grafana editor

The dashboard definitions can simply be exported and synchronized with other instances via Grafana’s HTTP API. Sharing graphs is not a problem anymore, you can even choose between UTC and the browsers time zone.

But there is still work to do: instead of only creating metrics via Splunk queries, we measure them directly in the application. This makes the whole pipeline less error prone but the drawback is that the aggregation has to take place in InfluxDB or the application. In addition, the need for graphite templates to extract metadata from the metric name means frequent config changes when we change or add new metrics. So, we are currently figuring out what’s the best way to get support for tags and fields into Kill Bill directly.

Groupon Hosting Palo Alto iOS Meetup on tvOS

at November 9th, 2015

We are excited to host the lastest Palo Alto iOS Meetup next week. See details below and hope to see you there!

What: Palo Alto iOS Meetup on Introduction to tvOS
When: Tuesday, November 17
Time: 6:30-9pm
Where: Groupon Office, 3101 Park Blvd, Palo Alto

Details: Eric Hyche from Groupon will be giving a talk on tvOS. Eric has been on the team investigating a possible Groupon tvOS app, and will share his experience with the new platform. He will focus on the differences between tvOS and iOS, and will build a sample tvOS app during the talk.

About the speaker:
Eric has been with the Groupon iOS team since April 2013, and prior to that was a Principal Engineer with Seattle-based RealNetworks. Eric holds a B.S. in Electrical Engineering from Tennessee Tech University and a M.S. in Electrical Engineering from Georgia Tech.

Groupon adds Redis support to the open source Presto analytic engine.

at October 22nd, 2015

Groupon recently added support for Redis to Facebook’s Presto exabyte scale SQL query engine.

Presto is an open source distributed SQL query engine for running interactive analytic queries against various data sources. Redis is an open-source in-memory key-value data server.

The Redis connector allows querying of live data stored in Redis via SQL queries. This can be used to join data between different systems like Redis and Hive in a single SQL statement.

The Redis/Presto capability allows Groupon to provide value for both consumers and merchants by providing real-time insights via matching real-time customer data such as store foot traffic against historical data stored in Apache Hadoop.

Last year Groupon also added Kafka support to Presto.

Groupon Hosting Meetup on Networking in iOS

at October 20th, 2015

We are thrilled to be hosting this month’s Meetup on Networking in iOS NSSURLSEssion and NSStream. The featured speaker will be Manjula Jonnalagadda and she will go in depth about NSURLSession, how it works as well as NSStream APIs and when to use them. There will be plenty of time for questions, networking and tasty drinks and snacks. Please join us! Festivities start tonight (October 20th) at 6:30pm. Meetup Invite, here.