Maven and GitHub: Forking a GitHub repository and hosting its Maven artifacts

at July 2nd, 2014

Two tools that the Android team frequently uses here are GitHub and Maven (and soon to be Gradle, but that’s another story). Groupon has one of the most widely deployed mobile apps in the world, and with more than 80 million downloads and 54% of our global transactions coming from mobile, it’s important that we have the right tools to keep our operation running smoothly all the time.

The Groupon Android team recently tried to move our continuous integration builds from a Mac Mini on one of our desks to the companywide CI cluster. We got held up during the process because the build simply wouldn’t run on the new VMs. We discovered the problem was a bug in the Maven Android plugin, which neglected to escape the filesystem path when executing ProGuard and caused our build to fail for paths that contained unusual characters. Instead of looking to change the paths of all of our builds company wide, we focused on what it would take to fix the bug in the plugin. Read on for details on how we managed this process.

Fixing the bug itself was easy. The harder part was figuring out how to distribute the fix while we waited for the plugin maintainer to review and merge it in. One of the great advantages of GitHub is that it makes it easy to fork a project and fix bugs. However, if you’re using Maven, you also need a way to host your fork’s artifacts in a Maven repo, and there isn’t a way to automatically do that with GitHub.

You could host your own Nexus server, but that’s a lot of overhead if you don’t already have one for just a simple fork. You could set up a cloud storage solution to hold your Maven repo, but the internet is already littered with defunct and broken Maven repo links – why add one more that you’ll have to maintain it forever?

A better solution to this problem is to use GitHub to host your Maven repositories. A Maven repository is, at its heart, just a structured set of files and directories that are publicly available via http, and GitHub allows you do this easily with its raw download support. The same technique is used by GitHub itself to serve up GitHub Pages websites.

The basic solution involves three steps:

  1. Create a branch called mvn-repo to host your Maven artifacts.
  2. Use the Github site-maven-plugin to push your artifacts to Github.
  3. Configure Maven to use your remote mvn-repo as a maven repository.

There are several benefits to using this approach:

  • It ties in naturally with the deploy target so there are no new Maven commands to learn. Just use mvn deploy as you normally would.
  • Maven artifacts are kept separate from your source in a branch called mvn-repo, much like github pages are kept in a separate branch called gh-pages (if you use github pages).
  • There’s no overhead of hosting a separate Maven Nexus or cloud storage server, and your maven artifacts are kept close to your github repo so it’s easy for people to find one if they know where the other is.

The typical way you deploy artifacts to a remote maven repo is to use mvn deploy, so let’s patch into that mechanism for this solution.

First, tell maven to deploy artifacts to a temporary staging location inside your target directory. Add this to your pom.xml:

<distributionManagement>
    <repository>
        <id>internal.repo</id>
        <name>Temporary Staging Repository</name>
        <url>file://${project.build.directory}/mvn-repo</url>
    </repository>
</distributionManagement>

<plugins>
    <plugin>
        <artifactId>maven-deploy-plugin</artifactId>
        <version>2.8.1</version>
        <configuration>
            <altDeploymentRepository>internal.repo::default::file://${project.build.directory}/mvn-repo</altDeploymentRepository>
        </configuration>
    </plugin>
</plugins>

Now try running mvn clean deploy. You’ll see that it deployed your maven repository to target/mvn-repo. The next step is to get it to upload that directory to github.

Add your authentication information to ~/.m2/settings.xml so that the github site-maven-plugin can push to github:

<!-- NOTE: MAKE SURE THAT settings.xml IS NOT WORLD READABLE! -->
<settings>
  <servers>
    <server>
      <id>github</id>
      <username>YOUR-USERNAME</username>
      <password>YOUR-PASSWORD</password>
    </server>
  </servers>
</settings>

(As noted, please make sure to chmod 700 settings.xml to ensure no one can read your password in the file.)

Then tell the github site-maven-plugin about the new server you just configured by adding the following to your pom:

<properties>
    <!-- github server corresponds to entry in ~/.m2/settings.xml -->
    <github.global.server>github</github.global.server>
</properties>

Finally, configure the site-maven-plugin to upload from your temporary staging repo to your mvn-repo branch on github:

<build>
    <plugins>
        <plugin>
            <groupId>com.github.github</groupId>
            <artifactId>site-maven-plugin</artifactId>
            <version>0.9</version>
            <configuration>
                <message>Maven artifacts for ${project.version}</message>  <!-- git commit message -->
                <noJekyll>true</noJekyll>                                  <!-- disable webpage processing -->
                <outputDirectory>${project.build.directory}/mvn-repo</outputDirectory> <!-- matches distribution management repository url above -->
                <branch>refs/heads/mvn-repo</branch>                       <!-- remote branch name -->
                <includes><include>**/*</include></includes>
                <merge>true</merge>                                        <!-- don't delete old artifacts -->
                <repositoryName>YOUR-REPOSITORY-NAME</repositoryName>      <!-- github repo name -->
                <repositoryOwner>YOUR-GITHUB-USERNAME</repositoryOwner>    <!-- github username  -->
            </configuration>
            <executions>
              <!-- run site-maven-plugin's 'site' target as part of the build's normal 'deploy' phase -->
              <execution>
                <goals>
                  <goal>site</goal>
                </goals>
                <phase>deploy</phase>
              </execution>
            </executions>
        </plugin>
    </plugins>
</build>

The mvn-repo branch does not need to exist, it will be created for you.

Now run mvn clean deploy again. You should see maven-deploy-plugin “upload” the files to your local staging repository in the target directory, then site-maven-plugin committing those files and pushing them to the server.

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building DaoCore 1.3-SNAPSHOT
[INFO] ------------------------------------------------------------------------
...
[INFO] --- maven-deploy-plugin:2.5:deploy (default-deploy) @ greendao ---
Uploaded: file:///Users/mike/Projects/greendao-emmby/DaoCore/target/mvn-repo/com/greendao-orm/greendao/1.3-SNAPSHOT/greendao-1.3-20121223.182256-3.jar (77 KB at 2936.9 KB/sec)
Uploaded: file:///Users/mike/Projects/greendao-emmby/DaoCore/target/mvn-repo/com/greendao-orm/greendao/1.3-SNAPSHOT/greendao-1.3-20121223.182256-3.pom (3 KB at 1402.3 KB/sec)
Uploaded: file:///Users/mike/Projects/greendao-emmby/DaoCore/target/mvn-repo/com/greendao-orm/greendao/1.3-SNAPSHOT/maven-metadata.xml (768 B at 150.0 KB/sec)
Uploaded: file:///Users/mike/Projects/greendao-emmby/DaoCore/target/mvn-repo/com/greendao-orm/greendao/maven-metadata.xml (282 B at 91.8 KB/sec)
[INFO] 
[INFO] --- site-maven-plugin:0.7:site (default) @ greendao ---
[INFO] Creating 24 blobs
[INFO] Creating tree with 25 blob entries
[INFO] Creating commit with SHA-1: 0b8444e487a8acf9caabe7ec18a4e9cff4964809
[INFO] Updating reference refs/heads/mvn-repo from ab7afb9a228bf33d9e04db39d178f96a7a225593 to 0b8444e487a8acf9caabe7ec18a4e9cff4964809
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.595s
[INFO] Finished at: Sun Dec 23 11:23:03 MST 2012
[INFO] Final Memory: 9M/81M
[INFO] ------------------------------------------------------------------------

Visit github.com in your browser, select the mvn-repo branch, and verify that all your binaries are now there.

mvn-repo

Congratulations!

You can now deploy your maven artifacts to a poor man’s public repo simply by running mvn clean deploy

There’s one more step you’ll want to take, which is to configure any poms that depend on your pom to know where your repository is. Add the following snippet to any project’s pom that depends on your project:

<repositories>
    <repository>
        <id>YOUR-PROJECT-NAME-mvn-repo</id>
        <url>https://raw.github.com/YOUR-USERNAME/YOUR-PROJECT-NAME/mvn-repo/</url>
        <snapshots>
            <enabled>true</enabled>
            <updatePolicy>always</updatePolicy>
        </snapshots>
    </repository>
</repositories>

Now any project that requires your jar files will automatically download them from your Github Maven repository.

Here at Groupon, we used this technique to fix our bug in Maven Android plugin, and then easily shared our fork with the wider Android community until the fix was incorporated upstream. This works particularly well for projects that are no longer being maintained since those projects may never get around to merging in your pull requests.

We hope this makes forking Maven repos in GitHub as easy for you as it has been for us!

No Tags


2 thoughts on “Maven and GitHub: Forking a GitHub repository and hosting its Maven artifacts

  1. wonderful article mike... can maven repo be hosted on github when the hithub repo is a private repo?

    by Azitabh on October 13, 2014 at 1:23 am
  2. Excellent! I wonder if now would be a good time to update this with a gradle example. I for one would really appreciate it. Thanks you!

    by Chris on June 10, 2015 at 2:49 pm

Leave a Reply

Your email address will not be published. Required fields are marked *