The Fedora project COPR service builds/hosts free RPM package repositories and provides useful automation hooks for your Devopsing.  You can combine COPR and any standard Git source (Github, Gitlab, etc) with webhooks for automated package builds when your source repository is updated.

Our Setup Example: QUADS Project
In the last year I’ve been working on QUADS, an upstream Open Source project that helps us schedule future, automated large-scale network and systems fleet provisioning.

This is our example as we recently setup automated RPM package builds triggered when the master branch is updated.  While QUADS can always be consumed by cloning the repository we wanted to provide a system package version as well.  Here’s what our workflow will look like:

• Patch submitted to Gerrit code review
• Jenkins CI container pipelines run checks and tests and vote +/-
• Code passes peer code review, gets merged to master branch on Github.
• Github webhook pushes build request to COPR, creating a new RPM package.

Getting Started – The Git-friendly RPM .spec
Before you can setup builder automation you first need a nicely working RPM .spec file that builds locally with no issue.  There will be some key differences between a traditional .spec file and one modified to work with external sources like Github, Gitlab, etc.  We’re going to look at the QUADS RPM .spec for these examples.

Here are the major differences:

Sources will need to point to your Git repository

Source0: https://github.com/redhat-performance/quads/archive/master.tar.gz#/%{name}-%{version}-%{release}.tar.gz

You need to use autosetup.

%prep
%autosetup -n %{name}-master

Some RPM macros may need workarounds to work in COPR.  In our usage with COPR we could not perform shell expansion to make the current YYYY-MM-DD become our version.  We instead needed a workaround like the following (thanks @hguemar)

%define build_timestamp %{lua: print(os.date("%Y%m%d"))}
Release: %{build_timestamp}

Always pick library version for oldest distribution you will support.  Because we want to support CentOS/RHEL (7.x +) as well as cutting-edge versions of Fedora we set our package library version to match the current version of the oldest distribution.

Requires: PyYAML >= 3.10
Requires: ansible >= 2.3
Requires: expectk >= 5.0
Requires: python2-aexpect >= 1.4
Requires: python-requests >= 2.6

These are the main changes we had to make but your experiences may differ, feel free to borrow anything from our .spec file here as needed.  This is by no means an exhaustive RPM .spec file guide, consult official documentation to fill in any gaps.

Testing Locally – Rpmbuild
Once you’ve got your .spec file setup you’ll want to test building locally before it’s ready to let the COPR builders handle it.  Since you’re dealing with remote sources you’ll first need some system prep to build a traditional package.

yum install rpmdevtools rpm-build -y

Next you need to prep your .spec with spectool.

spectool -g -R quads.spec

Now you can build the package.

rpmbuild -ba quads.spec

COPR Integration and Builds
Your Git source should contain a sub-directory with your .spec file once it’s ready and builds successfully on your local machine.  Make sure that is in place before integrating COPR in the next steps.

If you don’t have a Fedora FAS account you should also create an account so you can use the COPR service.  Once that’s sorted you can create your first COPR project.

Create New COPR Project

Next you need to fill out some basic information about your new project.

You’ll also want to select the architectures and distributions you want to target.  QUADS is a noarch Python project but we selected x86_64 here.  Choose epel if you want to also build for CentOS / RHEL respectively.

The last settings at the bottom are important, you need to have enable internet access during builds enabled as your source is a remote Git platform.

Click submit.

Adding a New Package for the Project
Navigate to the packages area and add a new package.

Next you’ll need to add a build location, select the following:

• source type: SCM
• Webhook rebuild: enabled
• subdirectory: where your .spec file lives if it’s not in your repo root.
• rpkg: choose this for the SRPM mechanism

Setup Your Git Webhook

Note the webhooks settings hyperlink in the previous screen, you’ll need to setup a webhook on your Git platform now.  You can access this via settings -> webhooks.  There will be instructions for several Git platforms to use – in this case we’re using Github.

In our case we added this inside Github settings for our repository but Gitlab is also supported.

Pulling it all Together
At this point you’re ready to test everything.  Pushing to the master branch should trigger a new package build.  You can use the following markdown in your README.md to highlight the current build status too:

![rpmbuild](https://copr.fedorainfracloud.org/coprs/youruser/yourproject/package/yourpackagename/status_image/last_build.png)

You should see something like  

You should also see your successful builds on the builds page.

Note: the build image is cached by Github so if it isn’t accurately reflected you can use curl against the Github SDN to remove the image cache.

curl -X PURGE https://camo.githubusercontent.com/201a0fe7eXXXX

Where /201a0fe7eXXXX is the full URL when you hover on your README.md over the build icon itself.

Testing via Src RPM or .Spec
You can also do a manual build by uploading a src.rpm or .spec file directly into COPR.  We used this method to test that our .spec file was COPR-friendly.  There is also no conflict between uploading a .spec or .src.rpm and still keeping your webhook automation in place.

Consuming RPM Packages
When you’re happy with your setup people can consume packaging with one or two easy commands, substitute below for your COPR account and project name.

On Fedora:

dnf copr enable quadsdev/QUADS -y
dnf install quads -y

On CentOS / RHEL:

yum install yum-utils -y
yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/quadsdev/QUADS/repo/epel-7/quadsdev-QUADS-epel-7.repo
yum-config-manager --enable quadsdev-QUADS-epel-7
yum install quads -y

Package updates from your source will now be integrated into your operating system package management.

Troubleshooting Build Failures
COPR exposes all the logs from the builders, you can find these by click on the individual build and drilling into the chroot / release directory on the lower left-hand side.

From here you can see an HTTP index with all the relevant build logs.

Further Git and COPR Automation
Webhooks aren’t the only automated way to use the awesome COPR Fedora build service, you can also trigger building a src.rpm and uploading it to COPR via their API using tito.  Miroslav Suchy has a good blog post that covers this.

Dominic Cleal has a good post on using tito and git-annex to do reproducible COPR build automation as well.