Mirror git repositories and serve them with cgit

Published:
Tags: git, self-hosting

We’ll start with our repositories hosted at an external service like GitHub and end up mirroring and serving our repos on our own server. We’ll use cgit to serve git data, a very fast (it’s written in C) and widely used git web frontend. Among its notable users are Linux and GNU.

This tutorial assumes a Debian 9+/Ubuntu 16.04+ system, replace the commands with apt with the equivalent for your package manager.

Mirror

First we’ll need to retrieve our git repos: git clone --mirror all of them. Instead of manual labor we’ll use a tool called giternity to do this work for us. We give it a GitHub username or repo and it fetches and updates them. It also retrieves metadata to display with cgit. Let’s install it:

sudo apt install python3-pip git
sudo pip3 install giternity

Its configuration is at /etc/giternity.toml, create the file:

# path where to keep the git mirrors
git_data_path = "/srv/git/"

# path for checkouts of the git mirrors (optional)
checkout_path = "/srv/git_checkout/"

# URL of your cgit instance (optional)
cgit_url = "https://git.cpu.re/"

[github]
repositories = [
    "octocat",
]

Set git_data_path to where you want to keep your gits, we’ll assume it’s set to /srv/git/ for the rest of this tutorial. Giternity will populate this directory with bare git repositories: the data kept in .git when you normally clone a repo. If you also want to have the actual files, set checkout_path to where to store them. cgit_url should hold the public URL where your repos will be browsable, comment out that line if you won’t use cgit. Replace octocat with your GitHub username, or any other user/organization whose (non-fork) repos you want to mirror. In the repositories list you add usernames/organizations by their name and individual repos as owner/repo, for example syl20bnr/spacemacs. You may add as many as you like.

Now you can be lazy and do sudo giternity --configure or read this whole paragraph and do it manually instead. Create a separate system user to run giternity with: sudo adduser giternity --system --no-create-home. This is to have some security by isolation: if giternity would turn bad, it would only be able to touch the git data. If it was running as root, it could ruin everything. Create the data directory: sudo mkdir -p /srv/git/ and give giternity access to it: sudo chown -R giternity /srv/git/. Do the same for the checkout_path if you’re using it. Finally create a cron job at /etc/cron.d/giternity to run giternity every hour (make sure the cron job ends with a newline, otherwise it won’t run!):

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
0 * * * * giternity giternity

Perform the initial mirroring once: sudo -u giternity giternity. If everything went well, this should fill up /srv/git/. Thanks to the cron job our local repos stay in sync and new repos are picked up when they appear on GitHub.

Do you want to mirror repos hosted on a different service than GitHub? I’ve planned to add support for GitLab and git repos hosted at arbitrary locations, leave a comment on the issue tracker if you need either of those features.

Serve

With our git repos arranged we can now serve them. Cgit is a CGI application, it needs a web server to talk with web browsers. Here we’ll use Nginx. First install cgit and some recommended packages:

sudo apt install cgit python3-pygments python3-markdown python3-docutils

We’ll use fcgiwrap with which Nginx and Cgit can communicate:

sudo apt install nginx fcgiwrap
sudo systemctl enable nginx.service fcgiwrap.socket
sudo systemctl start fcgiwrap.socket

The cgit package places its data in /usr/share/cgit/, there you’ll find the default css, logo and favicon. The CGI application is in /usr/lib/cgit/. Place this Nginx server conf in /etc/nginx/sites-enabled/cgit.conf:

server {
    listen 80;
    server_name git.cpu.re;

    location /cgit-css/ {
        alias /usr/share/cgit/;
        expires 24h;
        try_files $uri =404;
    }

    location / {
        include /etc/nginx/fastcgi_params;
        fastcgi_param PATH_INFO $uri;
        fastcgi_param QUERY_STRING $args;
        fastcgi_param SCRIPT_FILENAME /usr/lib/cgit/cgit.cgi;
        fastcgi_pass unix:/run/fcgiwrap.socket;
    }
}

Change server_name to the domain name or IP address controlled by your server and customize the config as needed. (For Apache web servers there is a sample configuration available at /etc/apache2/conf-available/cgit.conf, with Apache you don’t need fcgiwrap.)

Reload Nginx: sudo systemctl restart nginx.service.

Now we can configure cgit. Place the following at /etc/cgitrc:

# cgit config
# see cgitrc(5) for details

css=/cgit-css/cgit.css
logo=/cgit-css/cgit.png
favicon=/cgit-css/favicon.ico

# Highlight source code with python pygments-based highlighter
source-filter=/usr/lib/cgit/filters/syntax-highlighting.py

# Format markdown, restructuredtext, manpages, text files, and html files
# through the right converters
about-filter=/usr/lib/cgit/filters/about-formatting.sh

## Search for these files in the root of the default branch of repositories
## for coming up with the about page:
readme=:README.md
readme=:README.rst
readme=:README.txt
readme=:README

## List of common mimetypes
mimetype.gif=image/gif
mimetype.html=text/html
mimetype.jpg=image/jpeg
mimetype.jpeg=image/jpeg
mimetype.pdf=application/pdf
mimetype.png=image/png
mimetype.svg=image/svg+xml
mimetype-file=/etc/mime.types

root-title=Stupid Content
root-desc=Git Repositories
virtual-root=/

enable-commit-graph=1
enable-index-links=1
enable-log-linecount=1
enable-http-clone=1
enable-index-owner=0
repository-sort=age
case-sensitive-sort=0

# Enable caching of up to 1000 output entries
cache-size=1000

agefile=info/web/last-modified
section-from-path=1
scan-path=/srv/git/

You should customize the root-title and root-desc. Cgit has many options, look at the man page man cgitrc for more info. Cgit should now be visible at your server_name address. The git repositories mirrored by giternity should all be visible, this is thanks to the scan-path, which should be the same as your earlier git_data_path.

The above example config uses the earlier installed python3-pygments package to highlight source code and the python3-markdown python3-docutils packages to render Markdown and reStructuredText documents.

git.cpu.re

My cgit instance is available at https://git.cpu.re. Here are the exact configuration files it is using.

/etc/giternity.toml:

git_data_path = "/srv/git/"
checkout_path = "/srv/git_checkout/"

cgit_url = "https://git.cpu.re/"

[github]
repositories = [
    "rahiel",
    "sunsistemo",
    "TeMPOraL/nyan-mode",
]

/etc/nginx/sites-enabled/git-cpu-re.conf:

server {
    listen 443;
    server_name git.cpu.re;

    location /cgit-css/ {
        alias /usr/share/cgit/;
        expires 24h;
        try_files $uri =404;
    }

    location / {
        include /etc/nginx/fastcgi_params;
        fastcgi_param PATH_INFO $uri;
        fastcgi_param QUERY_STRING $args;
        fastcgi_param SCRIPT_FILENAME /usr/lib/cgit/cgit.cgi;
        fastcgi_pass unix:/run/fcgiwrap.socket;
    }

    location /.well-known/ {    # Let’s Encrypt
        root /home/rahiel/cpu.re/public/;
        try_files $uri =404;
    }
}

server {
    listen 80;
    server_name git.cpu.re;

    # redirect to https
    return 301 https://git.cpu.re$request_uri;
}

/etc/cgitrc:

# cgit config
# see cgitrc(5) for details

css=/cgit-css/cgit.css
logo=/cgit-css/cgit.png
favicon=/cgit-css/favicon.ico

# Highlight source code with python pygments-based highlighter
source-filter=/usr/lib/cgit/filters/syntax-highlighting.py

# Format markdown, restructuredtext, manpages, text files, and html files
# through the right converters
about-filter=/usr/lib/cgit/filters/about-formatting.sh

## Search for these files in the root of the default branch of repositories
## for coming up with the about page:
readme=:README.md
readme=:README.rst
readme=:README.txt
readme=:README

## List of common mimetypes
mimetype.gif=image/gif
mimetype.html=text/html
mimetype.jpg=image/jpeg
mimetype.jpeg=image/jpeg
mimetype.pdf=application/pdf
mimetype.png=image/png
mimetype.svg=image/svg+xml
mimetype-file=/etc/mime.types

root-title=git@cpu.re
root-desc=Git repositories hosted at cpu.re
virtual-root=/

enable-commit-graph=1
enable-index-links=1
enable-log-linecount=1
enable-http-clone=1
enable-index-owner=0
repository-sort=age
case-sensitive-sort=0

# Enable caching of up to 1000 output entries
cache-size=1000

agefile=info/web/last-modified
section-from-path=1
scan-path=/srv/git/

RSS Contact