Git-Teleport: Send Your Data to the Mirror Universe

git-teleport mirrors Git repositories between sites which cannot connect directly via SSH, HTTPS, or any other Git networking protocol. If

  • you have a Git repository in one place,
  • you need it to be in another, and
  • you can't use the normal Git transfer protocols to get it there,

then git-teleport is for you. Git-teleport allows you to mirror one or more local repositories to a remote system that is not accessible over the network. Updates can be transmitted via any binary-safe medium, including "sneakernets" if need-be.

Git-teleport builds on and simplifies the existing "bundle" functionality that is built into Git. Multiple repositories can be mirrored en masse—reliably—with a single command. Git-teleport can update test equipment and computers in the field, without a working internet connection. It can also be used to collaborate across restrictive firewall environments.

To get started, see "Install."

Git-Teleport Project Logo

Theory of Operation

Git allows you to make bundle files, which are binary archives of commits. Mirrors can fetch from these bundles, just as if they were regular remotes.

Complete bundles of some repositories are very large. This makes them inconvenient to transfer. Fortunately, bundles can be incremental: i.e., they can contain only a range of commits. As long as your bundle has commits in common with the mirror, then the mirror can fetch it and be updated.

Creating bundles which are minimal—yet still contain enough commits to do the job—is a non-trivial process. It requires a careful pruning of the history, which is difficult to do without knowing what the destination already has.

git-teleport solves this problem by keeping a record of all the "refs" it has already sent. Refs are named objects, like branches and tags. You can list the refs in any repository with

git show-ref -d

When selecting commits (and tags) to bundle, one can use an old list of refs to tell git: "Bundle everything EXCEPT commits reachable from X, Y, and Z." If the destination repository already contains commits g00d, b33f, and a tag 1.0.0, then you can exclude these with something like

git bundle create repo.bundle --all ^g00d ^b33f ^1.0.0 ...

See git help rev-list for more information about how commits are specified.

Git-teleport saves the output of show-ref to keep a record of what has already been transferred. It provides this list to future calls to bundle. This makes future updates much, much smaller. Git-teleport also provides automation for mirroring many repositories at once.

You may obtain an offline copy of this guide from our package server. You can also build the documentation yourself.

Installing

To obtain git-teleport, clone this repository to any directory you own.

cd ~   # or another suitable location
git clone -b master https://gitlab.com/git-teleport/git-teleport.git

Alternatively, you may also download a git-teleport tarball from our releases page.

Add git-teleport to your $PATH by running:

cd git-teleport
./bin/git-teleport setup --path

This will add git-teleport to your ~/.bashrc file.

If you prefer to do this manually, add a line like the following to ~/.bashrc or a file that is sourced by it, such as ~/.bash_aliases:

PATH="${PATH}:$HOME/git-teleport/bin"

Either way, this change will take effect for the next shell you start. Run

source ~/.bashrc

to make it take effect immediately for the current shell.

If you can run

git teleport

then you are all set!

System Requirements

git-teleport is portable to most operating systems which have Git and bash installed. It runs out of your home directory; no special privileges or compilers are required. We recommend:

  • git >= 1.8.3
  • bash >= 4.2

which should be satisfied by most distributions which are CentOS 7 and newer.

Linux

  • Some very minimal installations, like containers, are missing binaries for find and cmp. These are usually available as packages named findutils and diffutils. Most desktops will have these programs installed.

  • If your /home directory is mounted with the noexec option, you will need to find somewhere else to install git-teleport.

Windows

  • git-teleport is tested against the current version of Git for Windows in Chocolatey. You must run git-teleport commands from the "Git Bash" prompt.

Building Documentation

HTML-formatted documentation may be built by running

mdbook build

from the repository root. Your workstation must have mdbook installed. Documentation will be output to share/git-teleport/html by default.

Running the Unit Tests

To verify that git-teleport will function correctly on your system, you may run

./test/run.sh

to run the unit tests.

If any test fails, and you are sure your system meets the above dependencies, please submit a bug report on Gitlab.

Sending Repositories: Your First Mirror Set

This page describes sending repositories with git teleport pack. The following sections will walk you through creating your first Mirror Set and generating your first set of bundles. If you already have a Mirror Set that you want to use, skip to "Repeating the Process."

Before You Begin

You may want to run some variation on the below Git commands

# basic configuration (you've probably already done this)
git config --global user.name 'Iam Chroot'
git config --global user.email 'Iam.Chroot@mycorp.example'

# if you connect via http(s), and you have no credential helper enabled
#    On linux, see the /usr/share/doc/git/contrib/credential directory
#    for additional credential stores. Git for Windows uses the Windows
#    Credential Store by default, so there is no need to set this.
git config --global credential.helper cache

# if you connect via http(s), and you need to specify a username
git config --global \
  credential."https://internal.mycorp.example".username iamchroot

You will probably want the last two lines if you connect over HTTP(S). Turning on a credential cache will greatly reduce the number of times that you have to type your password. Since git-teleport URLs are shared, they do not have usernames in them. If your repositories are stored on an HTTP(S) server, you may need to specify your login username manually.

Bootstrapping

In order to provide fast, incremental updates, git-teleport must keep a history of what has already been sent. git-teleport stores its database files in (where else?) a Git repository. This repository is largely managed by git-teleport, and it is referred to as the Mirror Set. The Mirror Set repository is NOT transferred to the destination system.

New Mirror Sets are created with bootstrap. Choose a non-existing directory BDIR on your workstation. Run:

git teleport bootstrap BDIR

A new Git repository will be created in that location.

Create a unique Mirror Set for each of your destinations. Never re-use a Mirror Set for different destinations; make a new one. This is because git-teleport needs to remember what the destination side has received, and it can't do that if you mix-and-match destinations.

Defining What to Mirror

The newly-bootstrapped repository has a text file, Beamdownfile, which is used to define the list of repositories to bundle. Let's assume that you want to send three repositories, and they reside on two different servers:

  1. https://extranet.mycorp.example/git/ProjectAlpha.git
  2. ssh://git@internal.mycorp.example/ProjectBravo.git
  3. ssh://git@internal.mycorp.example/mysubfolder/ProjectCharlie.git

The first lines in your Beamdownfile should be:

# BEAMDOWNFILE
# …
# Comments get ignored
# …

# root definitions
! https://extranet.mycorp.example/git
! ssh://git@internal.mycorp.example

Lines with a leading exclamation point ("!") are repository roots. These root definitions list the servers and path prefixes where repositories are hosted. List all the servers you intend to use in this Beamdownfile. Exclude subdirectories which are important to your project structure, such as "mysubfolder" in the last URL. "mysubfolder" is not part of the root definition.

Next, list the full absolute URLs of all the repositories you want to mirror.

# repository list
https://extranet.mycorp.example/git/ProjectAlpha.git
ssh://git@internal.mycorp.example/ProjectBravo.git
ssh://git@internal.mycorp.example/mysubfolder/ProjectCharlie.git

When git-teleport generates bundles, it assumes that the destination system will have mirrors for them at

${SOMEWHERE}/ProjectAlpha.git
${SOMEWHERE}/ProjectBravo.git
${SOMEWHERE}/mysubfolder/ProjectCharlie.git

where ${SOMEWHERE} is an arbitrary location on the destination system. The ${SOMEWHERE} will be selected later, when the bundles are unpacked.

The repository root definitions ("!") exist to separate physical storage—i.e., where the repositories are kept—from their logical storage. Logical storage is the sub-folder hierarchy that is important to your project. As shown above, git-teleport will preserve subfolders which are not matched by the root definitions.

The repository roots must be a prefix match on the URL. The match must be byte-for-byte, including case. All entries must have a matching repository root definition.

You may use any URL scheme supported by git, including file://. If a unique username is required, always set it in your git config as described above. Only include usernames if they are common to all clients, like "ssh://git@mygitlab/repo.git." Common usernames are typical of hosting software like Gitlab. You can use "SCP syntax" too: "git@mygitlab:repo.git."

Don't Forget Git-Teleport!

You will need git-teleport on the destination system in order to unpack these bundles. Consider including it in your Beamdownfile.

# root definitions
!https://gitlab.com/git-teleport

# repositories
https://gitlab.com/git-teleport/git-teleport.git

When you are finished, save this file and commit your changes.

Packing the Bundles

Now you are ready to create the bundles. Run:

git teleport pack

The pack command downloads all repositories and outputs new bundles to bundles/. A new subdirectory in bundles/ is created for each and every update. The current bundles are always available in bundles/latest. This directory is a symbolic link on linux and a copy on Windows.

Only repositories which have updated will generate new bundle files.

Collect the latest bundle files together for transfer to the destination system. Use any binary-safe file transfer method, such as email, shared folders, USB drives, optical media, or carrier pigeons. You need all the files in bundles/latest.

Once you have finished transferring the bundles, it is safe to delete them. You can remove the entire bundles/ directory if you want.

If you are sharing your Mirror Set, you should push up your changes so that others will see what the latest bundle should contain.

If you examine the output bundles/latest directory, you will notice that git-teleport has removed all special characters and converted all path-names to lower-case. Alpha.git becomes alpha.bundle. This is to support deficient filesystems which do not adequately support these characters.

In addition to the .bundle files, you will find a matching .bundle.refs file for each. This is the refs bundle. It contains only commits which are refs: i.e., tags, branch heads, and other commit references. This additional file is used to prune the destination repository of refs which no longer exist, such as merged topic branches.

Special Considerations

Some repositories are more difficult to mirror than others.

Bundles Too Large?

If you are using email attachments or floppy diskettes to send your bundles, file size can be a problem. If the bundles are too large for your chosen transport mechanism, consider splitting them like this:

cd bundles && \
  tar --dereference --create 'latest' | split -b1024M - allbundles.tar.

which splits the bundles into 1024 MiB files. Split files can be reconstituted later with

cat allbundles.tar.* | tar -xv

Git Submodules

git-teleport is not aware of submodules. If any of your repositories contain submodules, you will need to manually specify them in your Beamdownfile.

In order for submodule checkouts to work on the destination system, submodules must use only relative paths. On the sending side, you can switch submodules to relative urls by editing your .gitmodules file and replacing absolute URLs with relative ones. A git submodule sync will bring your working copy up-to-date.

[submodule "libfoo"]
path = src/foo
url = https://internal.mycorp.example/repo/foo.git

If your project is also located in https://internal.mycorp.example/repo/, you should switch to using relative paths instead.

url = ../foo.git

If you have received submodules which use absolute paths, and the upstream project is not under your control, you can use Git's internal URL-rewriting functionality to fix them:

git config --global \
  url.'file:///net/git/'.insteadOf \
  'https://internal.mycorp.example/repo/'

With the above configuration directive in place, URL prefixes which match https://internal.mycorp.example/repo/ will be automatically replaced with file:///net/git/ instead. This works for git submodules or any other attempt to git clone the given URL. It can also be useful for package managers which use libgit or the git executable to clone repositories. Rust's cargo, for example, can use dependencies cloned from git repositories.

If you find yourself using a lot of submodules, you may want to consider if git-subtree workflows are more appropriate. Subtrees can greatly reduce the number of individual repos you need to mirror.

Issue Trackers

Generally, issue tracker content is not exposed to Git, and git-teleport cannot mirror it. While some issue trackers offer APIs, accessing issues and interpreting them for the destination system is beyond the scope of git-teleport.

git-bug is a distributed issue tracker that resides in your existing Git repository. It is not supported in git-teleport yet, but it could be with a little effort. See git-teleport/git-teleport#8.

Large File Storage

Git Large File Storage (LFS) is just "hyperlinks to things not stored in Git." git-teleport does not support LFS. As an alternative, consider keeping your files in Git itself and using either

to limit the amount of disk and network bandwidth consumed by working copies.

git-teleport still needs to mirror all the files in your repository.

Next Steps

Mirroring repositories is only the first step in transferring your build environment. Read more tips on mirroring a complete build environment in "Mirroring 'Everything Else'."

Receiving Repositories: How To Unpack

Ensure that the destination system has git-teleport installed. You can do without it if necessary, but it will make things much easier.

Creating New Receive Mirrors

While the bundle files are useful for transit, they are not a permanent home for your data. Each repository you mirror needs a corresponding repository on the receiving system. These repositories, which are called Receive Mirrors, are read-only mirrors. They are intended to represent the remote repositories just as if you had cloned them.

The Receive Mirrors are intended to be exact duplicates of the repositories on the sending side. Crucially, you MUST NOT let your users push novel commits to the Receive Mirrors. Receive Mirrors must be modified only with git-teleport. Failure to adhere to this requirement will cause problems.

You can keep your Receive Mirrors anywhere that Git can access, including your local disk, a shared filesystem, or behind a git remote accessible via SSH or HTTP(S). Wherever you keep them, you will need to set aside space in your hierarchy for these mirrors. If you usually store your repositories on NFS in /net/git, and the mirrored repositories originated from your corporate intranet, then a path of

/net/git/mirrors/intranet

might be appropriate.

Here, the mirrors directory is dedicated to Receive Mirrors only. This helps keep them separate from your normal, writable repositories that include original commits and content. The intranet subdirectory further segregates these mirrors from repositories that originate from other places, such as your corporate partners.

Once you have selected a storage location, you will need to create the Receive Mirror repositories.

Examine the directory of bundle files you created during the previous step. (If you compressed them, extract them.) In the root directory of the bundles, you will find a Beamupfile. This text file is automatically generated by git-teleport. Continuing with our example from above, you will find the following entries:

# GIT-TELEPORT BEAMUPFILE
# …
# This file was automatically generated by git-teleport.

ProjectAlpha.git
ProjectBravo.git
mysubfolder/ProjectCharlie.git

This file lists the repository names you must create. The names for these repositories on your system must match these strings exactly, including case. Continuing with the above example, you will need to create:

  • /net/git/mirrors/intranet/ProjectAlpha.git
  • /net/git/mirrors/intranet/ProjectBravo.git
  • /net/git/mirrors/intranet/mysubfolder/ProjectCharlie.git

Your repositories may or may not have the ".git" extension—this extension is optional. Use the same convention as the sender.

The creation instructions are different for repositories stored on your local filesystem versus on SSH/HTTP(S) network remotes. Follow the instructions in the applicable sub-section.

Receive Mirrors on Local Filesystem

This includes shared filesystems, like NFS or SMB.

git teleport skeleton \
  /net/git/mirrors/intranet/ProjectAlpha.git \
  /net/git/mirrors/intranet/ProjectBravo.git \
  /net/git/mirrors/intranet/mysubfolder/ProjectCharlie.git

The skeleton command will create the above repositories, if they don't already exist, and configure them for use with git-teleport. This script takes the following actions:

  1. Creates the repository in --bare mode
  2. git config core.sharedRepository true, which helps with linux permission issues on shared folders
  3. Adds a pre-receive hook which denies pushes from clients other than git teleport. This helps prevent accidental pushes.
  4. Adds a teleport-mirror file to the root of the repository directory. This file tells git-teleport that the repository is safe to use for mirroring.

If this command succeeds, you are ready to go.

Receive Mirrors on Remote Server

You can store your repositories on a "smart" server, like Gitlab, or on a "dumb" directory over SSH. The important difference from "local filesystem" mode is:

THE NAMING CONVENTION IS MANDATORY, and git-teleport will enforce it.

  • ✔ git@mygitlab:mirrors/intranet/ProjectAlpha.git
  • ✔ https://mygitlab/git/mirror/intranet/ProjectAlpha.git
  • ✖ git@mygitlab:mystuff/ProjectAlpha.git (invalid)

Your Receive Mirrors must be stored in a subdirectory named either "mirror" or "mirrors" (plural), exactly. It does not matter which path component is named "mirrors." Regardless of which variation you choose, it is vital that you keep only Receive Mirrors in the mirrors/ directory.

We also recommend that you use a subdirectory like mirrors/intranet, in case other sites also start sending you teleport bundles.

  • If you have access to the remote filesystem, you can use git teleport skeleton as above.

  • If you are using a smart host, like Gitlab, you must create the repositories yourself via your smart host's UI. Create an empty repository, without a README file or other templates. Everyone who updates bundles must have write permission.

Smart repository hosts might restrict push access to certain branches, like master or develop. This will prevent you from receiving updates. To avoid these restrictions on Gitlab, ensure that there are no protected branches or other push rules. Gitlab will usually auto-protect these branches after the first git-teleport update. Be sure to check the settings page after your first update and unprotect all branches.

git-teleport will set the CONFIRM_MIRROR_UPDATE push option when pushing to its mirrors. If your repo hosting software permits it, you may want to require this push option to prevent accidental updates to the mirrors.

Unpacking the Bundles

Now that your Receive Mirrors exist, it's time to put them to use. Change directory to the location of your bundle files and run:

cd path/to/bundle/files    # must contain a Beamupfile
git teleport unpack /net/git/mirrors/intranet

The argument to unpack is the URL or path prefix where your Receive Mirrors are stored.

Alternatively, you may run

git teleport unpack path/to/bundle/files /net/git/mirrors/intranet

without changing directory.

When this command completes successfully, the repositories in mirrors/intranet will be exact duplicates of the sending side. They will have all branches, tags, and notes—just as they are in the upstream repository.

If git-teleport reports an error, your repositories haven't been completely mirrored. See "Troubleshooting."

Repeating the Process (The Easy Part)

Installing a shell script, creating a Mirror Set, and unpacking bundles may sound like a lot of effort just to mirror a Git repository. It is.

The goal of git-teleport is to make the update process easy, efficient, and repeatable. With git-teleport, preparing and ingesting updates takes a matter of minutes. Use it weekly—or even daily—to accelerate your development process.

Send Bundles

You must have the Mirror Set Git repository for your intended destination.

Obtain the Mirror Set for your destination. If the Mirror Set is shared, update it.

cd mirror/set/location
git pull

If you need to add or remove a repository from the mirrors, edit the Beamdownfile. Otherwise, just pack the bundles:

git teleport pack

Send the contents of the bundles/latest directory to the receiving system.

When you are finished, push the Mirror Set repository if it is shared.

git push

By pushing to the Mirror Set, you are assuming responsibility for delivering the bundles. Transmit them promptly.

Receive Bundles

You must know the URL prefix to your Receive Mirrors for this sender.

Unarchive the bundle files if necessary. Change directory to them and run unpack.

cd unpacked/bundles
git teleport unpack /net/git/mirrors/intranet

Replace /net/git/mirrors/intranet with the prefix path or URL to your Receive Mirrors. If you keep your mirrors on a smart repository host, your unpack command might look something like:

git teleport unpack https://mygitlab/git/mirror/intranet

If the sender has added new repositories, this command will fail. Read the error message, identify the new repositories, and create mirrors for them. Then run the unpack command again.

Next Steps?

There are no next steps. You're finished!

If something went wrong, see "Troubleshooting."

Troubleshooting: Things Which Can Go Wrong

  • If the sending side added a new repository, the unpack will fail. Create the missing repository and run the unpack step again. The unpack command is safe to re-run.

  • If you have multiple updates to apply at once—perhaps because you let them accumulate without applying them—be sure to apply them in the order in which they were generated. Otherwise, the unpack will fail.

  • If a bundle won't apply, run git bundle verify on the bundle file. Determine the commits you are missing. You will probably need the sender to generate new bundles. You can:

    • Force full bundles to be generated with git teleport pack --full; OR

    • Revert the Mirror Set (on the sending side) to a previous version and try normal incremental bundles again.

    • Have the sender check and be sure that the bundle contains the required commit(s)

  • Whenever git-teleport needs to fetch or push to a remote repository, it caches it locally. On linux, caches are stored in ~/.cache/git-teleport/. Cache files are transient, and you may delete them to save space.

  • By default, git-teleport will mirror all branches, tags, and notes—but nothing else. This is because some repo hosts, like Gitlab and Github, use extra "read only" refs to pin certain commits and/or store other information. Since these refs are not useful, they are excluded.

    • You may adjust what is mirrored by setting the GTELEPORT_MIRROR_REFS environment variable.

    • This variable must be set on both the sending and receiving side.

  • Git-teleport generates two different bundle files: the incremental update (.bundle) and the refs bundle (.bundle.refs). The first bundle is used to update changed refs. The second is used to delete (or "prune," as git calls it) refs which are no longer in the source repository. If git-teleport fails to generate a refs bundle, it will print a warning and continue. If you see this warning, please report this as a bug.

Tips for Distributed Development

You can help avoid merge conflicts and nasty, nasty bugs by adhering to the following recommendations:

  1. Submit all relevant changes to the upstream project with a patching workflow.

  2. Keep patches small. Massive formatting changes will quickly wreck a project's merge-ability. Review your changes with git diff --patch …. If you don't absolutely need to touch that line of code, leave it alone.

  3. Keep topics isolated. If your changes do two separate things that do not depend on one another, consider splitting them into two separate feature branches. This allows the maintainer to merge what is ready and to iterate on the rest.

  4. Start your topic branches from the right place. If your site is the maintainer, topic branches should probably start from origin/develop or origin/master. If your site is the fork, topic branches should probably start from upstream/develop or upstream/master.

  5. Exchange bundles as frequently as possible. More frequent integration is much easier to do: there is less to merge at once, and everyone remembers what they've been doing.

  6. Talk about the changes you plan to make, especially if they are big or breaking.

  7. Test ferociously until there is nothing left to test. Write simple test-cases and automate their execution.

Mirroring "Everything Else"

checking whether build environment is sane...

... build environment is grinning and holding a spatula.

– #371

git-teleport can only mirror Git repositories. A typical application—and many atypical applications—have numerous dependencies. At bare minimum, you'll probably want a compiler toolchain… but which compiler toolchain?

This is a curated guide to mirroring all the other things you need to make your software run instead of shambling about aimlessly like a zombie. If you don't see your language or platform listed, see "Something Else" below.

General Advice

Write down instructions for reproducing your build environment and dependencies.

  • The best documentation for this activity is executable: think shell scripts, Containerfiles, and the like.

  • But that's no excuse to skimp on well-organized, greppable prose documentation.

It is much easier to mirror source code than binaries, but there is a balance: to build Linux from scratch you must first invent the universe. Regardless of what form your dependencies take, you should pin them.

  • For source code, specify exact versions or vendor the code.
  • For binaries, consider using containers.
  • For toolchains and operating systems, does their lifecycle match your product? Consider "Long Term Support" releases where appropriate.

The more of your build environment you can ship, the better…

  • … but if you're shipping a 12 GB virtual machine disk image, you should probably ask if there's a better way.
  • Your CI/CD environment and job runners count. Have a plan to mirror these as well.

Are you prepared in case $CRITICAL_DEPENDENCY vanishes from the face of the Internet tomorrow?

Operating Systems

Debian and Ubuntu

Debian repositories can be mirrored with ftpmirror.

Fedora and RHEL-based

dnf/yum repositories can be mirrored with createrepo.

Container Platforms

For best results, make your images as small as possible. Take steps to avoid unnecessary programs, unused files, or excessive layering in your container images.

  • Gitlab CE features a container registry that may be used in place of cloud registries like DockerHub.

  • Quay's mirror-registry script can help you stand up your own mirrors of internet registries.

Programming Languages

The more languages you use, the more complicated it will be to set up a build environment. Your build environment should also include the following tools, as applicable:

  • Autoformatters
  • Documentation generators
  • "Linters" and other static analyzers
  • Dynamic analyzers (i.e., memory-safety checkers)
  • Test frameworks and test-cases
  • Security- and other compliance-checkers, such as license checks

Be especially careful about forcing dependencies on IDEs, such as Eclipse, IntelliJ, or Visual Studio, on your downstream systems. When possible, use build systems which are not tied to a particular IDE.

C/C++

We recommend vendoring your source dependencies and building your entire application from source. Avoid operating system packages unless you have some other way to pin and store them, like containers.

  • CMake offers ExternalProject or FetchContent for making "superbuilds" of your project and its dependencies. Read these approaches carefully; they are very different.

Rust

Rust has standalone installers available, but these only cover the core cargo toolchain. cargo packages themselves can be mirrored with cargo-vendor.

Storage

Storage technologies include POSIX-like filesystems and object stores. The following programs can help with file retrieval:

  • rclone can be used to mirror cloud storage platforms like S3 and many others.
  • rsync can make incremental copies of filesystems over the network.

If you are attempting to mirror or publish a scientific dataset, ask yourself:

  • Is your dataset standardized? Does it have open-standard storage formats, a standardized canonical representation, and a stable API for access?

  • Is your dataset documented? Are there user manuals and other information that a user needs to know?

  • Is your dataset organized so that others may use it? Does it have a searchable catalogue or index?

  • Is your dataset curated? Does it have a maintainer that enforces quality and standards-compliance?

If the answer to any of the above questions is "no," a substantial investment may be required before the dataset can be mirrored.

Something Else?

Contributions are sought which expand this guide to other languages and systems. Open an issue or merge request today! The usual rules in CONTRIBUTING.md apply.

  • This guide covers only free and open-source software (F/OSS) only.

    • Mirroring tools must be open-source.
    • Documentation for closed-source products will be accepted only if the product has a substantial open-source version.

  • When possible, link to existing guides instead of writing your own. Hyperlinks should be to official maintainer websites or vendor/maintainer documentation only.

  • Please alphabetize your entry by software title—unless that doesn't make sense.

Forking a Receive Mirror

git-teleport Receive Mirrors are READ-ONLY. You MUST NOT make any commits to git-teleport mirrors. Instead, they must be treated like external repositories for which you lack write permission.

If you want to make changes to a mirrored project, you need to fork it. "Forking" a repository creates a new one with the same commits as the upstream project. This process is identical to that used by Github et. al.: A copy of the original repository is made. This copy, called the "fork," is a place where you can make your changes. Forks are kept up-to-date by manually merging in changes from the original repository, which is called the "upstream."

Create the Fork

git-teleport does not create or manage forks. If you want a fork, you must create one "by hand."

On a filesystem

Make a new server-style repository for your forked changes. It will start empty. For a mythical Receive Mirror stored at

/net/git/mirrors/intranet/ProjectAlpha.git

you can create the mirror as follows

cd /net/git

# you should probably ensure the fork is group read/write
umask 002

# create a new bare repo and mirror
git init ProjectAlpha.git --shared --bare
cd ProjectAlpha.git
git fetch \
    file:///net/git/mirrors/intranet/ProjectAlpha.git \
    '+refs/*:refs/*'

# --shared ensures your repository is always group read-write,
# but it also disables force-push by default. You probably
# want to do this:
git config --local receive.denyNonFastForwards false

On a smart host

If you use smart repository hosting software, it likely has a "fork" option. You can use this option to duplicate a Receive Mirror into another repository. If your Receive Mirror is named

mirrors/intranet/ProjectAlpha.git

then your fork should probably be named

ProjectAlpha.git

On some hosting software, like Gitlab, using the "fork" button will make merge requests target the Receive Mirror (mirrors/intranet/ProjectAlpha.git) by default. This is not what you want! You can't merge anything into the Receive Mirror because it's read-only. Adjust your options to ensure that all merge requests target the fork itself.

Merges Happen in a Working Copy

Your fork will probably diverge from its parent project. To keep your fork up-to-date, you need to merge changes from the outside world into your fork.

Git merges require a working copy, so let's make one of those.

cd "$SOMEWORKDIR"
git clone file:///net/git/ProjectAlpha.git

cd ProjectAlpha
git remote add upstream file:///net/git/mirrors/intranet/ProjectAlpha.git
git remote set-url --push upstream /dev/null

The working copy now has two remotes:

  1. origin, which refers to the forked repository. You will use this remote on a day-to-day basis.

  2. upstream, which points to the git-teleport mirror from the remote system. This is a read-only mirror. Like a Github project which belongs to someone else, you can't change it. The set-url command above will ensure that you cannot accidentally push to upstream.

Whenever you receive git-teleport updates, you should merge them into your fork. To merge their master branch into your master branch, you can run something like:

git fetch --all
git checkout master
git rebase # origin/master
git merge --no-ff upstream/master

# … resolve conflicts if necessary …

git push --follow-tags origin master

The above merging workflow is only one possible flow. Here are a few more:

  • Fast-forward only: You can reserve the master branch for use by the upstream project only. In this case, all merges can be --ff-only fast-forward merges. Use a different branch for your site's local changes.

  • Subtree merges: If the upstream project is a small piece of your project—i.e., a dependency—you can keep it in a subdirectory of your repository like project_alpha/. In this case, you may not want a fork at all. Consider if a git-subtree workflow is the right approach.

  • Independent projects: In some cases, you might want to make permanent changes to the project which are never going to be ported upstream. These might include CI/CD jobs, build-system tweaks, or even code changes. You might also want to make your own tagged releases. This workflow is discussed in the next section.

Regardless of the workflow you choose, you should be sure to document which branches get merged, and when.

Independent Projects

If you are making a long-lived fork, chances are you will want to make your own releases. This section assumes that you will have:

  1. A branch named master that is only for releases. Every merge into master represents a release.

  2. A branch named develop for integrating changes for the next release.

  3. Other branches for features and bugfixes.

This is the git-flow workflow used by git-teleport and many other projects. You can use a simpler, single-branch workflow without a develop branch if that is more to your liking.

To begin, don't make a fork. Instead, create a new empty server repository (like file:///net/git/ProjectAlpha.git) and working copy.

cd "$SOMEWORKDIR"

git init ProjectAlpha
cd ProjectAlpha

# you can also use `git flow init -d` if installed
git commit --allow-empty -m 'Initial commit'
git checkout -b develop

# populate the server-side repository, wherever you put it
git remote add origin file:///net/git/ProjectAlpha.git
git push --set-upstream origin master develop

# add the upstream project mirrored by git-teleport
git remote add upstream file:///net/git/mirrors/intranet/ProjectAlpha.git
git remote set-url --push upstream /dev/null

If you are using a git-flow workflow, you probably want to periodically merge one of the upstream project's version tags, or their master branch, into your develop branch.

git fetch --all
git checkout develop
git rebase
git merge --no-ff v1.0.4  # or upstream/master

This merge workflow ensures that:

  1. All commits in upstream/master end up in your project.
  2. Upstream updates are easy to identify in your history.

If your project will make its own tagged releases, make sure to pick a different tag prefix than the upstream project.

Be sure to document your chosen approach.

Submit Your Changes Upstream

Over time, a long-running fork can diverge from its parent project. This will invariably lead to merge conflicts and other maintenance issues—all of which may require substantial effort to fix.

One of the most helpful and productive things a software engineer can do is to find problems and fix them. An even more productive thing to do is to fix them upstream. By submitting your changes back to the original maintainer, you can reduce your maintenance burden and also improve the software for everyone.

Changes may be submitted to an upstream repository via a patch workflow.

Submitting Patches to Upstream

In disconnected environments, the best way to submit your changes to an upstream project is to use a patch workflow. Patch files are a set of instructions for making changes to a document: i.e.,

go to line 107, strike the word "is," and replace it with the word "was."

These instructions are both human- and machine-readable, which makes them an ideal way to review patches. Patch files are a text-only format, so they can even be printed on paper.

Git can create patches with the format-patch command and apply them with the dubiously-named am command. "am" stands for "apply mailbox." (Patch files are frequently sent over email.)

More complete information on this topic can be found in Distributed Git - Contributing to a Project: see the "Public Project over Email" section. This document summarizes the most important points.

Preparing a Patch

Let's assume you have:

  • A feature branch named feature/mycoolthing, which has been fully tested and is ready to submit; and

  • A remote named upstream which tracks a git-teleport Receive Mirror.

Prior to submitting a patch, you must ensure that your work is rebased on a commit that the receiving side has. If you have forked the project, you might have your own integration branch and a great number of commits which are either not ready, or not intended, for submission. If necessary, run something like:

git fetch --all
git checkout feature/mycoolthing
git checkout -b submit/mycoolthing
git rebase upstream/develop   # or upstream/master

We use the git-flow develop branch in our example, but your target branch may be different. Very often, a project's CONTRIBUTING.md will tell you which branch to use. If it doesn't, and you're not sure, upstream/master is probably a safe bet. As an alternative to rebasing, you can also create your submission branch with git cherry-pick.

Now is a good time to review your work. Did you touch only the lines of code that you intended? Did you include anything in your commits that shouldn't be there?

git diff --histogram upstream/develop

  • The --histogram diff algorithm can often make the output of git-diff easier to read.

  • Use --word-diff or --color-words for prose text

  • Use --ignore-all-space if you changed a bunch of indents

  • Use --find-copies-harder -M -B to tweak rename detection. Both -M and -B take optional thresholds.

  • If you need more context…

    • Use -Ux to see x lines of context.

    • Use --inter-hunk-context=x to merge adjacent diff hunks.

    • Use --function-context to see an entire file or function as context lines.

There are many other diff options that you can use to get a better view into your changes. Experiment!

Check your branch for whitespace errors:

git diff --check upstream/develop   # should print nothing

Whitespace errors introduce needless churn and conflicts, and most projects won't appreciate them. Make your patches look professional by eliminating the trailing whitespace from your code.

Recompile and re-test if necessary.

Now you are ready to create a patch.

git format-patch -M --histogram --base=upstream/develop upstream/develop

  • This command produces one file per commit. You can roll them all into one file with --stdout and an output redirection.

  • If you are submitting changes to binary files, you must add --binary.

  • You can add many other diff options, like --histogram, to make the patch file itself smaller. --find-copies-harder -M -B can be particularly useful.

Always REVIEW YOUR OWN PATCHES before submitting them. Read the patch files and make sure they do what you want.

Transmitting a Patch

Patches are just files, and you can transmit them via any utf8-safe medium.

Via Email

When emailing patches, be advised that many email clients and servers will "helpfully" alter (read: destroy) the content of your messages. Exchange, for example, is particularly bad about altering line endings. Thunderbird will re-wrap lines in your message body. git send-email and git imap-send can both prepare email messages for you if you have SMTP or IMAP server access.

On Paper

Sometimes, you may want to print a patch file for purposes of review. You might also need to apply the patch by hand. When printing, use an editor which supports syntax highlighting for patch files. Print in color. Use a readable, monospaced font.

To reduce the size of the printed document, try generating a patch with options like:

git format-patch -U2 -M -D --ignore-all-space

to reduce context, make deletes irreversible, and ignore indent changes. This patch cannot be applied with Git, but that's okay.

Applying a Patch

To apply a patch file to your project, first create a branch off of your project's default integration branch.

git fetch --all
git checkout origin/develop
git checkout -b feature/mycoolthing

Then run

git am /path/to/patch/files/*.patch

This applies the patch(es) to the current branch. If the patch does not apply, try

git am --3way /path/to/patch/files/*.patch

to perform a three-way merge. This only works if the patch records a base-commit: line. If the submitter followed the instructions above, it should.

You can then review the branch just as you would any other topic branch. If the changes are acceptable, you can merge them.

If you are using git-teleport to mirror your repositories, consider sending out a fresh batch of bundles after you merge.

Monitoring the Status of your Patches

Patches create fresh commits when applied. This means that, although the commit messages and the changes will be identical, the resulting Git commits will have different hashes.

The git cherry command is useful for determining which of your patches have been merged into the upstream repository.

Alternatives to Patch Workflows

Git-teleport only works in one direction at a time. With that said, you can book a round-trip flight for your data by purchasing two separate one-way tickets.

If you have forks of a repository that you receive via git-teleport, you can also use git-teleport to send your fork back to the maintainers. Simply create a Mirror Set which includes your fork and send your bundles off.

Although this approach works, it has serious limitations:

  • git-teleport bundles are typically "point to point" links. If you submit your changes this way, only one destination can receive them. Patches, due to their self-contained nature, can be sent to many recipients at once.

  • git-teleport will send everything in your fork by default. This is not always desirable. Sending large repositories this way can use a lot of storage and transfer space.

  • Bundling can add sometimes more work for the maintainer.

    • Branches from bundles might need rebasing and other work before they are useful. A patch process, with a pre-submission review, can be cleaner.

    • For a large repository with many branches, the sender must communicate which branches are "merge requests" (and which aren't).

  • Bundles require a binary-safe transport mechanism.

With that said, both patches and bundles are workable solutions. With practice, and a commitment to authoring good commits, either route can lead to success.

The setup command

path/to/git-teleport/bin/git-teleport setup <options>

The setup command prepares your workstation for the first run of git-teleport. At the moment, the only thing it does is add git-teleport to your $PATH.

You should only ever have to run this command once. After that, if you can invoke git-teleport with

git teleport help

then git-teleport is ready for use.

--path

Add git-teleport to your $PATH by modifying your ~/.bashrc file. After this command completes, restart your shell.

--remove

Remove git-teleport from your $PATH. Undoes --path.

The bootstrap command

git teleport bootstrap <directory>

Creates a new Mirror Set in <directory>. The specified <directory> must not exist. If necessary, a new a Git repository is initialized to hold the Mirror Set.

If you run git teleport bootstrap site_one, the freshly-created Mirror Set will contain the following:

site_one/
├── .gitignore
├── Beamdownfile
└── Beamdownstate/
    └── (initially empty)

Populate the Beamdownfile with the list of repositories to send. The template file contains basic instructions. Commit your changes. Then run git teleport pack to prepare your first set of bundles for sending.

Sample Beamdownfile

# Beamdownfile for Mirroring Git-Teleport Itself

# Repository root prefixes start with "!"
!https://gitlab.com/git-teleport

# URLs are repositories to mirror
https://gitlab.com/git-teleport/git-teleport.git

# The relative URL of each repository in the mirror is its full
# absolute URL with any matching root prefix stripped off.

Multiple Mirror Sets in One Git Repository

If you bootstrap a directory within an existing repository, git-teleport will re-use that repository for your new Mirror Set. If you send bundles to many places, you can store them all within a single repository to keep them together. For example:

git init my_mirror_set
git teleport bootstrap my_mirror_set/site_one
git teleport bootstrap my_mirror_set/site_two

creates the following:

my_mirror_set/
├── .git/
│
├── site_one/
│   ├── Beamdownfile
│   └── Beamdownstate/
│
└── site_two/
    ├── Beamdownfile
    └── Beamdownstate/

Be sure to cd my_mirror_set/site_one to select the Mirror Set you would like to use.

The pack command

git teleport pack [<options>] [--] [<beamdownfile>]

Generates new outgoing bundle files. A local mirror is created and/or updated for each repository in <beamdownfile>. The mirrors are then packed into bundle files. By default, the bundles will be incremental: that is, they will not contain the commits or refs from previous invocations of pack.

If the Beamdownfile is unspecified, the current working directory is searched.

Generated bundles are output in the bundles/latest directory, which is relative to the Beamdownfile. On symlink-capable platforms, this directory will be a symbolic link.

--full

Generate bundles which contain a complete history. This is useful if the recipient needs a completely fresh copy to resolve unpack issues.

--no-save

Run without saving the incremental history. The next pack will contain the commits which were also packed into these bundles. See "Differential Bundles."

--offline

Run without accessing the network. Local caches of mirrored repositories will not be updated. This permits you to create bundles without network access. Bundling will fail if you do not already have a cache for each mirror, and your bundles may be potentially out-of-date.

Exit Codes

  • 0: bundles successfully created
  • 1: failed to create bundles
  • 2: nothing new to bundle. No bundles were generated.

Environment Variables

  • GTELEPORT_CACHE: caches of mirrored repositories are stored in this location. If left empty, a platform-appropriate storage location is automatically selected.

  • GTELEPORT_BEAMDOWNFILE: an alternative way to specify the Beamdownfile to use.

  • GTELEPORT_MIRROR_REFS: an allowlist of Git refs to pack. Set to a space-delimited list of Git refs to mirror, like

    refs/heads/* refs/tags/* refs/notes/*

    If you wish to bundle more than the default refs, both the sending and receiving side need the same value of GTELEPORT_MIRROR_REFS. You may use negative refs like ^refs/notthis/*, but Git 1.8 does not support them.

    For further reading on refs, see "Git References."

Incremental Bundles

In the standard mode of operation, pack creates incremental bundles. Each bundle excludes everything from all previous bundles. As a result, the receiving side needs to apply all the bundle files, in order. If you run pack at three different times A, B, and C, the receiver must unpack them in order:

git teleport unpack bundles_A "$DEST"
git teleport unpack bundles_B "$DEST"
git teleport unpack bundles_C "$DEST"

  • If the receiving side mistakenly unpacks bundles_C first, the operation will fail.

  • If the receiving side unpacks bundles_A after bundles_B, git-teleport will discard some of the Git history graph. The receiver must unpack B again before proceeding on to C.

This mode is efficient for point-to-point transfers and minimizes storage during the transfer. It is inefficient for one-to-many transfers, however, as receivers are obligated to download and process every pack.

Differential Bundles

In differential bundles, only two packs of bundles ever need to exist at a time:

  1. The first is a --full bundle that contains all history.
  2. The second contains the history from the last --full bundle to "now."

This is optimized for one-to-many transfers. New receivers can obtain the complete history by downloading only two sets of bundles. No matter how far behind a receiver is, it only has to download two bundles. If a receiver misses an update, the next differential bundle will still contain everything they need.

To make differential bundles, first make a "full" bundle with complete history.

git teleport pack --full

Ensure that these bundles are uploaded somewhere to persistent storage. New receive sites may want them.

At a later time,

git teleport pack --no-save

This creates a differential bundle that is relative to your last --full bundle. Upload these bundles alongside your last --full bundle. Receive sites that have already applied the --full bundle only need your --no-save differential bundle.

You can create as many differential bundles as you want. When the differential bundles become "too large" in size to be convenient, make a new --full bundle.

In the future, this mode will be better automated.

For further reading, see "Sending Repositories."

The skeleton command

git teleport skeleton [--] <repo1> [<repo2> .. <repoN>]

Creates new git repositories that are compatible with the unpack command. The new repositories are created in the <repoN> directories on your local filesystem.

git-teleport can use any Git repository as a Receive Mirror so long as its filename or URL prefix is permitted. The skeleton command provides an additional layer of protection: it discourages users from modifying (i.e., by git push) the repo outside of git-teleport. Repositories created with skeleton:

  • Have core.sharedRepository set, which helps preserve access on shared filesystems.

  • Have a pre-receive hook that can prevent accidental pushes.

  • Have a special marker file which indicates to unpack that the repository is safe to use.

The skeleton command is safe to rerun on existing mirror repositories.

Limitations

  • Only use skeleton for creating mirrored repositories. skeleton must not be used to create regular, read/write repositories that contain original commits. Create these with git init instead.

  • githooks only run if permitted by safe.directory restrictions.

Required URL Prefix

git-teleport will only unpack into repository URLs which have a path component named "mirror" or "mirrors" exactly. For example,

git@mygitlab:mirrors/intranet/
https://mygitlab/mirror/extranet/
/net/share/git/mirror/public/

are all valid URL prefixes.

You should also create subdirectories, like intranet, to separate mirrors received from different sources.

The URL prefix is optional for repositories created with skeleton and shared on a local filesystem. The prefix is required for network protocols, like https.

The unpack command

git teleport unpack [--] [<beamupfile>] <dest_prefix>

Reads incoming git-teleport bundles from <beamupfile> and mirrors their contents to Git repositories at <dest_prefix>. The <dest_prefix> must be an URL or filename prefix.

If you have just un-archived a collection of bundle files in ~/Downloads/latest, and the root directory on your server for these repositories is /net/mirrors/pub/, then run

git teleport unpack \
    ~/Downloads/latest \
    /net/mirrors/pub/

If the Beamupfile is unspecified, the current working directory is searched.

It is safe to unpack the same Beamupfile multiple times—but only if those bundles are the most recent available. See "Precautions" and "Troubleshooting."

Environment Variables

  • GTELEPORT_CACHE: caches of mirrored repositories are stored in this location. If left empty, a platform-appropriate storage location is automatically selected.

  • GTELEPORT_MIRROR_REFS: an allowlist of Git refs to pack. Set to a space-delimited list of Git refs to mirror, like

    refs/heads/* refs/tags/* refs/notes/*

    If you wish to bundle more than the default refs, both the sending and receiving side need the same value of GTELEPORT_MIRROR_REFS. You may use negative refs like ^refs/notthis/*, but Git 1.8 does not support them.

    For further reading on refs, see "Git References."

Required URL Prefix

git-teleport will only unpack into repository URLs which have a path component named "mirror" or "mirrors" exactly. For example,

git@mygitlab:mirrors/intranet/
https://mygitlab/mirror/extranet/
/net/mirrors/pub/

are all valid URL prefixes.

You should also create subdirectories, like intranet, to separate mirrors received from different sources.

The URL prefix is optional for repositories created with skeleton and shared on a local filesystem. The prefix is required for network protocols, like https.

STORE ONLY MIRRORS in this URL prefix. Otherwise, git-teleport may inadvertently overwrite your original projects and commits.

Destination Mirror Not Found?

If any of the repositories specified in the Beamupfile do not exist, the unpack command will report an error like:

error: destination mirror "/net/mirrors/pub/foo/bar.git" not found

The unpack command requires that each entry in the Beamupfile have a matching repository under your selected <dest_prefix>. If your Beamupfile contains an entry named:

# GIT-TELEPORT BEAMUPFILE

foo/bar.git

then you must create an empty repository named "foo/bar.git." The name must match exactly, including case. The .git extension is optional but is highly encouraged.

If your mirrors reside on your local filesystem, you can create a new repository with skeleton. After creating all necessary repositories, run unpack again.

Precautions

The unpack command does not check that the bundles you are attempting to unpack are the most current version.

Incremental bundles—which are the default mode of operation—must be unpacked in the proper order. Every set of bundles that the sending side packs must be unpacked in the same order. To determine the order, use the datestamp on the bundle directory.

If an outdated bundle file is inadvertently unpacked, your mirrors will be forcibly overwritten with an outdated set of refs. To correct this condition, unpack more recent bundle files in the correct order.

For further reading, see "Receiving Repositories" and "Troubleshooting."