Sending Repositories: Your First Mirror Set

This page describes sending repositories with git teleport pack. The following sections will walk you through creating your first Mirror Set and generating your first set of bundles. If you already have a Mirror Set that you want to use, skip to "Repeating the Process."

Before You Begin

You may want to run some variation on the below Git commands

# basic configuration (you've probably already done this)
git config --global user.name 'Iam Chroot'
git config --global user.email 'Iam.Chroot@mycorp.example'

# if you connect via http(s), and you have no credential helper enabled
#    On linux, see the /usr/share/doc/git/contrib/credential directory
#    for additional credential stores. Git for Windows uses the Windows
#    Credential Store by default, so there is no need to set this.
git config --global credential.helper cache

# if you connect via http(s), and you need to specify a username
git config --global \
  credential."https://internal.mycorp.example".username iamchroot

You will probably want the last two lines if you connect over HTTP(S). Turning on a credential cache will greatly reduce the number of times that you have to type your password. Since git-teleport URLs are shared, they do not have usernames in them. If your repositories are stored on an HTTP(S) server, you may need to specify your login username manually.

Bootstrapping

In order to provide fast, incremental updates, git-teleport must keep a history of what has already been sent. git-teleport stores its database files in (where else?) a Git repository. This repository is largely managed by git-teleport, and it is referred to as the Mirror Set. The Mirror Set repository is NOT transferred to the destination system.

New Mirror Sets are created with bootstrap. Choose a non-existing directory BDIR on your workstation. Run:

git teleport bootstrap BDIR

A new Git repository will be created in that location.

Create a unique Mirror Set for each of your destinations. Never re-use a Mirror Set for different destinations; make a new one. This is because git-teleport needs to remember what the destination side has received, and it can't do that if you mix-and-match destinations.

Defining What to Mirror

The newly-bootstrapped repository has a text file, Beamdownfile, which is used to define the list of repositories to bundle. Let's assume that you want to send three repositories, and they reside on two different servers:

https://extranet.mycorp.example/git/ProjectAlpha.git
ssh://git@internal.mycorp.example/ProjectBravo.git
ssh://git@internal.mycorp.example/mysubfolder/ProjectCharlie.git

The first lines in your Beamdownfile should be:

# BEAMDOWNFILE
# …
# Comments get ignored
# …

# root definitions
! https://extranet.mycorp.example/git
! ssh://git@internal.mycorp.example

Lines with a leading exclamation point ("!") are repository roots. These root definitions list the servers and path prefixes where repositories are hosted. List all the servers you intend to use in this Beamdownfile. Exclude subdirectories which are important to your project structure, such as "mysubfolder" in the last URL. "mysubfolder" is not part of the root definition.

Next, list the full absolute URLs of all the repositories you want to mirror.

# repository list
https://extranet.mycorp.example/git/ProjectAlpha.git
ssh://git@internal.mycorp.example/ProjectBravo.git
ssh://git@internal.mycorp.example/mysubfolder/ProjectCharlie.git

When git-teleport generates bundles, it assumes that the destination system will have mirrors for them at

${SOMEWHERE}/ProjectAlpha.git
${SOMEWHERE}/ProjectBravo.git
${SOMEWHERE}/mysubfolder/ProjectCharlie.git

where ${SOMEWHERE} is an arbitrary location on the destination system. The ${SOMEWHERE} will be selected later, when the bundles are unpacked.

The repository root definitions ("!") exist to separate physical storage—i.e., where the repositories are kept—from their logical storage. Logical storage is the sub-folder hierarchy that is important to your project. As shown above, git-teleport will preserve subfolders which are not matched by the root definitions.

The repository roots must be a prefix match on the URL. The match must be byte-for-byte, including case. All entries must have a matching repository root definition.

You may use any URL scheme supported by git, including file://. If a unique username is required, always set it in your git config as described above. Only include usernames if they are common to all clients, like "ssh://git@mygitlab/repo.git." Common usernames are typical of hosting software like Gitlab. You can use "SCP syntax" too: "git@mygitlab:repo.git."

Don't Forget Git-Teleport!

You will need git-teleport on the destination system in order to unpack these bundles. Consider including it in your Beamdownfile.

# root definitions
!https://gitlab.com/git-teleport

# repositories
https://gitlab.com/git-teleport/git-teleport.git

When you are finished, save this file and commit your changes.

Packing the Bundles

Now you are ready to create the bundles. Run:

git teleport pack

The pack command downloads all repositories and outputs new bundles to bundles/. A new subdirectory in bundles/ is created for each and every update. The current bundles are always available in bundles/latest. This directory is a symbolic link on linux and a copy on Windows.

Only repositories which have updated will generate new bundle files.

Collect the latest bundle files together for transfer to the destination system. Use any binary-safe file transfer method, such as email, shared folders, USB drives, optical media, or carrier pigeons. You need all the files in bundles/latest.

Once you have finished transferring the bundles, it is safe to delete them. You can remove the entire bundles/ directory if you want.

If you are sharing your Mirror Set, you should push up your changes so that others will see what the latest bundle should contain.

If you examine the output bundles/latest directory, you will notice that git-teleport has removed all special characters and converted all path-names to lower-case. Alpha.git becomes alpha.bundle. This is to support deficient filesystems which do not adequately support these characters.

In addition to the .bundle files, you will find a matching .bundle.refs file for each. This is the refs bundle. It contains only commits which are refs: i.e., tags, branch heads, and other commit references. This additional file is used to prune the destination repository of refs which no longer exist, such as merged topic branches.

Special Considerations

Some repositories are more difficult to mirror than others.

Bundles Too Large?

If you are using email attachments or floppy diskettes to send your bundles, file size can be a problem. If the bundles are too large for your chosen transport mechanism, consider splitting them like this:

cd bundles && \
  tar --dereference --create 'latest' | split -b1024M - allbundles.tar.

which splits the bundles into 1024 MiB files. Split files can be reconstituted later with

cat allbundles.tar.* | tar -xv

Git Submodules

git-teleport is not aware of submodules. If any of your repositories contain submodules, you will need to manually specify them in your Beamdownfile.

In order for submodule checkouts to work on the destination system, submodules must use only relative paths. On the sending side, you can switch submodules to relative urls by editing your .gitmodules file and replacing absolute URLs with relative ones. A git submodule sync will bring your working copy up-to-date.

[submodule "libfoo"]
path = src/foo
url = https://internal.mycorp.example/repo/foo.git
If your project is also located in https://internal.mycorp.example/repo/, you should switch to using relative paths instead.
url = ../foo.git

If you have received submodules which use absolute paths, and the upstream project is not under your control, you can use Git's internal URL-rewriting functionality to fix them:

git config --global \
  url.'file:///net/git/'.insteadOf \
  'https://internal.mycorp.example/repo/'

With the above configuration directive in place, URL prefixes which match https://internal.mycorp.example/repo/ will be automatically replaced with file:///net/git/ instead. This works for git submodules or any other attempt to git clone the given URL. It can also be useful for package managers which use libgit or the git executable to clone repositories. Rust's cargo, for example, can use dependencies cloned from git repositories.

If you find yourself using a lot of submodules, you may want to consider if git-subtree workflows are more appropriate. Subtrees can greatly reduce the number of individual repos you need to mirror.

Issue Trackers

Generally, issue tracker content is not exposed to Git, and git-teleport cannot mirror it. While some issue trackers offer APIs, accessing issues and interpreting them for the destination system is beyond the scope of git-teleport.

git-bug is a distributed issue tracker that resides in your existing Git repository. It is not supported in git-teleport yet, but it could be with a little effort. See git-teleport/git-teleport#8.

Large File Storage

Git Large File Storage (LFS) is just "hyperlinks to things not stored in Git." git-teleport does not support LFS. As an alternative, consider keeping your files in Git itself and using either

shallow clones; or
partial clones (better)

to limit the amount of disk and network bandwidth consumed by working copies.

git-teleport still needs to mirror all the files in your repository.

Next Steps

Mirroring repositories is only the first step in transferring your build environment. Read more tips on mirroring a complete build environment in "Mirroring 'Everything Else'."

The Git-Teleport Manual