Mirroring "Everything Else"

checking whether build environment is sane...

... build environment is grinning and holding a spatula.

– #371

git-teleport can only mirror Git repositories. A typical application—and many atypical applications—have numerous dependencies. At bare minimum, you'll probably want a compiler toolchain… but which compiler toolchain?

This is a curated guide to mirroring all the other things you need to make your software run instead of shambling about aimlessly like a zombie. If you don't see your language or platform listed, see "Something Else" below.

General Advice

Write down instructions for reproducing your build environment and dependencies.

  • The best documentation for this activity is executable: think shell scripts, Containerfiles, and the like.

  • But that's no excuse to skimp on well-organized, greppable prose documentation.

It is much easier to mirror source code than binaries, but there is a balance: to build Linux from scratch you must first invent the universe. Regardless of what form your dependencies take, you should pin them.

  • For source code, specify exact versions or vendor the code.
  • For binaries, consider using containers.
  • For toolchains and operating systems, does their lifecycle match your product? Consider "Long Term Support" releases where appropriate.

The more of your build environment you can ship, the better…

  • … but if you're shipping a 12 GB virtual machine disk image, you should probably ask if there's a better way.
  • Your CI/CD environment and job runners count. Have a plan to mirror these as well.

Are you prepared in case $CRITICAL_DEPENDENCY vanishes from the face of the Internet tomorrow?

Operating Systems

Debian and Ubuntu

Debian repositories can be mirrored with ftpmirror.

Fedora and RHEL-based

dnf/yum repositories can be mirrored with createrepo.

Container Platforms

For best results, make your images as small as possible. Take steps to avoid unnecessary programs, unused files, or excessive layering in your container images.

  • Gitlab CE features a container registry that may be used in place of cloud registries like DockerHub.

  • Quay's mirror-registry script can help you stand up your own mirrors of internet registries.

Programming Languages

The more languages you use, the more complicated it will be to set up a build environment. Your build environment should also include the following tools, as applicable:

  • Autoformatters
  • Documentation generators
  • "Linters" and other static analyzers
  • Dynamic analyzers (i.e., memory-safety checkers)
  • Test frameworks and test-cases
  • Security- and other compliance-checkers, such as license checks

Be especially careful about forcing dependencies on IDEs, such as Eclipse, IntelliJ, or Visual Studio, on your downstream systems. When possible, use build systems which are not tied to a particular IDE.

C/C++

We recommend vendoring your source dependencies and building your entire application from source. Avoid operating system packages unless you have some other way to pin and store them, like containers.

  • CMake offers ExternalProject or FetchContent for making "superbuilds" of your project and its dependencies. Read these approaches carefully; they are very different.

Rust

Rust has standalone installers available, but these only cover the core cargo toolchain. cargo packages themselves can be mirrored with cargo-vendor.

Storage

Storage technologies include POSIX-like filesystems and object stores. The following programs can help with file retrieval:

  • rclone can be used to mirror cloud storage platforms like S3 and many others.
  • rsync can make incremental copies of filesystems over the network.

If you are attempting to mirror or publish a scientific dataset, ask yourself:

  • Is your dataset standardized? Does it have open-standard storage formats, a standardized canonical representation, and a stable API for access?

  • Is your dataset documented? Are there user manuals and other information that a user needs to know?

  • Is your dataset organized so that others may use it? Does it have a searchable catalogue or index?

  • Is your dataset curated? Does it have a maintainer that enforces quality and standards-compliance?

If the answer to any of the above questions is "no," a substantial investment may be required before the dataset can be mirrored.

Something Else?

Contributions are sought which expand this guide to other languages and systems. Open an issue or merge request today! The usual rules in CONTRIBUTING.md apply.

  • This guide covers only free and open-source software (F/OSS) only.

    • Mirroring tools must be open-source.
    • Documentation for closed-source products will be accepted only if the product has a substantial open-source version.

  • When possible, link to existing guides instead of writing your own. Hyperlinks should be to official maintainer websites or vendor/maintainer documentation only.

  • Please alphabetize your entry by software title—unless that doesn't make sense.