Making a local Yocto mirror

The Yocto build process involves downloading the source-code for all needed packages for every new build. This can be slow and inefficient, depending on your internet connection. To fix this, a Yocto source-code mirror can be configured inside the local network.

Starting a build with mirroring flags

The first step is to set up the build environment like you’d do for a normal build. In my case I’m starting a default build, but you can adjust this step to match your build configuration:

git clone https://git.yoctoproject.org/git/poky -b dunfell
source ./poky/oe-init-build-env build

Then, edit the configuration such that the source-code repositories get stored as tarballs (BB_GENERATE_MIRROR_TARBALLS):

vim conf/local.conf
#
# Add this at the end of the file:
# BB_GENERATE_MIRROR_TARBALLS = "1"
#

Now you can start a build, and specify that only the “fetch” step gets run. You can adjust this command to match your build-target.

bitbake core-image-minimal --runall=fetch

This will download all the needed archives and place them in the “downloads” directory.

Preparing the mirror directory

Not all the files from the “downloads” directory are needed. The following commands are just for clean-up.

# If the a downloaded archive is detected as being corrupted (its hashes do not
# match), it is renamed to something like
# fldigi-4.1.19.tar.gz_bad-checksum_8715e7109d2a674d80b742c97743fe7cb8997166b3c6ddef622c8cd8779d6e7f
# and saved anyway. We can remove these files.
find ./downloads -name "*bad-checksum*" -exec rm -f {} \;

# Repositories are first cloned into the ./downloads/git2/ or ./downloads/svn/
# directories and then compressed into the ./downloads directory. The mirror won't
# need the uncompressed files, so we delete them.
rm -rf ./downloads/git2
rm -rf ./downloads/svn

# For every downloaded archive, there is a "*.done" file used as a filesystem lock,
# to mark to other local processes that the archive is complete and can be read.
# Mirrors don't use this feature, we can delete the lock files.
find ./downloads -name "*.done" -exec rm -f {} \;

Now the “downloads” directory can be shared using HTTP and used across the network.

Using the mirror

If the “downloads” directory has been published on a HTTP server, under an URL like http://mirror.local/downloads, then all new Yocto builds can use the following two lines of configuration in the conf/local.conf file:

SOURCE_MIRROR_URL = "http://mirror.local/downloads/"
INHERIT += "own-mirrors"

You can start a new build and check the web-server’s logs that the files are pulled from there.