Methods for speeding up build time in a project using bitbake?

This is a very broad question!

First, here is a summary on how to inspect your build performanceand dependencies when using the openembedded/yocto project. This answers the first part of the question.

What packages take more time to build?

Use the buildstats with the pybootchartgui tool produce a build chart.

Details:

Set USER_CLASSES += "buildstats" in your $BUILDIR/conf/local.conffile. This will dump detailed performance data in$BUILDDIR/tmp/buildstats/<DATE>. Next use the pybootchartgui.py script (inpoky/scripts/pybootchartgui) to generate the chart. This will help youlocalize possible bottlenecks in the build. Of course, if you have alot of recipes to bake, your chart will be huge. To remove some noiseuse the -m MINTIME command line option.

For example:

poky/scripts/pybootchartgui/pybootchartgui.py -m 600 $BUILDDIR/tmp/buildstats/201312310904

will only display tasks (do_compile, do_fetch, etc.) that take longerthan 10 minutes (600 seconds) to run.

How to check for package dependencies?

To explore the dependencies of a particular package use the depexputility. For example, to explore the dependencies of eglibc use:

bitbake -g -u depexp eglibc

This will give a better understanding of what each recipe depends onat both run and compile time.

How to check if there are any circular dependencies and how to solve them?

bitbake automatically detects circular dependencies and prints an error message when such a thing happens.The error message contains the name of the packages causing this circular dependency.

How to check if there are recipes which aren't used and how to safely remove them?

bitbake calculates dependencies automatically and won't buildpackages which aren't needed by your target. If you find some unwanted packages in your image and you wish to remove them:

1. use bitbake -g -u depexp <TARGET> to inspect how the package gets pulled in
2. modify the needed recipes in your layer (by creating a bbappend for example) to eliminate the dependency manually

Improving overall build performance

Finally, some tips on how to improve the overall build performance. This answers the second part of the question.

• Clean up your dependencies (bitbake -g -u depexp <TARGET> is your friend). Building less stuff takes less time.
• bitbake can automatically cache the build outputs and use it forfuture builds, this cache is called the "shared-state cache" and iscontrolled with the SSTATE_DIR variable in your local.conf.
• Set the BB_NUMBER_THREADS and PARALLEL_MAKE variables in your local.conf to match yourmachine's resources. These variables control how many tasks are runin parallel and how many processes 'make' should run in parallel (-joption) respectively.
• Put the 'build' directory on its own disk.
• Use the ext4 filesystem without journaling and with these mountoptions: noatime,barrier=0,commit=6000. WARNING: This makes yourhdd unreliable in case of power losses. Do not store anything ofvalue on this hdd.
• building images with -dev and/or -dbg packages increases thedo_rootfs task time considerably. Make sure you enable them (seeEXTRA_IMAGE_FEATURES in your local.conf) only when needed.
• openembedded and yocto both support icecream (distributed compile). See the icecc class and this post.
• Buy a faster machine ;)

References:

Yocto Build Performance Wiki

Bitbake GUI tools

I've tried the distributed compile way years ago ,but the server environment config is not that flexible for CI agent work ,and the build is not that suit for muti-people .

I've tried analyse with the build time with buildstats ,and found the build time is nearly cost all by compiling the 3-rd party opensource components, which I didn't modify at all.

So The most simple and effective way is to use "sstate-cache" to avoid unmodified components to build again.

The way I use in my work is to trade space for time

1. Compile the whole bitbake project ,you could find many .tgz files under "build/sstate-cache"

2. Add and commit all this files to git for tracking the file modification

3. Compile the whole project again without clean it

4. cd to your "build/sstate-cache" ,git status and found the files which is modified, delete them and commit to git

5. Then clean project without .tgzs under "sstate-cache" , The Trade of space for time is done

The "unmodified" files list could be modified according to your own project .

The build time reduce for me is from 1 hour to within 10 minutes

Hope this would be helpful