English
资源说明：Distributed bulk build tool for pkgsrc
distbb - distributed bulk builds framework for pkgsrc (www.pkgsrc.org).
DISTBB main features:
- DISTBB is not fork of PBULK and original pkgsrc Bulk Build Framework.
Everything in it was written from scratch.
- DISTBB can build packages in an isolated environment, e.g. chroots
virtual machines etc. That is working system is kept intact.
- DISTBB was written with portability in mind.
Though, currently only NetBSD, Linux, Interix and Solaris are tested.
- Several machines in a network or chroots can be used for
building packages. That is, independent packages can be built
in parallel using all available power (CPUs and/or machines).
- Either all available packages (default) or only specified packages
(distbb -f) can be built. Individual packages may also be built
easily (distbb -p).
- Collecting an information about source packages
(pkg_src_summary stage) is also distributed.
Different packages are analysed in parallel .
- DISTBB is _really_ modular. Tool for running things in parallel
is completely independent from DISTBB.
It is a separate (parallel/paexec)
universal tool which is developped and packaged separatedly.
Another thing separated from DISTBB is a collection of tools
for manipulating so called packages summary. See pkg_summary(5)
and pkgtools/pkg_summary-utils package for more information.
Other helper tools are devel/pipestatus and lang/runawk.
- DISTBB is small. It is as small as Hubert Feyrer's
bulk build framework and is more than 30% smaller than
Joerg Sonnenberger's pbulk. At the same time
it provides more features.
- DISTBB is written completely in SHELL and AWK. "C" language is
not used at all. This is just wrong tool for things like DISTBB.
PERL is not used too. It is too big and I don't like it :-)
- DISTBB is very flexible. There are two very important things
about how distbb works. The first is building options
(options passed to bmake). The second is a list of targets
(fetch, configure, build etc.). All building options and targets
are specified in the configuration file (distbb.conf) and may
be easily changed. DESTDIR support mentioned above is not something
special. This is a result of just few changes in distbb.conf.
Another reason why DISTBB is so flexible is that it is possible
for users to define their own targets in distbb.local.mk config file.
For example, see dbb_canbebuilt, dbb_excluded and dbb_vars targets.
And finally one may set building options and target for individual
package. This gives you a way to, for example, build bootstrap packages
with USE_DESTDIR=full or to use different (not default) compiler
for individual packages (qemu and festival don't like gcc-4,
no problem, build them using gcc-3 if it is available on your system).
Another example is to set/unset "stack smashing protection"
for individual packages.
- DISTBB is able to upload building logs to the external server.
- DISTBB is able to upload binary packages.
Uploading packages is aware of NO_BIN_ON_CDROM
and NO_BIN_ON_FTP restrictions. See UPLOAD_PKGS_FILES variable
in distbb.default.conf.
- Bulk build results can be sent to the user via
email (or jabber, icq etc.).
- Bulk build logs include (optionally) difference with previous
bulk build and "global" progress sections. This is very powerful
feature absent (at the time of writing) in bbf and pbulk.
Command like tool for comparing results of two bulk builds
is also available.
- In distributed mode distbb is immune to client hosts' failures.
In case client host fails, this host is marked as broken
and distbb continue to work with the rest clients
moving broken package (if any) to other alive client.
- Normally DISTBB receives packages in PKGPATH form, e.g. archivers/gzip,
pkgtools/distbb etc. One can also use so called enriched PKGPATHs.
Enriching here means additional building options in the following form
category/package[:var1=value1[,var2=value2[,...]]]
For example,
textproc/dictem:EMACS_TYPE=xemacs214
textproc/dictem:EMACS_TYPE=emacs22
www/ap2-wsgi:PKG_APACHE=apache22,PYTHON_VERSION_REQD=25
sysutils/mc:PKG_OPTIONS.mc=-slang~-charset,PKGNAME=mc-light-4.6.1nb4
sysutils/mc:PKG_OPTIONS.mc=samba,PKGNAME=mc-full-4.6.1nb4
sysutils/mc:PKG_OPTIONS.mc=-x11,PKGNAME=mc-nox11-4.6.1nb4
See NEWS file about 0.19.0 release for details.
- DISTBB generates comprehensive logs.
* META/report.html - HTML version of the report
* META/report.txt - Plain text version of the report
* META/packages_built_prev.txt - list of packages that was
built previously and not rebuilt.
* META/packages_built.txt - list of packages really rebuilt
and succeeded.
* META/packages_built_total.txt - list of packages ready for use.
* META/packages_failed_scan.txt - list of packages for which
pkg_src_summary failed, e.g. due to syntax errors in Makefile.
* META/packages_failed_notavail.txt - list of packages marked
as not available (NOT_FOR_PLATFORM, ONLY_FOR_COMPILER etc.)
* META/packages_failed.txt - list of packages really failed.
* META/packages_failed_deps.txt - list of packages failed
due to broken dependencies.
* META/packages_failed_total.txt -
failed_scan + failed_notavail + faild + failed_deps
* META/pkg_src_summary.txt - src_summary for all analysed packages
* build_src_summary_log.txt - bmake stderr collected during
building the src_summary
(see META/pkg_src_summary.txt and META/packages_failed_scan.txt)
===========================================================================
CONFIGURATION/TUTORIAL:
1) bootstrap your working pkgsrc (to /usr/pkg, for example)
Example:
# cd /usr/pkgsrc/bootstrap/
# ./bootstrap
...
# edit /usr/pkg/etc/mk.conf
# cd /
# tar -czf /usr/pkgsrc/packages/bootstrap.tgz /usr/pkg /var/db/pkg
#
Example for mk.conf:
...
SKIP_LICENSE_CHECK= yes
# PKG_DEVELOPER= yes
ALLOW_VULNERABLE_PACKAGES= yes
# I'd strongly recommend to use local directory for building.
# Building packages on NFSed directories is slooooooooow.
WRKOBJDIR= /tmp/pkgsrc_obj
# The following line makes future changes in setting easier
.sinclude "/usr/pkg_distbb/etc/distbb.local.mk"
2) Prepare distbb
- bootstrap pkgsrc for distbb (to /usr/pkg_distbb, for example)
Example:
# cd /usr/pkgsrc/bootstrap/
# ./bootstrap --prefix /usr/pkg_distbb
- install pkgtools/distbb in it
Example:
# cd /usr/pkgsrc/pkgtools/distbb
# /usr/pkg_distbb/bin/bmake install
- install other software you may need (rsync, nbpatch etc.)
- edit /usr/pkg_distbb/etc/distbb.conf configuration file
========================================
For "distributed" bulk builds:
3d) On master host:
- make /usr/pkg_distbb and /usr/pkgsrc directories NFS mountable
for all your slave hosts.
4d) On all slave hosts:
- mount -t nfs -o ro master:/usr/pkg_distbb /usr/pkg_distbb
mount -t nfs -o rw master:/usr/pkgsrc /usr/pkgsrc
All slave hosts need they own PREFIX (e.g. /usr/pkg)
and PKGDBDIR (e.g. /var/db/pkg). These dirs will be deleted
on slaves.
NOTE:
Only /usr/pkgsrc/packages and /usr/pkgsrc/distfiles
subdirectories need to be writeable and therefore can be
mounted separately. /usr/pkgsrc can be mounted read-only.
5d) On all slave hosts:
Make sure PATH environment variable includes /usr/pkg/bin
(edit .profile or .login files for this).
6d) On the master host:
- /usr/pkg_distbb/bin/distbb -h
# for full bulk build
/usr/pkg_distbb/bin/distbb
# for partial bulk builds
/usr/pkg_distbb/bin/distbb -f packages.txt # for partial bulk build
# for building individual packages
/usr/pkg_distbb/bin/distbb -p 'meta-pkgs/xfce4 multimedia/mplayer'
# for (re)building installed packages
/usr/pkg_distbb/bin/distbb -i
NOTE:
- in "distributed" mode master's /usr/pkg is NOT changed.
Neither pkg_add nor pkg_delete is called on master host.
- /usr/pkgsrc/packages/All/pkg_summary.txt (PKG_SUMMARY),
/usr/pkgsrc/packages/All/pkg_all_summary.txt (PKG_ALL_SUMMARY) and
/usr/pkgsrc/pkg_src_summary.txt (PKG_SRC_SUMMARY) are created/updated
from master host
- Everything under ~/bulk_log directory (REPORTS_DIR)
is managed by master host
- Uploading logs (UPLOAD_LOGS_CMD), packages (UPLOAD_PKGS_CMD)
and sending a report (REPORT_CMD) is run on master host.
- On ALL slave hosts PREFIX (/usr/pkg) and PKGDBDIR (/var/db/pkg)
will be removed and recreated from bootstrap tarball.
- DistBB can be run on master host by unprivileged user
even if packages are built using root privileges
- Master host and slave hosts don't have to run the same OS.
Slave hosts should run a target operating operating system.
- By default ssh is used a transport program (TRANSPORT)
between master and slaves. You can use any ssh-like program
for this purpose.
- chrooted environment (ideally read-only chroots)
can also be used as slave.
========================================
For single-node bulk build:
3s) Make sure PATH environment variable includes
/usr/pkg_distbb/bin and /usr/pkg/bin
4s) /usr/pkg_distbb/bin/distbb -h
/usr/pkg_distbb/bin/distbb # for full bulk build
/usr/pkg_distbb/bin/distbb -f packages.txt # for partial bulk build
NOTE:
- /usr/pkg (and PKGDBDIR) will be removed and recreated from
bootstrap tarball.
===========================================================================
Notes.
A)
There are the following stages of the bulk build (BUILD_STAGES):
init - initialization, including running
pkg_update_src_summary and pkg_update_summary
build - actual package building
gen_report - report generation
update_best - update information about "virtual best" bulk build
($REPORTS_DIR/.best_bb)
summary - generating pkg_summary(5)
checksum - generating SHA512.txt
upload_logs - uploading bulk build logs
upload_pkgs - uploading binary packages
report - sending email or other type of notification
They are run in order.
All stages except 'init' are rerunable. They can be rerun like this
/usr/pkg_distbb/libexec/distbb/stage_upload_logs
BUILD_ID is printed at the very beginning of bulk build process
B) Full list of configuration variables is available
in share/distbb/distbb.default.conf
C) ?
===========================================================================
Aleksey Cheusov
本源码包内暂不包含可直接显示的源代码文件，请下载源码包。
