Skip to content
This repository has been archived by the owner on Jan 15, 2025. It is now read-only.

Support required to build puppet-runtime outside of puppet infrastructure. #784

Closed
maureen-lawless opened this issue Jan 23, 2024 · 9 comments
Labels
bug Something isn't working help wanted Issue has been reviewed & PRs are welcome triaged Jira issue has been created for this

Comments

@maureen-lawless
Copy link

maureen-lawless commented Jan 23, 2024

Hi,

I'm trying to build the puppet-runtime project "agent-runtime-main" for el-8-x86_64 from version 202312201.

Intially, I got an error from component libxml2 trying to dowload libxml2-v2.10.3.tar.gz from the 'puppet internal buildsources url'

_shared-agent-settings.rb
-------------------------------------
proj.setting(:artifactory_url, "https://artifactory.delivery.puppetlabs.net/artifactory")
proj.setting(:buildsources_url, "#{proj.artifactory_url}/generic/buildsources")

I replaced the libxml2 url to: https://gitlab.gnome.org/GNOME/libxml2/-/archive/v2.10.3/libxml2-v2.10.3.tar.gz
along with the urls for libxslt, ansicon as these also referenced the puppet internal buildsources url:

https://gitlab.gnome.org/GNOME/libxslt/-/archive/v1.1.39/libxslt-v1.1.39.tar.gz
https://github.com/adoxa/ansicon/archive/refs/tags/v1.86.tar.gz

Then the build failed with the following error:

gunzip -c "libxml2-v2.10.3.tar.gz" | tar xf -
touch libxml2-unpack
touch libxml2-patch
[ -d libxml2-v2.10.3 ] || mkdir -p libxml2-v2.10.3
export LDFLAGS="-L/opt/puppetlabs/puppet/lib -Wl,-rpath=/opt/puppetlabs/puppet/lib,-z,relro,-z,now" && \
export CFLAGS="-fstack-protector-strong -fno-plt -O2" && \
cd libxml2-v2.10.3 && \
./configure --prefix=/opt/puppetlabs/puppet --without-python 
**/bin/bash: line 3: ./configure: No such file or directory**
make: *** [Makefile:443: libxml2-configure] Error 127
bundler: failed to load command: vanagon (/usr/bin/vanagon)
/usr/share/gems/gems/vanagon-0.44.0/lib/vanagon/utilities.rb:275:in `block in local_command': Local command ((cd /var/tmp/tmp.47KsmYNMcZ; /usr/bin/make )) failed. (RuntimeError)
	from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler.rb:404:in `block in with_clean_env'
	from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler.rb:708:in `with_env'
	from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler.rb:404:in `with_clean_env'

The ./configure file is missing, and I see the following comment in the puppet-runtime/configs/components/libxml2.rb file:

Newer versions of libxml2 either ship as tar.xz or do not ship with a configure file
and require a newer version of GNU Autotools to generate. This causes problems with
the older and esoteric (AIX, Solaris) platforms that we support.
So we generate a configure file manually, compress as tar.gz, and host internally.

Same comment in puppet-runtime/configs/components/libxslt.rb

Can you share the modified/re-packaged tarballs of libxml2-v2.10.3.tar.gz, libxslt-v1.1.39.tar.gz, and ansicon v1.86.tar.gz (if this has been modified and hosted internally)
or share the Configure files that were generated manually and indicate where they should be placed in the tarball for a successful build?

@joshcooper
Copy link
Contributor

@maureen-lawless you might give #765 a try to see if that gets you closer. We welcome pull requests to make it easier to build the runtime outside of our environment.

@joshcooper joshcooper added the help wanted Issue has been reviewed & PRs are welcome label Jan 23, 2024
@maureen-lawless
Copy link
Author

maureen-lawless commented Jan 25, 2024

@joshcooper, I tried #765 but getting a different error now:

cd libselinux-2.9 &&
export VENDORARCHDIR=/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/3.2.0/x86_64-linux &&
install -d ${VENDORARCHDIR} &&
install -p -m755 _rubyselinux.so ${VENDORARCHDIR}/selinux.so &&
/usr/bin/make -e clean
make[1]: Entering directory '/var/tmp/tmp.5eArMYQitj/libselinux-2.9'
Package libpcre was not found in the pkg-config search path.
Perhaps you should add the directory containing libpcre.pc' to the PKG_CONFIG_PATH environment variable Package 'libpcre', required by 'virtual:world', not found Package libpcre was not found in the pkg-config search path. Perhaps you should add the directory containing libpcre.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libpcre', required by 'virtual:world', not found
configure.ac:42: error: require Automake 1.16.3, but have 1.16.1
autoreconf: automake failed with exit status: 1
make: *** [Makefile:439: libxml2-configure] Error 1
bundler: failed to load command: vanagon (/usr/bin/vanagon)
/usr/share/gems/gems/vanagon-0.44.0/lib/vanagon/utilities.rb:275:in block in local_command': Local command ((cd /var/tmp/tmp.5eArMYQitj; /usr/bin/make )) failed. (RuntimeError) from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler.rb:404:in block in with_clean_env'
from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler.rb:708:in with_env' from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler.rb:404:in with_clean_env'
from /usr/share/gems/gems/vanagon-0.44.0/lib/vanagon/utilities.rb:281:in clean_environment' from /usr/share/gems/gems/vanagon-0.44.0/lib/vanagon/utilities.rb:264:in local_command'
from /usr/share/gems/gems/vanagon-0.44.0/lib/vanagon/engine/local.rb:36:in dispatch' from /usr/share/gems/gems/vanagon-0.44.0/lib/vanagon/driver.rb:152:in run'
from /usr/share/gems/gems/vanagon-0.44.0/lib/vanagon/cli/build.rb:56:in block in run' from /usr/share/gems/gems/vanagon-0.44.0/lib/vanagon/cli/build.rb:53:in each'
from /usr/share/gems/gems/vanagon-0.44.0/lib/vanagon/cli/build.rb:53:in run' from /usr/share/gems/gems/vanagon-0.44.0/lib/vanagon/cli.rb:86:in run'
from /usr/share/gems/gems/vanagon-0.44.0/bin/vanagon:7:in <top (required)>' from /usr/bin/vanagon:25:in load'
from /usr/bin/vanagon:25:in <top (required)>' from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler/cli/exec.rb:58:in load'
from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler/cli/exec.rb:58:in kernel_load' from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler/cli/exec.rb:23:in run'
from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler/cli.rb:486:in exec' from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run'
from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command' from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler/vendor/thor/lib/thor.rb:392:in dispatch'
from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler/cli.rb:31:in dispatch' from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler/vendor/thor/lib/thor/base.rb:485:in start'
from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler/cli.rb:25:in start' from /usr/local/share/gems/gems/bundler-2.3.26/exe/bundle:48:in block in <top (required)>'
from /usr/local/share/gems/gems/bundler-2.3.26/lib/bundler/friendly_errors.rb:120:in with_friendly_errors' from /usr/local/share/gems/gems/bundler-2.3.26/exe/bundle:36:in <top (required)>'
from /usr/local/bin/bundle:23:in load' from /usr/local/bin/bundle:23:in

'

Seems to be pointing to needing a higher version of automake?

configure.ac:42: error: require Automake 1.16.3, but have 1.16.1
autoreconf: automake failed with exit status: 1
make: *** [Makefile:439: libxml2-configure] Error 1

Do you have any further suggestions to solve this, preferably without having to update the automake version? 1.16.1 seems to be the the latest automake version available on el 8.8.

@joshcooper
Copy link
Contributor

Sorry I'm not sure what to suggest other than to update automake to a newer version.

@maureen-lawless
Copy link
Author

maureen-lawless commented Feb 8, 2024

@joshcooper

My RHEL8 platform versions of the autotools are:

automake 1.16.1
libtool 2.4.6-25
autoconf 2.69-29
m4 1.4.18

The advantage of using the platform components is that they have all the necessary security patches and you know they are compatible together.

The only other option I see is to install the tools from source, i've tried the following:

automake 1.16.3
libtool-2.4.6
autoconf-2.70
libtool-2.4.6
m4-1.4.19
pkg-config-0.29

Can you advise if those versions are good versions to choose?

After installing those from source and applying a number of local patches the puppet-runtime component built.

I then successfully built the puppet-agent project version 8.2.0.
However there are some differences in my generated rpm for rhel8 in comparison to the puppet provided rpm: http://yum.puppet.com/puppet8/el/8/x86_64/puppet-agent-8.2.0-1.el8.x86_64.rpm

I notice differences in a few C++ header files (I can provide details if neccessary), i'm wondering if that would be expected, and if it could be due to my using different versions of the gnu autotools?

Also, is the puppet-agent pxp-agent component only necessary if using puppet orchestrator?

@joshcooper
Copy link
Contributor

Can you advise if those versions are good versions to choose?

This is what we use on RHEL8 to build agent-runtime-7.x

Installing:
 libtool                    x86_64 2.4.6-25.el8     localmirror-appstream 709 k
 autoconf                   noarch 2.69-29.el8      localmirror-appstream 711 k
 createrepo_c               x86_64 0.17.7-6.el8     localmirror-appstream  89 k
 rpm-build                  x86_64 4.14.3-26.el8    localmirror-appstream 174 k
 automake                   noarch 1.16.1-8.el8     localmirror-appstream 714 k
 gcc-c++                    x86_64 8.5.0-20.el8     localmirror-appstream  12 M
 cmake                      x86_64 3.26.5-1.el8_9   localmirror-appstream  14 M
 rsync                      x86_64 3.1.3-19.el8_7.1 localmirror-base      410 k
 rpm-sign                   x86_64 4.14.3-26.el8    localmirror-base       81 k
Upgrading:
 cpp                        x86_64 8.5.0-20.el8     localmirror-appstream  10 M
 gcc                        x86_64 8.5.0-20.el8     localmirror-appstream  23 M
 ima-evm-utils              x86_64 1.3.2-12.el8     localmirror-base       64 k
 python3-libdnf             x86_64 0.63.0-3.el8     localmirror-base      777 k
 dnf-plugins-core           noarch 4.0.21-4.el8_5   localmirror-base       70 k
 libdnf                     x86_64 0.63.0-3.el8     localmirror-base      701 k
 python3-dnf-plugins-core   noarch 4.0.21-4.el8_5   localmirror-base      234 k
 python3-hawkey             x86_64 0.63.0-3.el8     localmirror-base      116 k
 yum                        noarch 4.7.0-8.el8      localmirror-base      202 k
 dnf                        noarch 4.7.0-8.el8      localmirror-base      541 k
 dnf-data                   noarch 4.7.0-8.el8      localmirror-base      155 k
 make                       x86_64 1:4.2.1-11.el8   localmirror-base      498 k
 python3-dnf                noarch 4.7.0-8.el8      localmirror-base      545 k
 libarchive                 x86_64 3.3.3-5.el8      localmirror-base      360 k
 librepo                    x86_64 1.14.2-4.el8     localmirror-base       93 k
 python3-librepo            x86_64 1.14.2-4.el8     localmirror-base       54 k
 python3-rpm                x86_64 4.14.3-26.el8    localmirror-base      155 k
 rpm                        x86_64 4.14.3-26.el8    localmirror-base      544 k
 rpm-build-libs             x86_64 4.14.3-26.el8    localmirror-base      157 k
 rpm-libs                   x86_64 4.14.3-26.el8    localmirror-base      345 k
 rpm-plugin-selinux         x86_64 4.14.3-26.el8    localmirror-base       78 k
 rpm-plugin-systemd-inhibit x86_64 4.14.3-26.el8    localmirror-base       79 k
 elfutils-libelf            x86_64 0.189-3.el8      localmirror-base      232 k
 elfutils-libs              x86_64 0.189-3.el8      localmirror-base      303 k
 libgcc                     x86_64 8.5.0-20.el8     localmirror-base       81 k
 libgomp                    x86_64 8.5.0-20.el8     localmirror-base      208 k
 libstdc++                  x86_64 8.5.0-20.el8     localmirror-base      455 k
 libsolv                    x86_64 0.7.20-6.el8     localmirror-base      376 k
Installing dependencies:
 libatomic_ops              x86_64 7.6.2-3.el8      localmirror-appstream  38 k
 gc                         x86_64 7.6.4-3.el8      localmirror-appstream 109 k
 guile                      x86_64 5:2.0.14-7.el8   localmirror-appstream 3.5 M
 libipt                     x86_64 1.6.1-8.el8      localmirror-appstream  50 k
 zstd                       x86_64 1.4.4-1.el8      localmirror-appstream 393 k
 drpm                       x86_64 0.4.1-3.el8      localmirror-appstream  68 k
 libuv                      x86_64 1:1.41.1-1.el8_4 localmirror-appstream 156 k
 vim-filesystem             noarch 2:8.0.1763-19.el8_6.4
                                                    localmirror-appstream  50 k
 createrepo_c-libs          x86_64 0.17.7-6.el8     localmirror-appstream 116 k
 gdb-headless               x86_64 8.2-20.el8       localmirror-appstream 3.7 M
 libstdc++-devel            x86_64 8.5.0-20.el8     localmirror-appstream 2.1 M
 cmake-data                 noarch 3.26.5-1.el8_9   localmirror-appstream 1.9 M
 cmake-filesystem           x86_64 3.26.5-1.el8_9   localmirror-appstream  45 k
 cmake-rpm-macros           noarch 3.26.5-1.el8_9   localmirror-appstream  44 k
 m4                         x86_64 1.4.18-7.el8     localmirror-base      223 k
 libzstd                    x86_64 1.4.4-1.el8      localmirror-base      266 k
 libmodulemd                x86_64 2.13.0-1.el8     localmirror-base      233 k
 libbabeltrace              x86_64 1.5.4-4.el8      localmirror-base      200 k
 elfutils                   x86_64 0.189-3.el8      localmirror-base      553 k
 emacs-filesystem           noarch 1:26.1-11.el8    localmirror-base       70 k
 tpm2-tss                   x86_64 2.3.2-5.el8      localmirror-base      275 k
Installing weak dependencies:
 elfutils-debuginfod-client x86_64 0.189-3.el8      localmirror-base       75 k

Also, is the puppet-agent pxp-agent component only necessary if using puppet orchestrator?

Correct and you can omit that by setting export NO_PXP_AGENT=true

@maureen-lawless
Copy link
Author

@joshcooper, I'm trying to build agent-runtime-main (which I assume is applicable for puppet agent 8), not agent-runtime-7.x.
However, I assume the same versions of those tools will be in use for both agent-runtime-7.x and agent-runtime-main, on RHEL8 yes?

Right, those versions you have provided above (including automake 1.16.1) are identical to what I had in my baseline initially which is not surprising given we're both building on RHEL8.

I feel a bit like i'm going round in circles now, you were telling me to upgrade automake to 1.16.3 but you are using the older version in your own build.

Just to re-cap:

My initial problem was the usage of internal puppet buildsource urls for libxml2 and libxslt components which are not available to me. It looks like you have patched those tarballs internally, generating your own .configure files - rather than upgrading autotools in your RHEL8 build env.

Comment in libxml2.rb:

Newer versions of libxml2 either ship as tar.xz or do not ship with a configure file
and require a newer version of GNU Autotools to generate. This causes problems with
the older and esoteric (AIX, Solaris) platforms that we support.
So we generate a configure file manually, compress as tar.gz, and host internally.

It seems to me the best way forward would be for you to share the configure files for both libxslt and libxml2 so I can create my own tarball and host this locally - or share the steps to do so, so I get an identical build.

The alternative you suggested was to pull this PR: #765 which is forcing the download of those tarballs from github and these require the higher version of automake 1.16.3 and other associated tools which are not available on the RHEL8 platform.

As I said already, with the higher versions of autotools installed from source I appear to be seeing differences in C++ header files in my puppet-agent rpm, I've no idea why this is or if it's of consequence, - could it be due to the usage of the higher versions installed from source?

I feel the only way to find out is to try the alternate I suggested above.

@h0tw1r3
Copy link
Contributor

h0tw1r3 commented Feb 14, 2024

@maureen-lawless the required libxml2 and libxslt sources depend on automake 1.16.3 because of this bug related to python 3.10 detection. As we're building these libraries without python support, the newer automake is not needed and you can safely set it to 1.15 in configure.ac.

I updated #765 to include this patch. In my local test it configures and builds libxml2 and libxslt now on EL8.

It seems to me the best way forward would be for you to share the configure files for both libxslt and libxml2 so I can create my own tarball and host this locally - or share the steps to do so, so I get an identical build.

If you prefer not to use the PR or patch, the steps are quite straightforward. On any system that includes an autotools package of at least 1.16.3, download and extract the libxml2 and libxslt source archives from the vendor, run autoreconf -fi, then tar the directory up, use it as the source for the component.

@maureen-lawless
Copy link
Author

Thanks a lot @h0tw1r3 for updating the patch #765 and for the detail on the issue.

@joshcooper joshcooper added the triaged Jira issue has been created for this label Apr 4, 2024
Copy link

github-actions bot commented Apr 4, 2024

Migrated issue to PA-6375

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working help wanted Issue has been reviewed & PRs are welcome triaged Jira issue has been created for this
Projects
None yet
Development

No branches or pull requests

3 participants