#200 getopt.test test fails on macOS

[Ryan Carsten Schmidt]

The test getopt.test fails on macOS:

creating getopt.def in /tmp/autogen-5.18.16/autoopts/test/getopt-testd
creating getopt1-base.c
creating getopt2-base.c
dyld: Library not loaded: /usr/local/lib/libopts.25.dylib
  Referenced from: /tmp/autogen-5.18.16/agen5/.libs/autogen
  Reason: image not found
Abort trap: 6
Killing AutoGen 64742
FAILURE REASON:  COMMAND FAIL: /tmp/autogen-5.18.16/agen5/.libs/autogen -bgetopt-bn -L/tmp/autogen-5.18.16/autoopts/tpl  -Toptions.tpl getopt.def

The reason this happens is that /usr/local/lib/libopts.25.dylib hasn't been installed yet.

To use a library you just built but haven't installed yet, you have to set the DYLD_LIBRARY_PATH environment variable to the path where the just-built library is located. For security reasons, DYLD environment variables are not passed on to subshells so you have to set it as close as possible to the point where it's actually going to be used (i.e. right when you try to run the agen5/.libs/autogen program).

[Bruce Korb]

I've never heard of DYLD environment variables before. Must be some OS/X-ism. I'm going to guess that the "run-ag.sh" script should hunt down the libopts..dylib from somewhere in the top_builddir and then set it. The proper fix is for the LibTool stuff figure it all out. the $top_builddir/agen5/autogen file is actually a script that is supposed to do the right thing. The binary that needs a mess of loader hints is named $top_builddir/agen5/.libs/autogen

[Ryan Carsten Schmidt]

You're right, dyld is the macOS-specific dynamic library loading mechanism, and the DYLD environment variables that can be used to modify its behavior are also macOS-specific.

You're also right that libtool should handle this. libtool creates a script agen5/autogen which sets DYLD_LIBRARY_PATH and then launches the compiled executable agen5/.libs/autogen.

I think the problem is related to the fact that autogen relaunches itself. When the getopt test fails, macOS generates two crash reports. Usually there would be only one log per crashed process. In this case there are two autogen processes that crash. The more recently launched one crashes when it's unable to load the dynamic library, and the first one crashes with a different error, presumably because the other autogen that it launched didn't exit cleanly. This tells me DYLD_LIBRARY_PATH is set properly by the libtool wrapper for the initial launch but not for the subsequent sublaunch.

I was trying to figure out where this sublaunch occurs and I spent some time fixated on this block in main() in agen5/autogen.c:

    {
        char const * lc = getenv("LC_ALL");
        if ((lc != NULL) && (*lc != NUL) && (strcmp(lc, "C") != 0)) {
            static char lc_all[] = "LC_ALL=C";
            putenv(lc_all);
            execvp(argv[0], argv);
            fserr(AUTOGEN_EXIT_FS_ERROR, "execvp", argv[0]);
        }
    }

But that's not where it's happening. I added enough printf statements to convince myself it's getting to AG_SCM_BOOT_GUILE, so the sublaunch is somewhere in there. If whatever AG_SCM_BOOT_GUILE is doing to sublaunch autogen would pass DYLD_LIBRARY_PATH along when it does so, that might fix it.

[Bruce Korb]

It all depends on when the DYLD stuff gets stripped out. I suspect it's the loader so that autogen the binary wouldn't find it, tho it could be the shell. Anyway, I think the proper thing would be to have our LibTool friends hide duplicates into MACOS_DYLD_xxx variables. Then I can add some magic OS/X specific code in a replacement exec() funciton.

[Ryan Carsten Schmidt]

DYLD environment variables are available to the autogen process, they're just not automatically propagated to subprocesses anymore. This is a new security measure introduced in OS X 10.11 as part of a feature Apple calls System Integrity Protection. Apple isn't trying to prevent programs from passing DYLD variables along manually, they just don't do so automatically anymore since that was a security vulnerability.

Running the test suite on OS X 10.10 which does not have that protection, the test gets further, failing with:

## /private/tmp/autogen-5.18.16/autoopts/test/getopt-testd
## ld: library not found for -lintl
## clang: error: linker command failed with exit code 1 (use -v to see invocation)
Killing AutoGen 75895
FAILURE REASON:  cannot compile /tmp/autogen-5.18.16/autoopts/test/getopt-testd/getopt-tmpd/.ag-DMdgFg/test_getopt.c

This seems like another flavor of the problem I reported in bug #191, where in this case the -lintl flag is being passed along but the -L/opt/local/lib flag is not.

[Ryan Carsten Schmidt]

An alternative to messing with DYLD_LIBRARY_PATH, if it's too difficult to manually propagate it to everywhere it's needed, is to build the library with an install_name pointing to its location in the build tree. Then the tests should run just fine, since the library will be where its install_name says it is. Upon installation, relink the library with the install_name pointing to the final install location and relink all the programs that had been linked with the library during the build.

The install_name of a macOS dynamic library is similar to the soname of a UNIX shared library except that the install_name is the absolute path where the library will be installed. Any programs linked against the library get the install_name of the library baked in, and will look for the library at that location (unless directed otherwise by DYLD_LIBRARY_PATH).

I think libtool might have a mode where it can do this for you. Something about --mode=relink. I'm not very familiar with libtool though.

[Bruce Korb]

It's too difficult to figure out when I cannot build GNU style source
packages on my Mac. How do you do it? I tried to build my password
manager, but it's insisting on a libintl.h header which is nowhere to be
found.

On 9/16/20 12:29 PM, Ryan Schmidt wrote:

An alternative to messing with DYLD_LIBRARY_PATH, if it's too
difficult to manually propagate it to everywhere it's needed, is to
build the library with an install_name pointing to its location in the
build tree. Then the tests should run just fine, since the library
will be where its install_name says it is. Upon installation, relink
the library with the install_name pointing to the final install
location and relink all the programs that had been linked with the
library during the build.

The install_name of a macOS dynamic library is similar to the soname
of a UNIX shared library except that the install_name is the absolute
path where the library will be installed. Any programs linked against
the library get the install_name of the library baked in, and will
look for the library at that location (unless directed otherwise by
DYLD_LIBRARY_PATH).

I think libtool might have a mode where it can do this for you.
Something about |--mode=relink|. I'm not very familiar with libtool
though.

I was familiar with libtool, but that was 20 years ago. :) I'm not anymore.

[Ryan Carsten Schmidt]

libintl.h is part of gettext, which Apple does not include in macOS; you would have to install gettext yourself, either by building it from source or installing it using a package manager like MacPorts or Homebrew.

I assume the password manager you're referring to is gnu-pw-mgr? I could try adding it to MacPorts for easier installation.

[Ryan Carsten Schmidt]

gnu-pw-mgr should be available in MacPorts within the hour:

sudo port install gnu-pw-mgr

[Bruce Korb]

Thank you! I'm actually making a few updates to it, but it's not quite
ready for prime time tho.
So I've had X-Code installed since I first got my Mac and I've just
installed MacPorts, but:

Bruces-iMac:~ brucekorb$ sudo port install gnu-pw-mgr
Password:
sudo: port: command not found

Obviously not installed correctly. How do I diagnose it? :) Thank you!

On Thu, Sep 17, 2020 at 5:45 PM Ryan Schmidt ryandesign@users.sourceforge.net wrote:

gnu-pw-mgr should be available in MacPorts
https://www.macports.org/install.php within the hour:

sudo port install gnu-pw-mgr

---

• [bugs:#200] https://sourceforge.net/p/autogen/bugs/200/ getopt.test
  test fails on macOS*

Status: open
Group: autogen
Created: Tue Sep 15, 2020 08:24 PM UTC by Ryan Schmidt
Last Updated: Fri Sep 18, 2020 12:14 AM UTC
Owner: Bruce Korb
Attachments:

• test-suite.log
  https://sourceforge.net/p/autogen/bugs/200/attachment/test-suite.log
  (1.6 kB; application/octet-stream)

The test getopt.test fails on macOS:

creating getopt.def in /tmp/autogen-5.18.16/autoopts/test/getopt-testdcreating getopt1-base.ccreating getopt2-base.cdyld: Library not loaded: /usr/local/lib/libopts.25.dylib
Referenced from: /tmp/autogen-5.18.16/agen5/.libs/autogen
Reason: image not foundAbort trap: 6Killing AutoGen 64742FAILURE REASON: COMMAND FAIL: /tmp/autogen-5.18.16/agen5/.libs/autogen -bgetopt-bn -L/tmp/autogen-5.18.16/autoopts/tpl -Toptions.tpl getopt.def

The reason this happens is that /usr/local/lib/libopts.25.dylib hasn't
been installed yet.

To use a library you just built but haven't installed yet, you have to set
the DYLD_LIBRARY_PATH environment variable to the path where the
just-built library is located. For security reasons, DYLD environment
variables are not passed on to subshells so you have to set it as close as
possible to the point where it's actually going to be used (i.e. right when
you try to run the agen5/.libs/autogen program).

---

Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/autogen/bugs/200/

To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/

[Bruce Korb]

Back to the exec() problem. it seems to be in getdefs. That program invokes autogen as a server process. I think the right solution is to add code to ensure the DYLD stuff gets exported under a "#ifdef apple" guard.

[Bruce Korb]

I could use a hint here. I've not figured out how to build the stuff under OS/X and I'm not completely certain where the DYLD environment stuff gets dropped.