1

Hi, I’m cross posting from here to reach more people possible. I’m trying to compile a C/C++ project that targets an old Linux system. To have the old libraries at hand, I made a copy of the target root fs, and since GCC has hardcoded the library path I cannot really cross-compile against an old version of libc. Therefore I am using Clang + lld. The point is that when linking it complains about

[ 71%] Linking CXX executable single_lidar_listener
Ubuntu clang version 14.0.0-1ubuntu1.1
Target: arm-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6
Found candidate GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0
Selected GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0
Candidate multilib: .;@m32
Selected multilib: .;@m32
 "/usr/bin/ld.lld" --sysroot=/home/abertulli/box-root-inst -pie -EL -z relro -X --hash-style=gnu --build-id --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o single_lidar_listener /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/Scrt1.o /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crti.o /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtbeginS.o -L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0 -L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../lib -L/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf -L/home/abertulli/box-root-inst/lib/../lib -L/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf -L/home/abertulli/box-root-inst/usr/lib/../lib -L/home/abertulli/box-root-inst/lib -L/home/abertulli/box-root-inst/usr/lib CMakeFiles/single_lidar_listener.dir/single_lidar.cpp.o CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o libtoojpeg.a -ldl -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtendS.o /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crtn.o
ld.lld: error: undefined symbol: __dlopen
>>> referenced by dlopen.o:(dlopen) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a

ld.lld: error: undefined symbol: __dlclose
>>> referenced by dlclose.o:(.text+0x0) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a

ld.lld: error: undefined symbol: __dlsym
>>> referenced by dlsym.o:(dlsym) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/single_lidar_listener.dir/build.make:114: single_lidar_listener] Error 1
make[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/single_lidar_listener.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

Analyzing the content of libc.a as suggested in this answer, I see that these symbols are present. So why can’t Clang find them? Note that sysroot is set, and the GCC installation found seems to be the correct (target-rooted) one. thanks!

my toolchain file

# the name of the target operating system
set(CMAKE_SYSTEM_NAME Linux)

include(CMakePrintHelpers)
cmake_print_variables(CMAKE_DL_LIBS)

# where is the target environment located
set(root_fs_dir /home/abertulli/box-root-inst)

set(CMAKE_FIND_ROOT_PATH ${root_fs_dir})
set(CMAKE_SYSROOT ${root_fs_dir})


# which compilers to use for C and C++
set(CMAKE_C_COMPILER   "clang")
set(CMAKE_CXX_COMPILER "clang++")

set(CMAKE_LINKER       "ld.lld")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld -v")
set(CMAKE_C_FLAGS   "${CMAKE_C_FLAGS}   --gcc-toolchain=${CMAKE_SYSROOT}/usr")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} --gcc-toolchain=${CMAKE_SYSROOT}/usr")

set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) # to avoid trying to link and execute test programs
set(CMAKE_C_COMPILER_TARGET   arm-unknown-linux-gnueabihf)
set(CMAKE_CXX_COMPILER_TARGET arm-unknown-linux-gnueabihf)


# adjust the default behavior of the FIND_XXX() commands:
# search programs in the host environment
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)

# search headers and libraries in the target environment
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
2

Try to add -ldl to your linker line. Ideally the CMake scripts would detect that the platform needs to link to libdl, but many don’t. The proper way to do this is to add CMAKE_DL_LIBS to link flags.

3

I think I linked it correctly in the CMakeLists with
target_link_libraries(${PROJECT_NAME} ${CMAKE_DL_LIBS})
but also you can see in the output I posted (my newlines):

"/usr/bin/ld.lld" --sysroot=/home/abertulli/box-root-inst -pie -EL -z
relro -X --hash-style=gnu --build-id --eh-frame-hdr -m
armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o
single_lidar_listener
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/Scrt1.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crti.o
/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtbeginS.o
-L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0
-L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../lib
-L/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf
-L/home/abertulli/box-root-inst/lib/../lib
-L/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf
-L/home/abertulli/box-root-inst/usr/lib/../lib
-L/home/abertulli/box-root-inst/lib
-L/home/abertulli/box-root-inst/usr/lib
CMakeFiles/single_lidar_listener.dir/single_lidar.cpp.o
CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o
libtoojpeg.a -ldl -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc
/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtendS.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crtn.o
4

Try -Wl,-t,-y,__dlopen. libdl.a should define __dlopen.

See Analysis and introspection options in linkers | MaskRay for -t and -y.

You might relink with -Wl,--reproduce=/tmp/rep.tar and upload the tarball in a file host website.

5

Thanks for the suggestion. Apparently libdl.a gets checked, and a reference is found, but not the definition:

[ 42%] Linking CXX executable single_lidar_listener
Ubuntu clang version 14.0.0-1ubuntu1.1
Target: arm-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6
Found candidate GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0
Selected GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0
Candidate multilib: .;@m32
Selected multilib: .;@m32
 "/usr/bin/ld.lld" --sysroot=/home/abertulli/box-root-inst -pie -EL -z relro -X --hash-style=gnu --build-id --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o single_lidar_listener /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/Scrt1.o /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crti.o /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtbeginS.o -L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0 -L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../lib -L/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf -L/home/abertulli/box-root-inst/lib/../lib -L/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf -L/home/abertulli/box-root-inst/usr/lib/../lib -L/home/abertulli/box-root-inst/lib -L/home/abertulli/box-root-inst/usr/lib -t -y __dlopen -t -y __dlopen CMakeFiles/single_lidar_listener.dir/single_lidar.cpp.o CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o libtoojpeg.a -ldl -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtendS.o /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crtn.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/Scrt1.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crti.o
/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtbeginS.o
CMakeFiles/single_lidar_listener.dir/single_lidar.cpp.o
CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlopen.o)
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlopen.o): reference to __dlopen
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlclose.o)
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlsym.o)
/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/libstdc++.so
/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/libgcc_s.so.1
/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/libc.so.6
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libc_nonshared.a(elf-init.oS)
/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/ld-linux-armhf.so.3
/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/libgcc_s.so.1
/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtendS.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crtn.o
ld.lld: error: undefined symbol: __dlopen
>>> referenced by dlopen.o:(dlopen) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a

ld.lld: error: undefined symbol: __dlclose
>>> referenced by dlclose.o:(.text+0x0) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a

ld.lld: error: undefined symbol: __dlsym
>>> referenced by dlsym.o:(dlsym) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/single_lidar_listener.dir/build.make:114: single_lidar_listener] Error 1
make[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/single_lidar_listener.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

checking again with

readelf /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a -s

I find

File: /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlopen.o)

Symbol table '.symtab' contains 12 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 00000000     0 NOTYPE  LOCAL  DEFAULT    1 $t
     2: 00000000     0 NOTYPE  LOCAL  DEFAULT    5 $d
     3: 00000000   130 OBJECT  LOCAL  DEFAULT    5 __evoke_link_war[...]
     4: 00000000     0 SECTION LOCAL  DEFAULT    1 .text
     5: 00000000     0 SECTION LOCAL  DEFAULT    3 .data
     6: 00000000     0 SECTION LOCAL  DEFAULT    4 .bss
     7: 00000000     0 SECTION LOCAL  DEFAULT    5 .gnu.warning.dlopen
     8: 00000000     0 SECTION LOCAL  DEFAULT    6 .note.GNU-stack
     9: 00000000     0 SECTION LOCAL  DEFAULT    7 .ARM.attributes
    10: 00000001    12 FUNC    GLOBAL DEFAULT    1 dlopen
    11: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __dlopen

File: /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlclose.o)
[...]

Maybe the problem is that __dlopen is not defined as a function?

The libraries tar is here

6

Yes.

--why-extract=a.txt shows that libdl.a(dlopen.o) is extracted to satisfy a reference to dlopen. But __dlopen needed by it is not satisfied by any input file, therefore the error.

reference       extracted       symbol
home/abertulli/two_lidars/build/CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o       home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlopen.o)      dlopen
home/abertulli/two_lidars/build/CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o       home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlclose.o)     dlclose
home/abertulli/two_lidars/build/CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o       home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlsym.o)       dlsym
home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/Scrt1.o        home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libc_nonshared.a(elf-init.oS)  __libc_csu_init

In newer glibc, some libdl.a symbols have been moved to libc.a. __dlopen is available as a definition.

% nm --quiet -A /usr/lib/x86_64-linux-gnu/libc.a | grep __dlopen
/usr/lib/x86_64-linux-gnu/libc.a:vtables.o:                 w __dlopen
/usr/lib/x86_64-linux-gnu/libc.a:dlopen.o:00000000000000e0 T ___dlopen
/usr/lib/x86_64-linux-gnu/libc.a:dlopen.o:0000000000000080 T __dlopen
/usr/lib/x86_64-linux-gnu/libc.a:rtld_static_init.o:                 U __dlopen
7

Thank you, but I’m not so expert in the linking process, and I can’t understand this:

Shouldn’t Clang already be compiling against libc.a, and anyway, can I do something to make it find the functions resolve the error?

8

I think this is unlikely an issue of Clang or lld. Your libc doesn’t provide __dlopen.

% tar xf rep.tar
% cd rep
% cat home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libc.so   # change the absolute paths
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-littlearm)
GROUP ( home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/libc.so.6 home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libc_nonshared.a  AS_NEEDED ( home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/ld-linux-armhf.so.3 ) )
% sed -i '/--sysroot/d' response.txt
% arm-linux-gnueabi-ld @response.txt
...
arm-linux-gnueabi-ld: home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlopen.o): in function `dlopen':
(.text+0x8): undefined reference to `__dlopen'
arm-linux-gnueabi-ld: home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlclose.o): in function `dlclose':
(.text+0x0): undefined reference to `__dlclose'
arm-linux-gnueabi-ld: home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlsym.o): in function `dlsym':
(.text+0x8): undefined reference to `__dlsym'

GNU ld reports similar errors.

To have the old libraries at hand, I made a copy of the target root fs, and since GCC has hardcoded the library path I cannot really cross-compile against an old version of libc. Therefore I am using Clang + lld. The point is that when linking it complains about

Cross compilation with GCC is indeed more tricky… And its --sysroot= is not that useful in contrast to Clang. If you specify a custom specs file with gcc -specs= you can customize the linker command line. GCC specs: an introduction

gcc -dumpspecs

9

I see, thank you! So basically GCC uses the specs to find the correct symbols, and that configuration is not used by Clang, so it cannot find them?

But then my question is: why does it work on the native machine? Clang on the target system, when doing native compilation, seems to find the symbols just fine, and those are defined only in libc.a, which should then be searched for correctly? Is there something, when doing cross-compilation / using sysroot, that prevents Clang from searching them?

$ for lib in $(find /opt/box-root-fs -name \*.a) ; do echo $lib ; nm $lib 2> /dev/null | grep __dlopen | grep -v " U "  ; done

[...]

/opt/box-root-fs/usr/lib/arm-linux-gnueabihf/libc.a
         w __dlopen
00000081 T __dlopen

In other words, Clang should use the libc provided in the sysroot, that does provide __dlopen?

10

Some guessing:

  1. Clang doesn’t detect the correct libraries when cross-compiling because the driver isn’t smart enough. By this I mean, there’s probably a pattern that recognizes native libraries that do not extend to independent sysroot environments. If this is true, then this is a driver bug and should be fixed, but we need to have a reproducer that this is the case.
  2. GCC native versus cross builds are different. This would be a GCC issue, not necessarily a bug. It’s possible (though I’m just guessing) that their driver has similar issues as Clang, in that the code that discovers where libraries, headers and tools are is a mess. In this case, it’s possible (still guessing) that the build system has co-evolved with the driver and created a disparity that GCC “doesn’t see” because it’s hand-crafted.

The main difference above is the age-old Clang-native-cross-compiler versus GCC-combinatorial-build-cross-compiler, where the decisions are taken at run time for Clang and compile time for GCC.

Even if neither of these guesses are true, I hope this can at least enlighten the search for the actual problem.