Unable to link openal during compilation

[Alberto Salvia Novella]

I'm writing a recipe to package the game for Linux, and it to work system wide. The recipe compiles everything from source-code, as usually required by Linux distributions for security.

If I compile as it is, everything is fine. But if I also try to compile the third party libraries by myself, even when they compile successfully, the process fails when linking to openal:

Quote

/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALu.c.o):(.rodata+0x3bf60): multiple definition of `FuMa2N3DScale'; ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALc.c.o):(.rodata+0xf40): first defined here
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALu.c.o):(.rodata+0x3bfa0): multiple definition of `SN3D2N3DScale'; ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALc.c.o):(.rodata+0xf80): first defined here
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALu.c.o):(.rodata+0x3bfe0): multiple definition of `N3D2N3DScale'; ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALc.c.o):(.rodata+0xfc0): first defined here
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(bformatdec.c.o):(.rodata+0x2e0): multiple definition of `SN3D2N3DScale'; ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALc.c.o):(.rodata+0xf80): first defined here
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(bformatdec.c.o):(.rodata+0x320): multiple definition of `N3D2N3DScale'; ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALc.c.o):(.rodata+0xfc0): first defined here
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(bformatdec.c.o):(.rodata+0x2a0): multiple definition of `FuMa2N3DScale'; ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALc.c.o):(.rodata+0xf40): first defined here
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(panning.c.o):(.rodata+0x1360): multiple definition of `FuMa2N3DScale'; ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALc.c.o):(.rodata+0xf40): first defined here
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(panning.c.o):(.rodata+0x13e0): multiple definition of `N3D2N3DScale'; ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALc.c.o):(.rodata+0xfc0): first defined here
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(panning.c.o):(.rodata+0x13a0): multiple definition of `SN3D2N3DScale'; ThirdParty/cmake_find_package/../artefacts/openal/lib/lnx64_s_gcc10_rel_stdcpp/libopenal.a(ALc.c.o):(.rodata+0xf80): first defined here
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/TheDarkMod.dir/build.make:8570: thedarkmod.x64] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/TheDarkMod.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

There's also a second problem. "ThirdParty/cmake_find_package/tdm_find_package.cmake" expects gcc to be version 5, but my Linux distribution only offers gcc 10:

Quote

if (UNIX)
    if (CMAKE_SIZEOF_VOID_P EQUAL
        set(PACKAGE_PLATFORM "lnx64_s_gcc5_rel_stdcpp")
        set(PACKAGE_PLATFORM_DEBUG "lnx64_s_gcc5_rel_stdcpp")
    else ()
        set(PACKAGE_PLATFORM "lnx32_s_gcc5_rel_stdcpp")
        set(PACKAGE_PLATFORM_DEBUG "lnx32_s_gcc5_rel_stdcpp")
    endif ()

If I compile libraries by myself, that gcc dir is "gcc10". Although this is easy fixed by substituding the string with:

Quote

sed "s|_gcc[0-9]*_|_gcc${gccVersion}_|g" "ThirdParty/cmake_find_package/tdm_find_package.cmake"

 

Edited May 24, 2021 by Alberto Salvia Novella

[stgatilov]

Yes, it is rather strange how it works with prebuilt libs.
The major version of compiler is automatically included in the platform name when artefacts are built, but it is hardcoded when artefacts are used.

There are two options:

1. Use CMAKE_CXX_COMPILER_VERSION to deduce correct platform name when using artefacts.
2. Remove major version of compiler from platform name when building artefacts.

The problem with first approach is that it will force everyone without GCC 5 to build artefacts himself, which removes the idea behind storing artefacts at all.
The problem with second approach is that static libraries on VC are incompatible between versions. At least they were before VS2015: the newest three versions should be compatible.

It boils own to defining some tag of binary compatibility, and in current implementation it should be generated both in conan and cmake and be the same.

P.S. If you want to always build everything from sources, you can simply use platform override. It is described as "Unsupported platform" section in ThirdParty/readme.md. In this case you will get away completely from these weird lnx32_s_gcc5_rel_stdcpp names.

[cabalistic]

10 hours ago, Alberto Salvia Novella said:

If I compile libraries by myself, that gcc dir is "gcc10". Although this is easy fixed by substituding the string with:

There's an override variable specifically for this purpose: THIRDPARTY_PLATFORM_OVERRIDE. You can set it when running CMake, and it will override the defaults which are meant to fit the prepackaged binaries.

[stgatilov]

30 minutes ago, cabalistic said:

There's an override variable specifically for this purpose: THIRDPARTY_PLATFORM_OVERRIDE. You can set it when running CMake, and it will override the defaults which are meant to fit the prepackaged binaries.

Yes, but it is rather unintentional that people with Linux and GCC cannot use our prebuilt artefacts if they have version different from 5.
They should be perfectly usable.

Do you have any idea how to resolve this?

[stgatilov]

Ok, people can use prebuilt artefacts, they just can't build and replace them if they have different GCC.

I like two possibilities:

1. Remove compiler version from platform name.
2. Leave everything as is and say that THIRDPARTY_PLATFORM_OVERRIDE should be used in this case.

[Alberto Salvia Novella]

Perhaps the answer is that libraries are rebuilt automatically when building the game, and there's no such thing as a sub-folder for platform.

[Alberto Salvia Novella]

Using a custom platform name doesn't prevent the linker to fail.

The building process I'm following is as explained in the instructions. There should be something missing, either in the instructions or in the source code. Have you tested that (still) builds for you?

The code I'm using is:

buildLibs () {
    so "-buildLibs: removeArtefacts" rm --recursive "artefacts"
    yes "yes" | so "-buildLibs: exportCustomRecipes" python "1_export_custom.py"
    so "-buildLibs: compile" conan install . --build --options platform_name="linux"
}

buildEngine () {
    if [[ ! -d "${buildEngine}" ]]; then
        cp --recursive "${buildWorkspace}" "${buildEngine}..."
        cd "${buildEngine}.../darkmod_src"
        
        so "-buildEngine: cmake" cmake \
            -DCMAKE_BUILD_TYPE="Release" \
            -DTHIRDPARTY_PLATFORM_OVERRIDE="linux" \
            -DCMAKE_EXE_LINKER_FLAGS="-no-pie"
        so "-buildEngine: makeMax" makeMax
        
        mv "${buildEngine}..." "${buildEngine}"
    fi
}

Maybe it's related with that "DCMAKE_EXE_LINKER_FLAGS". Without it I got a different error:

/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libavcodec.a(me_cmp.o): relocation R_X86_64_32 against symbol `ff_pb_1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libavcodec.a(sbrdsp.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libavcodec.a(xvididct.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libavcodec.a(aacpsdsp.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libavutil.a(float_dsp.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libswresample.a(audio_convert.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libswresample.a(rematrix.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libswresample.a(resample.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libswscale.a(hscale_fast_bilinear_simd.o): relocation R_X86_64_32S against `.text.unlikely' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libswscale.a(swscale.o): relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libswscale.a(input.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libswscale.a(output.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libswscale.a(rgb2rgb.o): relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libswscale.a(rgb_2_rgb.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libswscale.a(scale.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/linux/libswscale.a(yuv2rgb.o): relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/TheDarkMod.dir/build.make:8570: thedarkmod.custom] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/TheDarkMod.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

 

[stgatilov]

Yes, no-pie is added only for 64-bit Linux for some reason...

@cabalistic, I guess we can enable it for everywhere?
For instance, do check like this:

check_cxx_compiler_flag(-no-pie CXX_NO_PIE_FLAG)
if (CXX_NO_PIE_FLAG)
    set_target_properties(TheDarkMod PROPERTIES POSITION_INDEPENDENT_CODE OFF)
    set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -no-pie")
endif()

By the way, it is weird to see two commands doing the same
 

[Alberto Salvia Novella]

I have found the culprit: same as in this bug report.

[Alberto Salvia Novella]

How to fix it, in source code:

https://wiki.gentoo.org/wiki/Gcc_10_porting_notes/fno_common

[Alberto Salvia Novella]

I opened a bug report about this.

[stgatilov]

If you go to ThirdParty\custom\openal\CMakeLists.txt and add a line before add_subdirectory:

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fcommon")

Then run 1_export_custom.py, 2_build_all.py, and TDM build again.

Will the problem go away then?

[Alberto Salvia Novella]

It does.

Is it possible to pass that flag to Conan as a command?

[stgatilov]

2 hours ago, Alberto Salvia Novella said:

Is it possible to pass that flag to Conan as a command?

I don't know.

Conan has some "options", but I don't think you can pass arbitrary flags via it.

[stgatilov]

On 5/24/2021 at 11:50 PM, stgatilov said:

Ok, people can use prebuilt artefacts, they just can't build and replace them if they have different GCC.

I like two possibilities:

1. Remove compiler version from platform name.
2. Leave everything as is and say that THIRDPARTY_PLATFORM_OVERRIDE should be used in this case.

I have chosen approach 1.

Starting from svn rev 9396, version of compiler is not included in the platform name for third-party artefacts.
It is just "win64_s_vc_rel_mt" and "lnx64_s_gcc_rel_stdcpp" now.

[Alberto Salvia Novella]

Thank you

[nbohr1more]

Rev 9398 now "I" cannot compile:

/usr/bin/ld: ../ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/lnx64_s_gcc_rel_stdcpp/libswscale.a(gamma.o): relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: ../ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/lnx64_s_gcc_rel_stdcpp/libswscale.a(hscale.o): relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
CMakeFiles/TheDarkMod.dir/build.make:8094: recipe for target 'thedarkmod.x64' failed
make[2]: *** [thedarkmod.x64] Error 1
CMakeFiles/Makefile2:94: recipe for target 'CMakeFiles/TheDarkMod.dir/all' failed
make[1]: *** [CMakeFiles/TheDarkMod.dir/all] Error 2
Makefile:102: recipe for target 'all' failed
make: *** [all] Error 2

So all platforms now need to run the python fix before compiling now?

[Alberto Salvia Novella]

That's a different error.

[nbohr1more]

1 hour ago, Alberto Salvia Novella said:

That's a different error.

Thank you! I am back to compiling

[Alberto Salvia Novella]

You are welcome

[Alberto Salvia Novella]

That said the creation is finished.

[stgatilov]

7 hours ago, nbohr1more said:

Rev 9398 now "I" cannot compile:

/usr/bin/ld: ../ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/lnx64_s_gcc_rel_stdcpp/libswscale.a(gamma.o): relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: ../ThirdParty/cmake_find_package/../artefacts/ffmpeg/lib/lnx64_s_gcc_rel_stdcpp/libswscale.a(hscale.o): relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
CMakeFiles/TheDarkMod.dir/build.make:8094: recipe for target 'thedarkmod.x64' failed
make[2]: *** [thedarkmod.x64] Error 1
CMakeFiles/Makefile2:94: recipe for target 'CMakeFiles/TheDarkMod.dir/all' failed
make[1]: *** [CMakeFiles/TheDarkMod.dir/all] Error 2
Makefile:102: recipe for target 'all' failed
make: *** [all] Error 2

So all platforms now need to run the python fix before compiling now?

Hey!

Do you have CMake newer than 3.14?
It should pass -no-pie automatically to linker.

[stgatilov]

Try to change this line OFF to FALSE in the line with POSITION_INDEPENDENT_CODE

Try to add line after minimum required version: cmake_policy(SET CMP0083 NEW)

Post your GCC version.

[stgatilov]

1 hour ago, Alberto Salvia Novella said:

That said the creation is finished.

Does it support updating existing installation?

If people will uninstall + install the game every year, they will generate unnecessary traffic spikes.

UPDATE: By the way, I don't think system-wide installation is a good idea.
TDM installation is designed as user-only: it does not need admin rights, and is installed to one directory. Trying to turn this into system-wide installation is fragile and pointless.

[nbohr1more]

3 minutes ago, stgatilov said:

Try to change this line OFF to FALSE in the line with POSITION_INDEPENDENT_CODE

Try to add line after minimum required version: cmake_policy(SET CMP0083 NEW)

Post your GCC version.

gcc --version

gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

cmake --version

cmake version 3.19.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).