Latest Posts

Topic: Linking time

jmoerschbach
Avatar
Topic Opener
Joined: 2019-09-26, 21:40
Posts: 24
Ranking
Pry about Widelands
Posted at: 2020-05-05, 11:11

Hi, this is a somewhat technical question regarding the toolchain...

On my machine (Windows 10, MSys MinGW 64, i5-9600@3.7GHz, 6 Cores, 16GB RAM) it takes 6-7min to link the final widelands executable. So developing gets really frustrating very fast because when I change only one line of code, I wait ~6min before I can see any effects.

Question: Do you encounter similar times when linking or am I doing something terrible wrong (apart from being on windows instead of linux, of course... face-wink.png ) Is there any way we may improve this? I searched the internet but did not come up with any straight-forward solutions...

I use the compile.sh like

compile.sh -w -t -s -a

Output is then:

###########################################################
#     Script to simplify the compilation of Widelands     #
###########################################################

  Because of the many different systems Widelands
  might be compiled on, we unfortunally can not
  provide a simple way to prepare your system for
  compilation. To ensure your system is ready, best
  check https://www.widelands.org/wiki/BuildingWidelands

  You will often find helpful hands at our
  * IRC Chat: https://www.widelands.org/webchat/
  * Forums: https://www.widelands.org/forum/
  * Mailinglist: https://www.widelands.org/wiki/MailLists/

  Please post your bug reports and feature requests at:
  https://github.com/widelands/widelands/issues

###########################################################

Using 5 core(s).

Any website-related code will be OMITTED in the build.
Make sure that you have created and tested a full
build before submitting code to the repository!

Translations will not be built.

Tests will not be built.

###########################################################

Creating a Debug build. Use -r to create a Release build.

Will build without AddressSanitizer.

Basic XDG Base Directory Specification will be used on Linux
if no existing $HOME/.widelands folder is found.
The widelands user data can be found in $XDG_DATA_HOME/widelands
and defaults to $HOME/.local/share/widelands.
The widelands user configuration can be found in $XDG_CONFIG_HOME/widelands
and defaults to $HOME/.config/widelands.
See https://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
for more information.


###########################################################

Call 'compile.sh -h' or 'compile.sh --help' for help.

For instructions on how to adjust options and build with
CMake, please take a look at
https://www.widelands.org/wiki/BuildingWidelands/.

###########################################################

-- SDL2_INCLUDE_DIR is C:/msys64/mingw64/include/SDL2
-- SDL2MAIN_LIBRARY is C:/msys64/mingw64/lib/libSDL2main.a
-- Not using AddressSanitizer.
-- Detected mingw32-w64
-- Version of Widelands Build is r69[b7fbd40@fix_multiplayer_ui_many_players](Debug)
-- Configured windows resource file
-- Configuring done
-- Generating done
-- Build files have been written to: C:/msys64/home/Jonas/widelands2/widelands-clion/build

I tried providing --clang and --gcc but did not see any major differences. Compiling the sources to object files is no problem at all, it is the linking. the ld process is only using one core which can clearly be seen in the task manager. But parallel linking is not supported/possible I guess.

Can we do anything about it?

Thanks

Jonas


Top Quote
Nordfriese
Avatar
Joined: 2017-01-17, 18:07
Posts: 1929
OS: Debian Testing
Version: Latest master
Ranking
One Elder of Players
Location: 0x55555d3a34c0
Posted at: 2020-05-05, 11:24

This is mainly a hardware problem I think. On my old machine linking initially took one hour per executable. (And that was before -s and -j were implemented…). Then I upgraded the machine with more RAM and linking time decreased to around 10 minutes per executable. On my new dev machine, linking takes only 1-2 minutes.

Does your machine have enough memory? If it uses Swap space, linking time shoots up. Other than that, I can only advise you to get a faster CPU…

EDIT: Ah I see, 16 GB and not that bad CPU… perhaps MinGW is generally slow? Using linux really would be an improvement face-wink.png

Edited: 2020-05-05, 11:26

Top Quote
hessenfarmer
Avatar
Joined: 2014-12-11, 23:16
Posts: 2646
Ranking
One Elder of Players
Location: Bavaria
Posted at: 2020-05-05, 12:20

Hi,
I don't think it is an issue of MinGW it is rather due to linking a debug build which takes much longer then a release build due to some debug interfaces to be considered and our circular dependencies. This is the same on linux an MacOS as can be seen on travis as well (debug 35 min release 18 min). so 6 minutes for a debug doesn't sound that bad in comparison to how long it takes on appveyor.
If you use the compile.sh on MinGW be aware that this script spares one core from compiling and linking (as linking modules is done in parallel to compiling) so you could speed up the whole thing by using cmake and make or ninja a bit.


Top Quote
jmoerschbach
Avatar
Topic Opener
Joined: 2019-09-26, 21:40
Posts: 24
Ranking
Pry about Widelands
Posted at: 2020-05-05, 13:49

wow, hessenfarmer's suggestion switchting from debug to release build reduces the link time to ~1 second (yes, from 6 minutes to few seconds Oo) Could have tried that before... That's what I'm talking about face-grin.png

Of course, this way I cannot debug but I think I will still be much more productive just because I have so many more tries/time slot^^

Would like to know if the difference between debug vs release linking time is on your machine as significant as on my machine... face-wink.png

Also, does anybody have more in-depth knowledge why the difference is so big? Some valid explanations/hypothesis? face-smile.png


Top Quote
hessenfarmer
Avatar
Joined: 2014-12-11, 23:16
Posts: 2646
Ranking
One Elder of Players
Location: Bavaria
Posted at: 2020-05-05, 13:53

jmoerschbach wrote:

wow, hessenfarmer's suggestion switchting from debug to release build reduces the link time to ~1 second (yes, from 6 minutes to few seconds Oo) Could have tried that before... That's what I'm talking about face-grin.png

Of course, this way I cannot debug but I think I will still be much more productive just because I have so many more tries/time slot^^

Would like to know if the difference between debug vs release linking time is on your machine as significant as on my machine... face-wink.png

It is, as I have a comparable setup. Normally for testing if it works I do release builds. For chasing errors I insert debug logs in the code providing me information which are printed in release as well. Only to test some lua issues i use a debug as we have the lua debug console in it.

Also, does anybody have more in-depth knowledge why the difference is so big? Some valid explanations/hypothesis? face-smile.png

I think TinoM and GunChleoc might know in depth why this is the case.

Edited: 2020-05-05, 13:53

Top Quote
Nordfriese
Avatar
Joined: 2017-01-17, 18:07
Posts: 1929
OS: Debian Testing
Version: Latest master
Ranking
One Elder of Players
Location: 0x55555d3a34c0
Posted at: 2020-05-05, 14:05

Would like to know if the difference between debug vs release linking time is on your machine as significant as on my machine... face-wink.png

Wow face-grin.png On my machine the difference is like 1 min (release) to 2 mins (debug)…


Top Quote
hessenfarmer
Avatar
Joined: 2014-12-11, 23:16
Posts: 2646
Ranking
One Elder of Players
Location: Bavaria
Posted at: 2020-05-05, 14:39

Nordfriese wrote:

Would like to know if the difference between debug vs release linking time is on your machine as significant as on my machine... face-wink.png

Wow face-grin.png On my machine the difference is like 1 min (release) to 2 mins (debug)…

That is corresponding to the values in Travis. to some extent and 32 bit appveyor as well. (x64 in Appveyor is artificially long due to not having much Ram there and therefore not linking in parallel at all)


Top Quote
Tibor

Joined: 2009-03-23, 23:24
Posts: 1377
Ranking
One Elder of Players
Location: Slovakia
Posted at: 2020-05-05, 14:40

Just watch your swap space usage before, during and after linking. I have 8 GB RAM and prefer to compile in 2 or even 1 thread to avoid RAM issues during linking...


Top Quote
hessenfarmer
Avatar
Joined: 2014-12-11, 23:16
Posts: 2646
Ranking
One Elder of Players
Location: Bavaria
Posted at: 2020-05-05, 14:45

Tibor wrote:

Just watch your swap space usage before, during and after linking. I have 8 GB RAM and prefer to compile in 2 or even 1 thread to avoid RAM issues during linking...

Yes nice graphs to watch in Windows Task Manager. face-wink.png I have 16 GB and it is possible to use all cores for compiling as there aren't that many parallelm linking jobs but in the end it really needs nearly all of my 16GB for linking all the exe files.


Top Quote