Latest Posts

Topic: Memory leak?

tothxa
Avatar
Joined: 2021-03-24, 12:44
Posts: 84
Ranking
Likes to be here
Posted at: 2021-04-19, 21:08

When I play for a long time, Widelands slowly eats up all available memory. Even when I leave it on pause while doing something else. However, when I quit, AddressSanitizer only reports a 152 byte leak, the same as when I just start the program and quit immediately.

Is this a known issue? If not, what should I do to debug it?

(I think it may be related to the handling of sound files: When I leave it on pause, stdout is full of info messages about loading song files, with some ALSA underrun messages at times. So I once tried it with music turned off, and -- this is very subjective -- I think the memory growth was much slower.)


Top Quote
Nordfriese
Avatar
Joined: 2017-01-17, 18:07
Posts: 1433
Ranking
One Elder of Players
Location: 0x55555d3a34c0
Posted at: 2021-04-19, 22:15

That's out of my comfort zone, I always have sound disabled… face-wink.png

I think that such a problem is not known though. Could you attach a logfile with those error message? Also, does it happen reproduceably or only sometimes? How much memory does Widelands approximately consume extra after a certain time?


Top Quote
kaputtnik
Avatar
Joined: 2013-02-18, 20:48
Posts: 2083
Ranking
One Elder of Players
Location: Germany
Posted at: 2021-04-20, 10:11

We had some of this ALSA issues in the past

They were closed as wontfix because it is an issue of ALSA. Anyway i do not have such ALSA underruns anymore... i remember doing some adjustments in my ALSA/PulseAudio configs, but i can't remember exactly what i did. I guess i have adjusted this settings in "~/.config/pulse/deamon.conf":

; default-fragments = 5
; default-fragment-size-msec = 2

Ref: https://wiki.archlinux.org/index.php/PulseAudio#daemon.conf

But i think this is not the issue. widelands have to maintain much things. On games with many players i think all the roads, carriers, worker programs needs some memory which just increases with every conquered terrain.

@ tothxa: Is this a release or development build?


Top Quote
tothxa
Avatar
Joined: 2021-03-24, 12:44
Posts: 84
Ranking
Likes to be here
Posted at: 2021-04-20, 22:54

It happens all the time, just by leaving the game alone in the main menu as well as if I start a game and leave it paused.

I did some data gathering. It seems it's not the sound. The rate of the memory creep is the same with the music disabled. Here's the graph of RSS over time with the game just started and left in the main menu:

diagram1

The thick line shows a session when I did other stuff on the computer (during the gap in the diagram), while Widelands was sitting in the main menu.

My current test version is a "release" build of af1afc6d80af17, but it happened with my earlier debug build as well.

Edited: 2021-04-20, 23:15

Attachment: wl_mem_menu.png (39.4 KB)

Top Quote
tothxa
Avatar
Joined: 2021-03-24, 12:44
Posts: 84
Ranking
Likes to be here
Posted at: 2021-04-20, 23:00

And here's the diagram when a (small) game is loaded, paused immediately, and left alone:

diagram2

The steady state rate is around 0.63MiB/min in both cases.

Edited: 2021-04-20, 23:09

Attachment: wl_mem_game_paused.png (36.0 KB)

Top Quote
Nordfriese
Avatar
Joined: 2017-01-17, 18:07
Posts: 1433
Ranking
One Elder of Players
Location: 0x55555d3a34c0
Posted at: 2021-04-20, 23:36

Inexplicable… I started a Widelands instance of latest master (release) now and let it sit around in the main menu without doing anything for 15 minutes; the memory usage goes to 102.9 MiB within the first minute and from then on stays exactly constant. Perhaps it's a bug in some graphics driver or other 3rd party library? Some advanced profiling would be needed to see where the memory goes, I wouldn't know how to do that though…


Top Quote
tothxa
Avatar
Joined: 2021-03-24, 12:44
Posts: 84
Ranking
Likes to be here
Posted at: 2021-04-21, 11:28

I was able to resolve the problem of constant memory creep, and I can vaguely see some pattern for an explanation:

Your 102.9MiB vs. my ~600MiB made me think. I already wondered why I get ASAN reports after each run when I disabled it in my compile.sh. My first build with this setup was a debug build (as set on github), and I didn't run make clean. (didn't know how to at the time, and assumed that config changes would propagate anyway)

  • First I tried a make clean in the build directory and a rebuild with compile.sh. It didn't change anything. (executable size stayed ~22MB)
  • Then I checked with my flatpak build. Its memory usage is normal, no creep. (executabe size is 10MB)
  • Finally I removed the entire build directory, and rebuilt. Memory usage normal, no creep. (executable size is 9.6MB)

So the problem was cleaning up for the build, and it seems I still only know the most heavy-handed way to do it. face-wink.png Is this how it is supposed to work?

There remains one problem though (maybe worth a new thread?): When I quit a game, and go back to the main menu, memory usage does not go down, and if I repeat loading a game (same one) and quitting, then memory usage still keeps growing with every turn.


Top Quote
Nordfriese
Avatar
Joined: 2017-01-17, 18:07
Posts: 1433
Ranking
One Elder of Players
Location: 0x55555d3a34c0
Posted at: 2021-04-21, 11:48

Good that it works now face-smile.png

ASan is a bit aggressive. The only way to get rid of it when recompiling is to clean the entire CMake build cache. make clean is often not enough for this. There's probably some CMake command for this, but deleting the build directory is the easiest way.

The memory usage is expected to remain higher than before after playing, because graphics are loaded dynamically when first needed and then remain cached in memory until closing Widelands. But the memory for the map objects etc is released when returning to the main menu. I can confirm though that repeated loading and quitting of the same savegame results in some unexpected accumulation of memory – not critically much, but I'll check anyway what may be the cause of this…


Top Quote
tothxa
Avatar
Joined: 2021-03-24, 12:44
Posts: 84
Ranking
Likes to be here
Posted at: 2021-04-21, 12:33

I think the need for removing or changing (as per https://www.widelands.org/wiki/Building%20Widelands/#building-with-cmake-manually-advanced) the build directory when compile options are changed should be documented. As far as I can tell, https://www.widelands.org/wiki/Building%20Widelands/ is the right place. I think it would be best added as subsections under both #building-widelands-with-compile-script-fastest-way and #building-with-cmake-manually-advanced, with the appropriate level of explanation for each.

I can do it, but I'll wait for comments.


Top Quote
Nordfriese
Avatar
Joined: 2017-01-17, 18:07
Posts: 1433
Ranking
One Elder of Players
Location: 0x55555d3a34c0
Posted at: 2021-04-21, 17:06

There's a brief mention of this in the README but +1 for adding it to the wiki page as well face-smile.png

Regarding the memory overhead, gdb shows that in my testcase nearly all of the extra memory goes to the graphics cache as expected, with an additional ~ 10 MiB for each load-and-quit cycle that should not happen. I'm curious to find out where it goes.


Top Quote