Latest Posts

Topic: widelands-media better findable/useable

knarf
Avatar
Topic Opener
Joined: 2024-01-03, 20:20
Posts: 45
OS: Debian
Ranking
Pry about Widelands
Posted at: 2024-01-03, 20:33

Hi,

I am not very familiar with widelands and its development and as such, I have the viewpoint of a typical potential future contributor. However, I just had quite a hard time figuring out where to find the blender files. Everything seems to be on git nowadays and the mirror-bot is cool, but after quite some searching, I found a short note on a widelands git help page that everything "but widelands-media" is now on git. That lead me again after some digging to some old bazaar repos and finally to what I wanted:

https://launchpad.net/widelands-media

I hope this is correct, is it?

Now my real question and a few remarks:

  • Could you not move the files there to git, too? The way things are currently, potential contributors might know git already, but most won't know bazaar, and some would have to get to know two version control systems for working on the media.
  • Please make the location of things a little clearer. There are quite some places that speak of the 'widelands-media' repo, but do not link to it and do not mention where that lives in the first place. Most people would assume right beside the widelands-repo, but will not find it that way.

Please do not take these points as complaining. I just wanted to put some of my first impressions while looking "behind the scenes" into writing in the hope it helps to improve things that might naturally not be visible to long-term contributors.

Edited: 2024-01-03, 20:41

Top Quote
hessenfarmer
Avatar
Joined: 2014-12-11, 22:16
Posts: 2684
Ranking
One Elder of Players
Location: Bavaria
Posted at: 2024-01-03, 23:31

+1 for having everything ccesible in one (maybe mirrored) place.


Top Quote
kaputtnik
Avatar
Joined: 2013-02-18, 19:48
Posts: 2480
OS: Archlinux
Version: current master
Ranking
One Elder of Players
Location: Germany
Posted at: 2024-01-04, 08:37

The reason to keep the media-repo on launchpad was just the size of the repo (~9GB). As far i remember github didn't allow to store such a big repo for free. If i understand this correctly github allows only 500MB for free.

Regarding finding where all the media is found: It's in our wiki, Graphics development. This page can be found by e.g. clicking on "Development" (on the above List) -> "Contribute" -> Chapter "Graphics".

Edit: Just saw that you modified the wiki yesterday and added the links. Very nice, thank you!

Edited: 2024-01-04, 09:17

Fight simulator for Widelands:
https://wide-fighter.netlify.app/

Top Quote
knarf
Avatar
Topic Opener
Joined: 2024-01-03, 20:20
Posts: 45
OS: Debian
Ranking
Pry about Widelands
Posted at: 2024-01-04, 10:28

Would it be an option to have the media repo only on codeberg? They state on https://docs.codeberg.org/getting-started/faq/:

We do not have a fixed limit, as long as your use case is not impacting the quality of our services for other users and projects.

9 GB is large, but it should still not impact others and for media that size is not unreasonable. I do work with repos that size regularly, although not on codeberg.


Top Quote
tothxa
Avatar
Joined: 2021-03-24, 11:44
Posts: 452
OS: antix / Debian
Version: some new PR I'm testing
Ranking
Tribe Member
Posted at: 2024-01-04, 11:07

The main problem with the media repo is that it's so big that it's practically impossible to check out. So first of all it should be split into manageable segments. I don't know the sizes of music and sound, but I know that graphics must be further split up for each tribe and a general section.

The second, still quite big problem is that media files are not well fit for version management, but we shouldn't throw away history either. At least the blender models went through several sweeping automatic or semi-automatic updates without actually checking whether they still render properly, so they do need the history, at least until they are all tested and either repaired or reverted to the last working version.

My re-export efforts help with the last point, but my repo grew bigger than desirable too (for different reasons, mine has too many intermediate files). And I really need to convert it to some automated build system.


Top Quote
knarf
Avatar
Topic Opener
Joined: 2024-01-03, 20:20
Posts: 45
OS: Debian
Ranking
Pry about Widelands
Posted at: 2024-01-04, 12:17

My experience is that large binary files in git are ok as long as they do not change often. 9GB do take a while to checkout, but in the end that is the size of the data and whatever you do, you will need to download those data in all cases. Splitting into graphics and sound might make sense, as rarely you will need both at the same time, and even if you do, cloning two repos instead of one isn't that much more work.

Also, maybe git-lfs could be an option?


Top Quote
kaputtnik
Avatar
Joined: 2013-02-18, 19:48
Posts: 2480
OS: Archlinux
Version: current master
Ranking
One Elder of Players
Location: Germany
Posted at: 2024-01-04, 12:46

I have the media-repo here, unfortunately updated in year 2019 face-shock.png

Anyway, checking the size of directories shows:

$:> du -h --max-depth 1 | sort -rh
5,2G    ./graphics
24M     ./sounds
7,6M    ./.bzr
1,7M    ./music

So the biggest part is the graphics directory (5,2GB) and sounds needs just 24MB. Digging in the graphics directory tree:

$:> du -h --max-depth 1 graphics/ | sort -rh
4,7G    graphics/tribes
290M    graphics/worlds
81M     graphics/website
57M     graphics/user_interface
49M     graphics/stock_sources
40M     graphics/campaigns
22M     graphics/Misc
7,4M    graphics/tools
68K     graphics/immovables
20K     graphics/wiki

graphics/tribes is the biggest directory, and there:

$:> du -h --max-depth 1 graphics/tribes/ | sort -rh
4,7G    graphics/tribes/
3,1G    graphics/tribes/frisians
656M    graphics/tribes/barbarians
555M    graphics/tribes/atlanteans
381M    graphics/tribes/empire
32K     graphics/tribes/worker level indicator icons

graphics/frisians is the biggest directory.

Not sure if this repo can be split into smaller parts.


Fight simulator for Widelands:
https://wide-fighter.netlify.app/

Top Quote
Nordfriese
Avatar
Joined: 2017-01-17, 17:07
Posts: 1985
OS: Debian Testing
Version: Latest master
Ranking
One Elder of Players
Location: 0x55555d3a34c0
Posted at: 2024-01-04, 17:09

For completeness here are the most stats of the most recent revision r285 from 2023-08-20:

14G graphics/tribes/
8,0G    graphics/tribes/amazons
3,7G    graphics/tribes/frisians
672M    graphics/tribes/barbarians
579M    graphics/tribes/atlanteans
382M    graphics/tribes/empire
32K graphics/tribes/worker level indicator icons

Maybe a lot of space could be gained by removing the prerendered hi-res PNGs for Fri and Amz from the repo.

git-lfs is evil, since it is built around storing the actual content on some remote server outside the version control system itself.

Splitting off the sound and music folders into a separate repo on Codeberg/GH would be easily doable – when and if someone actually wants to work with this part of the repo, which was last modified in 2011…

Edited: 2024-01-04, 17:09

Top Quote
knarf
Avatar
Topic Opener
Joined: 2024-01-03, 20:20
Posts: 45
OS: Debian
Ranking
Pry about Widelands
Posted at: 2024-01-04, 20:26

If no development happened on sounds/music for that long, the number by @kaputtnik should still be accurate and then we talk about 26 MB compared to 14 GB. I don't think it makes sense to split this off at this point.

Edited: 2024-01-04, 20:27

Top Quote
blind3rdeye
Avatar
Joined: 2020-03-26, 07:47
Posts: 74
Ranking
Likes to be here
Posted at: 2024-01-04, 22:35

Nordfriese wrote:

Maybe a lot of space could be gained by removing the prerendered hi-res PNGs for Fri and Amz from the repo.

I don't know much about the graphics stuff, but I'd say this is a good idea. In my view, files that are generated from some other source in the repo probably shouldn't be included in the repo themselves (unless there is a special reason for it). So if the PNGs can be recreated from the other files, then we probably don't need to keep all their historical version.


Top Quote