View unanswered posts | View active topics It is currently 20 Sep 2019, o 07:46



Reply to topic  [ 15 posts ] 
 How to pack art assets for faster boot times 
Author Message
RI Team
User avatar

Joined: 2 May 2009, o 15:44
Posts: 841
Location: Olympia, WA, USA
Post How to pack art assets for faster boot times
If you have followed the instructions for the Subversion repository and are setup for beta testing, the first thing you'll probably notice is that the mod takes a really long time to boot. This post describes what causes the issue and how, with some effort up front, you can pack your own art assets and greatly reduce your load times.


Preamble
Before continuing, please make sure that you are using Subversion 1.7 (Tortoise SVN 1.7) or later. If you have an older version, you should update to a current version first and then try to run the game. It's possible that the speed improvement from this alone will be satisfactory for you. Note that you'll have to upgrade the working copy if you update from an old version of the Tortoise SVN software.


The Problem
When Civilization starts up a mod, it scans over every file in the mod directory. It probably does this to compute a checksum of the mod's files so that it can detect file differences between players in multiplayer games, which helps to prevent cheating. However, our mod has a lot of files so this scan can take quite a while. The Art directory alone contains ~28,000 files and ~3,100 folders. Non-art assets account for ~4,000 files and ~265 folders. The game appears to hang while this scan is occurring, and it can be very confusing for new beta testers.

The problem is further compounded by the fact that our mod is backed by a Subversion repository: Subversion creates a large number of hidden metadata files in a .svn directory. These files must be scanned in addition to the actual mod files. This problem used to be much worse, but a recent redesign of Subversion greatly reduced the number of files in its metadata database. Even so, when backed by the Subversion repository a complete checkout contains ~60,000 files and ~3,700 folders at the time of this writing.


A Solution
One solution is to pack most of the art assets into a single Firaxis Pack (.FPK) file. You can think of Firaxis Pack files as specialized .zip files. The advantage to packing the art assets is that the game only has to scan ~8,000 files at load time instead of ~60,000.

Firaxis always packed their art assets for the official game and we always pack our art assets when we do major releases. However, we do not provide packed files while we are in development/beta testing. This is because packing takes time and can get in the way of our development work. You can, however, pack your own art assets if you are willing to take the time to do so.

The advantage of packing your art assets is that the game will boot much, much faster. The disadvantage is that you will have to re-pack your art assets every time you do an SVN Update that pulls down updated art files. The packing process itself takes 5-20 minutes depending on your system's performance.

Packing is a bit complicated and not for the faint of heart, but if you are really bothered by the startup time and are up to the challenge read on.

The Goal
What we want to do here is check out the art assets from the Subversion repository to some location that the game won't scan, pack them into a single .FPK file, and then copy that .FPK back to the mod's assets folder.

At the same time, we want to REMOVE those art assets from the mod's directory, because they will now be stored in the .FPK and keeping those files around means the game will continue to scan them.

Two Checkouts
When we're done we'll end up with two different copies of the Subversion repository checked out:
  • The mod in "Beyond the Sword\Mods\Realism", but now with the art assets removed
  • A new checkout with just the art assets, stored somewhere else on the hard drive where the game won't scan them

Take a look at the RI Subversion Repository thread for setting up the mod checkout.

In the future, updating to newer revisions will involve an "SVN Update" of BOTH checkouts and then re-packing the art assets.

An Introduction to Sparse Checkouts
One of the new features introduced in Subversion 1.7 is the ability to do a "sparse" checkout. This means that instead of checking out an entire directory, you can pick and choose which items you want to include in your checkout. This is hugely useful to us. We'll use this feature to exclude the art assets from the mod folder.


A Note About Movies
The movie files are already compressed and won't work if they are packed along with the other assets in the art directory, so we'll have to be careful to leave the movies in the mod directory and not to pack them. The sparse checkout will once again be very useful in accomplishing this, and the instructions below will help you navigate this technicality.


Setup Instructions
Here's an overview of the steps that we're going to take to accomplish our goal:
  1. Create a place somewhere on the hard drive to hold the unpacked files
  2. Perform a sparse checkout of the art assets (except for movies) to that new place
  3. Perform a sparse checkout of the mod (this removes the art assets except for movies which are unpackable)
  4. Get the packing utility
  5. Get the current revision number
  6. Pack the art assets
  7. Copy the newly created .FPK file to the Realism Assets folder
  8. (Optional) Defragment the hard drive
  9. Boot the game

Step 1: Create a place somewhere on the hard drive to hold the unpacked files
To setup and maintain the packed art assets we'll need a working space that is completely separate from the Civilization IV directory. This can be any place on the hard drive or even on a different hard drive. It won't be used at run-time but is merely a place to do our packing work.

I typically set it up with a folder structure like so:

  • Realism Asset Packing
    • Source
      • Art
    • Versions

The Source directory will hold a checkout of the art files to be packed in the Art folder. The versions folder will hold the outputted .FPK versions. I usually keep the last couple .FPK versions that I've output in case I need to roll back for testing purposes. This will probably be unnecessary for anyone but developers. You can omit the Versions folder if you don't feel like hanging on to old versions.

Step 2: Perform a sparse checkout of the art assets (except for movies) to that new place
We're going to get a complete copy of the mod's art assets, except for the movies. Later we'll end up packing everything that we checkout in this step.

Right click on the newly created Art folder and select "SVN Checkout". A checkout window will pop up. These are the appropriate settings:

URL of repository
Code:
https://svn.code.sf.net/p/civ4mods/code/realism/bts/trunk/mod/Assets/Art

Checkout directory
(The folder you just created is automatically selected)

Checkout Depth
Custom depths (this will be set automatically after selecting Choose Items...)

Choose items...
Click on the "Choose items..." button and a repository browser window will open up. It should list the art directory at the top of the tree and all of the Art directories below it.

Uncheck the Movies directory, but leave everything else checked.

Click OK to close the repository browser window.

NOTE: If you go back to choosing items again you will have to uncheck the Movies directory each time.

Omit externals
No (not checked)

Revision
HEAD revision

Click OK to begin the checkout process. The operation can take a long time. Have some tea :)

You should end up with a "Realism Asset Packing\Source\Art" folder when you're done. That folder should contain all of the Realism: Invictus art directories except for the Movies.

Step 3: Perform a sparse checkout of the mod (this removes the art assets except for movies which are unpackable)
Even if we pack all of the art assets and put the newly packed file in the appropriate place, BtS will still scan all of the unpacked art files if we leave them anywhere in the mod's file tree. To improve the our load times we must remove all of the Art subfolders (except for Movies).

Right click on your "ROOT CIV FOLDER\Beyond the Sword\Mods\Realism" folder and navigate to "TortoiseSVN -> Update to revision..." in the TortoiseSVN submenu.

Under the "Update Depth" section, select the "Choose items..." button.

A repository browser dialog will appear.

Expand the Assets folder and then the Art folder. Be careful not to uncheck anything accidentally.

Individually uncheck every item in the Art folder except for Movies.

Click OK to close the repository browser dialog. The Update Depth should now be listed as "Custom depths".

Make sure that the "Make depth sticky" option is checked.

Click OK to execute the command. At this point all of the art files except for the movies should be removed from the mod's directory.

Step 4: Get the packing utility
Packing is done with a special utility that Firaxis released called PakBuild. You can download it from our repository. Save it in your Realism Asset Packing folder.

Step 5: Get the current revision number
I recommend putting the revision number of the art assets you pack into the name of the pack file. This will help you to remember what version of the assets you last packed. To do this you need to know what revision your unpacked files are at. The revision is simply a number like 3056 that helps us (and you) keep track of the mod version during development. Here are the steps to get it:
  1. Navigate to your Realism Asset Packing\Source\Art folder
  2. Right-click on Art and select "Properties"
  3. The windows "Art Properties" dialog pops up
  4. Click on the "Subversion" tab
  5. Near the top of the dialog you'll see a "Last commit revision:" field with a number after it. That number is the revision number of your unpacked art assets.
  6. Note the revision number somehow (e.g. selecting it and copying it)
  7. Close the dialog

Step 6: Pack the art assets
Run Realism Asset Packing\PakBuild.exe

PakBuild is designed to let you create and save layout files that describe .FPK archives, but we won't actually be saving the layout file. We'll just use it to produce the .FPK archive itself. Let's go through the process step by step:
  1. From the top menu select File->New (we're making a new layout file based on an existing folder structure)
  2. Select the Realism Asset Packing\Source folder (note that we don't select the art folder but just the source)
  3. The Raw Files panel on the left now shows an input folder at the top and the Art folder in a list
  4. Check the "Store full paths" option in the bottom right
  5. Click the "New PAK File" option in the middle
  6. The right panel is activated and shows an empty Packed Files list for a Pak0.FPK
  7. Click the "Rename" button in the right panel
  8. A dialog pops up asking you to "Enter new name for Pak0.FPK"
  9. Enter RealismArt.<REVISION>.FPK (replace <REVISION> with the version number from step 5, so it's end up like this: RealismArt.3065.FPK)
  10. Hit "OK"
  11. From the top menu select "PAK->Build"
  12. A Building PAK Files dialog appears
  13. Wait 5-20 minutes while the .FPK file is created. It may appear unresponsive while it is working.
  14. The Building PAK Files dialog disappears when it done and you'll see a list of files in the right panel, indicating that they have been packed.
  15. Close the PakBuild utility (it will ask you if you want to save but you don't need to save a layout file, the .FPK file has already been saved)

Step 7: Copy the newly created .FPK file to the Realism Assets folder
The newly created .FPK file will appear in your Realism Asset Packing\Source folder.

If you aren't keeping a series of versions just move the newly created .FPK file from Realism Asset Packing\Source folder to the ROOT CIV FOLDER\Beyond the Sword\Mods\Realism\Assets folder.

Important Note: The .FPK file goes in the Assets directory, not the Art directory.

If you do want to store a series of versions move the newly created .FPK file from Realism Asset Packing\Source to the Realism Asset Packing\Versions folder and then copy it from there to the ROOT CIV FOLDER\Beyond the Sword\Mods\Realism\Assets folder.

Step 8: (Optional) Defragment the hard drive
This previous process can thrash your hard drive with all of the file operations. At this point you may want to empty your trash and check to see if your disk needs to be defragmented.

Step 9: Boot the game
The game should now boot in much less time and without error.

Happy testing!

-Josh


22 Jul 2010, o 15:40
Profile
RI Team
User avatar

Joined: 2 May 2009, o 15:44
Posts: 841
Location: Olympia, WA, USA
Post How to update to the latest revision when using packed asset
Updating Instructions
After you've got the game running off of your .FPK file you'll need to follow a more involved process to update your game files to the latest revision.

To update use the following process:
  1. Perform an "SVN Update" on the Realism Asset Packing\Source\Art folder.
  2. If you don't see any changed files come down that means that no Art assets have actually changed and you probably don't need to repack your .FPK file (skip the next two steps (step 3 and step 4)
  3. Delete your old .FPK file from the ROOT CIV FOLDER\Beyond the Sword\Mods\Realism\Assets folder
  4. Create and copy a new .FPK file using steps 5, 6, and 7 in the previous post
  5. Navigate to ROOT CIV FOLDER\Beyond the Sword\Mods
  6. Select the Realism folder, right-click and select SVN Update; let it update and then close the dialog.


22 Jul 2010, o 15:47
Profile
Noble

Joined: 16 Jun 2008, o 19:59
Posts: 47
Location: Seattle, WA
Post Re: How to pack art assets for faster boot times
See the reason I love this mod is the people. No where else in the world I come from are people this complete. If I knew how to say this it would take me a day maybe two to think it out even if its second nature to you. I've changed my mind I'm doing this printing steps as I type right now. At first I figured this looked like a lot to do but I meant no disrespect after mentioning the wait was worth it...

Thanks for taking the time to do this I appreciate it a lot. RC - nicro

EDIT spelling and grammer


24 Jul 2010, o 19:58
Profile YIM
RI Team
User avatar

Joined: 2 May 2009, o 15:44
Posts: 841
Location: Olympia, WA, USA
Post Re: How to pack art assets for faster boot times
Great people art files are packable now due to some improved code. This actually happened quite some time ago but I forgot to update this post. The post has now been updated.

-Josh


21 Sep 2011, o 13:27
Profile
Noble

Joined: 10 Sep 2011, o 16:27
Posts: 34
Post Re: How to pack art assets for faster boot times
I have followed the tutorial, but I don't manage to get the packaging right (everything appears as pink and the game crashes after a turn or two). I have no clue of what I might be doing wrong: I have followed it to the letter three times... Any ideas? :/


22 Sep 2011, o 03:29
Profile
RI Team
User avatar

Joined: 2 May 2009, o 15:44
Posts: 841
Location: Olympia, WA, USA
Post Re: How to pack art assets for faster boot times
Some quick thoughts off the top of my head:

  • Are you checking the "Store full paths" check box?
  • Are you putting the .FPK file in the Assets directory? (not the Art directory)
  • Are you packing the entire Art directory that you copied by selecting the directory above it?
  • Can you unpack the file the FPK that you created and do the files look correct?

-Josh


22 Sep 2011, o 03:40
Profile
Noble

Joined: 10 Sep 2011, o 16:27
Posts: 34
Post Re: How to pack art assets for faster boot times
Josh wrote:
Are you putting the .FPK file in the Assets directory? (not the Art directory)

That was my mistake :D

Now everything looks as expected, and it opens in a reasonable time. Thanks! :)


22 Sep 2011, o 04:25
Profile
RI Team
User avatar

Joined: 2 May 2009, o 15:44
Posts: 841
Location: Olympia, WA, USA
Post Re: How to pack art assets for faster boot times
Great! Glad you got it working!


22 Sep 2011, o 05:10
Profile
Site Admin
User avatar

Joined: 10 Jul 2007, o 17:19
Posts: 543
Location: Warsaw, Poland
Post Re: How to pack art assets for faster boot times
Josh wrote:


Step 2: Copy the Art folder to that new place(...)
You should end up with a Realism Asset Packing\Source\Art folder when you're done. When it's finished right-click on the new copy of the Art folder and select Tortoise SVN->Clean up. This makes sure that Tortoise has its wits about it.


No such option here on my PC. Should I first do SVN Checkout for the new copy of Art folder?
I do have Tortoise SVN->Cleanup option only on ROOT CIV FOLDER\Beyond the Sword\Mods\Realism\Assets\Art folder.
I've skipped that step of SVN->Cleanup and everything seems to work fine.

Quote:
Step 3: In the original Realism Art folder delete all but one directory (the movies folder which is unpackable)


If we leave movies folder because it's unpackable, then why we still pack it in Step 6?
What's the point of having this duplicate that is possibly extending the working time?
Shouldn't we delete 'Movies' from Realism Asset Packing\Source\Art ?


Josh wrote:
Some quick thoughts off the top of my head:

(...)Are you putting the .FPK file in the Assets directory? (not the Art directory)


Please highlight it somehow in your instruction. I've made the same mistake.

_________________
The only worse thing than being talked about is... not being talked about.


18 Oct 2011, o 11:18
Profile WWW
RI Team
User avatar

Joined: 2 May 2009, o 15:44
Posts: 841
Location: Olympia, WA, USA
Post Re: How to pack art assets for faster boot times
Vitez wrote:
Josh wrote:
Step 2: Copy the Art folder to that new place(...)
You should end up with a Realism Asset Packing\Source\Art folder when you're done. When it's finished right-click on the new copy of the Art folder and select Tortoise SVN->Clean up. This makes sure that Tortoise has its wits about it.


No such option here on my PC. Should I first do SVN Checkout for the new copy of Art folder?
I do have Tortoise SVN->Cleanup option only on ROOT CIV FOLDER\Beyond the Sword\Mods\Realism\Assets\Art folder.
I've skipped that step of SVN->Cleanup and everything seems to work fine.

It looks like Tortoise SVN had a major new release (1.7) which you've probably got. The new version no longer has a "Cleanup" action because it's been redesigned. These instructions will need to be updated for the new version.

Most importantly, the new version no longer creates the huge number of hidden .svn files that the old version did. I suspect that this will be a big help with boot times for SVN users.

Vitez wrote:
Quote:
Step 3: In the original Realism Art folder delete all but one directory (the movies folder which is unpackable)


If we leave movies folder because it's unpackable, then why we still pack it in Step 6?
What's the point of having this duplicate that is possibly extending the working time?
Shouldn't we delete 'Movies' from Realism Asset Packing\Source\Art ?

You certainly can delete it. I think I just didn't bother to in the original instructions because it complicated the updating process later.

Something else that's cool: the new version lets a person do "sparse" checkouts much more easily. A sparse checkout is where only part of the tree is checked out. When I update these instructions I'll probably take advantage of this new feature to simplify the process.

Vitez wrote:
Josh wrote:
Some quick thoughts off the top of my head:

(...)Are you putting the .FPK file in the Assets directory? (not the Art directory)


Please highlight it somehow in your instruction. I've made the same mistake.

Will do.


20 Oct 2011, o 14:56
Profile
RI Team
User avatar

Joined: 12 Jul 2007, o 09:23
Posts: 2913
Post Re: How to pack art assets for faster boot times
Josh wrote:
It looks like Tortoise SVN had a major new release (1.7) which you've probably got. The new version no longer has a "Cleanup" action because it's been redesigned. These instructions will need to be updated for the new version.

Most importantly, the new version no longer creates the huge number of hidden .svn files that the old version did. I suspect that this will be a big help with boot times for SVN users.


Yep, new version is a huge overall improvement. One significant consequence of the new arrangement is that with old SVN, any subfolder copied to a separate location retained all SVN functionality, while with new version any copy that doesn't include the root folder doesn't have any SVN files at all, and thus can no longer be updated or committed. In regards to speed, the new version is also much much faster to update and commit stuff. 1.6 versions have always demanded almost 100% of my system's resources when they updated/commited, while now it runs like a background process, without slowing anything.


21 Oct 2011, o 04:33
Profile
King
User avatar

Joined: 7 Sep 2007, o 18:27
Posts: 402
Location: Brazil, Latin America
Post Re: How to pack art assets for faster boot times
On my system the 1.7 version is + - 50% more faster. The new version is really great.


21 Oct 2011, o 13:41
Profile
RI Team
User avatar

Joined: 2 May 2009, o 15:44
Posts: 841
Location: Olympia, WA, USA
Post Re: How to pack art assets for faster boot times
The instructions in this thread have been updated to reflect the changes made in TortoiseSVN 1.7.


5 Jun 2012, o 15:45
Profile
King
User avatar

Joined: 7 Sep 2007, o 18:27
Posts: 402
Location: Brazil, Latin America
Post Re: How to pack art assets for faster boot times
A tip / caution: do not create the folder that contains art files in a folder path too long, the pakbuild will not read all the way and not will create the final file.


12 Jul 2013, o 13:06
Profile
Warlord

Joined: 6 Nov 2007, o 15:08
Posts: 72
Post Re: How to pack art assets for faster boot times
*transfers Realism Invictus to laptop for trip*

*forgets to also copy over source/art folder*

*needs to update art*

*has poor internet connection*

*watches screen sloooowwwwllly fill with assets*


Estimated time of completion: Sometime after I get back home.


Lesson: Don't forget to copy over the folder you put the art in when moving files to new devices.


6 Nov 2014, o 11:50
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 15 posts ] 

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.