resource fork to file structure expander

Poll: Res or Rez (2 member(s) have cast votes)
what file format is this needed for?
Res
(1 votes [33.33%])
Percentage of vote: 33.33%
Rez
(2 votes [66.67%])
Percentage of vote: 66.67%

Hi everyone,
This is a thread for discussing the development of a utility that expands/compresses a EVN rez file into/from a directory structure and files, representing the resources stored in the rez.

User process
1. Expand rez file to disk
2. Edit expanded data with tools of choice. ie text and graphics editors
3. Compress data into a new rez file

Perceived benefits
version control and backup of individual resources and data within them
- online project hosting ref:http://code.google.com/hosting/
- collaborators and contributors working on one repository
- branches
- cross over TC's
tools of choice may allow for greater creativity

code
i would like this to be an open source project. i am currently working on a syst map editor: http://www.ambrosiasw.com/forums/index.php?showtopic=128516
and am able to read a rez file into realbasic classes. i will reuse this code to make a rezfile IO library to use in this utility.

data formatting
to be decided upon
-xml

This post has been edited by Tycho : 19 May 2010 - 11:30 PM

This is how MissionComputer worked in some of the 3.x versions (it goes back to an idea I had back when I was trying to imagine a Mac OS 8-style replacement for ResEdit). It proved to be basically reliable, although the dependence on the file system made the conversion back and forth rather slow — that's part of why I switched back to the Resource Manager when it became viable to do so. I still think storage in memory is the most elegant solution, and have been pushing towards that goal since mid-way through developing 4.0.

Btw, RezEditor 0.1 supports exporting/importing the data part of a single resource to disk.

The problem with making a filesystem of it is that you have to store the type (directory, I expect) and name, which may contain invalid characters for the filesystem (In a different context, this broke building RezEditor when moving development from Windows to Linux).
You probably want to use human-readable names for the directories/files, so hex is unsatisfactory.
But, I wouldn't embed them in the file either, as basic editors should not have to care about that, and advance editors should use the filesystem to find out the information.
I guess you could use a percent-encoding scheme for characters that can cause problems:
control characters 0-31 and 127, nonASCII-characters 128-256, and non-alphanumeric characters (I was going to list which ones were interpreted specially by various filesystems and/or shells, but the only ones left were comma, dash, underscore).

Also, if the ID becomes the first part of the filename, then you have to remember that numbers won't appear in numeric order, but instead alphabetic order for most people. Do you prepend zeros? Or write the ID in hex, prepending zeroes?

thanks for the replies.

disk vs memory
the idea is more like compiling.
ie the rez is decompiled once(if working from a rez already created)
all work is done on the data, leaving out the details of ID's, etc.
the rez is compiled, only when you want to test something or build the final plug.
in fact, it won't always be necessary to compile the entire plug.
-you may only want the systems and spobs

structure
i was thinking along the lines of:
plug directory
/syst/
/syst/128/
/syst/128/syst.plist
/syst/129/
/syst/129/syst.plist
/syst/130/
/syst/130/syst.plist
/spob/
/spob/spobGlobals.plist
/spob/128/
/spob/128/spob.plist
/spob/128/spob.pict
/spob/129/
/spob/129/spob.plist
/spob/129/spob.pict
- etc

spob Globals.plist would define any global value for all spobs. this will be overridden by values in each spobs own plist, if they are there. eg you could make all syst's have a connection to the Rome syst by adding the connection in the global syst file. all syst's will then inherit this connection if they have fewer than maxConnections. ie all hyperjump-roads lead to Rome.

i've used filenames like spob.plist, but realistically it can be anything as long as the extension remains the same.
the nature of xml even allows multiple files to be read as one , so you could have resource.plist and patch.plist in the same directory and they would be merged on compile. if values from resource.plist were to be overridden with values from patch.plist then, the filenames would have to be in alphanumeric order of priority or a priority encoded in the xml.

i never thought of using strings from the plug as filenames. they aren't unique( am i correct in this assumption? ). only the ID's are. appending the name after the ID would just be annoying, surely?. i hate it when people add a number before a filename to control it's sorting. makes no sense. integer+string=wtf?
what OS won't the alphanumeric filenames appear in numeric order? it's windows, isn't it? sheesh. i think i'd be better off writing a new window/file manager for windows. make them one that works.

This post has been edited by Tycho : 20 May 2010 - 09:37 PM

Tell me the name of syst 159 off the top of your head.
That's why I thought about doing id-name for the filename.

With your current view up there, having the individual resources be directories doesn't seem to be doing much good... Edit: I see the spob now, though I'd do it as spob/128.xml, spob/128.pict. The problem with actually representing the image as a separate file is the problem of coalescing duplicates; it's O(n^2) and the file comparisons themselves are expensive. Using symlinks is an option, but restricts your userbase.

Oh, you mean actually interpreting the resources themselves, like in EVNEW text (but in xml)? I don't really like that idea, but if you do it that way, you'll want to have the raw resource first. For example, you can then do a timestamp check (à la make) to see if you even need to recompile the individual resource.

And actually I think Windows Explorer will try to be smart and sort numerically (and have you heard of ExplorerXP?), but I was referring to the general case of being unpredictable:
It's difficult to know whether sorting numerically is the right thing to do; in linux you can always do
ls -1|sort -g
For example, how should the following be sorted?

CODE

11ff
2fff

To humans (or at least a subset of them), it is obviously a set of hexadecimal numbers, but windows explorer puts the greater number ahead of the lesser.
Therefore, a pure alphabetical sort is the only reasonable default.

That being said, I have thought that base 10, no fill would be preferred.

One last rant: why don't standard programming I/O libraries have print(int num, int base, int digits)? The C++ way is one of the things they only made worse from C. And Java doesn't even have a function for it, unless you go through Formatter.

This post has been edited by Nonconventionally Creative : 21 May 2010 - 10:56 PM

QUOTE (Nonconventionally Creative @ May 22 2010, 04:31 AM) <{POST_SNAPBACK}>

Tell me the name of syst 159 off the top of your head.
That's why I thought about doing id-name for the filename.

the user would have to open the relevant xml file to determine syst name.
the folder and filenames actually would have no impact on compiling. the xml could be merged and parsed. all info relevant to making a plug is in the xml. the directory structure would be a product of decompiling a plug or user creation. i think it's a personal thing to keep things organised in directories. the user would also be able to keep other files in the directories. ie a textual description of the resource, concept art, etc. these could be included in the plug repo.
i understand there will be redundancy in the xml, but i think it will normally be negligible.

eg.
/my systems/
/my systems/my system/
/my other systems/
/my other systems/my other system/
/my other systems/128/
/my other systems/Ronald/

QUOTE (Nonconventionally Creative @ May 22 2010, 04:31 AM) <{POST_SNAPBACK}>

With your current view up there, having the individual resources be directories doesn't seem to be doing much good...

i believe it's a personal preference. there's also the version control repository integration. a plug maker could easily check out directories of the plug, without specifying individual files.

QUOTE (Nonconventionally Creative @ May 22 2010, 04:31 AM) <{POST_SNAPBACK}>

Edit: I see the spob now, though I'd do it as spob/128.xml, spob/128.pict. The problem with actually representing the image as a separate file is the problem of coalescing duplicates.

images would be specified in the xml. names don't matter, the OS will handle duplicates. multiple xml files can reference a single image. the image will only be encoded once.

QUOTE (Nonconventionally Creative @ May 22 2010, 04:31 AM) <{POST_SNAPBACK}>

Oh, you mean actually interpreting the resources themselves, like in EVNEW text (but in xml)? I don't really like that idea,

yeah, that's the idea. what's not to like? you have beer, ya have tv, xbox or something, right? exactly.

QUOTE (Nonconventionally Creative @ May 22 2010, 04:31 AM) <{POST_SNAPBACK}>

but if you do it that way, you'll want to have the raw resource first. For example, you can then do a timestamp check (à la make) to see if you even need to recompile the individual resource.

good idea. do you want to layout the specifications for what you are thinking of? if you can think of anything more efficient or other ideas like that, i'd love to hear.

QUOTE (Nonconventionally Creative @ May 22 2010, 04:31 AM) <{POST_SNAPBACK}>

One last rant: why don't standard programming I/O libraries have print(int num, int base, int digits)? The C++ way is one of the things they only made worse from C. And Java doesn't even have a function for it, unless you go through Formatter.

that's a fair point. all i can think of is backward compatibility, but even that seems a long shot. POSIX compliance?