#248654 - 09/02/2005 08:35
empeg file structures
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
I am looking into creating my own empeg sync process - unfortunately mp3tofid doesn't quite fit the bill. But I am in need of info about the file formats and hope someone can help with some advice.
Is there somewhere I can find full and precise details of these formats?
Failing that:-
Does the tags ('1') file for a track HAVE to include all the tag info (my files have some different lines to what I've read elsewhere) and is the order important? e.g. is 'offset' necessary? The Rio Receiver defines this as a possible tag, but it is not supplied and hence is not required, but what about the RioCar (empeg)?
Likewise for a playlist tags file. If there is no appropriate artist etc, can this be simply left out?
What is the structure of the actual playlist ('0') file? It is not ascii (as are all the tags files). Are these stored in binary format? Any special requirements?
How can I create the /var/tags, playlists and database files?
tags appears to be ascii, but with a lot of blank lines. Do I need to touch this at all?
The database and playlists files appear to be binary, would they be dbm files by any chance (hope so:-). If not what?
Can anyone suggest how I can appropriately split the files across the 2 drives? Whatever method I use I need to be able to repeat the same split so that rsync will work efficiently. I thought I'd create 2 separate local fids dirs and simply rsync them, but what would make sense in deciding into which fids dir each file should be placed? I have some ideas, but hoped for some independent input.
Any help with this would be much appreciated?
|
|
Top
|
|
|
|
|
#248655 - 09/02/2005 12:31
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14491
Loc: Canada
|
The *1 tags files are not used at runtime by the player. The only reason they exist is for input when the player has to rebuild the binary database from scratch. The Hijack webserver also uses the *1 tags though, so it's a good idea to keep them around.
The *0 playlist files are just a series of 32-bit fids in binary form, in empeg byte order (little endian -- thanks Roger).
The database file format is just a catenation of most of the *1 tag fields for each fid, with separators etc.. easy to reverse engineer with a binary hex/ascii viewer/editor.
Cheers
Edited by mlord (09/02/2005 13:37)
|
|
Top
|
|
|
|
|
#248656 - 09/02/2005 13:02
Re: empeg file structures
[Re: mlord]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
in empeg byte order (big endian?).
Little-endian.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248657 - 09/02/2005 15:56
Re: empeg file structures
[Re: mlord]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
I can easily create the 1 tags files, so might as well. Is the order of the lines important?
Excuse my ignorance, but in the playlist files, are the fids all full 32 bits, so it's like a fixed length record structure? How else could it determine the individual fids?
Is the /var/playlists file of the same format/structure as the database file? What about the /var/tags file. Is their structure fully explained anywhere? It would help enormously if I could short circuit the reverse engineering of these. Surely someone has this info written down somewhere?
I confess, trying to manipulate binary files like this is right at the limit of my knowledge so I need to leverage any help I can get:-)
|
|
Top
|
|
|
|
|
#248658 - 09/02/2005 16:42
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Surely someone has this info written down somewhere?
No. The only public documentation for the format of these files exists in the emptool source code.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248659 - 09/02/2005 18:28
Re: empeg file structures
[Re: ukengb]
|
pooh-bah
Registered: 12/02/2002
Posts: 2298
Loc: Berkeley, California
|
Perhaps you could tell us what you're trying to accomplish in the end? There are quite a few tools available that perhaps could be adapted to suit your needs more easily than building something from the ground up.
Matthew
|
|
Top
|
|
|
|
|
#248660 - 09/02/2005 20:04
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14491
Loc: Canada
|
Yeah, the fids are all 32-bits in the playlist *0 files. You know how many there are by simply looking at the "length" tag in the *1 files, or just the Linux filesize in bytes.
The two files named playlists and tags are special: I don't remember what playlists is actually for, but tags gives the layout (field order) for each record in the database file -- it lists the tags that are actually known/used.
The database file then just has a record for each tune/playlist, with the variable-length fields in the order as given by tags, using NULLs (I think..) as field terminators.
Oh wait, I just looked at the database file, and it appears that each field within each record has two bytes in front: the first byte is the binary index of that field into lines in the tags file, and the second byte is the bytecount for the ASCII data that follows. So if "artist" is the fourth line in tags, then the artist data in the database for "Bare Naked Ladies" would have 0x03 0x11 immediately in front of the band's name, on each record within database that pertains to that band.
Cheers
Edited by mlord (09/02/2005 20:14)
|
|
Top
|
|
|
|
|
#248661 - 09/02/2005 20:07
Re: empeg file structures
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14491
Loc: Canada
|
MMm.. the playlists file is just another playlist: a bunch of 32-bit (binary) fid numbers, one after another, for all of the playlists that are visible under the first level Playlists menu on the player. I think.
Cheers
|
|
Top
|
|
|
|
|
#248662 - 09/02/2005 21:50
Re: empeg file structures
[Re: mlord]
|
addict
Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
|
Yes, it's a concatenation of all playlists in fid order.
Pim
|
|
Top
|
|
|
|
|
#248663 - 09/02/2005 21:56
Re: empeg file structures
[Re: ukengb]
|
addict
Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
|
Quote:
Can anyone suggest how I can appropriately split the files across the 2 drives?
When using (j)emplode, it's the player that decides on which drive a fid ends. The player knows the disk usage of both drives while the transfer is going on, so it probably takes the drive with the most space. This is as efficient as you can get.
mp3tofid cannot know the disk usage of the drives where the fids will end up, so it has to make a guess. You can specify the percentage of the number of fids that will be placed on the second drive. This does not have to correspond exactly with the percentage of disk space used, but it is close enough to allow you to fill both your drives to the limit.
Pim
|
|
Top
|
|
|
|
|
#248664 - 10/02/2005 10:46
Re: empeg file structures
[Re: Roger]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
As a NON C programmer, trying to read C source code is not a simple matter and there's lots of files in emptool. Can you be more specific about which files are worth looking at?
|
|
Top
|
|
|
|
|
#248665 - 10/02/2005 11:10
Re: empeg file structures
[Re: matthew_k]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Perhaps you could tell us what you're trying to accomplish in the end? There are quite a few tools available that perhaps could be adapted to suit your needs more easily than building something from the ground up.
OK.
I use solely iTunes to encode, maintain and control my music with the actual music data located on a linux server (a shared volume of). I then run a perl script (nightly) that reads the iTunes XML backup file and stores the relevant info in dbm files (actually uses DB_File) and this can be read and used by apache and some perl cgi scripts acting as the music server to several Rio Receivers around the house. Oh and I also use iTunes itself to play the music in my study/office.
So anything I do in iTunes is automatically available the following day in all the Rios (or earlier if I manually run the update) and it works perfectly.
Now I want to turn my hand to syncing the RioCar/empeg with the exact same music info (data and playlists) so that it all becomes a fully rounded iTunes/Rio music system solution, for me and anyone else who realises the benefits of using iTunes as their main control point.
None of the emplode type products comes close to doing what I want and are extraordinarily slow at doing what they can do. So I want to create a local fids structure and then use rsync.
Ah, "mp3tofid" I hear you say. Close, but no cigar. Very close in fact, but mp3tofid has to read the ID3 tags to get the data and I already have everything I need in the dbm databases, as well as the exact playlists I want to upload, which are already created and stored for use by the Rio Receivers. Actually writing the fids data will be relatively trivial and very fast, once I know what I need to write - hence my questions here.
As I've said elsewhere, I'm not a C programmer so trying to read C source files is a right PIA and in some cases I simply cannot make any sense out of them at all. Likewise trying to glean the file structures from sample files is also a non trivial task and prone to errors.
For these reasons I ask if there is any usable written definitions of these file formats to help me. Failing that I need to ask questions, but will ensure that the results of my quest are written up and made availabale for others.
However, in the meantime, I still need to know more about these files.
|
|
Top
|
|
|
|
|
#248666 - 11/02/2005 07:08
Re: empeg file structures
[Re: ukengb]
|
pooh-bah
Registered: 13/09/1999
Posts: 2401
Loc: Croatia
|
Hm, interesting approach. BTW, can you share your scripts that parse iTunes XML (I see (from WinAmp export) that the XML structure is quite straightforward, but I hate to rediscover the wheel (i.e. I am lazy  )) ? OK, I guess your best bet is to build all *0 (tune and playlist) and *1 (tag) files locally, transfer them over, remove database files (instead of bothering to recreate them) and let the player rebuild them from tag files. As already said, playlist *0 files are simply arrays of 32-bit numbers (member's FIDs), so they should be very simple to construct from perl of whatever (no C needed, but watch for endianness). AFAIR, tag files are pretty free-form: tags can be in any order, you can add your own (but they will not end up in the database).... Finally, maybe this is a good time to learn C? (says he, who is too lazy to parse a simple XML from Perl)  Cheers!
_________________________
Dragi "Bonzi" Raos
Q#5196
MkII #080000376, 18GB green
MkIIa #040103247, 60GB blue
|
|
Top
|
|
|
|
|
#248667 - 11/02/2005 07:09
Re: empeg file structures
[Re: pim]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
OK, I think I've got the file format/structures sorted, but I am puzzled as to the raison d'etre for the /var/playlists file. As Pim says, it seems to be a concatenation of actual playlist (0) fids, but why? What is its purpose? There's no delimiter separating each playlist's group of fids from the next group, so what exactly is it for?
Anyone enlighten me on this?
Oh, one more technical question on this. Does /var/playlists contain ALL the playlists (as Pim suggested) or just the top level (as I think Roger indicated) or some other combination?
Thanks.
|
|
Top
|
|
|
|
|
#248669 - 11/02/2005 08:57
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Does /var/playlists contain ALL the playlists (as Pim suggested)
No. Pim is right. /var/playlists contains ALL of the playlists. It's built by catenating the *0 files of each playlist, in FID order into one big file. It doesn't need delimiters because the length of each playlist is already in the database file.
It's an optimisation -- this way, the player doesn't need to load each *0 file when it starts up.
As Bonzi says, if you're just slapping a bunch of *0 and *1 files on the player, you can avoid generating anything in the /empeg/var directory, and let the player rebuild them itself.
_________________________
-- roger
|
|
Top
|
|
|
|
#248670 - 11/02/2005 17:28
Re: empeg file structures
[Re: ukengb]
|
journeyman
Registered: 08/08/2001
Posts: 51
Loc: CA, USA
|
Awesome project... I'd use (& test) it! Keep up the great work...
_________________________
Ryan here... Empeg [08000462] 40 Gig with Subaru WRX sport wagon attached... (still! pending memory + LED upgrade, sheesh, I've been sitting on my ass for years)
|
|
Top
|
|
|
|
|
#248671 - 12/02/2005 12:18
Re: empeg file structures
[Re: bonzi]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
I want to finish this RioCar sync part of the project and get it all tidied up then make it generally available to anyone who wants. You're not in a rush are you?
My XML parsing is all my code - I don't use any other modules as I had very precise requirements for what it did. In fact these have changed slightly and become a bit more general, but I've already done it so pointless going back.
I originally ran the Rio Receivers on the ARM software, but have now moved to tRio which offers greater possibilities. Had to sort out a few glitches due to tRio doing some things slightly differently and Paul (tRio's author) sorted a couple of issues and so my system can now run with either Reciever player software.
With regard to the /var/??? files - it makes sense to me that I create them at the server end and then rsync the whole lot. Building the playlists and database doesn't look that hard and the tags file can be left as is I think.
I'll post back here when I have some results, may be a little while as I want to get another RioCar running to do the testing on and need a new HD for that.
BTW, I had thought of using the fids subdirs that Mark Lord makes use of for fidsift, but am I right in thinking that this only offers speed improvements in rebuilding the database? So if I do that on the server, there's no gain with using subdirs.
Can anyone comment on this?
|
|
Top
|
|
|
|
|
#248672 - 12/02/2005 12:29
Re: empeg file structures
[Re: Roger]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
Does /var/playlists contain ALL the playlists (as Pim suggested)
No. Pim is right. /var/playlists contains ALL of the playlists. It's built by catenating the *0 files of each playlist, in FID order into one big file. It doesn't need delimiters because the length of each playlist is already in the database file.
Of course. Juggling with lots of unknowns at present and that slipped my mind. Makes sense to me now. Does it HAVE to be constructed in that order (i.e. playlists in their fid order)? Or could the playlists be strung together in any order?
BTW, thanks for the tip about the database records containing an index into the tags file to specify the tag for the following data item. It would have taken me some time to figure that one.
Quote:
As Bonzi says, if you're just slapping a bunch of *0 and *1 files on the player, you can avoid generating anything in the /empeg/var directory, and let the player rebuild them itself.
I understand your point, but I like the idea of having everything ready on the server for a simple rsync, which can be relatively quick compared to a database rebuild. I always seem to want to sync the player last thing before I have to go out with it and then I get complained at for the delay:-(
So my theory is keep the local copy up to date (part of my nightly updates that read in the iTunes XML file) and be ready for 'Instant Sync'. In fact I think I'll copyright that phrase:-)
|
|
Top
|
|
|
|
|
#248673 - 12/02/2005 12:34
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14491
Loc: Canada
|
Quote:
BTW, I had thought of using the fids subdirs that Mark Lord makes use of for fidsift, but am I right in thinking that this only offers speed improvements in rebuilding the database? So if I do that on the server, there's no gain with using subdirs.
The fids subdirs have a substantial, measurable impact (positive) on any and all operations dealing with the playlists, tracks, and tags files. They will help rsync as much as they help the internal database rebuild.
Many of us have 10000-20000 tracks on our players, for a total of perhaps 20000 to 40000 inodes in the /fids/ heirarchy. When those are all in a single directory, the clunky sequential search directory lookup mechanism (in the kernel) is very very slow, even just to play a selected track.
The fids subdirs really should be there by default in all software that accesses that data.
Cheers
|
|
Top
|
|
|
|
|
#248674 - 12/02/2005 12:59
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Does it HAVE to be constructed in that order (i.e. playlists in their fid order)?
Yes.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248675 - 12/02/2005 13:10
Re: empeg file structures
[Re: ukengb]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
OK, here's a wild thought.
If the 1 files and the 0 playlist files (i.e. all fids except the mp3 files) are only used to build the /var/??? files and I will create those from my own data anyway, are they really needed? IOW can I get away with just the 0 fid music files and the /var/??? files and nothing else?
Seems that the player doesn't actually use anything else so why create them and have to sync them? This would really simplify the maintenance scripts and also save space on the empeg which is always to be desired.
Please tell me I'm right.
|
|
Top
|
|
|
|
|
#248676 - 12/02/2005 13:16
Re: empeg file structures
[Re: mlord]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
BTW, I had thought of using the fids subdirs that Mark Lord makes use of for fidsift, but am I right in thinking that this only offers speed improvements in rebuilding the database? So if I do that on the server, there's no gain with using subdirs.
The fids subdirs have a substantial, measurable impact (positive) on any and all operations dealing with the playlists, tracks, and tags files. They will help rsync as much as they help the internal database rebuild.
Many of us have 10000-20000 tracks on our players, for a total of perhaps 20000 to 40000 inodes in the /fids/ heirarchy. When those are all in a single directory, the clunky sequential search directory lookup mechanism (in the kernel) is very very slow, even just to play a selected track.
The fids subdirs really should be there by default in all software that accesses that data.
Cheers
I have about 13,000 songs/tunes/tracks at present so that applies to me. If you say it will impact on other activity (which you're right, is to be expected) then it makes sense to use the subdirs. So I will.
Thanks for the quick reply.
|
|
Top
|
|
|
|
|
#248677 - 12/02/2005 13:26
Re: empeg file structures
[Re: Roger]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
Does it HAVE to be constructed in that order (i.e. playlists in their fid order)?
Yes.
OK, thanks for that. Does the database need to be in a similar order? Since it's basically a concatenation of all the 1 tags files, do they need to be in fid order too? I assume so, but confirmation would be good to have.
|
|
Top
|
|
|
|
|
#248678 - 12/02/2005 14:14
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Does the database need to be in a similar order?
Yep.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248679 - 12/02/2005 14:16
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
IOW can I get away with just the 0 fid music files and the /var/??? files and nothing else?
No. When deciding whether to rebuild the database, the player scans the fids directories to see if the numbers tally. If they don't (and with your scheme, they won't), it'll force a database rebuild -- which won't work, because the files are missing.
So, in short, just put them there anyway -- they're minute when compared to the MP3 data, so don't worry about it.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248680 - 12/02/2005 15:49
Re: empeg file structures
[Re: Roger]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
OK, so I need the 1 tag files for the music data 0 files (mp3 etc), what about the playlists? Couldn't I just add the playlist data to /var/playlists and NOT have any 0 or 1 playlist fids? There would still be the correct ratio of 0 : 1 files (for the music data) and with NO playlist files at all, either 0 or 1 fids, wouldn't it be happy?
Just trying to pare it down to the bare essentials.
|
|
Top
|
|
|
|
|
#248681 - 12/02/2005 16:23
Re: empeg file structures
[Re: Roger]
|
carpal tunnel
Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
|
Quote:
When deciding whether to rebuild the database, the player scans the fids directories to see if the numbers tally.
If the database loads successfully, the directories aren't examined. There's no way the player stats 13,000 files on every startup -- it'd take minutes. I think the proposed cheat would actually work, but I'd still counsel against implementing it -- you'll confuse every other bit of car-player software out there, including Emplode and Jemplode.
Peter
|
|
Top
|
|
|
|
|
#248682 - 12/02/2005 17:16
Re: empeg file structures
[Re: peter]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Agreed. Taking the simple extra step to preserve the tag files will allow the player to continue working normally with other software, thus providing a universal solution instead of a proprietary one.
|
|
Top
|
|
|
|
|
#248683 - 12/02/2005 22:03
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14491
Loc: Canada
|
Yes, the tags and playlists are needed. The web interface requires them, as do some of the various add-on apps. Best just to keep them there, as they take up practically no space at all.
Cheers
|
|
Top
|
|
|
|
|
|
Previous Topic
Index