Quite some time ago, the player software was enhanced to be able to deal with /fids/ directories that are broken up into smaller subdirs. This can speed up database rebuilds and filesystem checks etc, and is a worthwhile thing to have.

The Emplode and emptool software still just upload everything into /drive?/fids/, but the player software (and Hijack) can handle things in either format, including a mix of the two methods.

So.. here's my small script for (re)sifting the tune and tag files into appropriate subdirs, on a one or two drive RioCar/Empeg player.

To use, upload via FTP to /fidsort.sh, and set the exec permissions:

ftp myplayer

site rw

cd /

put fidsift.sh

chmod 0755 fidsift.sh

site ro

quit

Then connect via the serial port, hit control^C to gain control of the player, and do this command: /fidsift.sh

Then go and have coffee or cola, and when you get back it will eventually finish, and you can view the results in /drive0/fids/ and /drive1/fids/. As new tunes are later uploaded, unsifted files will acculumate again, and you might want to periodically rerun the script in the same way as the first time, to sift the newly uploaded tunes into their subdirs (not necessary, but nice).

The script is attached -- EDIT: there's a better version later in this thread..

Cheers

Want me to do a "before and after" timing of database rebuild? ~22,000 tracks I think.

Now that could be very educational!

I'm also wondering if the script will just crap out with that many fids.. go ahead and give it a whirl!

(it won't actually damage anything if it does crap out.. maybe I'll try a test here..)

I trust that if it craps, only part of the fids will be moved and emplode will just handle it?

Actually, the place where it might crap would just result in nothing happening.

And if it did only do it partially, no big deal -- emplode doesn't know nor care, and neither does the player or hijack. Everything still just works

Hang on a few minutes.. I'm generating a dummy fids directory with 22000 "tracks"..

I just tried it (on my laptop) with 25000 fids (over 50000 files). No problem. Not tried on the player (would take me too long), but it's safe.

Cheers

can hijack be modified to run this script automatically after uploading? like i would love it before the player restarts, so the player would still be expected to be in rw mode.

Seems like it might make (already long) sync's longer.

Okay, here's a slightly improved version.

This one shrinks the /fids/ directory afterwards to improve lookups further, and also copes correctly with a /fids/ directory that contains no files.

Cheers

Faster updates than HiJack!

This script, after the initial run, is actually quite fast. So it wouldn't really slow down most syncs at all (not noticeably), unless uploading several hundred tracks.

I thought about building it into hijack, but a better option is just to bind it into the menu system as a user app.. Gotta write that new menu thingie Real Soon Now (the "site exec" stuff and friends).

Cheers

chmod 0755 = what does this equate to?

I'm using WS_FTP and it doesn't do numbered chmods... only checkboxes.

rwxr-xr-x == 0755

User(owner): read/write/execute
Group: read/execute
Other: read/execute

Thanks for the edit too.

Update: The FidSift is currently taking longer to run (at least this first time) than the "pre sift" sync... will it be a photo finish or is there a long way to go?

The first sift will be slow. But you should be getting the odd progress indication in the form of a mkdir: created directory `_00000' style of message once in a while.

Cheers

Okay, I have 20,919 mp3s on the player (I always forget how many I have because the player doesn't always removed duplicates when repeating the root playlist like it used to) with 6.26GB free of 130GB. Anyway... the process:

1. Installed latest Hijack.

2. Connected via emplode and added one 4.54mb file.

3. Pre "FidSift" Sync
Start: 10:11
End: 10:28
Elapsed: 0:17 (I was surprised it was this quick!)

4. Installed and ran FidSift.
Started sifting drive0: 10:52
Started sifting drive1: 11:11
Ended: 11:34
Elapsed: 0:42
Drive0 went up to about _0004d and Drive1 went to _0005b. As Mark pointed out, this shouldn't take this long after the initial run.

5. Connected via emplode, replaced same 4.54MB file.

6. Post "FidSift" Sync
Start: 11:35
End: 11:43
Elaspsed: 0:08

That's over a 100% speed increase!

Ahh, good. I was hoping for some kind of measurable improvement there. I forgot to do the "before" timings on my own players, and I have not written a script to convert the files back to the old naming/layout scheme, so..

WooHoo!

(thanks for trying it out)

No, thank you! Database rebuilds were painfully slow but I can live with 8 minutes now! I assume that everyone can get rebuilds to be over twice as fast too.

Mmm.. I suppose I could have the script do both drives in parallel, which would cut the time for the first sift down by quite a bit. But that's a one-time only event (per player), so no big deal. And I don't have a two-drive unit here to test with, so I think I'll just leave well enough alone!

A subsequent "sift" should be near instantaneous, unless a thousand or so (or more) tracks are modified with emplode/emptool.

Cheers

Sweeet. Nice job Mark. So can we expect this to be built into hijack soon? If so i think i'll wait. I'm going to be completely rebuilding my player soon anyhow. It would rock if this could be run after a certain number of tracks have been added. Could Jemplode do this somehow??

Mmm.. Well, JEmplode over ethernet could use FTP for the file transfers, in which case it could just use the new subdir structure as it uploads tracks. Maybe Mike will add that someday.

I don't know about building it into Hijack -- so don't wait for that to happen. Once I someday implement the "run a command from the menu" feature it will be trivial to do. But I keep getting distracted here..

This is pretty simple to do as-is.. and even if it's only done once it will still make a difference. The only tricky bit is getting the script onto the player, but that only need be done the first time. From then on, it's still there and can be run just by hitting control^c and typing it's name (using the serial link). And it's only slow the first time!

Sleep time..

Cheers

Is this thing safe under the V3 alpha kernel?

-- Gary F.

Nice work Mark,.... people of the world (with obscenely large music collections) Rejoice!

Yes.

The "v3alpha kernel" is the same as the v2-final kernel, so.. yes, it's safe.

It is also just fine with the v3alpha player software, too.

-ml

Okay, there's a flaw with this scheme.

When syncing with Emplode or emptool, the player stores new/modified FIDS in the top level /driveX/fids/ directory, rather than the subdirs. Same when doing deletes. Nothing wrong with that, other than some duplication.

BUT.. the player software does NOT search the top level directory if it finds a subdir.. OUCH. I was under the impression that it was smart enough to look in the right order, but it is actually even smarter than that -- smart enough not to "waste" the effort. This is a good thing, but it means that you MUST re-run fidsift.sh after each sync(Ugh!), and even that won't properly delete things.

Okay, I'll fix this, in the Hijack kernel, so that the player does "the right thing" whether it wants to or not. Later today, or tomorrow.

The plan is have the kernel automatically remap ALL /fids/ accesses to the subdirs when they (subdirs) exist. This way, syncs which affect existing fids will work correctly. New uploads can still be dropped into the top level /fids/ directories, if no appropriate subdir exists at the time. Not an issue for now.

Cheers

my solution.

;@AC ;@EXEC /bin/fidsift.sh

cant use ;@EXEC_ONCE because that only executes after a reboot.

unless you uploaded over 500 files, the time diff is unnoticable.

That will almost work. Requires a re-sync after the first sync, to ensure the database is rebuilt using the newly sifted files.. thus the problem.

Cheers