Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Page 2 of 3 < 1 2 3 >
Topic Options
#183440 - 08/10/2003 12:20 Re: Oh no! A serious flaw! [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Mmm.. now that I've traced through it some, it appears that the player IS doing the RIGHT THING(tm), at least some of the time. I need to play with it more to make sure.

At database rebuild time, it correctly looks for (eg.) /drive0/fids/100 before it looks for the same file in a subdirectory (eg.) /drive0/fids/_00000/100

So, as long as it remembers where it found each file for later (part of the database?), then it should be fine as is.

I'll check that now. Meanwhile, I have a hacked hijack version that automatically ensures that new uploads end up in the right places. Not released yet.

Cheers

Top
#183441 - 08/10/2003 12:44 Re: Oh no! A serious flaw! [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Yeah, it's fine.

My worries were for naught. All is well.

But I may still release the changes I made to Hijack, which ensure that the /fids/ directory stays sifted after the initial run.. But then again, a userspace tool can still do this equally well, so why bloat Hijack?

Cheers

Top
#183442 - 08/10/2003 15:54 Re: Oh no! A serious flaw! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
I agree that there is no point in bloating HiJack. Could this user ap be executed via ethernet or ftp rather than serial?
_________________________
Brad B.

Top
#183443 - 08/10/2003 16:35 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
loren
carpal tunnel

Registered: 23/08/2000
Posts: 3826
Loc: SLC, UT, USA
Bloat hijack because it's one less thing we'll have to install and keep track of. And it'll be easier to run. Right? Wrong?
_________________________
|| loren ||

Top
#183444 - 08/10/2003 19:50 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
Daria
carpal tunnel

Registered: 24/01/2002
Posts: 3937
Loc: Providence, RI
Think of this as technical prose, and not really a good solution.

telnet empeg
rw
mv /empeg/bin/player /empeg/bin/player.disabled
cat > /empeg/bin/player << __EOF__
#!/bin/sh
echo "init will helpfully reboot if it can't run the player. Bad init. No biscuit."
return 0
__EOF__
chmod +x /empeg/bin/player
ps auxww|grep player|grep -v grep|awk '{print $2}'|xargs kill
sleep 3
mv /empeg/bin/player.disabled /empeg/bin/player
fidsift
kill -1 1

Top
#183445 - 09/10/2003 03:25 Re: Oh no! A serious flaw! [Re: mlord]
peter
carpal tunnel

Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
At database rebuild time, it correctly looks for (eg.) /drive0/fids/100 before it looks for the same file in a subdirectory (eg.) /drive0/fids/_00000/100

So, as long as it remembers where it found each file for later (part of the database?), then it should be fine as is.
It doesn't remember; it re-does the search every time.

Peter

Top
#183446 - 09/10/2003 05:15 Re: Oh no! A serious flaw! [Re: peter]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5683
Loc: London, UK
Yeah, but at database rebuild time, it does a directory-recursive search for FID files. At FID-lookup time, it has a specific strategy -- which might be the other way round, I don't recall.
_________________________
-- roger

Top
#183447 - 09/10/2003 05:44 Re: Oh no! A serious flaw! [Re: mlord]
Shonky
pooh-bah

Registered: 12/01/2002
Posts: 2009
Loc: Brisbane, Australia
OK Mark I just tried it on a 9436 song empeg - about 50Gb.

My force database rebuild consisted of changing the name of one playlist.

pre sift sync : 4:53
initial sift time was 18:27
post sift sync : 3:20

So not 100% like Brad but still faster. It obviously helps players with more files.

However, there was one problem! On my /drive1 it created a dodgy directory. This is a section of the inital sift...
Sifting /drive1/fids..

mkdir: created directory `_00000'
mkdir: created directory `_0'
mkdir: created directory `_00001'
mkdir: created directory `_00010'
mkdir: created directory `_1'
mkdir: created directory `_00011'
mkdir: created directory `_00012'
mkdir: created directory `_00013'
mkdir: created directory `_00014'
mkdir: created directory `_00015'
hijack: found new-style fids subdirectories
mkdir: created directory `_00016'
mkdir: created directory `_00017'
mkdir: created directory `_00018'


In the _0 directory is now a file called 10 which is the info type file. In the _1 directory is now a file 11 which is the matching mp3 file for the info file in the _0 directory.

According to a CSV export of my database (done after this song was upload I'm 100% certain) the fid of this track should have been 7172 decimal i.e. 0x1C04. So I have no idea what has happened and where it got possible 110 and 111 as the fids from. I don't understand regexps that well, so I can't see anything obviously wrong with your script.

I tried moving the files back to where they were and called them 1c040 and 1c041. I reran the sift program but this time it worked fine and put the files in the correct directory. So I have no idea what went wrong. I haven't listened to the particular song in awhile so it may not have been stored properly, although I didn't receive any sort of database errors via emplode.

Anyway thanks for doing it. It still sped up my rebuilds.
_________________________
Christian
#40104192 120Gb (no longer in my E36 M3, won't fit the E46 M3)

Top
#183448 - 09/10/2003 05:57 Re: Oh no! A serious flaw! [Re: Roger]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Yeah, I think it is the other way around. Not a big deal, that, though it can be the source of some confusion. Database rebuilds look for non-subdir'd files first, then for the subdirectory versions. At playback time, the player looks into the subdirectories first, and then for the flat file versions. I think.

Rather than theoretically "fixing" that someday, I would very much prefer that a future release of the player software merely include code to automatically write new FIDS to the subdirectories when uploading. Unconditionally would be fine, but one could gate it by requiring that at least one subdir already exists or that the player be empty.

I have modified Hijack-v344 to automatically redirect ALL acceseses to the subdirs, player or otherwise, but only when performed through the symlinks in /empeg/fids?/ (not when done via /drive?/fids/, but only if at least one (any) subdir already exists. I didn't bother with the "empty player" case. To be released soon.

Cheers

Top
#183449 - 09/10/2003 06:08 Re: Oh no! A serious flaw! [Re: Shonky]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Mmm.. Shonky, that's rather odd -- and theoretically impossible from the way the script is written. I suppose maybe /bin/bash got low on memory or something. I will modify the script to turn on swap before/after running, to alleviate any such memory pressures.

The upcoming Hijack-v344 kernel includes code to automatically keep a sifted drive sifted after future sync operations. This code also replaces some existing code that was in the http server, so overall it doesn't contribute much bloat at all.

Cheers

Top
#183450 - 09/10/2003 12:33 Re: Oh no! A serious flaw! [Re: Daria]
Daria
carpal tunnel

Registered: 24/01/2002
Posts: 3937
Loc: Providence, RI
everything except the kill -1 1 at the end (which i hoped would make init respawn the player) worked. fidsifted without use of a serial port last night.

Top
#183451 - 23/11/2003 12:54 Re: Oh no! A serious flaw! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Bump. I thought I'd bump this thread for people interested in fidsift now that the new HiJack is coming.
_________________________
Brad B.

Top
#183452 - 19/09/2004 22:24 help! fidsift [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
I decided to send fidsift.sh via TeraTerm (Hyperterminal type program) rather than ftp'ing since it would save me the extra step of hooking up an ethernet cable.

I did an "ro" and "rom" before changing to the root directory and using the "Send File" option.

Problem is, once I sent the file, it appears to have executed the file! Is this normal? It "looks" to be doing the right thing, but I didn't have a chance to be sure... I also never ran "chmod 0755 fidsift.sh".

Do I have anything to worry about? Just let it run? Why did it execute?

UPDATE: Looks like everything ran okay... but I see no trace of fidsift.sh being on the player... It opened the file but never saved it.. I've since ftp'd it to the root directory so it can be run in the future.


Edited by SE_Sport_Driver (19/09/2004 22:38)
_________________________
Brad B.

Top
#183453 - 19/09/2004 22:30 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Could this line be added to config.ini to run fidsift from HiJack?

[hijack]
;@MENUEXEC fidsift /fidsift.sh
_________________________
Brad B.

Top
#183454 - 19/09/2004 23:15 Re: help! fidsift [Re: SE_Sport_Driver]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
You probably sent an ascii dump rather than transferring the file via xmodem or similar, and since a .sh file is just a series of commands to the shell being run as an interactive shell, it just did the right thing.

What you should have done first is "cat > fidsift.sh", then ascii-dumped it. Then pressed Ctrl-D after it was done to tell cat that you're done sending. The problem with this is that the empeg's serial port uses no flow control, so if there's a buffer overrun or similar, it has no way to compensate, and you'll just lose data. xmodem or similar would be a lot better, but I don't think that there's any such stuff on the empeg to use.
_________________________
Bitt Faulk

Top
#183455 - 20/09/2004 10:23 Re: help! fidsift [Re: wfaulk]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5683
Loc: London, UK
Quote:
xmodem or similar would be a lot better, but I don't think that there's any such stuff on the empeg to use.


The developer image supports ZModem transfers. How do you think we used to get the player binaries on there ourselves?
_________________________
-- roger

Top
#183456 - 20/09/2004 12:05 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Quote:
Could this line be added to config.ini to run fidsift from HiJack?


Ahh... yes. And thank you for reminding me. I had been meaning to add that to my own config.ini once @;MENUEXEC was implemented.. like now!

Cheers

Top
#183457 - 20/09/2004 22:07 Re: Oh no! A serious flaw! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Mark, should I only have folders in my fids folder after running this? I see a bunch of loose fids in there after running it from the HiJack menu... Does the player software need to be turned off or does a "rw" need to be sent prior?
_________________________
Brad B.

Top
#183458 - 21/09/2004 04:51 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Ahhh.. yes, it probably has trouble running if the player s/w is active. Bummer.. but one could write a fancier shell wrapper for it, that does a seek and destroy on the player before running fidsift, and then restarts the player again after.

Bitt? (I'm overloaded here)

Top
#183459 - 21/09/2004 14:25 Re: Oh no! A serious flaw! [Re: mlord]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
Gosh, I don't have an empeg shell available right now, but it'd be something like:

Code:
kill `ps ax | grep player | grep -v grep | awk '{print $1;}'`

fidsift.sh
/player/bin/player


I'll double check that soon. I'm sure it's wrong right now (I don't remember the correct player path, or know that awk is installed, or, for that matter, that the empeg's ps puts PID in the first column), but should give someone a start. It should also be possible to fit it into one line for MENUEXEC, but I haven't done any research into how much I can throw at it. I think '&&'s work, so you should just be able to combine those three lines with them.

Anyway, I'll get back to you.
_________________________
Bitt Faulk

Top
#183460 - 21/09/2004 15:54 Re: Oh no! A serious flaw! [Re: wfaulk]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
I'm not sure that there's a ps command normally available. If not, then it'll have to parse through /proc/nnn/* to find the player threads.

Cheers

Top
#183461 - 21/09/2004 16:49 Re: Oh no! A serious flaw! [Re: mlord]
mcomb
pooh-bah

Registered: 31/08/1999
Posts: 1649
Loc: San Carlos, CA
Quote:
I'm not sure that there's a ps command normally available.


There is a ps, but no awk, split, eval, etc to do the parsing. This seems to work...

Code:

kill `for file in /proc/*/cmdline; do grep -v self $file>/dev/null && grep player $file>/dev/null && echo ${file:6:2} ; done`



but I am sure one of you can come up with something cleaner. Anyway, you still have the problem that the player automatically relaunches from init anytime you kill it.

-Mike
_________________________
EmpMenuX - ext3 filesystem - Empeg iTunes integration

Top
#183462 - 21/09/2004 17:07 Re: Oh no! A serious flaw! [Re: mlord]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
ps is there, at least on the developer install. I forgot, though, that it lists each thread as a different process. And awk is not available. Nor is tail. sed is, though. I'll have to figure out a good script for it. Also, you have to kill -2, otherwise it just restarts the player app. I'll get back to it later today.
_________________________
Bitt Faulk

Top
#183463 - 21/09/2004 19:15 Re: Oh no! A serious flaw! [Re: wfaulk]
Daria
carpal tunnel

Registered: 24/01/2002
Posts: 3937
Loc: Providence, RI
Quote:
ps is there, at least on the developer install. I forgot, though, that it lists each thread as a different process. And awk is not available. Nor is tail. sed is, though. I'll have to figure out a good script for it. Also, you have to kill -2, otherwise it just restarts the player app. I'll get back to it later today.


In the old pthreads (clone()) world, each thread gets a pid. The new world has some problems, it seems.

Top
#183464 - 22/09/2004 02:00 Re: Oh no! A serious flaw! [Re: Daria]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
someone can just compile busybox really quick and then you can have the killall command. i did it some time ago, but i can't seem to find the binary, and i don't have my player w/ me. maybe tommorow.

Top
#183465 - 22/09/2004 09:44 Re: Oh no! A serious flaw! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Mark, I forgot to mention (and was unable to measure) but the last hard drive upgrade that I did went WAY faster than I expected it to. And it was the largest yet - copying 60gb of data to an 80gb drive. Do you think sifted fids could have helped here? I started mid-evening and didn't have to wait for the copy over night per usual.

So even if people start using Jemplode's rebuld on PC option, it appears that fidsift has other benefits in player performance.
_________________________
Brad B.

Top
#183466 - 22/09/2004 12:39 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
That all makes very good sense -- the kernel in the Empeg just does linear searches through the directory when opening a file, and if the directory is very large, it will have to do multiple seeks/reads and longer searches to find each file..

But newer drives are faster nowadays too, which could also be a factor.

Cheers

Top
#183467 - 22/09/2004 14:51 Re: Oh no! A serious flaw! [Re: image]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
Eh, you just need to kill the lowest-numbered PID and all's well. No need to try to kill all of them.
_________________________
Bitt Faulk

Top
#183468 - 23/09/2004 12:52 Re: Oh no! A serious flaw! [Re: wfaulk]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
true. but the reason i said to use killall was because you don't have to specify the pid#, therefore achieving your goal trying to use awk, sed, etc. of course, if you have busybox, you'd have those utils, so...

btw, here is busybox


Attachments
233743-busybox.tar.bz2 (522 downloads)


Top
#183469 - 15/10/2004 20:34 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Anyone have any luck with this?
_________________________
Brad B.

Top
Page 2 of 3 < 1 2 3 >