Hi guys, I've arleady written squash, for the linux console:
http://freshmeat.net/projects/squash/

I always had in mind to port it to the Empeg. And I've spent the last week and a half with final success. Right now it can run without any interface and play both mp3's and ogg's. I'm going to be working on the interface next.

I have some questions though. What is the best way to get my player, squash, to run instead of the default player. The trick is, though that I would like the player to run first and then maybe go to the quit menu -- after the player exits I'd like squash to start up.

The reason I'd like the player to still run is mainly just so it runs dhcp, and also since squash isn't completely finished yet. I wouldn't mind having some sort of option settable through hijack, so long as it's fool proof (since for some reason my serial port stopped working, so if I don't have an IP address I'm pretty screwed). And obviously the player does have to go away, or else it eats too many resources.

Ok, so the second question is that I have found documentation for accessing the display, that's no problem. (Though I haven't found any code for rendering text, but I haven't looked hard). But I'm having problems finding info on the buttons. I just need the basic ones for right now. And do I have to (or is it better) to use it through hijack or not? (What little info I found has confused me here).

So if anyone can give me any pointers I should be able to finish this port soon. I'm very excited about it .

Oh, and excuse me if I'm stepping on anyone by wanting to replace the player. There are only two things that I have problems with the original player (and otherwise it is very very nice). One is the lack of OGG support. The second is the use of that annoying database, and the lack of good automatic song picking (I am not a fan of playlists).

So thanks for any info. I'm going to keep looking myself.

p.s. The current version of squash does not have the current empeg partial port -- if you want a copy let me know.

I always had in mind to port it to the Empeg. And I've spent the last week and a half with final success. Right now it can run without any interface and play both mp3's and ogg's. I'm going to be working on the interface next.

Awesome. Sounds pretty cool.

Oh, and excuse me if I'm stepping on anyone by wanting to replace the player. There are only two things that I have problems with the original player (and otherwise it is very very nice). One is the lack of OGG support. The second is the use of that annoying database, and the lack of good automatic song picking (I am not a fan of playlists).

I think most would agree that competition is a great thing, especially when it's open source!

Do keep in mind though, that there is a version 3 of the software that does play OGGs that is now in the alpha stages, although we may not see it beta for another three to six months (my estimation) due to the other (real, commercial) product obligations. With regard to the database, it seems rather necessary in some form to allow the very cool (IMHO) search functionality. But I haven't yet seen what you envision by "automatic song picking" either.

If you have not already, you might want to look at another attempt at a 3rd party player that has been worked on in the past called Rioplay. I'm not very familiar with it but maybe you could get some insight from it's author (grgcombs) at least in terms of obstacles overcome and experience gained. I don't believe he's released his source, but it might be worth asking nicely to see if you could look through it.

Though I haven't found any code for rendering text, but I haven't looked hard

Richard Kirby (kirkis) has written a nice graphics library called vfdlib for this. Try a BBS search and check out here.

I'll let others comment on their respective areas of knowledge, but that should at least give you some search terms to start with. Good luck!

John

Richard Kirby (kirkis) has written a nice graphics library called vfdlib for this. Try a BBS search and check out here.

Not to rain on the parade, but vfdlib is broken with v3 of the player software. I tried my hand at reverse-engineering the new format, but no luck so far.

Richard Kirby (kirkis) has written a nice graphics library called vfdlib for this. Try a BBS search and check out here.

Not to rain on the parade, but vfdlib is broken with v3 of the player software. I tried my hand at reverse-engineering the new format, but no luck so far.

I know, but I feel confident that it will get fixed before or at least soon after the time version 3 goes beta. (Given the issues with the alpha and the initial support for the new Karma / flash players, I don't think that time is exactly right around the corner.)

In any case, you can at least use vfdlib for testing / initial development.

John

Isn't it just the font support that is broken?

Sure, what, you don't like text?

Hi guys,

Thanks for your responses.

First, I used vfdlib with good success, now the display tells you the player is loading and how many songs have loaded. Then it says what song you are on. Now I'm trying to do buttons. And I assume it's an ioctl on/dev/display, from the code snippets I've seen.

However, every seems to have a copy of the hijack source -- but I cannot find it anywhere . What's with that? Am I that bad at googling?

Also, what do you mean vfdlib will be obsolete, exactly? Finally, new version of the player?? Ok, now I'm confused. I thought with Rio being closed down the developers had to go do other things?

I knew that the player -was- going to have OGG but the primary motivation for making this alternative was for that support. Now you guys are telling me that there is a 3.0 in the works? Well don't worry, I'm still going to work on this port. For the least it makes sending files to the player easier, since I don'twant to enforce that database structure for the files.

On that note, would it be a big deal to not support the database format? It would be easy for squash to read the directory, it already can. And with little modification I can have it tell if the file is an mp3/ogg or not (right now it just looks at the filename).

However, for mp3s (which is the only thing that will be in current users database list since there is no current OGG support) I would have to either read the metainfo in the database format.

Or else I would have to get id3lib to compile to read the tag directly (squash can do this on the PC, but the exectutable may grow quite a bit if I include a C++ library statically as well).

So do you all care for the ability to database tags/and or id3libsupport? Or is it ok for those that want to try it out that songs (mp3s not oggs) may all list as "Unknown".

Hmm, maybe that should have been broken down into smaller posts. Anyway can someone point me at the source for hijack (I assume that's where the docs are for the button ioctls) and/or can someone just point me to this missing documentation?

Thanks,
-Adam

p.s. I hope to have squash play-testable before friday evening, since I have a small roadtrip to take. If anyone wants a copy to try out let me know

p.p.s. I got squash to run after player rather simply by renaming the player to player.backup and then making a shell script that ran the player and then squash. (Duh. Don't know why it was puzzling me earlier).

Umm... The Hijack patch can be found by clicking on the bit that says Hijack right at the top of the page...

Well I went to that page several times. But I could not find the source. This time, undaunted, I decided to ctrl-f the html source itself. Well this time I did find the source (I'm aparantly blind), it's scattered over40 patches!

I'm ok with that. But am I still missing something? There isn't a tar of the currently source? I need to download 40 patches to get the latest version? That's fine since I can script it, but I have a feeling I'm still being a dope.

Thanks for the patience,
Adam 'the dope'

discussed here

and yeah, no current source. i think mark stopped because he hated to upload 17+ megs every update. looks like CVS is needed

See my post here for an easy way to retrieve the lastest and greatest hijack source

Thank you for the script (already did it myself though ).

Just like everyone to know I've gotten the display and buttons working. And I'm going to add volume control and start cleaning up the code next (hopefully before I take my short trip tommorrow).

Oh, and I got it reading the current database format. It happens that except for the kooky naming scheme for filenames, the database shares the same ideas of my player (that was an accident). Only some of the key names were different ("source" vs. "album").

Anyway that means you don't have to upload correctly named files to have the player work, you can use the files already on the player to try out squash. (Btw, I just assumed that a file ending in '0' might have meta info stored in a similar name, but with '1' on the end. I hope this is correct at least for recent versions).

One major issue needs to be resolved, I need to add a master list or else decipher the current player's master list (if it has one). Squash for a normal sized PC works pretty quickly without one -- and the collection would change too frequently to make a master list really useful. However on the empeg unit it's the opposite. So that's definately something that needs to be added. (Currently the player takes about a minute to start up with 3000 files).

Also, I'm having two curious problems. One is that when I run the original player from a shell script instead of directly, hijack refuses to start kftpd/khttpd. I assume it's because hijack has not loaded config.ini yet. Could it be possible to get hijack to load the ini file even without the player running?? I may not understand the dynamics completely.

The other issue is probably arm or my cross compiler. I keep getting llseek errors when using my ogg library (tremor). They are non fatal, but they do mean I don't know the length of ogg song files since the library cannot seek. I don't know where to start with that one. But right now it's a minor issue since the ogg files do play, you just don't know how long they are.

The last thing I'd like to mention is something I forgot to elaborate on earlier: the "smart picking" feature of squash. The deal is I don't like wasting my time with crafting playlists. So instead I just have squash record the number of successful plays versus the number of skips for each song. That gets turned into a rating and statistics are used to pick the next song based on the ratings' distribution. The only other thing I do is avoid picking the same songs frequently by having a count down timer. (That is a song must be "picked" by the statistics routine 10 times before it's actually picked and added to the playlist).

If you want to see what I mean, try out the linux console version of squash -- which has a much better interface at the moment than the empeg port.

Thanks for all the support guys.

Hi,

The current Hijack design is that it detects player startup in fs/exec.c and fs/read_write.c, and then reads config.ini in the player's context. That way, it can apply the various player-only macros (";@AC", ";@DC", ";@HOME", ";@WORK" ...) it supports to the player's buffered copy of config.ini, but not to anyone else's copy. Nor to subsequent reads of the file by the player (during a "sync", for instance).

I suppose that code could stay in place, and one could then also have Hijack read the config.ini file for itself on the first call to "do_execve" (a nice safe time, as the filesystems are guaranteed to be available by then).

Cheers

The llseek() errors may be due to misaligned args in memory. The strongarm CPU wants 64-bit values to always be stored on 64-bit boundaries (even multiples of 8 bytes) in memory, etc..

(?)

-ml

So instead I just have squash record the number of successful plays versus the number of skips for each song. That gets turned into a rating and statistics are used to pick the next song based on the ratings' distribution.

Cool idea, but I do have some concerns about how this concept works over a long period of time, and with multiple listeners. Having studied "smart" music algorithms for my masters project earlier this year, my group concluded that as the number of tunes becomes larger, skips vs. plays becomes less effective in truly "rating" your songs. If you happen to love a song, but not be in the mood to listen to it at a certain time, you skip it, and its chances of popping up again are decreased. If you happen to not like a song very much, but your significant other really wants to hear it, it will get played, its rating will be increased, and you'll hear it the next time you're in the car without her. I'm sure you have some algorithms built in to make it more "fuzzy" than that, but if skips and plays are the only inputs, I don't think you can approach a very robust shuffle mechanism. (Incidentally, the player app already supports skips and plays in its custom shuffle modes, but I think recording of "skips" hasn't been implemented yet, and it only gives you 5 seconds before it marks a track as "played.")

I think the way to go is to maybe use skips vs. plays as a component of a song's score, but to rely more heavily on input from the user. The simplest way would be to take in a rating from 1-10 from the remote (or front buttons if you make the interface easy enough.) This takes more manual effort to get going, but once it's done, you've really pinpointed how much a user likes songs based on their own preference, not necessarily how often they decided to listen to particular songs. Those ratings will still be subjective and based on the user's mood somewhat, but less so than a user's decision to skip/play a particular song at a particular time.

Also, whatever algorithm is used to calculate a song's score, it should use a decaying average of previous scores, so that the most recent score is weighted higher, but older scores are still taken into account somewhat. If I ranked a song as a "3" two years ago but rank it as a "7" now, the "7" should be the main component, but there was a reason for ranking it as a "3" back then, so it should also be included, just weighed less in the overall score calculation.

Incidentally, no matter how good the music selection algorithm is, I wouldn't use a player app that didn't have custom playlist functionality built in. Sometimes you just want to select a playlist, an album, etc and not worry about whether the player will play what you want.

Just some food for thought. Sounds like you've got something very cool brewing. Now if we could only figure out how to make the player app's visuals work with it.

Also, what do you mean vfdlib will be obsolete, exactly? Finally, new version of the player??

No, it's just that currently, vfdlib knows how to render the fonts that come with version 2 of the player, but not the fonts that come with version 3.

If you would distribute the old fonts with your player and use those, vfdlib will keep working.

On that note, would it be a big deal to not support the database format?

It would be a huge plus if the player could use the existing data as-is. It should not be to hard, as there already is open source software that reads and writes the needed data structure.

Or else I would have to get id3lib to compile to read the tag directly (squash can do this on the PC, but the exectutable may grow quite a bit if I include a C++ library statically as well).

You could use libid3tag which is written in plain C rather than C++. But there's no need to do this if you can read the existing tags and database.

Pim

>Currently the player takes about a minute to start up with 3000 files.

I wonder if maybe the dentry cache needs some tuning on the Empeg.

The long long database rebuild times have always bugged me, as has the amount of time (very similar) it takes for a complete root playlist download from the khttpd server. These are mostly just repetitive directory lookups, followed by disk reads.

If we prevented fs and block-layer read-aheads on all accesses to the metadata files, it would probably also show a huge impact.

EDIT: I wonder if our kernel has DIRECT_IO in it? This would also help big time here.

Mmm...

Oh, and I got it reading the current database format.

Cool. Make sure, though, that you can find fids both using the
old flat directory structure and the newer hierarchial structure
(/driveX/fids/_YYYYY/ZZZ).

I need to add a master list or else decipher the current player's master list (if it has one

You mean the root playlist? It's fid 100.

Pim

So instead I just have squash record the number of successful plays versus the number of skips for each song. That gets turned into a rating and statistics are used to pick the next song based on the ratings' distribution

wont that lead to eating into itself? itll tend to play first what most often played which would make the the next time it plays first what it played last time potentially leaving some music unplayed for huge amounts of time. and the gap would just continue to get bigger.
maybe theres more to it that im not seeing

The long long database rebuild times have always bugged me

This calls for another shameless plug for mp3tofid. mp3tofid will build the database on your PC which you then can upload to the player. This is much faster than having the player build the database.

I'm currenty working on a new version that will support not only mp3, but also wav, wma, ogg and flac files. Playing ogg and flac will require a v3 player, off course.

As far as I know, this will be the only way to get flac running on the player, as the version of emplode that comes with v3.00-alpha3 does not support uploading flac files, even though the player can play them.

Pim

If you would distribute the old fonts with your player and use those, vfdlib will keep working.

I have strong suspicions that doing that would be jumping over a line in the sand.

Or make your own fonts that use the same file format...

Well, I assume that once v3 is beta then vfdlib support will likely be forthcoming. I'd also point out that distribution shouldn't be necessary anyway - the vast majority of people have a source for the font files anyway that they could back up.

Maybe Rob is happy enough for soon-to-be-obsolete font files to be distributed with a third party app. I don't know. (Obviously.) I wouldn't want to make that assumption.

Well when squash becomes good enough of a port to stand on it's own, will having squash named '/empeg/bin/player' and having squash read 'config.ini' be sufficient to trigger the kernel?

Even so, what if I still want to use a script to launch squash (It makes the most sense to use other small utitlites to do one-time things like setup the network card, etc.). So the kernel would need to be modified. How receptive are hijack patches? No point in making on if I'll be the only one with that kernel.

Oh I just thought of something I will try. I wonder if I issue a shell command (not like cat) to read config.ini would work. The problem with the shell script seems to be that hijack doesn't detect children reading config.ini only the parent process. I'll try this out soon.

This could be it. Should I be compling tremor with a special config flag? I looked at the --help but didn't see anything that was interesting looking. Or is this a bug I should report to xiph? (Or both).

Well sorry for sounding pushy. Anyway, help is appreciated greatly .

Adam

Tony,

I'd like to start off by saying two things: that the whole point to the learning routine is to just improve over a regular shuffle. Not to be the pancea of all time and space. Also, squash will -eventually- have all those tweaking playlist thingamabobs. But there is little point in doing that until the baseline works (i.e. "adv." shuffle, and rest of the interface and guts).

Ok, so anyway. To address your concerns: Statistics already can be gathered for multiple people. But the only interface currently for switching people is to restart the player.

Concerning skipping a song you like or listening to a song you hate. That's why it's called statistics? You are likely to listen to the former more often than the latter, but not garuanteed to always listen to or to always skip. The more you like or dislike the more extreme the rating. I mean, isn't this all a definition how statistics work? There are people who have no children or 10 children even if the average is 2.5. That's just life.

As for manually scoring. I don't mind implementing this, but just like custormizing the playlist, this is alot of work for the user. (So in other words, I'd rather not bother with it personally -- though I understand others wanting to). So I don't have that feature in just yet, but don't worry it will go in eventually (feel free to contribute ).

Concerning decaying averages. You have a good point. However automatic data collection is -so slow- already. It has been 3 months and I have only listened to 70% of all my music once (of course 5% I skipped). So maybe in another month I'll be start round two of skip vs play. Decay for this kind of input rate would need to be very small or else you will lose all of you data. I will not be implementing decay soon, since it could easily destroy the whole system if done badly -- but I suppose it could be added safely if you are careful (I, as a user, will just turn it off though).

Again I will eventually have playlist manual manipulation, but that's like version 1 going to version 2 features. Not the version 0 to version 1 features.

Lastly remember that squash effectively works like an improved shuffle -- so very good songs just appear more frequently than good songs. And bad songs can appear too, but less frequently. See another post I'm about to right for some numbers. The important part is that it's shuffle of all the music, not some magic dijinn that knows exactly what you want to listen to right now.

Statistics can only play the songs you like better more frequently and vice versa. And the level of data that is being collected right now has no way of knowing what you are -in particular- would like to hear. Just what you in general like more than otherwise.

Thanks for your comments I like discussing this. And I plan on making it my master's thesis, so this helps me later in defending claims.

p.s. Thinking about the decaying average more. This seems like overkill, even if the data was collected at a quicker rate somehow. The current goodness function is: (p-s)/(p+s+1). Which "decays" if say you played the song the first 10 times, but then decide you hate it. Anyway, it may still be worthwhile -- but I think it needs to be very small.

Have you seen mserv? It relies on user ratings of songs, but does a very good job of weighting the shuffle towards good songs once it knows which they are. I used it all the time before I had any Empeg gear.

Peter

I may not have understood what I was looking at. But it looked like the current copy of vfdlib has the font's just as part of the code (static arrays). And this is added to the fact that the small font (the only one I have used so far) doesn't look the same as the player's small font (which is smoothed out). I don't mind including code since I already have to build statically (making a hefty 1meg executable!).

Ok well right now squash supports any "database" format as long as music is kept as plain files (just like the current player does). It does take a while to load the first time (since it must read the first bytes of the file to determine if it really is a mp3 or an ogg). But I just added a master list feature that generates a masterlist of all songs on the player. Then the next time we just use that list instead of walking the filetree and probing each file for songiness.

Load time was 2 minutes (still is for first time), now it's down to like 5 seconds.

Also, for any file that ends in a 0, I read a corresponding file that ends in a 1 for the meta info. (Normally I read a corresponding file ending in ".info"). That pretty much fills up support for the current database format. The only thing missing is interpeting the playlists, which is pointless since squash doesn't have playlists.

libid3tag I had problems with concerning special characters like 'ö' and a few other problems (if I remember correctly). Anyway, you have plenty of options for mp3 files:

1) You can create the ".info" files on the PC and then transfer them over.
2) You can rely on the data in the old player's database which squash will read.
3) You can use oggs!

I don't know what current "tuning" empeg has had to it's kernel. I just know that very little writing is enough to make squash skip. (Squash relies on the operating system to cache file reading sufficiently -- since it would be difficult to do this with the current decoding libraries, and the OS -should- be the one doing it). Right now squash is writing statistics to disk a few seconds after the song plays. The writes don't seem to get flushed until after about 30 seconds. But then you get a little hiccup in the song, because there isn't enough buffered data to read.

(The skip does not always happen, so we are probably close to enough read-ahead buffering). A few more kilobytes would probably be sufficent. But I may not be the best judge on what is needed in the kernel .

(The current player only uses 7 or 9 meg with 3,000 songs, and squash uses about 4 or 5 (which I have a scheme to reduce further if need be), so there are definately kilobytes to spare).

As for manually scoring. I don't mind implementing this, but just like custormizing the playlist, this is alot of work for the user.

My ideal interface would be hitting two buttons on my remote, the first saying "rate this song" and the second being a number from 0-9. With my steering wheel remote, I could easily rate songs as they're played, which is much less effort than manually creating playlists. More effort than your skips/plays setup, but not much.

The important part is that it's shuffle of all the music, not some magic dijinn that knows exactly what you want to listen to right now.

Yeah I understand that. I just think numerical ratings offer more flexibility and precision for getting a better shuffle with more "good" songs. With numerical ratings, after one pass through your entire collection, you'll have songs rated 8 or 9 showing up many more times than songs rated 2 or 3. With plays/skips, it seems you need to go through several times to build up a skipped vs. play count that would be meaningful in separating good vs. bad. I think you may have alluded to this with your point about the "input rate." I think it all boils down to a tradeoff between effort and accuracy, but both methods are better than a plain old random shuffle... Using both would be the best of both worlds.

In any event, I think your efforts to develop an open-source player for the empeg are commendable, and I'll be following your progress closely. Maybe if this thing takes off I will give manual ratings a shot myself.