I did some pretty massive ID3 tag editing on my backup mp3 hard drive, and want to re-upload my whole collection. I will be using ethernet, but was wondering which would be faster, or would it be the same.

Ethernet via Jemplode or Ethernet via emplode.

Good question.

In theory, for large single uploads, they should be pretty much equal in speed. I actually did some speed tests between the two (helping Mike with HSX-109 support in Jemplode) and discovered that they were more or less identical for the uploading of each individual song to the car player via ethernet.

I did not do any testing to see about the between-song latency and database-fiddling parts. So I don't know about that aspect.

Currently they should be _nearly_ identical ... The catch will be when I decide to hunker down and see if I can add Hijack _upload_ support ... I don't know if this is possible, but it would be really slick. I'd kind of have to side-step the protocol a bit to make it work, though.

Mike

Yeah, Hijack should be noticeably faster on upload/download, especially for bulk music (lots of megabytes). And the funny part is, it's not doing it the fast way yet.. (it should use mmap() instead of read/write).

Cheers

-ml

I imagine the speedup would be roughly equivalent to the switch from Empeg protocol download to Hijack download.

The Empeg guys should seriously consider an FTP-like model (control channel separate from download channel) and just dump raw bytes for the actual upload and download.

Mike

We do this already; Peter added it a while back. It's call fast channel and just has a separate TCP pipe the file data gets shoved down; I *think* it's been there since v1. ISTR it's only in one direction (ie, downloads to the player).

mmap() has issues. Fast channel used to use mmap(), but kernel bugs in 2.2.14 caused it to be unstable and result in lockups, hence it's not in the shipping code. Peter can probably fill you in on this one... it almost maxed out ethernet before falling over.

Hugo

The Empeg guys should seriously consider an FTP-like model (control channel separate from download channel) and just dump raw bytes for the actual upload and download.

That's exactly what 2.0 does (on Ethernet, at least), and why it's faster than 1.0.

Peter

PS. Some people use "download" to mean the PC-to-empeg direction, and some to mean the empeg-to-PC direction. ("Can I download songs from my player?"). Following at least two important Sonic Blue meetings where people got confused, I nowadays always use the words "store" and "fetch" to make it obvious which direction I'm talking about.

it should use mmap() instead of read/write

The 2.2.14 VM (and other VMs up to 2.2.17) fell over every time I tried that. I believe that 12Mbyte machine, no swap, 40Mb Orb track mmap'd, 100% CPU usage (Ethernet and PIO disk) constitutes "severe VM pressure" within the meaning of the act.

http://empeg.comms.net/php/showthreaded.php?Cat=&Board=bigs&Number=20328

Peter

PS. Hugo, you swine, stop posting my posts before I do!

Really?! Is this in the emptool code?

Mike

I responded to Huge's with the same response: "Really?? Does emptool have the code for this?" I never saw it anywhere ... This could really speed up storing. (p.s. i like the terminology and agree that the others are confusing)

Mike

Mmm... yes, userspace mmap() could do that in 2.2.xx..

For the "store-to-player" direction, I suppose the raw TCP channel that we have (Hijack, anyway) is likely close to as fast as it can get (except that there are two wasted copy_to/from_user() moves on that path).

For the "fetch-from-player" direction, I don't think anything is using sys_sendfile() yet.. which can eliminate the copy_to/from_user() overhead, even for a userland process.

Cheers

"Really?? Does emptool have the code for this?" I never saw it anywhere ...

In lib/protocol/protocolclient.cpp in WriteFidFromFile() where it goes GetFastConnection() at about line 576.

Peter

For the "fetch-from-player" direction, I don't think anything is using sys_sendfile() yet...

Mmm. When I was first writing this, the sendfile man page I had didn't mention that the source fd wasn't allowed to be a socket -- it was a bit disappointing to find that out.

There's no fundamental reason, of course, why sendfile couldn't optimise away the copies in the read-from-socket-to-mappable-file case, but no-one here has had the kernel-VM-hacking bravery to try it.

Peter

Cool -- Apparently I just look right past these method calls ... I'll put this one in.

In looking back at the comments you and Hugo made, are you saying that it _currently_ uses mmap and catches itself on fire, or it _used_ to use mmap and catch itself on fire? If it works now I'll just do a straight port, but if you don't think it will work out, then maybe I can use (read: steal) the same underlying design that you guys have but return a Hijack FTP connection instead of a fast tcp connection ....

Thanks Peter
Mike

Thanks for all the replies, it's good to see there's a lot of developement on this player to make it faster. I may just wait a little longer for the next jemplode release to see if anything is implemented in that release in regards to speed. From my personal test, when I do large file transfers via ethernet and emplode2b11, I get errors after about 2 Gigs, so I was hoping that jemplode w/ hijack would be more a little more stable, which it does look like so far. Thanks again.

it _used_ to use mmap and catch itself on fire

I don't think even the alpha team got a version that used mmap. The way it works now, er, works. You get it for free if you use WriteFidFromFile.

Peter

Sadly, since I rewrite all your code in Java, I don't get anything for free Thanks for the info...

Mike