Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Page 2 of 2 < 1 2
Topic Options
#312363 - 22/07/2008 11:45 Re: Running (mostly) from RAM (script) [Re: mlord]
andym
carpal tunnel

Registered: 17/01/2002
Posts: 3996
Loc: Manchester UK
Originally Posted By: mlord
Originally Posted By: mlord
Mine is using aufs ("another unionfs") rather than the older unionfs, but apart from that it's pretty much the same setup.

Continuing in the tradition of using the internet as a huge personal backup device, here is the little script I use to convert my notebook into a RAM-based kiosk at boot time, with read/write access still available under /.rw/ for when I need it.


Just tried this script on a little ALIX board with a 4GB CF card, running Ubuntu Server 8.04 LTS. It seems when starting up, amongst various errors from the ksyslogd init script I see this in dmesg:

Code:
[   58.058116] aufs au_opts_parse:957:mount[3486]: unknown option errors=remount-ro
[   58.065444] aufs au_opts_parse:957:mount[3487]: unknown option errors=remount-ro


I've install aufs-tools from apt but other than that it's a stock install.

mount yields the following:

Code:
/dev/sda1 on / type ext3 (rw,relatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
lrm on /lib/modules/2.6.24-16-generic/volatile type tmpfs (rw)
securityfs on /sys/kernel/security type securityfs (rw)


but cat /proc/mounts gives:

Code:
rootfs / rootfs rw 0 0
none /.rw/sys sysfs rw,nosuid,nodev,noexec 0 0
none /.rw/proc proc rw,nosuid,nodev,noexec 0 0
udev /.rw/dev tmpfs rw,relatime 0 0
fusectl /.rw/sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/disk/by-uuid/ac5dd425-ea8d-4173-86cf-fe8029fb0098 /.rw ext3 rw,noatime,nodiratime,relatime,data=ordered 0 0
/dev/disk/by-uuid/ac5dd425-ea8d-4173-86cf-fe8029fb0098 /.rw/dev/.static/dev ext3 rw,relatime,data=ordered 0 0
tmpfs /.rw/.tmpfs tmpfs rw,relatime 0 0
none / aufs rw,noatime,nodiratime,relatime,xino=/.rw/.tmpfs/.aufs.xino,br:/.rw/.tmpfs=rw:/.rw=ro 0 0
proc /proc proc rw,nosuid,nodev,noexec 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec 0 0
tmpfs /var/run tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /var/lock tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /lib/modules/2.6.24-16-generic/volatile tmpfs rw,relatime 0 0
none /dev/.static/dev aufs rw,noatime,nodiratime,relatime,xino=/.rw/.tmpfs/.aufs.xino,br:/.rw/.tmpfs=rw:/.rw=ro 0 0
udev /dev tmpfs rw,relatime 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
devpts /dev/pts devpts rw,relatime 0 0
tmpfs /var/run tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /var/lock tmpfs rw,nosuid,nodev,noexec 0 0
securityfs /sys/kernel/security securityfs rw,relatime 0 0


Any ideas Mark?
_________________________
Cheers,

Andy M

Top
#312365 - 22/07/2008 12:18 Re: Running (mostly) from RAM (script) [Re: andym]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Note that the system looks like it should be working fine, according to the stuff you posted above. But..

Quote:
[ 58.058116] aufs au_opts_parse:957:mount[3486]: unknown option errors=remount-ro

Okay, here are two other scripts that I modified to help the system run smoothly. The first is a hacked copy of the Ubuntu /etc/init.d/checkroot.sh script, from which that error above came.

The second is a replacement for the /etc/init.d/umountroot script to prevent similar complaints on shutdown.

You can use diff to compare these with the originals on your system to see what parts got hacked.

Cheers


Attachments
checkroot.sh (390 downloads)
Description: /etc/init.d/checkroot.sh

umountroot (182 downloads)
Description: /etc/init.d/umountroot




Edited by mlord (22/07/2008 12:24)

Top
#312367 - 22/07/2008 12:22 Re: Running (mostly) from RAM (script) [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
And here are some very fast halt/reboot scripts, which I linked into /etc/rc0.d/K01aufs_sysrq_halt and /etc/rc6.d/K01aufs_sysrq_reboot


Attachments
aufs_sysrq_halt (190 downloads)
Description: /etc/rc0.d/K01aufs_sysrq_halt

aufs_sysrq_reboot (170 downloads)
Description: /etc/rc6.d/K01aufs_sysrq_reboot



Top
#312368 - 22/07/2008 12:26 Re: Running (mostly) from RAM (script) [Re: andym]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Quote:
mount yields the following:
...
but cat /proc/mounts gives:
...
Any ideas Mark?

Nothing looks particularly wrong there to me -- the system should be working with those mounts.

The mount command simply parses /etc/mtab and shows only the mounts listed therein, whereas cat /proc/mounts always shows the real picture of things -- in this case with more detail than is useful. Thus the differences.

Cheers

Top
#312383 - 22/07/2008 17:48 Re: Running (mostly) from RAM (script) [Re: mlord]
andym
carpal tunnel

Registered: 17/01/2002
Posts: 3996
Loc: Manchester UK
Looks like it's done the trick.

Code:
none on / type aufs (rw,noatime,nodiratime)
/dev/sda1 on /.rw type ext3 (relatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
lrm on /lib/modules/2.6.24-16-generic/volatile type tmpfs (rw)
securityfs on /sys/kernel/security type securityfs (rw)


...and the bootlog is now clean.

I'm in over the VPN so I can't time the reboots with the new rc scripts. That'll have to wait until tomorrow.

These scripts are seriously useful, have you thought about making them more widely available?
_________________________
Cheers,

Andy M

Top
#312384 - 22/07/2008 18:08 Re: Running (mostly) from RAM (script) [Re: andym]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: andym
These scripts are seriously useful, have you thought about making them more widely available?

Yes, someday. smile

By coincidence, I'm giving a workshop session on "hacking the empeg" at OLS-2008 (Ottawa Linux Symposium) later this week. I basically picked that topic because I didn't have a better one at the time.

But something like this could be my ticket for next year, or perhaps some other conference. smile

The system shutdown is not 100% clean yet, though. It's safe, but it doesn't fully unmount everything as it should. I think another pivot_root (back to the underlying CF/disk) might be needed to correct it. Once I hack at that and get it working 100% correct, *then* I'll have something to give a workshop on.

Cheers


Edited by mlord (22/07/2008 18:12)

Top
#312404 - 24/07/2008 14:30 Re: Running (mostly) from RAM (script) [Re: mlord]
canuckInOR
carpal tunnel

Registered: 13/02/2002
Posts: 3212
Loc: Portland, OR
Originally Posted By: mlord
But something like this could be my ticket for next year, or perhaps some other conference. smile

The Open Source Convention is in Portland during late July...

Top
#312405 - 24/07/2008 14:48 Re: Running (mostly) from RAM (script) [Re: canuckInOR]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: canuckInOR
Originally Posted By: mlord
But something like this could be my ticket for next year, or perhaps some other conference. smile

The Open Source Convention is in Portland during late July...

Yeah, I think it is taking place *this week* already.

Maybe linux.conf.au in Tasmania, though.. smile

Top
#312435 - 25/07/2008 14:55 Re: Running (mostly) from RAM (script) [Re: mlord]
canuckInOR
carpal tunnel

Registered: 13/02/2002
Posts: 3212
Loc: Portland, OR
Originally Posted By: mlord
Originally Posted By: canuckInOR
Originally Posted By: mlord
But something like this could be my ticket for next year, or perhaps some other conference. smile

The Open Source Convention is in Portland during late July...

Yeah, I think it is taking place *this week* already.

It is -- but you did say next year, so I thought I'd throw it out there.
Quote:
Maybe linux.conf.au in Tasmania, though.. smile

Hmm... Oregon... Tasmania... Oregon... Tasmania.

Well, you got me beat. smile

Top
#313241 - 27/08/2008 17:13 aufs_commit.sh [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Here's a new (UPDATED AGAIN!) script for the collection.

This one, will commit changes made during an AUFS session back to the underlying persistent filesystem.

It only looks for added/deleted/modified regular files and directories, and skips changes to /tmp and /var/tmp contents.

Probably not perfect yet, but it does work for me here.
Code:
#!/bin/bash
#
# Script to "commit" aufs tmpfs changes back to the underlying persistent filesystem.
#

SCRIPT="`realpath $0`"
ROOT=/.rw
AUFS=$ROOT/.tmpfs
arg1="$1"

MIRRORDIR=/usr/bin/mirrordir
if [ ! -x $MIRRORDIR ]; then
        echo "$MIRRORDIR: not found; aborting." >&2
        exit 1
fi

COMMIT=""
if [ "$1" = "--commit" ]; then
        COMMIT="$1"
        shift
fi

function doit(){
        echo "$*"
        [ "$COMMIT" = "--commit" ] && $*
}

if [ "$1" = "" ]; then
        cd $AUFS || exit 1
        find . -type f -name .wh.\* | xargs -n1 $SCRIPT $COMMIT WHITEOUT
        doit sync

        cd $AUFS || exit 1
        find . | xargs -n1 $SCRIPT $COMMIT COPY
        doit sync

        [ "$COMMIT" = "--commit" ] || (echo ;echo "$SCRIPT: Dry run complete. Use --commit to do it for real"; echo)
        exit 0
fi

if [ "$1" = "WHITEOUT" ]; then
        cd $AUFS || exit 1
        dir="${2%%/.wh.*}"
        fn="${2##*/.wh.}"
        if [ "$dir" = "." ]; then
                [ "$fn" = ".rw" -o "$fn" = ".aufs" -o "$fn" = ".tmpfs" ] && exit 0
        fi
        if [ "$fn" = ".wh..opq" ]; then
                if [ -d "$ROOT/$dir" ]; then
                        if [ -d "$dir" ]; then
                                doit $MIRRORDIR -v --exclude-regexp '.*/[.]wh[.].*' "$dir" "$ROOT/$dir"
                        else
                                doit rm -rf "$ROOT/$dir"
                        fi
                fi
                exit 0
        fi
        if [ "$dir/.wh.$fn" = "$2" ]; then
                [ -e "$ROOT/$dir/$fn" ] && doit rm -rf "$ROOT/$dir/$fn"
        fi
        exit 0
fi

if [ "$1" = "COPY" ]; then
        cd $AUFS || exit 1
        if [ -d "$2" ]; then
                [ "$2" = "./.wh..wh.plink" ] && exit 0
                cd $AUFS || exit 1
                dir="${2%%/.wh.*}"
                if [ ! -e "$ROOT/$dir" ]; then
                        doit mkdir -p "$ROOT/$dir"
                        doit $MIRRORDIR -v --exclude-regexp '.*/[.]wh[.].*' "$dir" "$ROOT/$dir"
                fi
                exit 0
        fi
        [ "${2:0:6}"  = "./.wh."     ] && exit 0
        [ "${2:0:6}"  = "./tmp/"     ] && exit 0
        [ "${2:0:10}" = "./var/tmp/" ] && exit 0
        dir="${2%%/*}"
        fn="${2##*/}"
        [ "${fn:0:4}" = ".wh." ] && exit 0
        [ ! -e "$ROOT/$dir" ] && doit mkdir -p "$ROOT/$dir"
        if [ -f "$2" -a -f "$ROOT/$2" ]; then
                if diff -q "$2" "$ROOT/$2" >/dev/null ; then
                        [ "$2" -nt "$ROOT/$2" ] && touch -m -r "$2" "$ROOT/$2"
                        exit 0
                fi
        fi
        doit cp -dp "$2" "$ROOT/$2"
        exit 0
fi

echo "$SCRIPT: huh?  $*" >&2

Cheers


Attachments
aufs_commit.sh (180 downloads)
Description: aufs_commit.sh




Edited by mlord (28/08/2008 14:35)

Top
#313242 - 27/08/2008 17:21 Re: aufs_commit.sh [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
So with this new commit script, one can now do things like.. download/install some software, play around with it, and finally either "reboot" to discard everything, or run the script to retain the new software permanently.

Top
#313243 - 27/08/2008 17:25 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: gbeer]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: gbeer
Does this mean that at the end of a session you have the choice to commit the current ram overlay onto the CF, that it will become the new boot state?


With the newly posted aufs_commit.sh script, the answer is now YES! Except that it can be done as often as desired, not only at the end of a session.

Top
#313245 - 27/08/2008 19:00 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
In theory "cheap" SATA SSDs.

Looks pretty neat. I wonder what the price will be?

Top
#313278 - 28/08/2008 14:06 Re: aufs_commit.sh [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
I have now updated the aufs_commit.sh script in the original posting above, to reflect some bugfixes and improvements. Now requires mirrordir to be installed before using (it checks).

-ml

Top
#315589 - 27/10/2008 15:30 Pretec (true) 233X CF cards -- 32GB for US$114 [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Following up again here:

A seller on eBay is now offering Pretec brand 233X CF cards in 32GB capacity for US$114. These cards apparently read at ~35MBbytes/sec and write at ~25MBytes/sec, far faster than the early cheap CF cards.

I'll be replacing the 32GB card in my notebook with a Pretec card, to fix the agonizingly slow write speeds there, and the old slower card (reads at ~35MB/sec, writes at ~1MB/sec) will get moved to one of the empegs here.

Cheers

Top
#315616 - 27/10/2008 23:53 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: tman]
gbeer
carpal tunnel

Registered: 17/12/2000
Posts: 2665
Loc: Manteca, California
Originally Posted By: tman
In theory "cheap" SATA SSDs.

Looks pretty neat. I wonder what the price will be?


If I am reading between the lines correctly. They are getting higher speeds by striping across the chips inside the flash drive?
_________________________
Glenn

Top
#315639 - 29/10/2008 02:31 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: gbeer]
altman
carpal tunnel

Registered: 19/05/1999
Posts: 3457
Loc: Palo Alto, CA
Originally Posted By: gbeer
Originally Posted By: tman
In theory "cheap" SATA SSDs.

Looks pretty neat. I wonder what the price will be?


If I am reading between the lines correctly. They are getting higher speeds by striping across the chips inside the flash drive?


Pretty much all flash cards do this to some extent. Actual flash chips themselves are dog slow individually (well, below 10MB/sec to a single die, plus there's your flash translation layer overhead given that you can't erase single "sectors", hence there's a lot of logical remapping going on).

I think the faster ones have multiple buses and can have multiple concurrent operations per bus, plus enough buffer memory on the controller to be able to do things fairly optimally. It can get pretty hairy making NANDs go fast in my experience smile

Hugo

Top
#315960 - 06/11/2008 00:15 Re: Pretec (true) 233X CF cards -- 32GB for US$114 [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
I'll be replacing the 32GB card in my notebook with a Pretec card, to fix the agonizingly slow write speeds there, and the old slower card (reads at ~35MB/sec, writes at ~1MB/sec) will get moved to one of the empegs here.


The Pretec 32GB 233X card arrived today, and I've already migrated shrimp to it. Which in itself is a statement, because with the original 32GB card I'd still be waiting for the copy to finish now, instead of typing this post from the new card!

41MBytes/sec sequential reads, 10-20MBytes/sec writes. Good!

I'm happy! smile

Top
Page 2 of 2 < 1 2