#297177 - 17/04/2007 19:03
Quick linux question
|
addict
Registered: 29/06/2002
Posts: 531
Loc: Triangle, VA
|
im currently working on a bash script to backup my ubuntu servers web pages and mysql db. Im using a zip drive cause it just happens to have one and figure would be fun to try to do it..(Dont have alot of scriptting experience) Ive gotten my script to check if the zip is mounted, if not mounts it. Then I even have it strip the % of the disk in use from df. Now my question is:
Is there a way in linux to create a completely useless file at a specific size?
I am looking for an easy way to test my script with the amount of disk in use without having to copy random files over to this disk. would like to just be able to create a file that uses say 90% of it. A LONG time ago I had a program for DOS that did this Resize.exe i think it was called but long since lost that file. Thanks!
_________________________
-D
Modifying and Tweaking is a journey,
not a destination................................
MKIIa : 60gig - 040103286 - Blue - v2 + PCATS tuner
MKIIa : 20gig - 040103260 - Blue - v3a8 + Mark Lord Special Edition Cherry Dock
|
|
Top
|
|
|
|
#297178 - 17/04/2007 19:17
Re: Quick linux question
[Re: SonicSnoop]
|
carpal tunnel
Registered: 27/06/1999
Posts: 7058
Loc: Pittsburgh, PA
|
dd if=/dev/zero of=foo bs=1024 count=N
Where N is the size in kilobytes of the dummy file you want to create.
|
|
Top
|
|
|
|
|
#297179 - 17/04/2007 19:17
Re: Quick linux question
[Re: SonicSnoop]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Quote:
Is there a way in linux to create a completely useless file at a specific size?
Some Linuxes will have a utility called mkfile that does exactly that.
If yours does not, use dd.
Code:
dd if=/dev/zero of=useless-file bs=x count=y
This reads from /dev/zero (which produces an endless string of zeroes) and writes them to useless-file. The x argument to bs should be a number indicating the "block size" in bytes and the y argument to count is the number of blocks you want to copy. So if you want a 10MB file, I'd probably do
Code:
dd if=/dev/zero of=useless-file bs=1048576 count=10
|
|
Top
|
|
|
|
|
#297180 - 17/04/2007 19:28
Re: Quick linux question
[Re: wfaulk]
|
addict
Registered: 29/06/2002
Posts: 531
Loc: Triangle, VA
|
Thanks guys!!
_________________________
-D
Modifying and Tweaking is a journey,
not a destination................................
MKIIa : 60gig - 040103286 - Blue - v2 + PCATS tuner
MKIIa : 20gig - 040103260 - Blue - v3a8 + Mark Lord Special Edition Cherry Dock
|
|
Top
|
|
|
|
|
#297181 - 17/04/2007 19:43
Re: Quick linux question
[Re: SonicSnoop]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
Quote:
im currently working on a bash script to backup my ubuntu servers web pages and mysql db.
A command called mirrordir can be your friend here..
|
|
Top
|
|
|
|
|
#297183 - 17/04/2007 20:19
Re: Quick linux question
[Re: SonicSnoop]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
One thing I would check is to see if it's faster on the zip drive to create one big file or multiple small files. I don't really know for sure, but I wouldn't be at all surprised if you found a significant difference. You might also want to check to see if it's faster to create the file directly on the zip disk or create it in a temporary location and move it over.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#297184 - 17/04/2007 20:22
Re: Quick linux question
[Re: SonicSnoop]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Also, check out the ziptool program, if you haven't already. It's been a long time since I dealt with zip drives, but I seem to recall the tool having some nifty features.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#297185 - 18/04/2007 00:17
Re: Quick linux question
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
Quote:
One thing I would check is to see if it's faster on the zip drive to create one big file or multiple small files
Under Linux, there should not be much of a difference, if any. Unless the one big file is also compressed (usually it is compressed, 'z' or 'j' flag for 'tar'), in which case it will be faster simply due to less data to be copied.
Of course, that's only for the initial copy. Subsequent incremental/differential backups will be much faster using individual files normally, especially with mirrordir.
Cheers
Edited by mlord (18/04/2007 00:18)
|
|
Top
|
|
|
|
|
#297186 - 18/04/2007 01:19
Re: Quick linux question
[Re: mlord]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
You have to remember that both seek time and transfer rate on zip drives are really, really, slow. (29ms and 7.5MB/s, best case, according to specs.) The additional overhead of writing metadata for each file can become significant at those speeds. (Honestly, I'm not sure where metadata is kept in ext2/3. If I'm reading the spec properly, I think it's kept separate from the data blocks.) Of course, that assumes that he's going to use ext2 on the zip disk. If, for some reason, like so he can read the disk from other OSes, he wants the zip disk to be FAT, doing a file-by-file copy is probably a bad idea, as he'll lose all that metadata, plus have filename issues (the latter of which could be overcome by using VFAT, I suppose). This is actually important to point out, as I'm sure that zip disks still come preformatted, so he'll probably want to reformat it with ext2/3.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#297187 - 18/04/2007 01:57
Re: Quick linux question
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
Quote:
You have to remember that both seek time and transfer rate on zip drives are really, really, slow.
I also remember an awful lot of trivia about how Linux performs I/O, block device I/O in particular. It uses this thing we call the "page cache" in combination with "lazy writes". So the I/O is quite minimal, so long as it all is queued up within a short interval.
-ml
|
|
Top
|
|
|
|
|
#297188 - 18/04/2007 02:32
Re: Quick linux question
[Re: mlord]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Yeah, but zip disks are removable media. Which I guess doesn't make any difference under Linux. There's no bypassing the cache, is there?
Anyway, I think it's still worth benchmarking to make sure. I remember using zip disks under Linux ages ago and it was really painful. Marginally faster than a floppy, but 70 times the capacity. Any performance enhancement would have been very welcome.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#297189 - 18/04/2007 02:38
Re: Quick linux question
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
Quote:
Yeah, but zip disks are removable media. Which I guess doesn't make any difference under Linux.
Zip disks are "block devices", IDE/ATAPI drives actually, and are managed by the IDE drivers that Gadi & I once wrote for Linux. "Removable" doesn't make them any less efficient in this case.
Quote:
There's no bypassing the cache, is there?
Not without lots of effort. We have a misguided feature in the kernel that even Linus regrets, which could be used to bypass the page cache when less efficient I/O is mistakenly preferred (things like Oracle database software reputedly use it).
Cheers
|
|
Top
|
|
|
|
|
#297190 - 18/04/2007 02:57
Re: Quick linux question
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
Quote:
I remember using zip disks under Linux ages ago and it was really painful. Marginally faster than a floppy, but 70 times the capacity.
They're just as slow on a Mac (I have a G3 B&W here with a Zip drive), and under Windows: about half the speed of an empeg upload.
Pretty much any software can keep them saturated: 372KBytes/sec for one of the Zip drives I have here. Uses PIO mode-0 only, with 500ns minimum cycle times. Later revisions might be faster (?).
The one in the G3 box might be newer, but I don't have a Linux CD for that box right now.
Cheers
|
|
Top
|
|
|
|
|
#297191 - 18/04/2007 07:39
Re: Quick linux question
[Re: mlord]
|
addict
Registered: 11/01/2002
Posts: 612
Loc: Reading, UK
|
or, in a similar vein, look at this: http://www.mikerubel.org/computers/rsync_snapshots/(which also links to mirrordir) And my suggestion is to install "rsback" using apt-get. It uses the algorithm described and I use it happily. Basically on day 1 it will create a backup mirrored directory tree - so far so good. On day 2 it uses hard-links to create a 2nd copy of the tree. This uses (almost) no space. Then it removes and updates files in the original tree for the day 2 backup - this uses as much space as the 'delta'. It should also be very efficient at doing incremental copies - rsync is highly regarded. The good thing about it is that you have a complete snapshot history of your tree for browsing as a normal filesystem. For safety's sake you can make the backup disk read-only during the day and only switch it to read-write whilst the backup is running. It's better (IMHO) to use something that's seen widespread use and wrap it in some script to tailor it for your system than to write something from scratch and potentially get it wrong  The other nice thing about these is that they work over the network. What I've been thinking about doing is finding someone elsewhere in the world and set up a reciprocal backup (with encryption) for true off-site backups.
_________________________
LittleBlueThing
Running twin 30's
|
|
Top
|
|
|
|
|
#297192 - 18/04/2007 11:47
Re: Quick linux question
[Re: LittleBlueThing]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
Quote:
rsync is highly regarded.
Highly regarded for network copies over a relatively (to the CPU speed) slow connection, with intelligent/fast CPUs at both ends of the link.
Lousy choice for use with a Zip disk, though. Rsync requires reading/summing the existing data as well as the new data, so it will have to do slow reads from the existing data on the zip disk, which would only slow down this process.
Cheers
|
|
Top
|
|
|
|
|
#297193 - 18/04/2007 11:48
Re: Quick linux question
[Re: LittleBlueThing]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Quote:
On day 2 it uses hard-links to create a 2nd copy of the tree. This uses (almost) no space. Then it removes and updates files in the original tree for the day 2 backup - this uses as much space as the 'delta'.
That's very cool. I'll have to remember that one.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#297194 - 18/04/2007 11:51
Re: Quick linux question
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
re: rsync
I suppose it might be clever enough to read back only files whose size/timestamp has changed, though, so it could be good here after all. One would have to try it to find out, I guess.
Cheers
|
|
Top
|
|
|
|
|
#297195 - 18/04/2007 12:07
Re: Quick linux question
[Re: mlord]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
You can choose whether rsync checks file times/sizes, by default (in mirror mode at least) it won't bother to look at the contents of a file if the update time and size are the same in both folders.
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#297196 - 18/04/2007 14:15
Re: Quick linux question
[Re: wfaulk]
|
addict
Registered: 29/06/2002
Posts: 531
Loc: Triangle, VA
|
yeah the disks are showing up as using vfat for the type in mount. I hadnt even thought about the format on them. I dont have any plan on using these disks in a system other then my server so ill go ahead and look into putting ext2/3 on it.. it seems my linux system it self is using ext3, so ill probobly go with that for the disk, not sure if it even matters.
_________________________
-D
Modifying and Tweaking is a journey,
not a destination................................
MKIIa : 60gig - 040103286 - Blue - v2 + PCATS tuner
MKIIa : 20gig - 040103260 - Blue - v3a8 + Mark Lord Special Edition Cherry Dock
|
|
Top
|
|
|
|
|
#297197 - 18/04/2007 14:24
Re: Quick linux question
[Re: LittleBlueThing]
|
addict
Registered: 29/06/2002
Posts: 531
Loc: Triangle, VA
|
Im a little confused as to which to use, rsync or mirrordir? I personally was just gonna zip em up and throw em on the disk but if one of these would be better i have no problem playing with it and trying them out just confused which one to be trying  I like the idea of being able to do the incremental/differential type backups, would save space and allow me to do more backups then the way I had planed. Also with these it seems its just doing my html files right? im still going to have to export my mysql dbs to a .sql file and copy that over right? or do they include some kind of db backup as well? Thanks guys again for all this info!
_________________________
-D
Modifying and Tweaking is a journey,
not a destination................................
MKIIa : 60gig - 040103286 - Blue - v2 + PCATS tuner
MKIIa : 20gig - 040103260 - Blue - v3a8 + Mark Lord Special Edition Cherry Dock
|
|
Top
|
|
|
|
|
#297198 - 18/04/2007 14:39
Re: Quick linux question
[Re: SonicSnoop]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Quote:
im still going to have to export my mysql dbs to a .sql file and copy that over right? or do they include some kind of db backup as well?
You should probably read the MySQL manual section about backups. For versions 3 & 4, that's here. For version 5, here.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#297199 - 18/04/2007 14:54
Re: Quick linux question
[Re: wfaulk]
|
addict
Registered: 29/06/2002
Posts: 531
Loc: Triangle, VA
|
Im sorry yeah I already have a command for mysqldump to get the dbs in .sql format I ment did these other programs include a db backup feature. I guess what I could do is just dump the dbs to a dir as a sql file and include that dir in the backup
_________________________
-D
Modifying and Tweaking is a journey,
not a destination................................
MKIIa : 60gig - 040103286 - Blue - v2 + PCATS tuner
MKIIa : 20gig - 040103260 - Blue - v3a8 + Mark Lord Special Edition Cherry Dock
|
|
Top
|
|
|
|
|
#297200 - 18/04/2007 15:05
Re: Quick linux question
[Re: SonicSnoop]
|
addict
Registered: 11/01/2002
Posts: 612
Loc: Reading, UK
|
For mysql, in cron I run this command: Code:
/usr/bin/mysqldump -u mythtv -pmythtv mythconverg -c | /bin/gzip -9 > /everything/Backup/teak/mythtv_backup-$(date '+\%a').sql.gz
The date thing basically writes to files with the day in the name: Code:
mythtv_backup-Tue.sql.gz
mythtv_backup-Mon.sql.gz
mythtv_backup-Sun.sql.gz
mythtv_backup-Fri.sql.gz
mythtv_backup-Thu.sql.gz
mythtv_backup-Wed.sql.gz
mythtv_backup-Sat.sql.gz
So this overwrites after a week. Whilst I'm at it... for rsback I do this: Code:
0 2 * * 1-7 /everything/net/bin/rsback -v work-daily >>/var/log/rsback/work-daily.log
0 4 * * 6 /everything/net/bin/rsback -v work-weekly >>/var/log/rsback/work-weekly.log
0 5 1 * * /everything/net/bin/rsback -v work-monthly >>/var/log/rsback/work-monthly.log
Then the config file contains: Code:
tasks = work-daily \
work-weekly \
work-monthly
and Code:
[work-daily]
mode = rsync
source = /space/
destination = /space-backup
rotate = daily 7
lock = work-weekly work-monthly
# if_locked_retry = 3 30min
[work-weekly]
mode = link
source = /space-backup/daily.0
destination = /space-backup
rotate = weekly 12
lock = work-daily work-monthly
[work-monthly]
mode = link
source = /space-backup/weekly.0
destination = /space-backup
rotate = monthly 6
lock = work-daily work-weekly
I'll let you judge whether this looks suitable for you - I think it's pretty straightforward
_________________________
LittleBlueThing
Running twin 30's
|
|
Top
|
|
|
|
|
#297201 - 18/04/2007 15:28
Re: Quick linux question
[Re: SonicSnoop]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Quote:
I ment did these other programs include a db backup feature
To say less than LittleBlueThing: no. They copy files and that's it. You'll have to make sure that you have plain files that you can back up. You can't back up the database files directly because the database system may have them in an inconsistent state. That is, you can back up the files, but if you had to restore those files, you'd have to restore the database as if the computer had crashed.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#297202 - 18/04/2007 16:13
Re: Quick linux question
[Re: LittleBlueThing]
|
addict
Registered: 29/06/2002
Posts: 531
Loc: Triangle, VA
|
Thank you LittleBlueThing. That will help out alot.. Ill go ahead and try that out on my system.. It always amazes me how what takes me lots of commands & multiple lines to do something someone always has this one line cleaned up code that does more.. If I can just stay constant enough in playing with linux one day I hope I might be able to do that
_________________________
-D
Modifying and Tweaking is a journey,
not a destination................................
MKIIa : 60gig - 040103286 - Blue - v2 + PCATS tuner
MKIIa : 20gig - 040103260 - Blue - v3a8 + Mark Lord Special Edition Cherry Dock
|
|
Top
|
|
|
|
|
|
Previous Topic
Index