NekoWiki: LinuxOnCompactFlash

My attempts to wade through the various bits of information, bad and good, available on the net.

Important Update!!
Flash types
Wearing out NAND
Wear Levelling
Adapters
File system choice
Journal stuff
Swap stuff
Kernel IO Scheduler
Making things bootable
Attack Plan
1. /etc/sysctl.conf
2. Partition layout
Boot times
Links

Important Update!!

UPDATE

Okay guys, I have an interesting new data-point for this CompactFlash?-SSD scheme. It goes a little like this: don't bother.

I've been in China for about four weeks now, and I think I've been using this setup for maybe 2 months. And the other day, sumomo started to run out of writes on the CF card, resulting in lockups.

Sigh.

So, she's in a state of undeath right now. All the data is still there, except for a bit of filesystem damage when blocks were suddenly not being written as they should. Incidentally, the kernel ext2 driver panics a bit and causes the filesystem to be emergency-remounted read-only. No writability means I'm not able to start X, which is a bit of a bummer.

Anyway, bottom line:

No, you can't easily turn a CF card into a budget SSD. You can use it for an embedded system, if you carefully set everything up so there's practically NO writing to disk, like a LiveCD? - but my laptop doesn't really have the RAM to keep the whole system in memory.

You can't expect to use it as a general-purpose computer. Even though I'd been fairly careful in reducing writes, it still wasn't enough. Lockfiles, logfiles, caches, user preferences, mini-databases, all these tiny things contribute a lot to the write limit, and if I'm using it as a general purpose computer, I'm periodically copying video, mp3s, installing software, removing software, even -configuring- software will result in some writes.

So. It sucks, but there you have it. I'm not unhappy with my experience; it was interesting, and I get to tell you guys about it ;)

-Neko

Flash types

NOR flash, can be programmed a byte at a time, but is slow.
NAND flash, more gooder. Writes are fast, but require entire blocks to be rewritten.

Wearing out NAND

Reading from NAND is easy.
Writing to NAND is not so easy. An entire block must first be erased, then the 0's (or 1's) for that block replaced.

Wear Levelling

CompactFlash? presents itself with a traditional disk interface, and does it's own wear-levelling.
For this reason, it is better to use 'traditional' file systems on CompactFlash?, rather than specialist wear-levelling file systems.

Adapters

The card reader in my monitor can pull data from the card at 10M/s.
The PCMCIA adapter managed slightly less, if I remember right.
The 'ircube' CF-44pin IDE adapter was a hassle. It has two card slots, which is great in theory, but meant an extra permutation when trying to figure out which way was 'up'. It also had all 44 pins in place, annoying as the cable in my laptop has a blanked-out hole for the appropriate pin which is missing from the hard drive. I wasn't able to get this one to work, no matter how I permuted it in the USB-44pin IDE adaptor (which allows for all 44 pins to be inserted)
So, I bought another one. This one was cheaper, with only one socket (perhaps both my laptop and usb caddy were only expecting a single interface). Although the picture presented 43 pins, the actual device came with the full 44 again. Man. Anyway, it worked fine in the usb caddy, so I bent the appropriate pin away and attached it to the laptop. It recognised it as a hard drive, and boots, which is a huge relief. However, initial tests in the DSL install suggest a max transfer speed of 5.5M/s. Not great, although the lower hopefully access times will help. I'm going to restore the original system and see how that compares.

File system choice

Filesystems:

JFFS2: Not good. Designed for raw MTD interfaces. It does it's own wear levelling, meaning it stores stuff on disk as modifications to files, along with a sequence number and checksum. At start up, it iterates over the entire device looking for blocks and piecing things together. So it's not fast, and not ideal for what we want, and the wear-levelling would just be redundant.
Ext2: Maybe.
Ext3: Maybe. It has a journal. This may or may not be a good thing.
XFS: Again, Maybe. Some interesting looking XFS tweaks in a gentoo forum post here.
Something funky involving UnionFS?: Possibly... Would certainly be awesome to turn the thing into a knoppix-like setup, or even figure out how Puppy does it's thing, and present a dialog on shut down letting you commit changes.... although, remember, we're limited in RAM space...
Btrfs: It's still an alpha filesystem, but it could be very good eventually - it has an -o ssd option that clusters writes together.

Options:

With all cases, mounting the file system with noatime is a good idea, to reduce writes.
And you'd be crazy not to go with tmpfs for /tmp
/var is trickier though, since we've got e.g. /var/cache to deal with. But /var/log should definitely point to a tmpfs.

FUSE combinations:

What goodies can FUSE give us?

Block sizes:

Some slashdot post mentioned that the block size of NAND is around 128K, possibly even 256K. So if the block size of your underlying filesystem is only 4K, and the block size of ATA is 512B by default, you're in trouble because you'll see a lot of NAND block clear/rewrites.
- Ergo, we should investigate a filesystem with a gargantuan block size and any kernel parameters that let us write in nice big chunks.
- If we can do that, we should also ideally start the partition on a multiple of the block size, e.g. at 8M (Suggested by Samsung PDF study below)

Journal stuff

Journalling only metadata, and on the same device as our root, would be a bad idea. It would increase writes.
Journalling all data might work out for us, IF we put it somewhere else...
- Ramdisk? We lose the 'reliability' part. But it'd be helluah fast and would cut down on writes.
- Onto the second smaller CF card? It would wear out quicker, but that's okay.
Note that all of this is assuming that the journal is written to FIRST, then 'flushed' to the real device. We need to confirm this.

Swap stuff

Obviously, swapping to the CF would kill it pretty fast.
So for now, I've bought myself a 128M Memory Stick Pro, and I'm swapping to that.
- Doesn't matter if it dies - it was a $5 memory stick and can be swapped out easily.
- Doesn't matter that it's slow (in the vaio interface, I only get pre-Pro speeds. In the Dell cardreader, I get maybe 2.5M/s, the 'Pro' speed.) - Hey, it's swap, shit is hitting the fan anyway.
- I get a nifty little yellow light that blinks when I've exhausted my RAM.
- If I need more swap (say, a dist upgrade), I could always plug in an external drive and swapon that for a while.
Only downside is, if I want hibernation, I need swap that is bigger than my RAM. Maybe I can find a bigger stick, and maybe it will work in my non-Pro slot.

Kernel IO Scheduler

You can see the current scheduler by looking in sys:

cat /sys/block/sda/queue/scheduler

Adding a kernel parameter apparently lets you change the scheduler:

elevator=deadline

Making things bootable

SYSLINUX

To make my little CF1G card bootable, I had to:

Partition:
- 740M FAT32 "ubuntu8" BOOTABLE FLAG ON
- Remaining EXT2 "casper-rw" (The disk label is important)
Copy Ubuntu LiveCD? files over to ubuntu8 partition.
I renamed the 'isolinux' dir to 'syslinux', and created a syslinux.cfg similar in spirit to the isolinux.cfg file, but with a 'persistent' flag. Hell, let's include it here:

DEFAULT /casper/vmlinuz
GFXBOOT bootlogo
APPEND  file=/preseed/ubuntu.seed boot=casper initrd=/casper/initrd.gz quiet splash --
LABEL live
  menu label ^Run Ubuntu in Persistent Mode saving and restoring changes
  kernel /casper/vmlinuz
  append  file=/preseed/ubuntu.seed boot=casper persistent initrd=/casper/initrd.gz quiet splash --
LABEL live-install
  menu label ^Install Ubuntu
  kernel /casper/vmlinuz
  append  file=/preseed/ubuntu.seed boot=casper only-ubiquity initrd=/casper/initrd.gz quiet splash --
DISPLAY isolinux.txt
TIMEOUT 300
PROMPT 1
F1 f1.txt

I had to set up a default MBR that would look for 'bootable' partitions. This step seems often overlooked, as if your flash memory already has a DOS MBR, all will be fine... but if it doesn't!
- ms-sys --mbrsyslinux /dev/sdd
I had to set up syslinux on the first partition. Now, be careful here - some versions of syslinux can't handle FAT32, and the only version I've seen with the '-d' (directory) option is on Debian.
- syslinux -d syslinux /dev/sdd1

Done. However, check the pitfalls section.

Oh, and the version I've used doesn't seem to actually give a damn about the menu structure. (ahah, this is due to the version of SYSLINUX I used, plus the fact that the fancypants menu you see on the LiveCD? is actually done via "GFXBOOT", the special fancypants extension to SYSLINUX)

GRUB

Usually:

mkdir -p /boot/grub
copy stage files over, make a menu.lst
grub
- find /boot/grub/menu.lst
- root (hd4,0)
- setup (hd4)

However, I've got an "Error 15" (file not found) to contend with. More later.

Attempting to do it from yang (was very annoying because of GNOME treating 'eject' like 'throw away this disk I never ever want to see it again' rather than merely 'unmount'. Thanks GNOME!)

grub> find /8g_usb
 (hd1,0)
grub> geometry (hd1)
drive 0x81: C/H/S = 983/255/63, The number of sectors = 15794175, /dev/sdb
   Partition num: 0,  Filesystem type is fat, partition type 0xb
   Partition num: 1,  Filesystem type is ext2fs, partition type 0x83
   Partition num: 2,  Filesystem type is fat, partition type 0xb
grub> root (hd1,0)
grub> setup (hd1)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/fat_stage1_5" exists... yes
 Running "embed /boot/grub/fat_stage1_5 (hd1)"...  17 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd1) (hd1)1+17 p (hd1,0)/boot/grub/stage2
/boot/grub/menu.lst"... succeeded
Done.

I think I might need to run that 'embed' and 'install' command manually, and specify 'hd0' at the right place so that
- For installation, it uses the correct hd1 or hd4 to actually meddle with things, but
- The thing that gets installed is told to look for hd0 instead.
No, wait - this isn't the problem.
- I tried pulling the plug on yang's hard drive, booting off the USB key with syslinux (so that it is the ONLY hd in there), and then setting up GRUB from within the live environment... root (hd0,1), setup (hd0), same problem.
- It seems that grub is having trouble with the partitions?
- I also tried putting the syslinux ms-sys MBR on, and chainloading a grub I installed on (hd0,1). That seems to have messed with the partition, although I was at least able to get to the grub console this time.
Presumably, these are just problems for the USB booting - presumably, if I do this onto my 32G CF, and put that in the IDE adaptor, and put that into sumomo, everything will be fine... presumably.

Pitfalls

Although I set yang to boot "Removable" - "CDROM" - "Hard Drive", what I didn't realise was it was treating the card reader as a hard drive, which was set to a lower priority than the internal HDD.
SYSLINUX doesn't actually install code in the MBR, only the BR of the partition you point it at.
After zapping my USB stick with a slightly different GRUB, I had to go into the BIOS again to tell it that the USB stick had a higher boot priority than the local hard drive. I assume that the BIOS is taking a signature of the boot record of each drive, and believed the USB stick to be a brand new one.

Attack Plan

While we could copy the files off the internal disk drive first, either over the network or into the CompactFlash? card directly via the PCMCIA card, it might be better to put a minimal, bootable system in place on the card, then do the hardware swap, then later copy the original system over via the 2.5" USB caddy. It wouldn't even have to go over the network (though that might be a speedup, and we'd certainly want a backup)
DamnSmallLinux? would make a good 'bootstrap' OS, and a handy rescue system too.
- Partition for DamnSmallLinux? plus the eventual layout of the original.
- Put DSL onto the card as normal; Test that it is bootable (via cardreader and yang)
- Edit the bootloader and remove any sd*1 references, replacing them with hda1. Make sure init gets passed the correct location. And double check the contents of /etc/fstab
- Place CF card in IDE adapter, do open heart surgery on sumomo, put everything back in power on and hold your breath.

/etc/sysctl.conf

To minimise writing as much as possible a'la "laptop mode", and avoid using the HD for up to 10 minutes at a time:

vm.dirty_background_ratio = 20
vm.dirty_expire_centisecs = 60000
vm.dirty_ratio = 40
vm.dirty_writeback_centisecs = 60000

Partition layout

Rough guides to how much space I'll need:

sumomo's /usr is currently 5G, /var 2.5G /home is 3G /lib is 700M and everything else is in /local.
yin's /usr is 6.3G, /var is 1.7G /home is 14G /lib is 250M

32GB CompactFlash?:

190MB ext2 for DamnSmallLinux?.
8192MB ext2 ro "const" mounted as root. Data for /bin /boot /dev /etc(1) /initrd /lib /media /mnt /opt /proc /root(2) /sbin /srv /sys /usr goes here, we'll remount rw if we need to do an upgrade.
- (1): We need /etc for /etc/fstab so we can get the rw partition set up. But then how do we update our configuration? Even simple stuff like networking in user mode will probably write here.
- (2): Well, it's roots home dir, so it shouldn't be bad if it's ro, and if we are using it, then hey, we're probably doing weird stuff anyway and have mounted this partition rw.
Everything Else, ext2 rw "mutable" mounted as /sumomo /etc(3) /home /local /var(4)
- (3): Maybe... Maybe we could have a simple /etc in ro, and a more complex one mounted on top via rw.
- (4): Actually... was having some trouble with some things at boot time expecting /var to be there already. Hmm. For now, I've moved /var back to "const", and kept both rw.
tmpfs mounted as /tmp
tmpfs mounted as /var/log

Boot times

Using time read, from moment of Enter in GRUB to moment Ubuntu makes it's little "ready for login" noise

Before:-

real	2m23.968s
real	2m23.775s

After:-

real	1m59.903s
real	1m58.631s

LinuxOnCompactFlash

Contents