Haha! I help increase blog overpopulation!

I have now created a third blog. One has to separate one's interests you know. Anyway, old entries have been imported and published, will soon be gone from the old mixed blog. This one is my more public and global part of the "Speaker's corner", the others will return to being more personal. Oh and in swedish mostly... This one is english-only.

Tata for now!

My Windows vs Linux desktop fight

I really try to work with Linux, and for the most part it works ok. However I have issues with things not being stable and being difficult to get to work in a simple way.
With Windows everything almost always "just works". In addition, things are relatively stable, and when things do crash I may not know what caused the crash but it rarely happens again because I can avoid the condition for causing the crash. With Linux (ok, I don't use any of the directly end user centric distros (Arch, with KDE4.3) so I have myself to blame, but still) I can often cause crashes with subtly different effects just using the system as intended. And I still don't know what caused it!
One example is KDE and multiple desktop sessions with different users. It *should* work very simply, but every time it has caused window manager problems that forced me to at least log out and back in. Also, strange window manager behaviour in the second session.
So, you say, report the bugs! Sure I may do that - if I ever find the time to produce valuable bug reports. In this forum I can bitch and moan about problems in a slightly unstructured way, but it would be worthless as a bug report. I would like to help but it's difficult to find the time. I mean, I shouldn't even be writing this, I should be working!
Oh, and Adobe's flash plugin. Unfortunately they have basically monopolized dynamic content and media on the web so the closed-source plugin is completely mandatory on a desktop system. However, the Linux version crashes a lot - once it took down KDE, and once it *completely* locked up the whole machine. Comparing to the Windows Vista and XP computers I use my Arch Linux desktop crashes noticeably more frequently.
I'll continue to plug away at this, but it'll be quite some time before I can let me dear wife use it daily.
(You may wonder why I don't use Mint or Ubuntu or something similar? Well, I just got stuck on Arch since I could easily get a home server running with it, and Arch is so easy to manage. Desktop support is a bit trickier to get right it seems - but apart from the crashes it was easy to get up and running the way I wanted it to)

"BT spokesperson" suggesting tiered internet

This may or may not be news but I just saw it and I really became angry that this sort of misconception continues to be propagated. A BT spokesman (no name oddly enough) allegedly said the following:

Traffic is growing. The idea that ISPs will continue to pick up the bill is unsustainable. We want to work this out with the BBC and other content owners to come to some real-world compromise.

This person doesn't seem to understand that his industry is there to transport bits from point A to point B. It annoys me since the way it roughly works is this:

• Content provider pays for bandwidth from an ISP that charges based on usage (amounts of traffic up and down)
  
• Individuals/households pay ISP for internet access (based on certain speeds, sometimes with bandwidth caps)
  
• ISPs make arrangements to transport bits between each other so that if one ISP is sending much more data across another ISPs network it will end up paying for the used bandwidth

This means that usually ISPs share the cost of transporting data and end users (that is both content providers and consumers!) do pay for their activities. I assume that I may be quite wrong as there are likely lots of different deals between ISPs but I do think this is the generic model.

The "funny" thing that I guess this spokesperson does not realize is that both BBC and YouTube for instance are already paying, albeit not directly to BT in this case. In return, BBC gets to send data from its customers out to other ISPs without paying "extra". Now, if the costs are getting too high I suggest either increasing corporate efficiency or charging more for internet access. It's that simple. I just cannot understand how the spokesperson otherwise imagines things would work! Each individual ISP having to have agreements in place with each individual content provider that crosses some sort of arbitrary threshold in terms of the amount of traffic it causes in the network? Jeez, managing and negotiating that sort of dynamic scenario will cost more than increasing your bandwidth to just deal with it, I'm sure.

However I guess what they don't want is the increasingly symmetric internet, where the previous consumers are now producing content and will consume valuable uplink capacity since the ISP pays for that. Actually I suspect that they really are just afraid of the speed of change and want to slow that down to stay in control and maintain some sort of illusion of being in charge of their "products".

I have no problem paying more for internet access. As long as I know it is unlimited, I do not want to be limited in any way as to what sites I can or cannot visit, or about speed caps, or bandwidth caps. That is just silly and arbitrary censorship in my opinion. I will pay what it costs (within reasonable limits...), just don't come to me and say that "Yeah... you know, in order to access these sites that you seem to like so much we will need to charge an extra ten pounds... Yeah, sorry about that, but they're so popular, you know... Oh, and yes, we do charge them a pound each per individual user too of course. Mmm, yeah... that's in addition to what they already pay... Yeah..."

Now, luckily I don't live in the UK so BT is not a problem for me. However, any sign of such idiotic statements (well, I'll wait for them to actually implement it) from my ISP and I switch.

RANT: Man, I HATE Microsoft Word!!!

Oh my God how I hate MS Word. It is the spawn of the devil, a productivity vampire leech, the most vile and inconsequent piece of crap that has ever graced a computer with its idiocy.
And I am forced to use it at work because we're standardized on Word and the rest of MS Office of course.
At work I often write reports. They include simple things like headings, Table of Contents, headers, footers and pictures and tables with captions. Also, we often collaborate so we use change marks quite liberally, and also a versioning document storage system. There are internally developed macros that make it fairly easy to keep formatting consistent etc. However, picture handling is appalling. I become a nervous wreck every time I have to insert pictures because no matter how I try, the pictures float around willy-nilly, captions end up wherever in relation to the picture and settings for the picture get completely lost! If I set the text to wrap top and bottom and position it relative to line and margin it should stay that way, but nooooo, as soon as I resize or try do do any minor changes the settings are comlpetely forgotten and picture starts to play hide and seek again. I HATE IT!!!
I yearn for the simplicity of Framemaker (version 5 last I used it), where you tell the picture where it goes and it and its caption stay there until you change it. You can easily anchor it to the text and while the text may change and reflow, the picture (and its caption!) nicely follows along without destroying formatting.
We use Office 2003 and man do I hope that 2007 or 2010 are better because I have no doubt that no changes will be forthcoming regarding the approved office programs that my company uses.
Maybe I just need a course to learn how to work in Word again, but I think that Word 5 actually handled things allright, and ever since then the quality has been going down-hill.
By the way, I feel better now. Maybe I can actually go back and fight some more to get the pictures where I want them. Blogging is good for your heart ;-)

Adding LVM2 for flexible volume management on top of RAID

So as I mentioned a while back I also added LVM2 (Logical Volume Management) on top of my software RAID arrays (well, only one of them so far). This is really easy as LVM can use any existing partitions, be they actual regular partitions or RAID partitions. Here's what I did:
First tell lvm which existing partitions you want to use (pv stands for "physical volume"). In my case there are three RAID arrays

lvm pvcreate /dev/md0
lvm pvcreate /dev/md1
lvm pvcreate /dev/md2

You can probably leave out lvm, it seems to work anyway. The same goes for all commands below, I just included them for completeness.
Then you create an actual volume group out of each physical volume (I guess this adds data structures so that lvm can keep track of the logical volumes you create in the next step, and any changes later on). I only use one of the physical volumes due to the issue with hal that I mentioned in the previous post. data is my name for the volume group.

lvm vgcreate /dev/md0 data

I then add the logical volumes that I want. These are then used like ordinary partitions and must be formatted with a file system before use. You decide how large the logical volumes should be and their name. The final part of the command tells lvm which volume group should be used according to the name in the previous command.

lvm lvcreate --size 200G --name home data
lvm lvcreate --size 200G --name bkp data
lvm lvcreate --size 800G --name media data

These "partitions" will then show up in in /dev/mapper/home, /dev/mapper/bkp, /dev/mapper/media and can be formatted and mounted like any regular partition. Note that I did leave quite a bit of space unused, mainly this is for testing resizing later on. The nice thing with lvm is that you can eaily resize these things as disk space is added to the volume group, but of course the file system must be able to handle it.
Finally to get my Arch Linux system to recognize the volumes I modified the lvm line in /etc/rc.conf to read:

USELVM="yes"

And then I added a couple of hooks in /etc/mkinitcpio.conf, one for RAID and one for LVM2, so that the volumes are recognized when the system boots: mdadm lvm2
Finally I rebuilt the kernel and initrd to include the hooks by just letting pacman reinstall with the new settings:

pacman -Sy kernel26

This is probably the easiest way although you can rebuild manually as well of course, if you have specific requirements.
And then it was a breeze to install Arch, telling it to use the logical volumes home, bkp, and media wherever I wanted them in the file system. home is obviously used for home directories (currently mainly used for "backing up" home directories on other computers), bkp is used for backup storage of family photos and videos (and probably other stuff too in the future), while media is used to store recorded TV (Mythtv recordings as well as old movies) and our audio library (including iTunes audio - only iTunes Plus so it is portable). The setup seems to work very well, but we'll see if any problems arise in case I have all four tuners recording at the same time... :D
Btw, you can actually boot from a RAID/LVM2 partition as well but I did not want to do this, I want the system to boot even if there's something wrong with this slightly complex setup.

RAID can make hald segfault

As mentioned in my last post I created three RAID arrays. Turns out that at least on Arch64 several arrays can cause hald (Hardware Abstraction Layer daemon) to crash, and thereby causing the new Xorg to not work since it relies on HAL to figure out the keyboard and the mouse (of course you can go back to the old way of configuring that especially on a server machine such as this, but hey, I really like things to work the way they are intended to!). However, I made the following changes to my setup compared to the previous post:

• removed raid1 and raid456 as well as dm-mod from the MODULES() line in /etc/rc.conf (post on lvm setup still to come but I got derailed by the hald issue)
• removed raid1 and raid456 as well as dm-mod from the MODULES() line in /etc/mkinitcpio.conf
• changed the raid hook for the newer and apparently more capable mdadm hook in /etc/mkinitcpio.conf (not in the Arch wiki, but mentioned in a forum posting (more info). Seemed like a good idea although I haven't dug too deep.)
• commented out the two still unused arrays in /etc/mdadm.conf. I still would like to use them, but unless there's an update to hal I need to figure out some other use for those partitions, perhaps I'll just make logical volumes out of them without using RAID.
• rebuild the kernel/initrd (simplest thing is to run pacman -S kernel26, which will fix both the actual kernel image and the fallback image. Yeah I don't know the difference either...)

And now it works. I probably could have just commented out the two arrays in /etc/mdadm.conf, but I thought that it might be wise to prune the configurations in general. To be honest I have not understood how hooks and the different MODULES() lines interact so I usually probably add more stuff than needed, and I guess that might have a detrimental effect on functionality. I might also see if I can figure out wherein lies the fault, but there may be a fix in the works, see this bug report.

Software RAID on Linux with mdadm

I am building a home server with the intent of using it for

• Mythtv backend server
• Backup of documents, email, photos, videos etc that are created and used on other computers in the house
• Print server
• Offline syncing for remote backup (using another server at my parents' house, and possibly one at the house of my inlaws)
• Eventually keeping home directories here instead of locally - probably when the kids are old enough to get their own computer(s)

So I want to use RAID to improve resilience in case of disk errors and perhaps get some performance increase. In addition I want to use LVM2 to make it possible to easily extend the storage space, but that is the topic for another post - RAID and LVM are independent since RAID just creates storage devices that can then be used by LVM. Or not. I use Arch Linux and despite the excellent wiki I had trouble as there were some details lacking. If I dare I might try to contribute and clear it up a bit.

I have five disks, and I will use a normal root partition and a normal swap partition, see below. Note that the abomination of the Linux kernel or udev that causes disks to not end up as the same device on every boot means that the below list, being valid when preparing the disks is not valid when rebooting, causing some problems. Disks must be referenced by their uuid, completely non-userfriendly since the sysadmin MUST deal with them. No fun.
Anyway, I boot the Arch live CD and manually prepare the disks with cfdisk:

• sda: 400GB (/ swap 360GB) 1 2 3
• sdb: 400GB (360GB 40GB) 1 2
• sdc: 500GB (360GB 40GB 100GB) 1 2 3
• sdd: 400GB (360GB 40GB) 1 2
• sde: 500GB (360GB 40GB 100GB) 1 2 3

After preparing the disks I load the RAID modules:

modprobe raid1
modprobe raid5

So I create a RAID5 array with five 360GB partitions, a RAID5 array with four 40GB partitions, and a RAID1 array with two 100GB partitions:

mdadm --create /dev/md0 -raid-devices=5 --level=5 /dev/sda3 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm --create /dev/md1 -raid-devices=4 --level=5 /dev/sdb2 /dev/sdc2 /dev/sdd2 /dev/sde2
mdadm --create /dev/md2 -raid-devices=2 --level=1 /dev/sdc3 /dev/sde3

It is necessary to inform mdadm about the raid arrays so that they can be assembled at boot so I used:

mdadm --examine --scan >> /etc/mdadm.conf

Now I did not do this during install, and the installer did not bother doing it, which I discovered when rebooting and fsck could not find any of my nice logical volumes that I created. Doing it during install you must consider where the file is mounted, probably /mnt/etc/mdadm.conf.
Later during the installation, in /etc/rc.conf, I add the raid modules to the MODULES line, and in /etc/mkinitcpio.conf I add the raid HOOK. Also, it might be necessary to add kernel parameters to the kernel line in /boot/grub/menu.lst:

md=0,/dev/sda3,/dev/sdb1,/dev/sdc1,/dev/sdd1,/dev/sde1 md=1,/dev/sdb2,/dev/sdc2,/dev/sdd2,/dev/sde2 md=2,/dev/sdc3,/dev/sde3 to kernel parameter

I haven't figured out if that is really necessary since I use a normal root partition, but at least things work now. Also notice that since I got it working after first reboot I manually regenerated the kernel:

mkinitcpio -g /boot/kernel26.img

You can allocate the RAID arrays to whatever mount point you want now, in my case I could have a large 1,3TB /home directory (/dev/md0), and perhaps a 100GB /backups directory (/dev/md2).
However, see a fortcoming post on LVM2 where will I show how I collected all the RAID arrays into a gigantic Volume Group, and then split it up into logical volumes that are then mounted into the file system. These LVs can be shrunk or grown as needed and thus offers much more flexibility than RAID alone.
As a concluding remark I might add that my partitioning scheme was created to maximize the use of current disk space and redundancy since the disks aren't identical. I don't know if it is optimal in any way and comments are welcome. This is my first shot ever at software RAID, so any disclaimers you can imagine apply.
I haven't planned for the future since it seems this disk space should be enough for quite some time, but I am hoping that I will be able to exchange the disks for larger ones and just create same-sized partitions for complementing the old arrays and additional ones for new arrays and then just grow the volume group and the logical volumes as needed. I should for instance be able to fail all participating partitions on a particular disk (thus degrading all arrays having partitions on the disk, but still working), shut down, put in a new disk, boot and create new partitions that are the same size and then add those to the arrays and watch them rebuild. I could do this for all disks, one at a time and then create new arrays and add them to the volume group and then grow the logical volumes as necessary. Other growth/migration strategies are possible of course.

Invisible things that go bump in the cpu...

Joanna Rutkowska states some obvious things, but it is still valuable since I believe we need at least a little bit more paranoia in our increasingly digital world.

Syncing apps?

Been looking for sync apps for a long time. At work I use Outlook, at home I use Thunderbird/Lightning, and I have a Sony Ericsson smartphone for work and an iPhone 3G for personal calls. I do not want to have separate calendars/contacts/tasks for work and private use, I need all things in one place. I currently use Plaxo for contact syncing between Thunderbird (home) and Outlook (work) and Outlook Express (home) and to view my calendar/tasks/notes via the iPhone/Safari. I then let iTunes (home) sync with the Outlook Express address book to get contacts into the iPhone.

Some suggestions are

• Oggsync
• Goosync
• And of course there's Google's recently launched ability to directly sync the iPhone along with its ability to sync Outlook and other desktop apps.
• GCalDaemon
  

Trouble is that I don't really like to use online services, I would like to sync everything locally on the computers I use, so for me Plaxo is a klugy workaround. But it works since I can't sync the iPhone to more than one computer; getting the info to my home computer via Plaxo is ok. It would probably be better to use Google with the added functionality it offers, but I'm not giving up just yet ;-)

Also, I miss the ability to sync tasks to the iPhone. I know there are apps, but most seem to be self-contained and don't offer the syncing that I need. If anyone knows of any sync solutions for the above scenario, please comment!

Arch Linux and network profiles

Just as a small note to self. I have been setting up Arch Linux on a laptop and had trouble working with network profiles. So here's a reminder on what is needed (after supporting software like dialog and netcfg have been installed, and drivers for your wireless card are functioning):

1. Create the profiles in /etc/network.d/. Easiest process is to copy relevant examples from /etc/network.d/examples/ and modify as needed.
2. Add "net-profiles" to the DAEMONS line in /etc/rc.conf
3. Uncomment the NETWORKS line in /etc/rc.conf and set it to read "NETWORKS=(menu)"
4. Comment out the lines related to a fixed network in /etc/rc.conf, but keep HOSTNAME
   

Information is available in the Arch wiki, but I misread and thought that the daemon to add was "network-profiles". It took me a while to figure it out...

The !joys of Windows

Edit to my previous post. Being more than a little naive I posted before checking whether things would work or not. dd for windows actually finished, seemingly successfully, but alas, either I made a mistake regarding the proper command line or it just doesn't work under vista, because when I tried booting from this stick it did not work - and when I tried to check what the disk looked like with cfdisk (yes I rebooted into Arch...) it said there was something seriously wrong with the disk. So I did what I should have done from the start: I ran dd under Linux. And it was good! First it was about a billion times faster and second, it worked ;-). Simple:

dd if= of=/dev/sdb

Your device may be different of course...

Getting an image to a usb stick under Windows Vista is slooooow

Windows Vista might be fun!
I had to get my work laptop reinstalled due to various undefinable problems so as I had the option to go with Vista I thought it could be, well, interesting... So far it generally works as well as I expected. We have Vista Enterprise and due to my job I and my colleagues are admins on our computers so no big problems in terms of getting things done as I like - the rest of the company have well-locked down boxes - which is good for most of them!
Anyway, today I had the need to write an iso image (750MB) to a usb stick (Kingston 1GB). I could boot into my Arch Linux partition but as there are win32 tools for accomplishing this it would be unnecessary. The iso was bz2-compressed so I first needed to extract the image.
Tools needed:

• bzip2
• dd

I renamed the bzip2 executable to just "bzip2.exe" and in order to not having to mess with paths I dumped the executables in the same directory as the compressed file. Then, in a command line window I ran:

bzip2 -d

I then located the image among the extracted files and moved that to where bzip2 and dd executables were and ran:

dd if= of=\\.\g: --size --progress

Important! In order to find out which drive to use I first run

dd --list

as writing the image to the wrong drive could cause a lot of destruction!!! Be careful!
About the --size option - it's probably not necessary since I'm not reading from a usb stick, just writing to it, if I understand the dd documentation.
So, I'm still not sure if this will work in the end since the point of this post is that the process of writing to the stick takes frickin' ages! It ran for a while before I started writing this post and it is still running now that I am nearing the end. It might be the stick that's slow and I can't compare with win xp right now so I can't tell if that is any better, but HELLO! Vista is sure nice in terms of looks but it is definitely no performer, judging by other people's opinions on the wild web so I think it is mainly a Vista issue... If I had rebooted into Arch I think this would have been done by now. Sigh. 300MB now, just 45 minutes or so left...
Luckily I can do other important stuff in the mean time - like post this blog entry! Yay!

...from Mediahemmet