building a new linux lab in 3, err 4 hours


lets see if i can make this!

I’ve laid out some basic things for now

Obviously a subnet (

A firewall ( which will be installed first and then cloned for at least another system to save time

A combined dns/dhcp/cobbler/web server that will provide the fastest boostrap time for cobbler. I’m not delighted with this as it means I dont have easy integration with my underlying xen host and no good plan for migrating to a proper structured network with separation of install/infrastructure/client networks.

A iSCSI subnet in there, potentially vlan-based but rather another „virtual“ lan

A Filer Emu box (ontap 7.3)

A dynamips box

In a moment I’ll start the install for the firewall, but I’ll hold my horses till I got a script to provision the actual storage

For the sake of easyness this will ALL be based on CentOS for the infrastructre and my basic server will come with 5GB disk, additional 15GB to the filer emu box  and 20GB for the cobbler install server)

now, lets see: firstofall a new bridge needs to be build using my old XenDistro  bridge script ‚create_bridge‘ which builds actually WORKING bridges, other than all this new virsh crap.

i’d paste this here if opera didnt suck balls right now

ok the script ended up with some little bugs, wasting about 25 minutes and in the end i pasted it into my shell. doesnt really matter. the firewall is just doing it’s first boot, for it’s install i attached it to the „production“ lan via xenbr0. Other little mistakes was

  • selecting the wrong radio button when trying to disable DHCPv6. My own clumsyness, why isn’t my v6 stuff up and running fine anyway!
  • using the old device name of ‚xvda‘ which worked with rhel52 beta but doesnt work with centos52. yet another time you get actually punished for knowing how things used to be before everyone else figured them ;p I switched to sda/sdb and this should do now.
  • it came down to a stupid typo: using phy=vgxen/lvname instead of phy:vgxen/lvname :>

I disabled all package groups in the installer, short of spending a few hours on shrinking via .ks files this is the best i can do to get a lean install. for the gui-affectionate other hosts a few yum groupinstalls will be needed (eeek) and by then i’ll have my local mirror to speed this up.

The installation took about 3 minutes, main factors are

  • separate disks for the install repo (a loopback mounted dvd iso image
  • enough ram on server and client
  • ftp protocol

another glitch: lacking a sophisticated kickstart file both disk images ( os disk and swap ) ended up in VolGroup00 and got fully allocated – thus I can’t even pvmove the swap volume out. I’ll waste some time on this RIGHT now and disable the swap to see if i can fix this real quick.

swapoff, lvremove the lv (1GB in size was too much anyway), lvcreate -l 15 -n LogVol01 VolGroup00 xvdb1  and mkswap it and swapon -a and we’re there. Why the fuzz? because this allows me to use different physical disks on the host for swap and os data.

This is basically _it_., and now i’ll have to decide whether to really copy the lvs or the use the .ks file. I think i’ll play safe and use the .ks file as this ensures I won’t get in trouble due to duplicate UUIDs for various bits and pieces.

So lets copy off the kickstart file to a random box with an http server and install the new dhcp & stuffs host.

Now this was a big failure I didnt convice the install kernel to actually read the kickstart file, I can just wonder as to why. It does have the –preload xennet ramdisk so this should just work.  Anyway, i gave in and fired off another install but I suppose i’ll have to recheck this later on. SOOO frustrating. This time I simply excluded the second disk device to conserve space for the later swap.

One more note: Always grab the autogenerated mac address in the interface configuration dialog.

Now the service host got a 3rd 20GB disk device dubbed lv<hostname>data so i can assign this a raid1 storage if needed and I attached it over to xenbr6, the lab network. Time to reduce the firewall’s ram size, add another vif on bridge6 and start routing.

I hand-edited ifcfg-eth0 to bootproto static with a new ip address, added ifcfg-eth1 with the correct mac address, added that to the xen config and enabled routing in sysctl.conf.

at the very start I tried to use system-config-something as it came up from first boot, but it turned out too annoying so i used vi instead 🙂

On my  production side firewall I added a route to the test network. Per se the main firewall would drop traffic there, BUT with some luck the icmp redirects will do the trick anyway.

Big mistake, now a dhcp timeout gets in the way,  I should have changed eth0’s config on the service host on time,

Ok, fixed the configuration and the new box is reachable. Time to prep volumes REAL fast now. I added it to VolGroup01 after doing fdisk & pvcreate and  created a 10GB lv to hold the mirror for the centos updates. No doubt I’m running out of time now, maybe I should rename this post 😉


I got stuck in firewall issues and will chose sleep over reaching my goals now 🙂

The next day I solved the firewall mess when I figured there was no reason to put the lab firewall into my normal lan. Now it’s in parallel to my internal firewall and only allows incoming access from the internal lan to the firewall and a few services. Well some pieces dont properly work (I default deny, the fixes won’t even add to 10 minutes of thinking but right now I rather stop it for a short moment and restart after doing what I had to)

Cobbler worked like a charm after reapplying the security contexts and after finally remembering how to it actually worked – I used it in the 0.4x.x days and hadn’t gotten around to it again. My first successful install kept looping, I’ll have to look for both the option to turn that of as well as a better xen config handling. But Xen issues aren’t in the heart of this doc, nor my firewall troubles.

Other than that the big next things are:

  • properly mirroring  -updates,
  • double using the cobbler centos mirror as a local repo
  • fine tuning of kickstart, cfengine integration
  • noting filer emu options for silent install

I suppose a big head start would be doing this during the daytime, but right now insomnia has it’s full grasp on me and I’m not expecting to be *awake* during that time.

Above this line there’s notes and experiences, luckily I got a lot further by now


Below ths line there’s a more thorough manual, no details but up to date with what I have running


  • centos52 dvd media
  • xen dom0, disk space (minimum 50GB)
  • one ip in your production net unless you have a two-firewall setup

optimized order of steps:

  1. use script to make storage for firewall domU
  2. manually ks install firewall domU
  3. set up firewall, routing, nat
  4. storage for cobbler domU
  5. manually ks install cobbler host domU
  6. install dhcp, named, tftp, xinetd, cobbler, koan, python-cheetah
  7. open firewall ports
  8. cobbler check, configure dhcp
  9. set up 20GB filesystem for /var/www and relabel selinux attrs
  10. cobbler  import and setup  &
  11. tweak kickstart config (base os storage, cfgengine?)
  12. cobbler add host netapp emulator & others
  13. create storage for netapp emu host
  14. create storage for dynamips host
  15. create storage for clients
  16. create yum updates repo and include in profiles
  17. cobbler install netapp emu, dynamips host, clients (i had to supply ks config in xen cfg)
  18. set up 15GB filesystem for netapp sim
  19. configure netapp sim
  20. configure http (+licence), nfs exports & iscsi targets in netapp sim
  21. install dynamips, dynagen, pemu rpms on dynamips host, enable them, open firewall ports

missing steps:

  • ldap+kerberos for user auth, including cobbler ui and filer
  • ldap+kerberos early config
  • filer rc scripts + screen
  • filer fw ports
  • cfengine early config
  • pxe boot menus, one_time dhcp option to avoid install loop
  • assigning mac’s uuid ip’s automatically and before domU building and putting them in dns etc


So far i’m 9 hours into this effort, and the lab is about 80% up. I’ll strive to polish edges (i.e. why manually setup the filer if i can have a cfengine snippet) and I doubt I’ll give in until this whole process is scripted well enough to work within the set time limit.

In a few weeks I’ll have to deploy this, short of the Xen bit for an in-house VxVM workshop. By then this should be a reliable bunch of scripts and save me a lot of grey hair. On the other hand I’m in this job long enough to know that noone besides other people that have done a few kickstarts already will figure I’ve been quite fast. So in all honesty – here and now I’ll make this work as fast as possible, and then I’ll sit back for two days and tidy things so my boss at least *grasps* that building a full lab takes a little effort.

You actually have to work as slow as the rest or the rest won’t even get you’re working hard.


2 Responses to “building a new linux lab in 3, err 4 hours”

  1. 1 Uday Joshi

    Lord heigl !!!

    Simply you are gr8. You are doing it with very little efforts. It shows ur hardwork. I think u hav habit to work in calmness of night.

    Can you help me to build simple VxVM lab?

  2. 2 darkfader

    hi uday,

    it wasn’t without effort, it took well into the 9th hour, and I had started in the middle of the night. so it was a night without sleep and a lot of headache 🙂

    about VxVM: there is no simple lab for it, I sometimes even think there is no simple at all with VxVM. but i’ll try to find something.

Kommentar verfassen

Trage deine Daten unten ein oder klicke ein Icon um dich einzuloggen:

Du kommentierst mit Deinem Abmelden /  Ändern )

Google Foto

Du kommentierst mit Deinem Google-Konto. Abmelden /  Ändern )


Du kommentierst mit Deinem Twitter-Konto. Abmelden /  Ändern )


Du kommentierst mit Deinem Facebook-Konto. Abmelden /  Ändern )

Verbinde mit %s

%d Bloggern gefällt das: