So, at work we have a little single port IP KVM, it is a StarTech SV1107IPEXT - it's not a "bad" bit of kit, if you're completely configured for using it in the way they expect... When you're not, then it throws a NullPointerException when you try to connect with it's Java applet.

This morning (mostly because the monitor and keyboard I had been using for building stuff on my desk has gone missing) I decided I'd try again with the device, it's not on our default office network range, mostly because it might not always be in the office, so I quickly added an ip alias using ip addr add 192.168.0.59/24 dev wlan0 on my laptop to bring up a route to it. Navigated to the page and asked it to fire up the Java applet. Straight off, a NullPointerException happens - hmm. So looked at what was being logged by the JVM - it was trying to run ifconfig (no, really, the java applet ran ifconfig... with no path...) which, on my debian laptop, is not in the default path... so I went and make a quick wrapper in ~/bin for it... right - now it can run that... still a NPE, hmm. So look a little closer, it's looking for the MAC address assigned to the IP that you're coming from... no, really! WTF! So I went and hacked my little ifconfig wrapper to rewrite the address for wlan0 to be what the KVM expected it to be, low and behold, I can now run the java client (hoorah!).

Why the heck are they doing that though - that's just plain insane! On the plus side, I now have a work around, on the minus side, egad that's ugly!

So, as I'm not always sitting directly in front of machines that I'm working on I've been using various tools to get temporary network connections between completely seperate network blocks, for example, sitting on wireless outside of my home, connecting back in to the desktop at home, and then connecting to the virtual machines on that desktop - the virtual machines are on their own private network, which is bridged and then routed through the desktop. The desktop machine has an ipv6 tunnel setup using tinc to my main VPS. That in turn has a tunnel setup to he.net's ipv6 network... I then have my laptop configured to also use tinc to connect to my VPS, so lots of ipv6 over ipv4, really.

From that I can then get a 'direct' ssh connection between my laptop and my desktop (OK, so it's actually going over 2 ipv6 over ipv4 tunnels to get there, but it "looks" direct), so, that gets me as far as the external network on that box. From here, we can run a vde_switch which is connected to a tun/tap interface that's part of the internal bridge, once we've got that running, I bring up another vde_switch on my laptop attached to another tun/tap interface purely for that network with a static ip configure on that interface... using the magic of dpipe I then connect the two vde_switches together using vde_plug, and as if by magic, I then have access to the internal networking of my virtual machines, meaning that I can now ssh directly in to the virtual machines without having to do anymore interesting ssh bouncing and port forwarding.

It's a little sick, but it really does make remote working a lot easier to do, it's a vpn of sorts, I suppose, as all traffic is being thrown over ssh pipes or tinc (which in turn is encrypted using x509 certificates)... probably not the quickest of networks, but very very nice all the same.

ARGH! So, now I know the issue, the disk was created with a standard dos partition table... this won't extend past the bit that I'm at... it needs to be GPT, so there's going to need to be some downtime to sort it out. SIGH.

I'm having some issues with LVM, basically it appears that I can't assign over a certain amount of my pv, where that certain amount is somewhere around 1.5TB... output from all the relevant chunks are:

ensenada:~# lvdisplay
  --- Logical volume ---
  LV Name                /dev/main/root
  VG Name                main
  LV UUID                ZR33WS-glE2-bUn6-adCX-n50C-37eX-Fc7Pde
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                7.45 GB
  Current LE             1907
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0

  --- Logical volume ---
  LV Name                /dev/main/voldemort
  VG Name                main
  LV UUID                DrT0co-THRw-6Zcy-Q3PT-OiQj-4KBz-xfV0Un
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.37 TB
  Current LE             358400
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1

ensenada:~# vgdisplay
  --- Volume group ---
  VG Name               main
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  56
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               5.45 TB
  PE Size               4.00 MB
  Total PE              1428053
  Alloc PE / Size       360307 / 1.37 TB
  Free  PE / Size       1067746 / 4.07 TB
  VG UUID               ddVSIr-W9BU-XOWJ-gdSo-HZ4m-vw3M-S33Q3s

ensenada:~# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda3
  VG Name               main
  PV Size               5.45 TB / not usable 1.54 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              1428053
  Free PE               1067746
  Allocated PE          360307
  PV UUID               sFxRYg-Ua60-zpMT-DMqo-tpD5-d24W-qfW31p

ensenada:~# lvcreate -n test -L 74G main
  Logical volume "test" created
ensenada:~# pvs
  PV         VG   Fmt  Attr PSize PFree
  /dev/sda3  main lvm2 a-   5.45T 4.00T
ensenada:~# lvs
  LV        VG   Attr   LSize  Origin Snap%  Move Log Copy%  Convert
  root      main -wi-ao  7.45G
  test      main -wi-a- 74.00G
  voldemort main -wi-ao  1.37T
ensenada:~# lvremove /dev/main/test
Do you really want to remove active logical volume "test"? [y/n]: y
  Logical volume "test" successfully removed
ensenada:~# lvcreate -n test -L 75G main
  device-mapper: reload ioctl failed: Invalid argument
  Aborting. Failed to activate new LV to wipe the start of it.
  LV main/test in use: not deactivating
  Unable to deactivate failed new LV. Manual intervention required.
ensenada:~# lvremove /dev/main/test
Do you really want to remove active logical volume "test"? [y/n]: y
  Logical volume "test" successfully removed
ensenada:~# lvcreate -n test -L 74G main
  Logical volume "test" created
ensenada:~# lvremove /dev/main/test
Do you really want to remove active logical volume "test"? [y/n]: y
  Logical volume "test" successfully removed
ensenada:~#

Please, someone, put me out of my misery - this is driving me nuts!

Extra info:

ensenada:~# uname -a
Linux ensenada 2.6.26-2-amd64 #1 SMP Fri Aug 14 07:12:04 UTC 2009 x86_64 GNU/Linux
ensenada:~# dpkg -l lvm2
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name                  Version               Description
+++-=====================-=====================-==========================================================
ii  lvm2                  2.02.39-7             The Linux Logical Volume Manager
ensenada:~#
ensenada:~# cat /etc/debian_version
5.0.1

(Erm, just remembered that I haven't got comments enabled at the mo, so if you've got fixes, please mail me! iDunno@sommitrealweird.co.uk - thanks!)

So, summer has arrived - how can I tell? The heat is killing the servers in our office again, the poor little AC unit can't keep up when the temperature outside is getting to summer weather. So the door to the magical cupboard is open for the moment, as the office temperature is slightly lower (but still, not great), and it's keeping the cupboard at somewhere around 26, which is still a little higher than I'd like, but better than the 32/33 that knocks the main firewall out.

I'll be replacing the main firewall over the weekend with a lower power (and thus less heat generating) soekris box (which should have enough grunt to shift the packets), and means that we'll go from OpenBSD 4.3 -> OpenBSD 4.5 at the same time.

Also, it's time to finally get the workstations on to lenny - as they're all NFS root, it becomes slightly tricky to get the setup just right for them... hopefully we can start rolling it out to the developers early next week, and then we should be in business...

So, was supposed to end up in Hector's House playing pool with various people that I know, but all of 'em were out last night (erm, half of 'em with me) and didn't quite manage to make it... So after an hour of waiting for people to turn up I gave up and considered going back to my "local" (the hop poles) to have a swift beer or three... instead, it being a wednesday, and knowing that Band Aids was on at the Pav Tav, I wandered there... managed to get there before the first band played - they were Vier and were absolutely bloody fantastic. Stayed and watched/danced to the other 3 bands and all was well with the world.

Now it's time for the sleep and the getting everything else ready for the big move on Friday (yay for working Good Friday... or something!)

There's a new Bravia advert coming - watch the web!

Why does Oracle complete an import with... "Import terminated successfully with warnings." when what it actually means is... "I ran out of table space mid import, but I'm only going to tell you that in an obscure way in the log, and otherwise make you think that it was alllll OK"? Surely in this day and age it could just say "Import terminated unsuccessfully: ran the hell out of table space", wouldn't that be nice?

Mid import of a large database was thwarted by cron cheerfully having a job that I was unaware of shutdown the database server! Bah! Oh well, maybe now I should actually head home, because this could "take some time".

Last night we moved Paston's servers from Telehouse to Global switch, so we're now hosting in Mistrals space in Global Switch... Now, Global Switch is a reasonably pretty looking building (compared with Telehouse, at least), and the machine move went quite well, with ~ 30 -> 45 mins downtime, ish. Things that went wrong: given a network cable, plugged in to switch, got nothing, monkey in Global Switch couldn't work out what was wrong, left him to it for ~ 5 to 10 mins before I had a sudden thought of "hmm, I wonder, I wonder if that's a straight or crossover cable", plugged it in to the uplink port, pushed the magic button... Hey presto, we have connection... Confront monkey, Monkey says "Well, they don't tell me these things". NYARGH.

So, anyways, got the 2 machines on to the net, tested everything was running, tested that it worked from remote machines, you know, all the usual crap... Leave Global Switch at 12.45am ish, get back to Norwich for 3am ish, get to bed for ~ 3.20am ish... Get a phone call at 10am saying "erm, it's not working for this bunch of people", the bunch of people in question being our ADSL customers, who were all recently migrated to Mistral's network... the problem? 2 of Mistral's ADSL routers couldn't find our machines... Mistral's suggested solutions to the problem: (1) wait 7 days and we'll fix it or (2) we'll connect you to a different switch port. So, fair enough, not too bad, now apparently connected to a different switch port... but to add insult to it, apparently the servers where "put in the rack the wrong way round for the airflow", we put it in the same way round as the other server in that rack, and the way we were told to put it in by the afore mentioned monkey... so this morning our kit was turned off for 10 mins while they turned it round in the rack, it's all come back cleanly (thank god), but ffs, why do they hire entirely useless fucking staff that apparently get this wrong quite often?

Anyways - end of rant... pub tonight as listed in the other post :)

Why doesn't Win2k3 have a nice way of exporting "Local User and Groups" for importing to a different Win2k3 server? At the moment, this would be really handy to be able to do. Oh well, I'll have to see if doing a restore and not letting it reboot and then restoring the hardware configuration works.

Not long left now, though... only 9 more working days for Paston.

Yesterday our leased line from UUNet (MCI) was cut off by BT at around 1.30pm, 7 days early. It should have been cut off next week on the 30th, as such a lot of rushing around, praying and general chaos has taken over while we continue (with the machines now in this building on the end of an ADSL line, and not accessable to the outside world) to move anything remaining off of the servers.

Of course, that was also our primary DNS server, yay for BT :(

Well, I've been up since ~ 6.45 ish, and was in Telehouse from ~ 10.45am through till 4.40pm (ish). Yesterday, our server hosted by netkonect in Telehouse "magically" lost routing to the world, I repeatedly told them that they had a problem, that the machine was fine before they had the other outage, and that traffic was consistently being dropped at their gateway... They insisted on rebooting our server, claiming that that would probably fix it... when it rebooted, of course, the routes and interfaces where set up exactly as before. Still no network. At 9.30pm Friday night we gave up and I resigned my self to going to London Saturday morning, got to telehouse, got to the right room, discovered that I hadn't got a key to our rack, or the monitor and keyboard that I'd asked for. So, phone netkonect, get a "oh, they should have given you the key at reception", right, trundled back to reception, yes, got the key, rah, now have access to the rack, but still no sign of a monitor and keyboard, or the netkonect engineer that was supposed to visit me, or the mistral engineer that was supposed to visit me. OK, fine, out comes the laptop, crossover cable, and some fiddling at the back of the server. Rah, I have ssh to it. Right, set up the second ethernet card to have a private address just so that the laptop can sit on that port and talk to it. Plug back in the other network lead, hey presto, still nothing from our box to the outside world, all being dropped at this point by the switch (I couldn't ping between the box and it's RAC port, which are on the same switch). Right, phone netkonect, tell them this, wait for a response, get bored, it's now midday and I'm dying of thirst, of course, the telehouse coffee shop is shut sigh, so off I go in search of coffee, ended up in the Bar-Cafe of Travellodge, about 5 minutes wander from telehouse. Sat there for ~ 45 mins waiting for responses, go back, and yes, I can now ping between the machine and the RAC port, WOO! Next step, right, yes, packets are being dropped straight after that, fantastic, so, our default gateway is down... back on the phone, long conversation, hang up, wait 30 mins to hear anything back, finally get told that our default gateway should be a COMPLETELY different address to what it has always been set to, 12 IPs upwards from where it was previously, on an IP that (fortunately) we hadn't already used in our range. SIGH, not happy about this, confirm that if I set the gateway to that IP it does infact work, but switch it back straight after confirming because I want THEM to fix THEIR router mis-configuration, after 2 hours of backwards and forwards getting nowhere I eventually caved, but not before making damned sure that they knew exactly what I thought of this event, and that if it ever happens again I will be severely pissed off, and that I want it in writting that their network engineers will not randomly change our gateway without consulting us ever again. I have a signed fax in my bag from netkonect saying that in the event of needing to change our gateway for whatever reason they will have to contact us to get approval. (I would have thought that was a bloody standard requirement anyway).

Anyways - all dealt with, networking came back, left telehouse.... It wasn't till a little while later that John sent a message asking if I'd remembered to restart the chroot web server (oh bollocks), no, I hadn't, but given it was running nicely until netkonect rebooted our server, and given that I'd forgotten by the point I left telehouse, I gave rough destructions over the phone, and it was all back up shortly afterwards (rah! thanks john).

On a different note, it was good to see you piem, you mad french man you :)

Paston's main webserver and currently reasonably main mailserver (as well as primary DNS), went down this morning. The disks in the SCSI RAID array died (oops), and took the raid array out. John and Dave are now reinstalling all the packages after pulling the data backups from the USB disk (yay for backups!). Hopefully it'll all be up and running again RSN.

Today is not turning out to be a good day.

Right, that's all the sites moved off of the broken server on to the other server, aspseek set to index the right places (and in theory running) and everything appears to be working... time to leave it for a couple of weeks to make sure.

Many many thanks to Dave for the help during that.

Now, lets settle back down to what I was supposed to be working on and what I have a meeting about on Friday.

I've been fighting with one of our webservers for most of the afternoon, and slowly moving all of the sites from it on to a different machine, as it appears to be dying entirely. Hopefully we should have the remaining sites over there tommorow morning, then there's just the waiting till everyone has confirmed that they all work.

Many thanks to Dave who was also working till 6pm, I finished about 30 mins ago for the day, having had one real break of 30 mins, and a 20 min drive since ~ 3.30pm when I got to the server room to fit the other machine with a new disk to take the sites.

And now, I think I shall consider sleeping.

OK - so we've got a very simple backup set for windows boxes on our work network, an rsync of the smb shares happens daily, 3 shares for each machine. Now, our sales guy's machine has just dropped its disk, this in itself is not a big problem, as everything should have been backed up ~ 11am this morning (which it was).... UNFORTUNATELY since I set up the shares, someone decided to change his e-mail client (yes, OK, I know, he should be using imap and we wouldn't have this problem, that's the next issue), and so he's just lost ~ 3 months worth of e-mail, the most important of which being his sent-mail.

sigh.

On the plus side, this might finally get more people using IMAP to our server, and so this situation not recurring :)

I suppose the next problem will be finding him a machine to use in the mean time.

Neeed beer!

Well, at 4pm our shiny new Win2k3 webserver fell over, it appears that it's killed a disk, and that the motherboard is knackered. Won't even get to the BIOS anymore :(

Left work ~ 6pm after me and Dave had got the new system installing, hoping that we can install the secure cert on the server that is needed but we have no key for it (unless we can rescue one from the other disk, which is unlikely).

Ah well, nearly time for the pub now, might nip in to work tommorow to kick it some more, depending on if the MD calls or not :)

Temporarily assigned the IPs to a linux box which serves a page saying "it's broken, should be back monday", so at least there's something there.