So, more because I was intrigued than anything else, I've got a pi3 from Mythic Beasts, they're supplied with IPv6 only connectivity and the file storage is NFS over a private v4 network. The proxy will happily redirect requests to either http or https to the Pi, but this results (without turning on the Proxy Protocol) with getting remote addresses in your logs of the proxy servers, which is not entirely useful.

I've cheated a bit, because the turning on of ProxyProtocol for the hostedpi.com addresses is currently not exposed to customers (it's on the list!), to do it without access to Mythic's backends use your own domainname (I've also got https://pi3.sommitrealweird.co.uk/ mapped to this Pi).

So, first step first, we get our RPi and we make sure that we can login to it via ssh (I'm nearly always on a v6 connection anyways, so this was a simple case of sshing to the v6 address of the Pi). I then installed haproxy and apache2 on the Pi and went about configuring them, with apache2 I changed it to listen to localhost only and on ports 8080 and 4443, I hadn't at this point enabled the ssl module so, really, the change for 4443 didn't kick in. Here's my /etc/apache2/ports.conf file:

# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default.conf

Listen [::1]:8080

<IfModule ssl_module>
       Listen [::1]:4443
</IfModule>

<IfModule mod_gnutls.c>
       Listen [::1]:4443
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

I then edited /etc/apache2/sites-available/000-default.conf to change the VirtualHost line to [::1]:8080.

So, with that in place, now we deploy haproxy infront of it, the basic /etc/haproxy/haproxy.cfg config is:

global
       log /dev/log    local0
       log /dev/log    local1 notice
       chroot /var/lib/haproxy
       stats socket /run/haproxy/admin.sock mode 660 level admin
       stats timeout 30s
       user haproxy
       group haproxy
       daemon

       # Default SSL material locations
       ca-base /etc/ssl/certs
       crt-base /etc/ssl/private

       # Default ciphers to use on SSL-enabled listening sockets.
       # For more information, see ciphers(1SSL). This list is from:
       #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
       ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
       ssl-default-bind-options no-sslv3

defaults
       log     global
       mode    http
       option  httplog
       option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
       errorfile 400 /etc/haproxy/errors/400.http
       errorfile 403 /etc/haproxy/errors/403.http
       errorfile 408 /etc/haproxy/errors/408.http
       errorfile 500 /etc/haproxy/errors/500.http
       errorfile 502 /etc/haproxy/errors/502.http
       errorfile 503 /etc/haproxy/errors/503.http
       errorfile 504 /etc/haproxy/errors/504.http

frontend any_http
        option httplog
        option forwardfor

        acl is_from_proxy src 2a00:1098:0:82:1000:3b:1:1 2a00:1098:0:80:1000:3b:1:1
        tcp-request connection expect-proxy layer4 if is_from_proxy

        bind :::80
        default_backend any_http

backend any_http
        server apache2 ::1:8080

Obviously after that you then do:

systemctl restart apache2
systemctl restart haproxy

Now you have a proxy protocol'd setup from the proxy servers, and you can still talk directly to the Pi over ipv6, you're not yet logging the right remote ips, but we're a step closer. Next enable mod_remoteip in apache2:

a2enmod remoteip

And add a file, /etc/apache2/conf-available/remoteip-logformats.conf containing:

LogFormat "%v:%p %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" remoteip_vhost_combined

And edit the /etc/apache2/sites-available/000-default.conf to change the CustomLog line to use remoteip_vhost_combined rather than combined as the LogFormat and add the relevant RemoteIP settings:

RemoteIPHeader X-Forwarded-For
RemoteIPTrustedProxy ::1

CustomLog ${APACHE_LOG_DIR}/access.log remoteip_vhost_combined

Now, enable the config and restart apache2:

a2enconf remoteip-logformats
systemctl restart apache2

Now you'll get the right remote ip in the logs (cool, huh!), and, better still, the environment that gets pushed through to cgi scripts/php/whatever is now also correct.

So, you can now happily visit http://www.<your-pi-name>.hostedpi.com/, e.g. http://www.srwpi.hostedpi.com/.

Next up, you'll want something like dehydrated - I grabbed the packaged version from debian's jessie-backports repository - so that you can make yourself some nice shiny SSL certificates (why wouldn't you, after all!), once you've got dehydrated installed, you'll probably want to tweak it a bit, I have some magic extra files that I use, I also suggest getting the dehydrated-apache2 package, which just makes it all much easier too.

/etc/dehydrated/conf.d/mail.sh:

CONTACT_EMAIL="my@email.address"

/etc/dehydrated/conf.d/domainconfig.sh:

DOMAINS_D="/etc/dehydrated/domains.d"

/etc/dehydrated/domains.d/srwpi.hostedpi.com:

HOOK="/etc/dehydrated/hooks/srwpi"

/etc/dehydrated/hooks/srwpi:

#!/bin/sh
action="$1"
domain="$2"

case $action in
  deploy_cert)
    privkey="$3"
    cert="$4"
    fullchain="$5"
    chain="$6"
    cat "$privkey" "$fullchain" > /etc/ssl/private/srwpi.pem
    chmod 640 /etc/ssl/private/srwpi.pem
    ;;
  *)
    ;;
esac

/etc/dehydrated/hooks/srwpi has the execute bit set (chmod +x /etc/dehydrated/hooks/srwpi), and is really only there so that the certificate can be used easily in haproxy.

And finally the file /etc/dehydrated/domains.txt:

www.srwpi.hostedpi.com srwpi.hostedpi.com

Obviously, use your own pi name in there, or better yet, one of your own domain names that you've mapped to the proxies.

Run dehydrated in cron mode (it's noisy, but meh...):

dehydrated -c

That s then generated you some shiny certificates (hopefully). For now, I'll just tell you how to do it through the /etc/apache2/sites-available/default-ssl.conf file, just edit that file and change the SSLCertificateFile and SSLCertificateKeyFile to point to /var/lib/dehydrated/certs/www.srwpi.hostedpi.com/fullchain.pem and /var/llib/dehydrated/certs/ww.srwpi.hostedpi.com/privkey.pem files, do the edit for the CustomLog as you did for the other default site, and change the VirtualHost to be [::1]:443 and enable the site:

a2ensite default-ssl
a2enmod ssl

And restart apache2:

systemctl restart apache2

Now time to add some bits to haproxy.cfg, usefully this is only a tiny tiny bit of extra config:

frontend any_https
        option httplog
        option forwardfor

        acl is_from_proxy src 2a00:1098:0:82:1000:3b:1:1 2a00:1098:0:80:1000:3b:1:1
        tcp-request connection expect-proxy layer4 if is_from_proxy

        bind :::443 ssl crt /etc/ssl/private/srwpi.pem

        default_backend any_https

backend any_https
        server apache2 ::1:4443 ssl ca-file /etc/ssl/certs/ca-certificates.crt

Restart haproxy:

systemctl restart haproxy

And we're all done! REMOTE_ADDR will appear as the correct remote address in the logs, and in the environment.

Over the last week I've been working to get the patches for Pound in to Debian for the POODLE vulnerabilty, the new package in unstable has the DisableSSLv3 option available which allows for disabling SSLv3 (fundamental for POODLE), and is waiting to migrate to testing, once that's happen I'll be looking to get that version in to wheezy-backports.

In the mean time, I'm currently rebuilding my personal repository, and will update this post once it's available, along with the GPG public key that it's signed with, the repository will contain wheezy-backports versions of pound, built from the unstable sources, along with a newer version of pound from the current experimental (2.7d) release of pound.

I also raised a ticket for EPEL 7 to get them to update from the 2.7c version that they currently have in the repository to the 2.7d version, which mostly only has the added option to disable various versions of SSL, and that appears to currently be in progress.

Internally at Mythic Beasts we have a repository for managed customers which already has the patched versions for both CentOS and Debian Wheezy.

So, got a beep this morning from our work monitoring system. One of our customers domain names is hosted with livedns.co.uk (which, as far as I can tell, is part of the Fasthosts franchise)... It appears that Fasthosts have managed to entirely break their DNS:

brettp@laptop:~$ host www.fasthosts.com
;; connection timed out; no servers could be reached
brettp@laptop:~$ whois fasthosts.com | grep -i "Name Server"
   Name Server: NS1.FASTHOSTS.NET.UK
   Name Server: NS2.FASTHOSTS.NET.UK
Name Server: NS1.FASTHOSTS.NET.UK
Name Server: NS2.FASTHOSTS.NET.UK
brettp@laptop:~$ whois fasthosts.net.uk | grep -A 2 "Name servers:"
    Name servers:
        ns1.fasthosts.net.uk      213.171.192.252
        ns2.fasthosts.net.uk      213.171.193.248
brettp@laptop:~$  host -t ns fasthosts.net.uk 213.171.192.252
;; connection timed out; no servers could be reached
brettp@laptop:~$ host -t ns fasthosts.net.uk 213.171.193.248
;; connection timed out; no servers could be reached
brettp@laptop:~$

So, that's fasthosts core nameservers not responding, good start! They also provide livedns.co.uk, so lets have a look at that:

brettp@laptop:~$ whois livedns.co.uk | grep -A 3 "Name servers:"
    Name servers:
        ns1.livedns.co.uk         213.171.192.250
        ns2.livedns.co.uk         213.171.193.250
        ns3.livedns.co.uk         213.171.192.254
brettp@laptop:~$ host -t ns ns1.livedns.co.uk 213.171.192.250
;; connection timed out; no servers could be reached
brettp@laptop:~$ host -t ns ns1.livedns.co.uk 213.171.193.250
;; connection timed out; no servers could be reached
brettp@laptop:~$ host -t ns ns1.livedns.co.uk 213.171.192.254
;; connection timed out; no servers could be reached

So, erm, apparently that's all their DNS servers "Not entirely functioning correctly"! That's quite impressive!

So, I've changed jobs, I now work for the incredibly awesome Mythic Beasts, So far it's been great, we've hit a few problems, we've made sane plans, and we're moving forwards and making things easier to manage long term. I've mostly been involved with debugging mac-mini boot problems, and getting them working with (as far as possible) stock kernels, rather than our previous set of custom compiled kernels, so that it's easier to upgrade later, and so that we're not having to maintain a whole set of patches that we shouldn't need to! We're also starting to package our own software that we use internally so that actually deploying new servers is becoming a lot easier, and configuration of our previous packages is now (mostly) handled by debconf, meaning that actually setting up backups is becoming much easier.

We've also changed out main backup strategy from using a (slightly flaky) openvpn connection with loopback mounted ext3 filesystems to using rsync with the --fake-super option - the only thing that we've found slightly wrong with this so far is that --fake-super can't reconcile symlinks (it uses user level xattr to store the origional permissions, and with symlinks this doesn't work, because, apparently, symlinks should store user xattr attrbutes!).

We also offer ipv6 for all our dedicated and vps customers, as per the mythic blog - and our 2 authoritative name servers are both v6 enabled. We're working towards making all of our core services available over both ipv4 and ipv6, and we should be there soon.

It's fantastic to work for a company that has the technical knowledge and no-shit stance that Mythic Beasts has, we will hold our hands up if we do something wrong, and we'll tell you exactly what happened, why, and what we're doing to stop it from happening again.

So, 1 month in, and I'm absolutely loving working for them, and I've been a customer of theirs since black cat networks sold their domain name side to them, and shortly afterwards bluelinux sold their virtual servers service to them.

Our new VDS lite service is reasonably priced, and well spec'd. I thoroughly recommend our services, and obviously, we actually use our own vds service to provide some of our services - so it's as important to us that it keeps working as it is for our customers.

Wow, so that was a very salesy post. Erm, next time, back to whinging about authors not having written the books I'm waiting for yet! :)

So, my current place of employ (for another 3 weeks and 1 day) have a bunch of dedicated servers from a large hosting company in the UK, occasionally they break their already truly broken internal networking (yes, it defaults to blocking nearly all TCP ports between servers in their own data center, but if you want to hit those same ports from outside their network, that's absolutely fine!), I get to deal with the same thread at least once a quarter, when something in their network decides to block one of the few ports that are allowed... that port is 22, and ssh is used "rather a lot" between the approx 50 servers we have with them. Just for shits and giggles, all of their infrastructure filters ICMP making traceroute mostly totally useless. tcptraceroute ain't no better. Repeating the same question, asking for what they did when it magically all then works again, and getting a "we didn't do anything!" response is somewhat taxing, almost as taxing as watching them run commands on the servers and them not actually know what they're doing, but appear to have a vague grasp on iptables being a firewall, and that they can maybe read the rules and go, "oh, you weren't lying.". Their support is also so far lagged that it's ridiculous, other than an auto response, it took 3 hours (during office hours) to get a "oh, yeah, what you've said, that all looks good, now give us access so that we can diagnose further..." when you do give them that... it'll be between 4 and 6 hours before they bother logging in, and closer to 8 to 10 hours for them to go "oh, it works now, but I don't know why!". NYARGH. These guys make bedroom ISPs look professional to the max. The only thing they have almost going for them is that if you order a new dedicated server it is usually spun up within the hour.

And here endeth the vent.

So, at work we have a little single port IP KVM, it is a StarTech SV1107IPEXT - it's not a "bad" bit of kit, if you're completely configured for using it in the way they expect... When you're not, then it throws a NullPointerException when you try to connect with it's Java applet.

This morning (mostly because the monitor and keyboard I had been using for building stuff on my desk has gone missing) I decided I'd try again with the device, it's not on our default office network range, mostly because it might not always be in the office, so I quickly added an ip alias using ip addr add 192.168.0.59/24 dev wlan0 on my laptop to bring up a route to it. Navigated to the page and asked it to fire up the Java applet. Straight off, a NullPointerException happens - hmm. So looked at what was being logged by the JVM - it was trying to run ifconfig (no, really, the java applet ran ifconfig... with no path...) which, on my debian laptop, is not in the default path... so I went and make a quick wrapper in ~/bin for it... right - now it can run that... still a NPE, hmm. So look a little closer, it's looking for the MAC address assigned to the IP that you're coming from... no, really! WTF! So I went and hacked my little ifconfig wrapper to rewrite the address for wlan0 to be what the KVM expected it to be, low and behold, I can now run the java client (hoorah!).

Why the heck are they doing that though - that's just plain insane! On the plus side, I now have a work around, on the minus side, egad that's ugly!

So, as I'm not always sitting directly in front of machines that I'm working on I've been using various tools to get temporary network connections between completely seperate network blocks, for example, sitting on wireless outside of my home, connecting back in to the desktop at home, and then connecting to the virtual machines on that desktop - the virtual machines are on their own private network, which is bridged and then routed through the desktop. The desktop machine has an ipv6 tunnel setup using tinc to my main VPS. That in turn has a tunnel setup to he.net's ipv6 network... I then have my laptop configured to also use tinc to connect to my VPS, so lots of ipv6 over ipv4, really.

From that I can then get a 'direct' ssh connection between my laptop and my desktop (OK, so it's actually going over 2 ipv6 over ipv4 tunnels to get there, but it "looks" direct), so, that gets me as far as the external network on that box. From here, we can run a vde_switch which is connected to a tun/tap interface that's part of the internal bridge, once we've got that running, I bring up another vde_switch on my laptop attached to another tun/tap interface purely for that network with a static ip configure on that interface... using the magic of dpipe I then connect the two vde_switches together using vde_plug, and as if by magic, I then have access to the internal networking of my virtual machines, meaning that I can now ssh directly in to the virtual machines without having to do anymore interesting ssh bouncing and port forwarding.

It's a little sick, but it really does make remote working a lot easier to do, it's a vpn of sorts, I suppose, as all traffic is being thrown over ssh pipes or tinc (which in turn is encrypted using x509 certificates)... probably not the quickest of networks, but very very nice all the same.

ARGH! So, now I know the issue, the disk was created with a standard dos partition table... this won't extend past the bit that I'm at... it needs to be GPT, so there's going to need to be some downtime to sort it out. SIGH.

I'm having some issues with LVM, basically it appears that I can't assign over a certain amount of my pv, where that certain amount is somewhere around 1.5TB... output from all the relevant chunks are:

ensenada:~# lvdisplay
  --- Logical volume ---
  LV Name                /dev/main/root
  VG Name                main
  LV UUID                ZR33WS-glE2-bUn6-adCX-n50C-37eX-Fc7Pde
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                7.45 GB
  Current LE             1907
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0

  --- Logical volume ---
  LV Name                /dev/main/voldemort
  VG Name                main
  LV UUID                DrT0co-THRw-6Zcy-Q3PT-OiQj-4KBz-xfV0Un
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.37 TB
  Current LE             358400
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1

ensenada:~# vgdisplay
  --- Volume group ---
  VG Name               main
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  56
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               5.45 TB
  PE Size               4.00 MB
  Total PE              1428053
  Alloc PE / Size       360307 / 1.37 TB
  Free  PE / Size       1067746 / 4.07 TB
  VG UUID               ddVSIr-W9BU-XOWJ-gdSo-HZ4m-vw3M-S33Q3s

ensenada:~# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda3
  VG Name               main
  PV Size               5.45 TB / not usable 1.54 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              1428053
  Free PE               1067746
  Allocated PE          360307
  PV UUID               sFxRYg-Ua60-zpMT-DMqo-tpD5-d24W-qfW31p

ensenada:~# lvcreate -n test -L 74G main
  Logical volume "test" created
ensenada:~# pvs
  PV         VG   Fmt  Attr PSize PFree
  /dev/sda3  main lvm2 a-   5.45T 4.00T
ensenada:~# lvs
  LV        VG   Attr   LSize  Origin Snap%  Move Log Copy%  Convert
  root      main -wi-ao  7.45G
  test      main -wi-a- 74.00G
  voldemort main -wi-ao  1.37T
ensenada:~# lvremove /dev/main/test
Do you really want to remove active logical volume "test"? [y/n]: y
  Logical volume "test" successfully removed
ensenada:~# lvcreate -n test -L 75G main
  device-mapper: reload ioctl failed: Invalid argument
  Aborting. Failed to activate new LV to wipe the start of it.
  LV main/test in use: not deactivating
  Unable to deactivate failed new LV. Manual intervention required.
ensenada:~# lvremove /dev/main/test
Do you really want to remove active logical volume "test"? [y/n]: y
  Logical volume "test" successfully removed
ensenada:~# lvcreate -n test -L 74G main
  Logical volume "test" created
ensenada:~# lvremove /dev/main/test
Do you really want to remove active logical volume "test"? [y/n]: y
  Logical volume "test" successfully removed
ensenada:~#

Please, someone, put me out of my misery - this is driving me nuts!

Extra info:

ensenada:~# uname -a
Linux ensenada 2.6.26-2-amd64 #1 SMP Fri Aug 14 07:12:04 UTC 2009 x86_64 GNU/Linux
ensenada:~# dpkg -l lvm2
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name                  Version               Description
+++-=====================-=====================-==========================================================
ii  lvm2                  2.02.39-7             The Linux Logical Volume Manager
ensenada:~#
ensenada:~# cat /etc/debian_version
5.0.1

(Erm, just remembered that I haven't got comments enabled at the mo, so if you've got fixes, please mail me! iDunno@sommitrealweird.co.uk - thanks!)

So, summer has arrived - how can I tell? The heat is killing the servers in our office again, the poor little AC unit can't keep up when the temperature outside is getting to summer weather. So the door to the magical cupboard is open for the moment, as the office temperature is slightly lower (but still, not great), and it's keeping the cupboard at somewhere around 26, which is still a little higher than I'd like, but better than the 32/33 that knocks the main firewall out.

I'll be replacing the main firewall over the weekend with a lower power (and thus less heat generating) soekris box (which should have enough grunt to shift the packets), and means that we'll go from OpenBSD 4.3 -> OpenBSD 4.5 at the same time.

Also, it's time to finally get the workstations on to lenny - as they're all NFS root, it becomes slightly tricky to get the setup just right for them... hopefully we can start rolling it out to the developers early next week, and then we should be in business...

So, was supposed to end up in Hector's House playing pool with various people that I know, but all of 'em were out last night (erm, half of 'em with me) and didn't quite manage to make it... So after an hour of waiting for people to turn up I gave up and considered going back to my "local" (the hop poles) to have a swift beer or three... instead, it being a wednesday, and knowing that Band Aids was on at the Pav Tav, I wandered there... managed to get there before the first band played - they were Vier and were absolutely bloody fantastic. Stayed and watched/danced to the other 3 bands and all was well with the world.

Now it's time for the sleep and the getting everything else ready for the big move on Friday (yay for working Good Friday... or something!)

There's a new Bravia advert coming - watch the web!

Why does Oracle complete an import with... "Import terminated successfully with warnings." when what it actually means is... "I ran out of table space mid import, but I'm only going to tell you that in an obscure way in the log, and otherwise make you think that it was alllll OK"? Surely in this day and age it could just say "Import terminated unsuccessfully: ran the hell out of table space", wouldn't that be nice?

Mid import of a large database was thwarted by cron cheerfully having a job that I was unaware of shutdown the database server! Bah! Oh well, maybe now I should actually head home, because this could "take some time".

Last night we moved Paston's servers from Telehouse to Global switch, so we're now hosting in Mistrals space in Global Switch... Now, Global Switch is a reasonably pretty looking building (compared with Telehouse, at least), and the machine move went quite well, with ~ 30 -> 45 mins downtime, ish. Things that went wrong: given a network cable, plugged in to switch, got nothing, monkey in Global Switch couldn't work out what was wrong, left him to it for ~ 5 to 10 mins before I had a sudden thought of "hmm, I wonder, I wonder if that's a straight or crossover cable", plugged it in to the uplink port, pushed the magic button... Hey presto, we have connection... Confront monkey, Monkey says "Well, they don't tell me these things". NYARGH.

So, anyways, got the 2 machines on to the net, tested everything was running, tested that it worked from remote machines, you know, all the usual crap... Leave Global Switch at 12.45am ish, get back to Norwich for 3am ish, get to bed for ~ 3.20am ish... Get a phone call at 10am saying "erm, it's not working for this bunch of people", the bunch of people in question being our ADSL customers, who were all recently migrated to Mistral's network... the problem? 2 of Mistral's ADSL routers couldn't find our machines... Mistral's suggested solutions to the problem: (1) wait 7 days and we'll fix it or (2) we'll connect you to a different switch port. So, fair enough, not too bad, now apparently connected to a different switch port... but to add insult to it, apparently the servers where "put in the rack the wrong way round for the airflow", we put it in the same way round as the other server in that rack, and the way we were told to put it in by the afore mentioned monkey... so this morning our kit was turned off for 10 mins while they turned it round in the rack, it's all come back cleanly (thank god), but ffs, why do they hire entirely useless fucking staff that apparently get this wrong quite often?

Anyways - end of rant... pub tonight as listed in the other post :)

Why doesn't Win2k3 have a nice way of exporting "Local User and Groups" for importing to a different Win2k3 server? At the moment, this would be really handy to be able to do. Oh well, I'll have to see if doing a restore and not letting it reboot and then restoring the hardware configuration works.

Not long left now, though... only 9 more working days for Paston.

Yesterday our leased line from UUNet (MCI) was cut off by BT at around 1.30pm, 7 days early. It should have been cut off next week on the 30th, as such a lot of rushing around, praying and general chaos has taken over while we continue (with the machines now in this building on the end of an ADSL line, and not accessable to the outside world) to move anything remaining off of the servers.

Of course, that was also our primary DNS server, yay for BT :(

Well, I've been up since ~ 6.45 ish, and was in Telehouse from ~ 10.45am through till 4.40pm (ish). Yesterday, our server hosted by netkonect in Telehouse "magically" lost routing to the world, I repeatedly told them that they had a problem, that the machine was fine before they had the other outage, and that traffic was consistently being dropped at their gateway... They insisted on rebooting our server, claiming that that would probably fix it... when it rebooted, of course, the routes and interfaces where set up exactly as before. Still no network. At 9.30pm Friday night we gave up and I resigned my self to going to London Saturday morning, got to telehouse, got to the right room, discovered that I hadn't got a key to our rack, or the monitor and keyboard that I'd asked for. So, phone netkonect, get a "oh, they should have given you the key at reception", right, trundled back to reception, yes, got the key, rah, now have access to the rack, but still no sign of a monitor and keyboard, or the netkonect engineer that was supposed to visit me, or the mistral engineer that was supposed to visit me. OK, fine, out comes the laptop, crossover cable, and some fiddling at the back of the server. Rah, I have ssh to it. Right, set up the second ethernet card to have a private address just so that the laptop can sit on that port and talk to it. Plug back in the other network lead, hey presto, still nothing from our box to the outside world, all being dropped at this point by the switch (I couldn't ping between the box and it's RAC port, which are on the same switch). Right, phone netkonect, tell them this, wait for a response, get bored, it's now midday and I'm dying of thirst, of course, the telehouse coffee shop is shut sigh, so off I go in search of coffee, ended up in the Bar-Cafe of Travellodge, about 5 minutes wander from telehouse. Sat there for ~ 45 mins waiting for responses, go back, and yes, I can now ping between the machine and the RAC port, WOO! Next step, right, yes, packets are being dropped straight after that, fantastic, so, our default gateway is down... back on the phone, long conversation, hang up, wait 30 mins to hear anything back, finally get told that our default gateway should be a COMPLETELY different address to what it has always been set to, 12 IPs upwards from where it was previously, on an IP that (fortunately) we hadn't already used in our range. SIGH, not happy about this, confirm that if I set the gateway to that IP it does infact work, but switch it back straight after confirming because I want THEM to fix THEIR router mis-configuration, after 2 hours of backwards and forwards getting nowhere I eventually caved, but not before making damned sure that they knew exactly what I thought of this event, and that if it ever happens again I will be severely pissed off, and that I want it in writting that their network engineers will not randomly change our gateway without consulting us ever again. I have a signed fax in my bag from netkonect saying that in the event of needing to change our gateway for whatever reason they will have to contact us to get approval. (I would have thought that was a bloody standard requirement anyway).

Anyways - all dealt with, networking came back, left telehouse.... It wasn't till a little while later that John sent a message asking if I'd remembered to restart the chroot web server (oh bollocks), no, I hadn't, but given it was running nicely until netkonect rebooted our server, and given that I'd forgotten by the point I left telehouse, I gave rough destructions over the phone, and it was all back up shortly afterwards (rah! thanks john).

On a different note, it was good to see you piem, you mad french man you :)

Paston's main webserver and currently reasonably main mailserver (as well as primary DNS), went down this morning. The disks in the SCSI RAID array died (oops), and took the raid array out. John and Dave are now reinstalling all the packages after pulling the data backups from the USB disk (yay for backups!). Hopefully it'll all be up and running again RSN.

Today is not turning out to be a good day.

Right, that's all the sites moved off of the broken server on to the other server, aspseek set to index the right places (and in theory running) and everything appears to be working... time to leave it for a couple of weeks to make sure.

Many many thanks to Dave for the help during that.

Now, lets settle back down to what I was supposed to be working on and what I have a meeting about on Friday.