So, I backed the Gemini and received my shiny new device just a few months after they said that it'd ship, not bad for an indiegogo project! Out of the box, I flashed it, using the non-approved linux flashing tool at that time, and failed to backup the parts that, err, I really didn't want blatted... So within hours I had a new phone that I, err, couldn't make calls on, which was marginally annoying. And the tech preview of Debian wasn't really worth it, as it was fairly much unusable (which was marginally upsetting, but hey) - after a few more hours / days of playing around I got the IMEI number back in to the Gemini and put back on the stock android image. I didn't at this point have working bluetooth or wifi, which was a bit of a pain too, turns out the mac addresses for those are also stored in the nvram (doh!), that's now mostly working through a bit of collaboration with another Gemini owner, my Gemini currently uses the mac addresses from his device... which I'll need to fix in the next month or so, else we'll have a mac address collision, probably.

Overall, it's not a bad machine, the keyboard isn't quite as good as I was hoping for, the phone functionality is not bad once you're on a call, but not great until you're on a call, and I certainly wouldn't use it to replace the Samsung Galaxy S7 Edge that I currently use as my full time phone. It is however really rather useful as a sysadmin tool when you don't want to be lugging a full laptop around with you, the keyboard is better than using the on screen keyboard on the phone, the ssh client is "good enough" to get to what I need, and the terminal font isn't bad. I look forward to seeing where it goes, I'm happy to have been an early backer, as I don't think I'd pay the current retail price for one.

So, you're new to the tech industry, you've only just got your GCSEs, you're mostly, by law, at this point, able to have "teh sex" anytime you want... You've then signed up to a course that has a bunch of limitations of where you can be, and what you can do, at various (ok - all), points of the day. Isn't this exciting, you're going to learn new things. Probably.

So, here are my problems, first off, the rules state that you can't go in to the building or bedroom of anyone of the other sex. Are you fucking kidding me at this point? A shit load of my learning was based from the other sex - we weren't trying to fuck each other - we were bouncing ideas and talking to each other, as equals, not as a different sex, purely as people. Why the actual fuck would you separate people from each other as a base line? No. Really.

Then, whilst I read the rules, I had the other idea in my head, which will entirely infuriate the stupid rule makers - although it's against the rules for opposite sex couples to ever see each other, it's entirely within the rules for any gay couple to cheerfully stay in the rules and invite friends round if so wanted! So, a win for them, obviously.

Of course, at all points, these are hormonaly charged teenagers, so, your entire back line is to kick them out of your course... please, someone, tell me how this doesn't end up with a lack of mutual respect? This is fucking stupid. sigh.

So, more because I was intrigued than anything else, I've got a pi3 from Mythic Beasts, they're supplied with IPv6 only connectivity and the file storage is NFS over a private v4 network. The proxy will happily redirect requests to either http or https to the Pi, but this results (without turning on the Proxy Protocol) with getting remote addresses in your logs of the proxy servers, which is not entirely useful.

I've cheated a bit, because the turning on of ProxyProtocol for the hostedpi.com addresses is currently not exposed to customers (it's on the list!), to do it without access to Mythic's backends use your own domainname (I've also got https://pi3.sommitrealweird.co.uk/ mapped to this Pi).

So, first step first, we get our RPi and we make sure that we can login to it via ssh (I'm nearly always on a v6 connection anyways, so this was a simple case of sshing to the v6 address of the Pi). I then installed haproxy and apache2 on the Pi and went about configuring them, with apache2 I changed it to listen to localhost only and on ports 8080 and 4443, I hadn't at this point enabled the ssl module so, really, the change for 4443 didn't kick in. Here's my /etc/apache2/ports.conf file:

# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default.conf

Listen [::1]:8080

<IfModule ssl_module>
       Listen [::1]:4443
</IfModule>

<IfModule mod_gnutls.c>
       Listen [::1]:4443
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

I then edited /etc/apache2/sites-available/000-default.conf to change the VirtualHost line to [::1]:8080.

So, with that in place, now we deploy haproxy infront of it, the basic /etc/haproxy/haproxy.cfg config is:

global
       log /dev/log    local0
       log /dev/log    local1 notice
       chroot /var/lib/haproxy
       stats socket /run/haproxy/admin.sock mode 660 level admin
       stats timeout 30s
       user haproxy
       group haproxy
       daemon

       # Default SSL material locations
       ca-base /etc/ssl/certs
       crt-base /etc/ssl/private

       # Default ciphers to use on SSL-enabled listening sockets.
       # For more information, see ciphers(1SSL). This list is from:
       #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
       ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
       ssl-default-bind-options no-sslv3

defaults
       log     global
       mode    http
       option  httplog
       option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
       errorfile 400 /etc/haproxy/errors/400.http
       errorfile 403 /etc/haproxy/errors/403.http
       errorfile 408 /etc/haproxy/errors/408.http
       errorfile 500 /etc/haproxy/errors/500.http
       errorfile 502 /etc/haproxy/errors/502.http
       errorfile 503 /etc/haproxy/errors/503.http
       errorfile 504 /etc/haproxy/errors/504.http

frontend any_http
        option httplog
        option forwardfor

        acl is_from_proxy src 2a00:1098:0:82:1000:3b:1:1 2a00:1098:0:80:1000:3b:1:1
        tcp-request connection expect-proxy layer4 if is_from_proxy

        bind :::80
        default_backend any_http

backend any_http
        server apache2 ::1:8080

Obviously after that you then do:

systemctl restart apache2
systemctl restart haproxy

Now you have a proxy protocol'd setup from the proxy servers, and you can still talk directly to the Pi over ipv6, you're not yet logging the right remote ips, but we're a step closer. Next enable mod_remoteip in apache2:

a2enmod remoteip

And add a file, /etc/apache2/conf-available/remoteip-logformats.conf containing:

LogFormat "%v:%p %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" remoteip_vhost_combined

And edit the /etc/apache2/sites-available/000-default.conf to change the CustomLog line to use remoteip_vhost_combined rather than combined as the LogFormat and add the relevant RemoteIP settings:

RemoteIPHeader X-Forwarded-For
RemoteIPTrustedProxy ::1

CustomLog ${APACHE_LOG_DIR}/access.log remoteip_vhost_combined

Now, enable the config and restart apache2:

a2enconf remoteip-logformats
systemctl restart apache2

Now you'll get the right remote ip in the logs (cool, huh!), and, better still, the environment that gets pushed through to cgi scripts/php/whatever is now also correct.

So, you can now happily visit http://www.<your-pi-name>.hostedpi.com/, e.g. http://www.srwpi.hostedpi.com/.

Next up, you'll want something like dehydrated - I grabbed the packaged version from debian's jessie-backports repository - so that you can make yourself some nice shiny SSL certificates (why wouldn't you, after all!), once you've got dehydrated installed, you'll probably want to tweak it a bit, I have some magic extra files that I use, I also suggest getting the dehydrated-apache2 package, which just makes it all much easier too.

/etc/dehydrated/conf.d/mail.sh:

CONTACT_EMAIL="my@email.address"

/etc/dehydrated/conf.d/domainconfig.sh:

DOMAINS_D="/etc/dehydrated/domains.d"

/etc/dehydrated/domains.d/srwpi.hostedpi.com:

HOOK="/etc/dehydrated/hooks/srwpi"

/etc/dehydrated/hooks/srwpi:

#!/bin/sh
action="$1"
domain="$2"

case $action in
  deploy_cert)
    privkey="$3"
    cert="$4"
    fullchain="$5"
    chain="$6"
    cat "$privkey" "$fullchain" > /etc/ssl/private/srwpi.pem
    chmod 640 /etc/ssl/private/srwpi.pem
    ;;
  *)
    ;;
esac

/etc/dehydrated/hooks/srwpi has the execute bit set (chmod +x /etc/dehydrated/hooks/srwpi), and is really only there so that the certificate can be used easily in haproxy.

And finally the file /etc/dehydrated/domains.txt:

www.srwpi.hostedpi.com srwpi.hostedpi.com

Obviously, use your own pi name in there, or better yet, one of your own domain names that you've mapped to the proxies.

Run dehydrated in cron mode (it's noisy, but meh...):

dehydrated -c

That s then generated you some shiny certificates (hopefully). For now, I'll just tell you how to do it through the /etc/apache2/sites-available/default-ssl.conf file, just edit that file and change the SSLCertificateFile and SSLCertificateKeyFile to point to /var/lib/dehydrated/certs/www.srwpi.hostedpi.com/fullchain.pem and /var/llib/dehydrated/certs/ww.srwpi.hostedpi.com/privkey.pem files, do the edit for the CustomLog as you did for the other default site, and change the VirtualHost to be [::1]:443 and enable the site:

a2ensite default-ssl
a2enmod ssl

And restart apache2:

systemctl restart apache2

Now time to add some bits to haproxy.cfg, usefully this is only a tiny tiny bit of extra config:

frontend any_https
        option httplog
        option forwardfor

        acl is_from_proxy src 2a00:1098:0:82:1000:3b:1:1 2a00:1098:0:80:1000:3b:1:1
        tcp-request connection expect-proxy layer4 if is_from_proxy

        bind :::443 ssl crt /etc/ssl/private/srwpi.pem

        default_backend any_https

backend any_https
        server apache2 ::1:4443 ssl ca-file /etc/ssl/certs/ca-certificates.crt

Restart haproxy:

systemctl restart haproxy

And we're all done! REMOTE_ADDR will appear as the correct remote address in the logs, and in the environment.

Yay! So, it's a year and a bit on from the last post (eeep!), and we get the news of the Psion Gemini - I wants one, that looks nice and shiny and just the right size to not be inconvenient to lug around all the time, and far better for ssh usage than the onscreen keyboard on my phone!

So, what ever happened to Psion, and more importantly, the Psion 5. The Psion 5 was an awesome pocket sized machine, it had a full qwerty keyboard that was actually quite nice to type on, it had graphics that were way beyond it's time (640x320) in your pocket. It had a whole desktop suite of applications, including word processing and spreadsheets, and yet, psion died... The most scary bit of this is that mobile phones are getting bigger than the original Psion 5, and yet have sacrificed the keyboard for, well, who the hell knows what - the Psion 5s keyboard was awesome. Mobile phone manufacturers need to take a step back and work out how to get that keyboard on to a new gen phone that's touch only... The psion 5 also had a touch screen (yeah, ok, only single point, not multi finger action), was pocket sized, near indestructable, and hell, if one now existed that could make calls I'd seriously consider it.

Dear cold calling goits, will you please learn that the landline of this house is not the home owners, and that asking for Mr Parker will get you the question of "which". Asking for the one that's the home owner is likely to get you sworn at as that Mr Parker died getting close to 2 years ago.

Now, stop damned well calling me.

(OK - so I normally know on the basis that there is caller id on the landline, but it's still damned annoying!)

Over the last week I've been working to get the patches for Pound in to Debian for the POODLE vulnerabilty, the new package in unstable has the DisableSSLv3 option available which allows for disabling SSLv3 (fundamental for POODLE), and is waiting to migrate to testing, once that's happen I'll be looking to get that version in to wheezy-backports.

In the mean time, I'm currently rebuilding my personal repository, and will update this post once it's available, along with the GPG public key that it's signed with, the repository will contain wheezy-backports versions of pound, built from the unstable sources, along with a newer version of pound from the current experimental (2.7d) release of pound.

I also raised a ticket for EPEL 7 to get them to update from the 2.7c version that they currently have in the repository to the 2.7d version, which mostly only has the added option to disable various versions of SSL, and that appears to currently be in progress.

Internally at Mythic Beasts we have a repository for managed customers which already has the patched versions for both CentOS and Debian Wheezy.

So, I have a SEN account (it's part of the PSN), I have 2 videos with SEN, I have a broken PS3 so I can no deactivate video (you can only do that from the console itself, yes, really)... and the response from SEN has been abysmal, specifically:

As we take the security of SEN accounts very seriously, we are unable to provide support on this matter by e-mail as we will need you to answer some security questions before we can investigate this further. We need you to phone us in order to verify your account details because we're not allowed to verify details via e-mail.

I mean, seriously, they're going to verify my details over the phone better than over e-mail how exactly? All the contact details are tied to my e-mail account, I have logged in to their control panel and renamed the broken PS3 to "Broken PS3", I have given them the serial number of the PS3, and yet they insist that I need to call them, because apparently they're fucking stupid. I'm damned glad that I only ever got 2 videos from SEN, both of which I own on DVD now anyways, this kind of idiotic tie in to a system is badly wrong.

So, you phone the number... and now you get stuck with hold music for ever... oh, yeah, great customer service here guys. I mean, seriously, WTF.

OK - 10 minutes on the phone, and still being told "One of our advisors will be with you shortly". I get the feeling that I'll just be writing off the 2 videos that I no longer have access to.

I'm damned glad that I didn't decide to buy more content from that - at least you can reset the games entitlement once every six months without jumping through all these hoops (you have to reactivate each console that you still want to use, but hey).

So, got a beep this morning from our work monitoring system. One of our customers domain names is hosted with livedns.co.uk (which, as far as I can tell, is part of the Fasthosts franchise)... It appears that Fasthosts have managed to entirely break their DNS:

brettp@laptop:~$ host www.fasthosts.com
;; connection timed out; no servers could be reached
brettp@laptop:~$ whois fasthosts.com | grep -i "Name Server"
   Name Server: NS1.FASTHOSTS.NET.UK
   Name Server: NS2.FASTHOSTS.NET.UK
Name Server: NS1.FASTHOSTS.NET.UK
Name Server: NS2.FASTHOSTS.NET.UK
brettp@laptop:~$ whois fasthosts.net.uk | grep -A 2 "Name servers:"
    Name servers:
        ns1.fasthosts.net.uk      213.171.192.252
        ns2.fasthosts.net.uk      213.171.193.248
brettp@laptop:~$  host -t ns fasthosts.net.uk 213.171.192.252
;; connection timed out; no servers could be reached
brettp@laptop:~$ host -t ns fasthosts.net.uk 213.171.193.248
;; connection timed out; no servers could be reached
brettp@laptop:~$

So, that's fasthosts core nameservers not responding, good start! They also provide livedns.co.uk, so lets have a look at that:

brettp@laptop:~$ whois livedns.co.uk | grep -A 3 "Name servers:"
    Name servers:
        ns1.livedns.co.uk         213.171.192.250
        ns2.livedns.co.uk         213.171.193.250
        ns3.livedns.co.uk         213.171.192.254
brettp@laptop:~$ host -t ns ns1.livedns.co.uk 213.171.192.250
;; connection timed out; no servers could be reached
brettp@laptop:~$ host -t ns ns1.livedns.co.uk 213.171.193.250
;; connection timed out; no servers could be reached
brettp@laptop:~$ host -t ns ns1.livedns.co.uk 213.171.192.254
;; connection timed out; no servers could be reached

So, erm, apparently that's all their DNS servers "Not entirely functioning correctly"! That's quite impressive!

So, using the command:

root@new# ssh root@old dd if=/dev/vg/somedisk | dd of=/dev/vg/somedisk

appears to fail, getting a SIGTERM at some point for no discernable reason... however, using

root@old# dd if=/dev/vg/somedisk | ssh root@new dd of=/dev/vg/somedisk

works fine.

The pull version fails at a fairly random point after a fairly undefined period of time. The push version works everytime. This is most confusing and odd...

Dear lazyweb, please give me some new ideas as to what's going on, it's driving me nuts!

Update: solved...

A different daemon wasn't limiting it's killing habits in the case that a certain process wasn't running, and was killing the ssh process on the new server almost at random, found the bug in the code and now testing with that.

Thanks for all the suggestions though, much appreciated.

There's been a recent spate of spam from domains held by Communicado Ltd, to counter this Martin A Brooks has created a lovely list of domains to easily filter, see his blog post for information on the distribution and usage of this list.

Of the (entire!) 35 comments on my blog that I've approved, 5 of those are actually from ipv6 addresses! Well done you ipv6 early uptakers! You all deserve a pint. (If you ever happen to turn up in the same pub as me, you should probably remind me I said that :) )

So, I've changed jobs, I now work for the incredibly awesome Mythic Beasts, So far it's been great, we've hit a few problems, we've made sane plans, and we're moving forwards and making things easier to manage long term. I've mostly been involved with debugging mac-mini boot problems, and getting them working with (as far as possible) stock kernels, rather than our previous set of custom compiled kernels, so that it's easier to upgrade later, and so that we're not having to maintain a whole set of patches that we shouldn't need to! We're also starting to package our own software that we use internally so that actually deploying new servers is becoming a lot easier, and configuration of our previous packages is now (mostly) handled by debconf, meaning that actually setting up backups is becoming much easier.

We've also changed out main backup strategy from using a (slightly flaky) openvpn connection with loopback mounted ext3 filesystems to using rsync with the --fake-super option - the only thing that we've found slightly wrong with this so far is that --fake-super can't reconcile symlinks (it uses user level xattr to store the origional permissions, and with symlinks this doesn't work, because, apparently, symlinks should store user xattr attrbutes!).

We also offer ipv6 for all our dedicated and vps customers, as per the mythic blog - and our 2 authoritative name servers are both v6 enabled. We're working towards making all of our core services available over both ipv4 and ipv6, and we should be there soon.

It's fantastic to work for a company that has the technical knowledge and no-shit stance that Mythic Beasts has, we will hold our hands up if we do something wrong, and we'll tell you exactly what happened, why, and what we're doing to stop it from happening again.

So, 1 month in, and I'm absolutely loving working for them, and I've been a customer of theirs since black cat networks sold their domain name side to them, and shortly afterwards bluelinux sold their virtual servers service to them.

Our new VDS lite service is reasonably priced, and well spec'd. I thoroughly recommend our services, and obviously, we actually use our own vds service to provide some of our services - so it's as important to us that it keeps working as it is for our customers.

Wow, so that was a very salesy post. Erm, next time, back to whinging about authors not having written the books I'm waiting for yet! :)

So, apparently WHSmith are very upset with the appearance of "Unacceptable Titles" in their Kobo feed, and so, the obvious solution to this is to entirely take down their website and replace it with the following holding page

So, rather than doing the sensible thing, of just removing the kobo feed temporarily, and putting a note on the ebook selling part of their website, instead they've gone with the TAKE EVERYTHING DOWN!!1!!1!1!!one! approach. That seems somewhat stupid, and I'd be looking at whoever made that call and looking to find out how they've still got a job... unless WHSmith really do absolutely no online trade at all, this seems to be particularly stupid.

It appears that if you want to do dns lookups, the output of host is a pain to parse, and dig doesn't follow the systems normal search path. So, the solution is to use getent ahostsv4 <hostname> which has easier to parse output (if you're only looking for v4 addresses).

Apparently this is not a well known thing, so I thought I'd drop it here to tell the world :)

So, in the last couple of weeks I've (mostly) moved out of Brighton, got a new job, and generally the world has gone crazy.

The bank holiday weekend was spent at Debian UK BBQ, and was fantastic fun, including the reading corner on Sunday where we played through various arcs of To Be or Not To Be (small spoiler: PIRATES!).

Since then I've been mostly back and forth from Brighton moving stuff, buying a car, and starting a new job... As mentioned in my new employers blog there was a small stumbling block with my new work laptop that stopped me from getting a sensible operating system on it from the get-go, but it's now running a shiny Debian Wheezy install and alllllllllll is well with the world... well, mostly. I have just removed gnome-keyring because I hate that I can't see a way of relocking it, or infact relocking any of the individual keys that it had.

So, back to ssh-agent I went, and the world is good again. But that means that I haven't got a gpg agent running at the moment, though, unless I'm missing something, I can't see a way of forcing gpg-agent to forget a passphrase either (ARGH, why on earth wouldn't you have an option for that?!).

So, my current place of employ (for another 3 weeks and 1 day) have a bunch of dedicated servers from a large hosting company in the UK, occasionally they break their already truly broken internal networking (yes, it defaults to blocking nearly all TCP ports between servers in their own data center, but if you want to hit those same ports from outside their network, that's absolutely fine!), I get to deal with the same thread at least once a quarter, when something in their network decides to block one of the few ports that are allowed... that port is 22, and ssh is used "rather a lot" between the approx 50 servers we have with them. Just for shits and giggles, all of their infrastructure filters ICMP making traceroute mostly totally useless. tcptraceroute ain't no better. Repeating the same question, asking for what they did when it magically all then works again, and getting a "we didn't do anything!" response is somewhat taxing, almost as taxing as watching them run commands on the servers and them not actually know what they're doing, but appear to have a vague grasp on iptables being a firewall, and that they can maybe read the rules and go, "oh, you weren't lying.". Their support is also so far lagged that it's ridiculous, other than an auto response, it took 3 hours (during office hours) to get a "oh, yeah, what you've said, that all looks good, now give us access so that we can diagnose further..." when you do give them that... it'll be between 4 and 6 hours before they bother logging in, and closer to 8 to 10 hours for them to go "oh, it works now, but I don't know why!". NYARGH. These guys make bedroom ISPs look professional to the max. The only thing they have almost going for them is that if you order a new dedicated server it is usually spun up within the hour.

And here endeth the vent.

So, this weekend I headed up to Uncle Steve's and assisted in the laying of the flagstones for the patio for the BBQ - there's even at least one bit of photographic evidence of me in a non-supervisory capacity. Was a good weekend, and fun was had by all, even if we couldn't really express it due to Complete Exhaustion by the end of it.

Only another few weekends and it'll be the Debian UK Party 2013 which will, as it is every year, be absolutely awesome. I'll leave the thanks for the beer and food donations until after the party :)

I'm just replacing the code running planet alug from a (very old) Planet 2.0 to Venus, appears that the transition has mostly worked, but maybe it's time to think about what to replace that with too, it'd be nice, for instance, to use something shiny and django based so that some more people can add feeds (with out me having to go edit the file!).

So, does anyone have any clue what's happened, because I'm at a loss. Apparently I didn't blog about the loss of my Dad back in December, he was found in Union Canal on the 13th December 2012, with his dinner in a carrier bag and his bike. The 'official' date of death is the 13th December 2012, but as far as we can work out (and I'm fairly sure on it), he died between 1930 and 2030 on the 12th. I was on the phone to him at 1930, and I was trying to call him back at 2030 on leaving a pub and wandering back in to town, apparently I wasn't the only one trying to call him, but we all put it down to him being out of signal at the time, until I got a call from my brother on the 13th at just gone midday, saying that the police had been round to tell him they'd found Dad in the canal that morning.

The funeral was held on the 22nd January 2013, with a large turnout of friends and family, it was a good send off for the old chap, and we made damned sure that it was an even better wake - The Brewery Tap did a fantastic job for us, and all that turned up (ok, with some exceptions, but we can block them out!) were fantastic. The boy knew more people than we could possibly imagine, and most of them turned up to either the funeral, the wake or both.

Still miss him, don't expect that to change anytime soon, if only because I used to talk to him most evenings after work between the train station and the pub, and I really miss being able to talk shit with him and being able to relate. Next week we should finally have the Certificate of Representation so that we can actually empty his canal boat ready to sell, at which point it's likely to go back down to Wilton Marina to be sold off. It's not a bad little boat, but it was his sanctuary, and none of us feel comfortable keeping it.

In other news, work has been slightly a bit busy since then, and I'm still playing a bit of catchup from missing fair chunks of January (though, I was mostly still working from home when I could!).

I'm sure other stuff must have happened... Oh, yeah, I got older, but obviously no wiser. I went and visited The Brewery Tap for Father's Day (that was a good day, 7 hours trains for 5 hours in the pub, but well worth it).

Coming up: Cambridge Visitations! The August Bank Holiday Debian Party in Cambridge - now, that I am looking forward to!