Google Apps – Panacea or Headache?

The email on ebrahim.org is currently hosted on pair Networks, a great webhost, but one whose email solutions are lacking in flexibility. I want to move to a solution where I can sync Email/Contacts/Calendar over multiple devices, for a domain with 7 mailboxes.

I’m considering two options:

Rackspace
Pros: Has all the features I’d ever need, excellent support, even for small customers.
Cons: Relatively small quota, and completely out of budget (at least US$13/user/month), email migration into Rackspace is difficult for large datasets.

As Rackspace is out of budget, I didn’t really spend much time looking into it in too much detail.

Google Apps Premier
Pros: Within budget (US$50/user/year), wide ranging feature set.
Cons: Technical support lacking (mainly DIY), doesn’t care about small customers, only compatible with old software, and import into Google Apps is a nightmare scenario due to lack of compatibility of migration tools.

However, there are significant issues which block my migration to Google Apps at the moment, most of which are shocking, given Google’s desire to capture the enterprise messaging/collaboration market.

Let’s make a list of missing features:

  • Google Apps Sync does not support Outlook 2010
  • Google Apps Migration for Microsoft Outlook does not support Outlook 2010
  • Google Apps Migration for Microsoft Outlook does not support Windows 7
  • There is no supported way to import a mbox format mailbox into Google Apps (there is a workaround where you can use third-party software to import the mbox into Outlook, and then use the Google Apps Migration for Microsoft Outlook, but then the Google migration tool doesn’t support Windows 7 or Outlook 2010, so you’re back to square one)

Sales of Windows 7 began in October 2009, and Office 2010 was made available to volume licensing customers in April 2010. When everybody else already supports Windows 7/Outlook 2010, Google lags far behind and lose all credibility when they claim they are the best solution for enterprise customers.

Enterprise customers rely on predictability, but yet, when asked for a timeline for when the above configurations will be supported, Google replied “we do not have a release date as yet”.

I’m ready to spend money with Google, if only they’d deliver support for modern software. A year in the software world is an eternity, and for Google to not support Windows 7 is akin to a wannabe top-tier airport telling pilots to land using VFR because they’ve not installed an ILS yet.

Mailing Lists and Email Deliverability

I subscribe to a number of moderated lists, and one of the poor practices that I see is untimely moderation of email. When list messages are not moderated quickly, there are two major pitfalls that end users can experience of which many list moderators may not even be aware.

The first of these is that most users sort their mailboxes by the Date: header, not the date that the message was received at the user’s inbox. This means that messages which are a few days old and have just been let through the moderation queue may show up a couple of pages above the newest messages in the user’s email client or webmail. This means that if a message that is 3 days old is approved, it shows up near other messages that are 3 days old that have already been read, not near the most recent messages. It is very easy to miss these messages and not read them, especially if the user’ unread mail count is consistently greater than zero.

Second, and perhaps more significantly, if the Date: header on mail is significantly (usually 24 hours or more) older than the current time, this can actually affect deliverability of email because spam filters use the difference between the Date: header and the current time as a criteria to evaluate the likelihood that a message is spam. A common characteristic of spam messages is that the Date: header is incorrect. Here is a real world example:

X-Spam-Status: No, hits=2.3 required=3.5 tests=DATE_IN_PAST_96_XX autolearn=disabled version=3.002004

The above message was moderated more than 4 days after it was sent into the queue, and you can see that SpamAssassin gave it a score of 2.3 (out of a required 3.5 to categorise as spam). Another single rule triggered could have caused the message to get sent to the spam folder. Here’s an example of where that happened:

X-Spam-Status: Yes, hits=4.4 required=3.5 tests=DATE_IN_PAST_96_XX,HTML_IMAGE_ONLY_32,HTML_IMAGE_RATIO_06,HTML_MESSAGE,HTML_TAG_BALANCE_BODY autolearn=disabled version=3.002004

Had this message been moderated quickly, it would not have incurred a point score of 2.3 for being so old, and would have been below the threshold of 3.5 required to classify it as spam.

In short, the lesson to mailing list administrators is that it is crucial to moderate messages in a timely manner so that users can easily notice the mail, and also so that the mail actually gets delivered to an inbox rather than to a spam folder.

Goodbye Dreamhost

This site has always been hosted on pair Networks, where I host my email and other personal things which I don’t ever want to go down. In 2005 I also signed up for a Dreamhost account to host a couple of less important things. I knew Dreamhost to be less reliable than pair, but I figured that for what I was paying ($9.99/mo), I could tolerate a little bit of downtime.

Enter 2007. Dreamhost’s service seems to be getting more and more flaky. This year the following happened:

1) Mass power outages, with both primary and backup power failure
2) Complete DNS breakdown affecting all their customers, which Dreamhost classified as a “medium” severity problem

Enter May 2007. People’s sites start getting hacked into (by hackers compromising Dreamhost’s servers, not through client side hacks). Dreamhost doesn’t notify people until 6 June 2007 (!!!). I was one of the people notified of the breach, as my account had also been hacked into (and my files tampered with):

Hello –

This email is regarding a potential security concern related to your
‘XXXXXXX’ FTP account.

We have detected what appears to be the exploit of a number of
accounts belonging to DreamHost customers, and it appears that your
account was one of those affected.

We’re still working to determine how this occurred, but it appears
that a 3rd party found a way to obtain the password information
associated with approximately 3,500 separate FTP accounts and has
used that information to append data to the index files of customer
sites using automated scripts (primarily for search engine
optimization purposes).

Our records indicate that only roughly 20% of the accounts accessed –
less than 0.15% of the total accounts that we host – actually had
any changes made to them. Most accounts were untouched.

We ask that you do the following as soon as possible:

1. Immediately change your FTP password, as well as that of any other
accounts that may share the same password. We recommend the use of
passwords containing 8 or more random letters and numbers. You may
change your FTP password from the web panel (“Users” section, “Manage
Users” sub-section).

2. Review your hosted accounts/sites and ensure that nothing has been
uploaded or changed that you did not do yourself. Many of the
unauthorized logins did not result in changes at all (the intruder
logged in, obtained a directory listing and quickly logged back out)
but to be sure you should carefully review the full contents of your
account.

Again, only about 20% of the exploited accounts showed any
modifications, and of those the only known changes have been to site
index documents (ie. ‘index.php’, ‘index.html’, etc – though we
recommend looking for other changes as well).

It appears that the same intruder also attempted to gain direct
access to our internal customer information database, but this was
thwarted by protections we have in place to prevent such access.
Similarly, we have seen no indication that the intruder accessed
other customer account services such as email or MySQL databases.

In the last 24 hours we have made numerous significant behind-the-
scenes changes to improve internal security, including the discovery
and patching to prevent a handful of possible exploits.

We will, of course, continue to investigate the source of this
particular security breach and keep customers apprised of what we
find. Once we learn more, we will be sure to post updates as they
become available to our status weblog:

http://www.dreamhoststatus.com/

Thank you for your patience. If you have any questions or concerns,
please let us know.

– DreamHost Security Team

I dutifully changed all my passwords and fixed all the damage. Next day my account was compromised again and my files tampered with again. At this point I decided to jump ship and move all my sites to pair. I’ve totally lost confidence in Dreamhost.

A couple of other things to note:

1) Dreamhost was storing all user passwords in plaintext.
2) They had the audacity to blame users for the compromise.
3) Rather than fixing the broken FTP daemon through which accounts were compromised they have just added an option to disable plaintext FTP for user accounts.
4) They still haven’t notified me that my site was compromised a second time.
5) A lot of people I know have had their sites compromised and files changed. I’m rather sceptical of their claim that only 0.15% of their sites had files changed. Here is but a small sample.

I’m willing to tolerate some inconvenience to save a few bucks but seriously no savings is worth this kind of hassle.

So that’s it. Goodbye Dreamhost.

Server Name Indication (SNI)

I was browsing through Yusuf’s blog today and read in his post about enabling cheaper SSL hosting for the first time about Server Name Indication (SNI), as specified in section 3.1 of RFC3546.

Anyone who’s had to set up an TLS/SSL (let’s say secure) site knows that currently, a secure site must be hosted on a unique IP. If you need to host more than one SSL site, you need to have separate IPs for each secure site hosted. This requirement is present because pre-SNI, the server name is negotiated based on the DNS hostname only. SNI elegantly works around this requirement by adding another step to TLS negotiation. As part of the TLS handshake, the client tells the TLS server which hostname it is trying to connect to, and the hostname thus knows which certificate to present to the client. This is explained a lot more elegantly by Paul Querna.

SNI makes life better because secure hosting becomes more affordable. The cost of a secure certificate is often no longer the largest cost that secure sites must bear to be secure. One can get a certificate for $20/year. However, dedicated IPs are expensive. On a host such as Dreamhost, unique IPs cost $4.95/month. Add this up and it’s almost $60/year. If this extra cost can be eliminated a lot more businesses might be tempted to go secure, and this is a good thing for everybody.

So what’s the current state of browsers?

It’s no secret that as far as end users are concerned, backend features are not as sexy as features which are exposed in the UI, but I wonder whether if SNI support is added to Gecko/NSS before IE, if Firefox will suddenly become a lot sexier to businesses who don’t have an arbitrarily large IP space but are looking to standardise on a browser, or recommend one to their clients. Hey, it’s a much better solution than forcing an upgrade to Vista.

bandwidth spikes and cache friendly headers

Graph of Bandwidth Usage for blog.ebrahim.org from October 10 2004 to November 9 2004 I knew while writing my blog entry for the Firefox 1.0 release that I’d see a spike in my bandwidth usage because of the large images I had in my post, but I didn’t expect anything like this! Thanks for dropping by and taking a look! I hope you took the hint and grabbed Firefox 1.0.

According to my pair.com account manager, I pushed approximately 1.1GB of data yesterday, which is more than I’ve pushed before in any single month.

Yesterday, shortly after posting my blog entry, I also set up cache-friendly headers to serve everything under https://blog.ebrahim.org/media/ in a way that doesn’t hit my server too hard. In short, the way it works is that I’ve used the Apache Expires module to instruct clients and caching proxies to cache images for one month from the date of the client pull. That means that if someone on AOL pulls an image from me, for one month, everyone on AOL who tries to pull that image doesn’t touch my server, and pulls it directly from AOL. It’s the next best thing to multicasting. It’s also cool because it reduces server load, because the images don’t generate requests on the Apache server. Not a big issue for me, but huge for those who are load limited rather than bandwidth limited. Here is a sample implementation using an .htaccess file:

ExpiresActive On
ExpiresByType image/gif A2592000
ExpiresByType image/png A2592000
ExpiresByType image/jpeg A2592000
ExpiresByType video/x-ms-wmv A2592000
ExpiresByType video/mpeg A2592000
ExpiresByType application/pdf A2592000

For more detailed information, the Apache docs have good info for both 1.3 users and 2.0 users. Credit goes to yusufg for hooking me up with the above implementation. One can only wonder what my bandwidth graph would have looked like if every single person who viewed my blog entry hit the graphics.

howto: converting a 3rd level .name domain to a 2nd level .name domain

The .name TLD, run by the Global Name Registry (GNR) is somewhat abnormal in that they accept both 3rd level and 2nd level domain registrations in the same TLD. When they first opened up the .name TLD, only 3rd level registrations were available, and I registered ali.ebrahim.name. At the same time, anybody else could register foo.ebrahim.name. This was to make sure that the maximum number of people got domain names related to their name. Fast forward two years, and the 2nd level gets opened up on those domains that don’t have any 3rd level registrations.

At this point, I’m left with ali.ebrahim.name, and I have a hunch that nobody else has any other registrations on *.ebrahim.name. So I want to try and “convert” my 3rd level ali.ebrahim.name into the 2nd level ebrahim.name domain. I don’t know how to go about handling it, and there is no information online on this topic at all. It goes on the back burner. Fast forward another few months, and I get an email from Register.com asking for $30 to extend my ali.ebrahim.name registration for another year. As I balk at being extorted for this amount, I decide to once and for all see if I can put this matter to rest.

I email GNR at info [@] gnr.com asking what the procedure is for converting a 3rd level domain to a 2nd level domain. They reply and say that if I tell them what 3rd level domains I own on any given 2nd level, they can check if there are any other registrations on that 2nd level, and if there aren’t, they’ll let me know how to proceed. So I let them know that I only have ali.ebrahim.name. I get the following reply, which has full instructions on how the process works:

I can confirm that the only registrations on ebrahim.name are:
ALI.EBRAHIM.NAME.

This means that you can “convert” to using ebrahim.name, thought it would mean that you would be without a working domain name and email address for 5-6 days. The only way of “converting” third level registrations to a second level registration is by doing the following:

1) Issue deletion of third level registrations. The names will go into serverHold/pendingDelete status and will remain so for five days. On the sixth day they will be explicitly deleted by a batch process running at 4am UTC/GMT.

2) On the sixth day after issuing the delete commands, register the second level domain name (through the same or a different registrar).

A few comments on the above:

If someone meanwhile register another third level registration on ebrahim.name, you will unfortunately not be able to register the second level domain name ebrahim.name. Also, you have to make sure you are the first one to try and register ebrahim.name after the third levels have been explicitly deleted.

If you choose to go through this process, after you notify your
registrar that you want to delete the domain name and email forwarding, please let me know. Then I can tell you exactly when the two registrations will be explicitly deleted and thus when you should be able to register the domain name ebrahim.name.

So I went through the process as stated, and lo and behold, ebrahim.name is now mine. Currently there are no domain backorder services for the .name TLD, so as long as you know when your 3rd level domains are scheduled to expire, you should be able to be the first one to grab the 2nd level. I registered ebrahim.name with Gandi for €12/year.

Finally, kudos should go to the people at GNR for always replying to emails in a timely fashion, and being a big help all around. Special thanks goes to Asbjorn Steira, who personally handled my case. I’m so glad I didn’t have to deal with Verisign to get this done.

Hope that somebody out there finds this information of some use. I figured I could write the first HOWTO on this topic. 🙂

spreadfirefox server issues

Since the release of Firefox 1.0 PR, the SpreadFirefox site has taken quite a beating. yusufg, someone who deals with these kind of issues on a daily basis for sites that would put MoFo‘s relatively tiny usage to shame, writes about some ways that load problems could be alleviated. In particular, he mentions that static content (including images and other stuff) should be shunted off to another server or vhost that would run a lightweight, non-preforking server (that could host static content for all of MoFo’s stuff). He also talks about cache-friendly headers that could be used to help lessen the load too. For email woes, he notes that postfix is far better than sendmail.

ebrahim.org now at Gandi

My migration away from Network Solutions is now 100% complete. I’ve transferred my ebrahim.org domain away from them and moved it to Gandi SARL, located in France.

One of the cool things about Gandi is their contract (fr), which states:

1.
The Client owns the Domain Name registered. Gandi simply acts on the Client's behalf. Client acknowledges that Gandi services consist only of including in the shared Domain Names database, the Domain Name choosen by the Client, for the duration of the present contract and without prejudice, notably, that the Domain Name is available and that the Client respects terms and conditions of the present contract.

I have yet to find another registrar that states in such unequivocal terms that the registrant owns the domain. No longer am I a subject of the evil (and ridiculously expensive) empire of Network Solutions.

silly court rulings on IP addresses

What do you get when you take technical issues to court when the judges know nothing about internet standards? Silly rulings.

A court in New Jersey has issued a temporary restraining order allowing a company to take its IP address with them when they move hosts. Anyone who knows the first thing about IP allocations and DNS knows that this is a ridiculous ruling, and violates ARIN policies and well established practices. In fact, the RFC for IPv6 explicitly states that relocatable IPs are not permitted.

we’ve moved to a new host

After wanting to move away from Network Solutions for a long time now, I’ve finally moved my ebrahim.org domain over to pair Networks.

One of the things I get at pair that I never had before was a decent managed web hosting platform, with CGI, PHP, and other goodies. So now I’ve moved away from Blogger and set up Movable Type. While Blogger is great, it just doesn’t provide the same flexiblity as MT does, particularly when it comes to advanced features.

One of the best things about this new setup is that I can also create subdomains of ebrahim.org, which I wasn’t able to do before. I’m excited about this. I know most of you who have been with decent web hosting services have been able to do this for a long time now; but what can I say? NSI sucks.

The next step is to move my ebrahim.org registration away from NSI, probably to either Gandi or pairNIC.

In the meantime, enjoy the new MT-powered blog. Now you no longer need to register in order to comment with a name! The only one thing I wasn’t able to do was import comments from Blogger. If anyone knows of a good way to do this, please let me know.

Right now I’m still pretty much using the MT defaults, with a few minor additions and changes to the templates. Sooner or later the blog will get more pretty as I customise things further. Bear with me while that happens. 🙂