Weirdest problem you ever had to solve?

A common question in job interviews is "What is the weirdest problem you ever had to solve?". Every time I am asked this, I can never think of something on the spot. I vaguely remember debugging things like nutty mod_perl caching through apache restarts, issues with variables ending up with some info from other variables due to memory overflows and such.

But today, I am pretty sure I discovered the weirdest thing I have ever seen. FTP servers blocking my connection attempts, AFTER letting me put in the password.

In migrating my application to a new machine, located on Linode in the Fremont data center, I was obviously allocated a new IP address. Everything seemed hunky dory for a day or so. Then I started noticing an increasing number of users saying that FTP connections from that IP address to their webhost were being rejected with password failed. This is nothing new, failed passwords are extremely common for my application. But the rate at which these complaints were coming in were becoming hard to ignore.

So I started debugging. I grabbed a short list of 5 of the supposed bad credentials and tested them from the new server and sure enough, I got a bad password error. Not a "connection refused" error or anything like that, all was normal in the connection process, the FTP server headers appeared, the username was requested and given, etc. I then went over to the old server, and a few other random servers I had access to. All connections were absolutely fine from these other servers.

Now I was completely confused. If web hosts out there were blocking my new IP, why weren't they blocking at the network level with a "host unreachable" or "connection refused" response. Why let it connect only to reject the password. This would imply the blocking is being done in the FTP server software itself, which just doesn't sit right with me.

In any case, I could not figure it out. I asked the kind folk at Linode to change my IP address and they obliged. All FTP connections are now working perfectly again. A few customers will be annoyed at having to update their firewalls with new IP addresses twice in the last few days, but it was that or have a whole bunch of users not even able to connect.

What could it be? Surely the IP is not blocked at the FTP server level. Surely not that fast after only just setting up the machine 5 days ago. Perhaps that IP previously belonged to some hax0r and was already blocked, but as mentioned, why block with FTP servers instead of iptables. Hopefully I haven't been MITM'd somewhere along the way!

Edit: The second IP Linode gave me started experiencing the same problems after a day or 2. The end result was a few web hosts out there blocking my IP after supposed "suspect" activity. Connections work from other servers as a once off, but prolonged use from my server is what causes them to ultimately block the IP in the weird manner mentioned originally. The resolution is that affected clients will need to whitelist my IP in the FTP section of their web hosting control panel.

MySQL to PostgreSQL database transition

There are quite a few blog posts around about how to convert your database from mysql to postgresql. I feel none of them really cover the entirety of the problem, I think the majority of people are dealing with small datasets or simple data and don't see the issues I had. So I figured I'd document my findings. I'll leave the arguments about why you should use one database or another for a different time.

First you'll want to get your data out of mysql. Personally, I found it faster/easier to just rewrite my entire database schema from scratch for postgres. You will need to be careful with datatypes to make sure the data will match up ok. This is where I hit my first problem, mysql doesn't have a boolean type, it uses a tinyint with the data being 1 and 0. Since we will be dumping the raw data later, it means your postgres database will have to be also a tinyint for now (you can change it once the data is in).

Once you have your schema written up, it's time to dump the data. This is where I hit problem number 2. No matter what your default charset is on your tables, the data will be dumped with the database default. I assumed mysql tables were utf8, postgres tables were utf8, good 'nuff. Wrong. You'll want to dump with --default-charset=utf8 to make sure.

Since we already wrote our schema, we can dump with --skip-create-info. Some other handy options you will want so you don't lock your mysql or run out of memory are --skip-opt --quick --single-transaction. And there are others just so you don't get bloat in your dump file, which are --skip-comments --skip-dump-date.

Now the main dump args you want to make sure it's importable by postgres are --skip-add-locks --complete-insert --compatible=postgres.

Onto problem number 3, mysql uses backslash as an escape char in it's dump data, escaping a whole bunch of stuff like tabs, newlines etc, when it doesn't really need to. At first I thought I just needed to fix the escaped quotes of \' and replace with ''. This allowed me to fully insert the data and I thought i was done. But after a while, I noticed all the extra backslashes in my data. Ouch. I thought I was going to have to get my sed programming hat on and replace every single escaped mysql character (there are lots). But alas, thanks to some dudes in the postgresql IRC channel, all you need is PGOPTIONS='-c standard_conforming_strings=off -c escape_string_warning=off'.

Now we're good right? We have our huge data file, the escape chars are taken care of, encoding is fixed. Nope. Remember the datatypes I referred to earlier. Turns out, postgres doesn't allow the null character (not the NULL value), in text or varchar fields. Any of your data that contains the null char you are going to need to change that field to a bytea. And then guess what, your database dump is now useless, postgres can't nicely import that "string" content into a bytea.

So make a call, do you really care about all your null characters? If you do, you're gonna have to find another way to import your data, if you don't, strip them out with perl or similar, making sure to use a positive look behind so you strip \0 but not \\0 (escaped backslash followed by literal zero). Why perl? Cos sed doesn't support look arounds.

All in all, it's a bit of a nightmare to keep all your data intact, I hope your transition goes smoother than mine.

The future of gaming

A while back I was having a discussion about the future of PC gaming and there were arguments that it was on the improve due to the release of Starcraft II and the take up of professional gaming in America (it's been huge in South Korea for a long time). However, my argument was not about the amount of money PC games might bring in for development companies, but that the average age of PC gamers was only getting higher. In 2011 the average age reported was 37. That age hadn't changed in 5 years, I.e. in 2006 the average age was about 32. The only conclusion you can draw from this is that it's the same people buying/playing PC games now that bought them 5 years ago. The industry isn't gaining any new customers.

The reasoning behind this 5 years ago was (in my insignificant opinion) the increase in console gaming. The xbox 360 and PS3 had just come out, young 20 somethings would rather sit on the couch and play online against randoms with server side match making / ranking facilities that sony/microsoft made available. Something the PC gaming world severely lacked. Some games implemented it, such as the arena rankings in W.o.W. but it was left up to individual game developers rather than a platform wide system.

The console gaming has now taken a massive drop off, and PC gaming is now smashing consoles in sales. Surely this is due to the outdated hardware of the PS3 et al. But where are these gamers going? Not to the PC, as the average age and sales have not changed much.

Enter mobile gaming. The same mob that did the survey last year (linked above), did the study again this year and decided they had no choice but to start including mobile gaming. As such the average "gamer" age has dropped dramatically, to 30. Little Johnny from grade 5 can't afford $100 for the new modern warfare. But he can afford $1.99 for angry birds. Will mobile gaming mean the death of PC gaming? I highly doubt it, since existing PC gamers aren't going anywhere. But I do worry about the future of PC gaming if all these teens playing "song pop" aren't switching over to guild wars any time soon. In 10 years time, will the average age of a PC gamer be 47?

It's hard to say, but mobile gaming is clearly having an effect on PC gaming, much like console gaming did. 5 years from now will something kill mobile gaming the same way it killed console gaming? Maybe virtual gaming is making a come back, if John Carmack makes it, I'll play it.

In the mean time, grab your console, whack it behind your door cos it's only good for a doorstop, delete your games off your mobile cos they aren't really "games", install Steam and checkout what real games are all about.

MVC is dead..... NOT

So I stumbled across this post on the twitterverse claiming that "MVC is dead". After a bit of hacker talk with Iordy, it was discovered there was already a heated discussion going on here. It seems even on the HN thread people get confused and start contradicting themselves.

The original article lost me pretty early on when it said "you end up stuffing too much code into your controllers, because you don't know where else to put it". If you're stuffing code in your controllers, then you're doing it wrong already.

The problem here stems from Rails programmers thinking the M in MVC is tightly coupled to your database table (via ActiveRecord). You can in fact have as many models as you want, not all of them have to be database tables, and in fact you can even inherit from existing models that are database tables if you wanted to.

To expand on that theory, the last few projects I've been a part of, we've sort of evolved into what I will call SMVC. We took the "database" side of things out of the model and put them into "schemes". Your scheme would do all the get/set work as well as serialize/de-serialize. The model could then inherit from the scheme and contain all the business logic but none of the "easy" serialization stuff.

Your controller in this case should not have any business logic. If you wanted to do certain things depending on request/environment, you could pass that info to the model via intialize and handle it in the model. Since you decoupled your scheme, you can easily add new accessors/mutators or just private instance variables that you can then use later, without affecting ActiveRecord in any way. Since you inherited from ActiveRecord you can overload any of the update/save methods you want and check your intial request/environment settings before calling super if you need to.

There is also not much need for fixtures if you are testing, since you still have access to the raw scheme, you can easily manage database data through these objects bypassing HTTP specific environments if you ever needed to.

Looking back to the original article above "MOVE", the SMVC example above is pretty much the same thing. It's just that the original article used the term "model" when referring to the serialization of data, probably because they thought that's just what models do. They simply "evolved" into the "MOVE" pattern because they were using MVC wrong in the first place. I.e. MVC is not dead, you're just doing it wrong, in fact, your new pattern is MVC renamed and ever so slightly abstracted.

Now we've focused on "too much logic in your controller", I'd like you to think about how much logic you have in your views. If it's way too much, I encourage you to check out MVVM.

Best way to hack code

For the best part of 4 years now I've been doing the majority of my development on a virtual machine. This allows me to keep my development environment perfectly inline with production, same operating system, exact same package versions etc, and then easily hand that disk image around to other developers who might need to hack on it.

Over the years it seems others agreed. Along came vagrant to make this easier. Along came chef/puppet to make installing the packages easier. My point here is that a lot of people are now using VMs as their dev environment. Your desktop can be whatever the hell you want, since your code runs somewhere else. You can easily swap from one project to another without having to worry about that legacy project not working on your nice, up to date desktop.

I've also been using vim as my editor of choice for the past 13 years. I've dabbled with things like UltraEdit, Notepad++, TextMate, E-Editor, SublimeText 2 and I always palmed them off for whatever reason. I decided to check that style of developing out again and I've really come to like it.

TextMate (and in turn E-Editor since it supports all of TextMates plugins) and SublimeText have built quite a large community around them now and plenty of people are writing nice plugins so you can do all kinds of things right inside your text editor. With vim there are certain things I've been doing for years, that these newer editors can now do as well due to the work of these plugins. I'm talking things like git diff/status, compile check this selection of code or entire file. Run this selection through some sort of lint program etc.

However, the desktop editors can only do this if the machine you are using the editor on has the right stuff installed. Due to the VM dilemna mentioned above, the only place that can truly run any kind of proper compile checks or test runs and the likes, is the VM itself. Unfortunately none of the editors mentioned above have the functionality to run commands over SSH, all the plugins etc are just written to run whatever locally. This seems short sighted to me, as mentioned it seems a lot of people are going the VM route.

Cue X-Forwarding. It's a pain to install libgtk on a base VM that just runs ruby code, but I figured I'd try it out. This way I can run the editor directly on the VM and have the output on my desktop. All good in theory. Unfortunately SublimeText runs like an absolute dog over SSH for some reason. Time to start toying with other X based IDEs such as gVim? Or throw my mouse away and start writing more helpful vim macros?

Upgrading to Rails 3 - Part 1

So it's been about 4 years since we wrote CushyCMS in rails 2.1.3. Over the years it eventually got upgraded through 2.2 and 2.3 and is now running on the latest stable version in the 2.3 tree. But as we all know, leaving a site behind will only bite you later when you want to add new features but can't use new gems or find help for your old code etc etc. The smart thing to do is to upgrade to rails 3.

I figured a staggered approach would be best. I'll upgrade to rails 3.0.12, then 3.1.x and then 3.2.x if I still have any motivation left by the time I finish. Making sure everything still works and tests still pass at each step. Should be pretty easy right? There are a tonne of guides and railscasts out there on how to smoothly transition. Wrong.

I marked this blog post as Part 1, even though I don't even know what will go in Part 2, I'm sure there will be a need for one. Let's just start listing the problems I've already endured in the supposedly simple upgrade.

• helpers :all is now on by default. Rails 3.1 has a way of disabling it, but Rails 3.0 does not. This is only a problem if you have a collision in your helper names, which we do, and so do many others according to a quick google search.
• default_url_options is no longer passed a Hash of options that include the controller/action for which the URL is currently being generated. Meaning you can't easily come up with some "default URL options" for a specific route.
• content_tag(:div, '') returns you an ActiveSupport::SafeBuffer instance, which is html_safe, and if you concat a string on the end of it, it will still be html_safe providing said string doesn't have HTML tags in it. However if you concat in the opposite direction, the same is not true. E.g. ('' + content_tag(:div)).html_safe? != (content_tag(:div) + '').html_safe?.
• I18n translations now use %{} as the interpolation mechanism instead of {{}}
• I18n translations require the phrase key to end in "_html" if you want them to be html_safe?. If you host your translations elsewhere, then this is not something that can be easily updated en masse.

Now this is just a list of what I would consider non-standard issues I've hit. There are of course a bunch more "standard" issues that most people will already know about and are usually covered by the guides out there that help you upgrade. Such as:

• ActionMailer has completely changed it's API and you will need to rewrite your entire Notifier models.
• error_messages_for has been completely removed, with no replacement built in.
• You will need to completely rewrite your routes.rb file from scratch.
• If you're using any plugins/gems (and let's face it, who isn't), more than likely you will need to upgrade them along with rails. If you're lucky, they will be well maintained and not have any API changes. I can assure you, I was not so lucky. At least 50% of the plugins we use are no longer maintained and don't work in Rails 3.0. Yay. 

In short. To anyone out there thinking it's going to be easy to upgrade. Think again.