Aikidoka Prevents Ruby Namespace Collisions
Posted: May 10th, 2009 | Author: Daniel Higginbotham | Filed under: Uncategorized |Recently I fell victim to the Twitter-Mash / Extlib-DataMapper-Mash namespace collision. To get around this problem, I’ve created a new gem, Aikidoka.
Here’s what happened when I tried to use Twitter when Extlib had already been loaded:
irb(main):001:0> require 'Twitter'
=> ["Twitter"]
irb(main):002:0> Twitter::Search.new("bokken").fetch
SystemStackError: stack level too deep
Here’s what happens when you use Aikidoka:
irb(main):001:0> require 'aikidoka'
=> ["Aikidoka"]
irb(main):002:0> Aikidoka.rename("Mash" => "Twitter::Mash"){require 'twitter'}
=> ["Mash"]
irb(main):003:0> Twitter::Search.new("aikidoka").fetch
=> <Mash completed_in=0.052875 max_id=1754360060 next_page="?page=2&max_id=1754360060&q=aikidoka"
etc...
It works! What this does is namespace the Mash defined when I require the Twitter gem, so that Mash is now Twitter::Mash. Also, Extlib’s Mash is still there, untouched, so you don’t need to worry about that. Here’s how Aikidoka does its magic:
- It temporarily renames existing constants so that they don’t get clobbered. In this case, “Mash” is renamed to “AikidokaMash”. Right now this only works with top-level constants.
- It yields to the given block. This block should define the constants you want permanently renamed/namespaced. In this case, we’re requiring “twitter”, which in turn requires “mash”. “mash” defines the constant we want to rename, Mash.
- It creates modules as necessary to create the namespace. In this case, the module Twitter is already defined so that’s used. However, if we wanted to rename “Mash” to “Potatoes::Mash”, then a module named “Potatoes” would have been created.
- It assigns the object referred to by the old constant to its new constant. “Twitter::Mash” now refers to the same object that “Mash” refers to.
- Old constants are removed to clean up the namespace. The constant “Mash” no longer exists, the object it used to refer to lives on.
- The constants temporarily renamed in step 1 are now given their original names back. Extlib’s “Mash” is no longer “AikidokaMash”; it’s “Mash” again.
The code is very simple - a total of 67 lines in one file with decent specs - so hopefully it’s easy to dig into.
Right now Aikidoka is best at nesting an existing top-level constant within another constant of a different name. I haven’t tried doing something like Aikidoka.rename("Mash" => "Mash::Twitter") or Aikidoka.rename("ActiveRecord::Base" => "ARBase"), and those examples probably wouldn’t work.
All in all, it does what I want it to and seems to work OK :) You can install it with “gem install flyingmachine-aikidoka“. If you’re wondering about the name, aikido is a martial art designed to resolve conflict harmoniously, and an aikidoka is a student of aikido.
Interesting.
How Twitter knows that it should use Twitter::Mash instead of Mash(defined by Extlib) ?
Hi Marcin,
Here’s an excellent overview of how ruby looks up constants: http://books.google.com/books?id=jcUbTcr5XWwC&pg=PA261&lpg=PA261&dq=ruby+constant+lookup&source=bl&ots=fHEird4sjB&sig=8fkAgim76-yZZBmDwxAFZZL5OjU&hl=en&ei=UQ8ISs3FF-CJtgeK-pX4Bg&sa=X&oi=book_result&ct=result&resnum=6#PPA261,M1
“When a constant is referenced without any qualifying namespace, the Ruby interpreter must find the appropriate definition of the constant… Ruby first attempts to resolve a constant reference in the lexical scope of the reference. This means that it first checks the class or module that encloses the constant reference to see if that class or module defines the constant. If not, it checks the next enclosing class or module. This continues until there are no more enclosing classes or modules.”
In this case, all references to Mash within the Twitter code take place within the Twitter module defined by the Twitter gem. So when Ruby comes across references to Mash, it will look to see if it’s defined in Twitter::Search, Twitter::OAuth, or whatever class or module currently encloses it. When it doesn’t find Mash, it will then look in the Twitter module and find it since we put it there.
It would not work to rename “Mash” to, say, “Twitter::MockHash”. If you did, Twitter’s references to Mash would resolve to Extlib’s Mash. It is possible to extend Aikidoka so that this problem would not arise, i.e. references to Mash within the Twitter gem would return Twitter::MockHash, but that’s probably not worth the time because it’s not a good idea to do that anyway.
The takeaway is that it’s probably wise to have a good idea of how the constant you’re namespacing with Aikidoka is being used by other code you’re using.
I don’t think this is a maintainable way to fix namespace collisions. While it might resolve the error at hand, I feel that it actually makes the problem worse in the long run by hiding the namespace problem. Adding this would make me not trust all references of Mash because it’d be harder to track down the original intention.
If it’s namespace collision you’re worried about, I think forking the plugin or namespacing your own code is the right way to go.
That’s definitely one drawback of taking this approach, Jerry.
And as a matter of fact, I initially forked the twitter gem and changed it to use Mhash instead of Mash. My previous blog post links to the code.
However, for my project I thought it would be more work to fork the twitter gem and have to maintain that than to effectively sandbox the official version. For my project, I am not requiring, defining, or using Mash myself, so I am not really concerned with maintaining references to Mash. Rather, the twitter gem and Datamapper’s extlib both define and use Mash. Unfortunately, both twitter and extlib pollute the global namespace. If the problem arose from code I had written myself, my first option would be to just namespace my own code.
If my own code did start breaking because it relied on a Mash defined by either the Twitter gem or the Extlib gem, then I would strongly need to consider fixing that code. Relying on a library which is indirectly required by some other library is not maintainable, in my opinion.
If I want to use a Mash in the future, it may differ from one or both of the current implementations being used in my project. It might also be defined in the global namespace as well. At that point I would still have the option of forking and namespacing it, but this way I at least still have the option of just namespacing without altering any of the original code.
With both approaches I feel like you face essentially the same problem. Somewhere down the line someone says, “What is this ‘aikidoka’ thing doing?” or, “Why is this person using his own version of the gem instead of the original?” In both cases, good documentation and good communication go a long way to helping people understand and not waste their time figuring out what’s going on.
Also, I think that documentation would help prevent folks from “not trusting all references of Mash.” Aikidoka is a little weird, but it’s not difficult to understand if you understand Ruby and how it looks up constants. There’s nothing magical happening, and it’s perfectly safe to trust that using Aikidoka is not going to mess up the rest of your code.
Finally, one advantage that Aikidoka does have is that you only need to add one line of code to use it, as opposed to changing multiple lines over multiple files, which is potentially what you would need to do if you decided to fork someone else’s code. In that regard, it’s more maintainable.
Thanks for the comment, though. Ideally, those creating gems will be better about namespacing in the future. Especially when creating a data structure like Mash.