Recent work: Cloud Storage

Aug 1st

Posted by Jon Keating in Cloud

No comments

Since all lot of people ask me about what kind of work I do at Yahoo! Japan, I figured that I should write a post about it. Over the past 4 years I have actually been on 3 different teams and each team had its own challenging projects. The one that I decided to pick for this post was just released (firewall port was opened) this week.

One of the biggest buzz words these days is … (drumroll) the “cloud”.

Cloud

My place of employment, being one of the big internet players, has followed suit and wanted a cloud storage system. Now, having a massive file/object storage system is nothing new for us, but the current technology is a bit dated and is nearing its limit.

This is where my part comes in. I was tasked with designing the architecture of said storage platform. Being that this is a closed source project I shouldn’t go into great detail about the design and implementation (sorry, but I wish I could!), however I can say that it is very similar to other closed source storage platforms: Amazon S3, MobStor. The one thing that I am especially proud of is having built a plugin system that allows a 3rd party to extend the functionality of the API. For example, a plugin can be made that is hooked into running after a file is uploaded to perform an asynchronous virus scan check.

The other parts of the platform include: the REST API, metadata storage, file storage, file replication, and many minor subparts that make everything just work when used. Of course there are many extra features that I want to see become a part of the platform, but as always it is a battle against the clock.

Without further ado, here is an actual URL of OctFS (the platform). It is just a 404 error page, so perhaps curl output is more interesting:

  1. $ curl -i octfs.jp
  2. HTTP/1.1 404 Not Found
  3. Date: Sun, 01 Aug 2010 12:25:33 GMT
  4. x-oct-req-id: 0d264df6-44d3-4389-821d-65a92289c15a
  5. P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV"
  6. Content-Length: 134
  7. Connection: close
  8. Content-Type: text/xml; charset=utf-8

Missing SID 0d264df6-44d3-4389-821d-65a92289c15a

Specifying your own Qt plugin directory

Jun 26th

Posted by Jon Keating in Coding

No comments

One of the projects I work on, I wanted to add an icon to the About dialog, just to make things a bit pretty. I used Qt Designer to create the dialog, and got the icon done and everything looked good. However, when I ran the program, the icon was not being displayed on a machine that didn’t have the Qt SDK installed on it.

The problem was that there was no icon library getting loaded that Qt needed. So I found the Qico4.dll from the SDK and had to get it loaded somehow. So I created a plugins directory where my program is installed and added a sub-directory called “imageformats” and put the qico4.dll in there.

Now that I have the file, I need to tell Qt where the plugins are. Here is my solution:

  1. HKEY hKey;
  2. long result = RegOpenKeyEx(HKEY_CURRENT_USER, L"SOFTWARE\\CompanyName\\Product",
  3.                            0, KEY_READ, &hKey);
  4. if (result == ERROR_SUCCESS) {
  5.     TCHAR keyBuffer[MAX_PATH];
  6.     DWORD keyBufferSize = sizeof(keyBuffer);
  7.     ULONG err = RegQueryValueEx(hKey, L"Install_Dir", 0, NULL, (LPBYTE)keyBuffer, &keyBufferSize);
  8.     if (err == ERROR_SUCCESS) {
  9.         QString installDir = QString::fromWCharArray(keyBuffer) + "\\plugins";
  10.         QCoreApplication::addLibraryPath(installDir);
  11.     }
  12.     RegCloseKey(hKey);
  13. }

And I put this code in main.cpp, so it gets executed before any windows get shown. And now I just distribute any plugins that Qt needs to load.