Saturday, February 9, 2013

Beginning NServiceBus

I my day to day work we are using NServiceBus as our Enterprise Service Bus (ESB) implementation in an effort to advance our organization to a form of  service oriented architecture (SOA).  I am a newbie to SOA and NServiceBus, thus the journey has not always been smooth.  I plan on posting short articles about very specific issues that I have encountered with using NServiceBus.  This is the first such article.

I like many developers have to interface to legacy systems.  These systems are usually connected via a .NET assembly and DllImport.  NServiceBus comes with a program called NServiceBus.Host.exe that allows you to host your own service endpoints (basically a message handler).  NServiceBus.Host.exe is provided to make your life easier (and it does).

When using NServiceBus.Host.exe to host your own services, upon loading it will scan the directory for all DLLs, load them and look for implementations of IConfigureThisEndpoint. In certain cases, this may cause NServiceBus.Host.exe to exit with an exception.

  • Multiple implementations of IConfigureThisEndpoint – i.e. multiple endpoints
    • Put each endpoint in its own directory. Besides, you don’t want to couple all of your code together at the deployment level either in SOA.
  • Native DLLs in the directory
    • C++ code anyone? Me too.
  • Probably other reasons that I will soon discover

You can prevent this scanning by including an NServiceBus.Host.exe.config file that points directly to the assembly that implements IConfigureThisEndpoint.  Here is a sample:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<add key="EndpointConfigurationType" value="YourNamespace.YourTypeName, YourAssembly"/>
</appSettings>
</configuration>


NServiceBus.Host.exe will still scan DLLs for other interface implementations during configuration (like IWantCustomInitialization), even with the NServiceBus.Host.execonfig file in place.  So you will still have to exclude specific DLLs from the configuration scanning process.  Here is an example, the Init() method of an IWantCustomInitialization handler:



public void Init()
{
Configure.With(AllAssemblies.Except("legacy.dll").And("native.dll"))
.Log4Net()
.XmlSerializer();
}


You can keep chaining “.And” calls to cover all of the DLLs necessary. I hope this saves you some time and frustration. 



There is a lot of great documentation at http://support.nservicebus.com/

Sunday, December 11, 2011

Getting Started with Ruby Development on Windows

This post is about getting a usable Ruby environment setup and configured on your Windows computer.  I assume that you are familiar with some form of software development but that you are completely new to Ruby.  Download the Ruby installer and the Ruby Development Kit from the RubyInstaller site.  I am using Ruby 1.9.2 (rubyinstaller-1.9.2-p290.exe) and the current DevKit (DevKit-tdm-32-4.5.2-20110712-1620-sfx.exe) I chose to use Ruby 1.9.2 even though Ruby 1.9.3 is out because there is an issue using the Ruby-debug-base19x gem with Ruby 1.9.3.  Currently, you would have to build your own Ruby from source to be able to use the ruby-debug-base19x gem with Ruby 1.9.3 and that is beyond the scope of this blog post. If you are going to use an IDE like RubyMine or eclipse for your Ruby development, then you will want to be able to interactively debug your Ruby code using the ruby-debug gem.

Run the Ruby installer that you just downloaded. If you are planning on using multiple Ruby installs, don’t check any of the boxes on the Installation Destination and Optional Tasks installer page.  These options include:

  • Install Tcl/Tk support
  • Add Ruby executables to your Path
  • Associate .rb and .rbw files with this Ruby installation. 

I always choose to install my development tools into the C:\Dev folder.  So for this install I chose C:\Dev\Ruby192. Later we will be creating batch files that set up the proper environment for each Ruby installation.  Next we need to setup the development kit.  This kit will provide you with all of the C language tools that you may need to compile and build ruby gems.  Many gems come with native extensions written in the C programming language.  The executable, DevKit-tdm-32-4.5.2-20110712-1620-sfx.exe, is a self-extracting executable.  Just run the executable and it will ask you for the location to which you should extract the files.  I chose to use the folder C:\Dev\RubyDevKit. 

Launch a command window with the Ruby interpreter on the PATH. In the Start Menu, choose All Programs –> Ruby1.9.2-p290 –> Start a Command Prompt with Ruby. Navigate to the Ruby DevKit installation and execute theses commands:

ruby dk.rb init
ruby dk.rb install

Further documentation for the Ruby DevKit can be found here.  I use my own batch files to set up the Ruby environment.  I do this because I may have different command windows open with different Ruby environments. I find it easy to start up a command window (cmd.exe) and just run the batch file for the Ruby environment that I wish to use.  In the C:\Dev folder, create a text document name ruby192env.bat.  The contents of this file will look like this:

@echo off

@echo.  
@echo.  
@echo         Setting environment for Ruby 1.9.2 tools
@echo.

set Path=C:\Dev\Ruby192\bin;%Path%
call C:\Dev\RubyDevKit\devkitvars.bat

title Ruby 1.9.2

This file will insert the Ruby interpreter and DevKit tools at the beginning of your PATH environment variable when run.  I have similar files for all of the Ruby versions installed on my computer.  Let’s test the ruby installation, fire up a command window and execute the command:

c:\dev\ruby192env.bat

This gets our Ruby environment ready to go. Next run the interactive ruby shell, irb. At the irb prompt type:

puts ‘Hello, World!’

Below is the output on my computer:

irb(main):001:0> puts 'Hello, World'
Hello, World
=> nil

If you get similar output to the above, then everything is installed correctly. Press Ctrl + d to exit irb. Now that we have a fresh Ruby installation and the DevKit for building native extensions, let’s get some gems!  Gem is Ruby’s package management tool.  If you come from the Python world, it’s like pip only much better. Gem will allow you to find and install Ruby gems (libraries).  Gems are just libraries of code that you can download into your Ruby installation so that they are available for your use.  For example, Rails can be installed with the command:

gem install rails

I don’t recommend that you actually install rails at this point.  First we need the debugging packages that will allow us to step through our Ruby code in IDEs like RubyMine and eclipse.  Go back to your command window (or fire up a new one, don’t forget to run the ruby192env.bat file) and execute these commands:

gem install ruby-debug-base19x
gem install ruby-debug-ide

Here is the output of these commands on my computer:

C: >>gem install ruby-debug-base19x
Fetching: columnize-0.3.5.gem (100%)
Fetching: archive-tar-minitar-0.5.2.gem (100%)
Fetching: ruby_core_source-0.1.5.gem (100%)
Fetching: linecache19-0.5.12.gem (100%)
Building native extensions.  This could take a while...
Fetching: ruby-debug-base19x-0.11.29.gem (100%)
Building native extensions.  This could take a while...
Successfully installed columnize-0.3.5
Successfully installed archive-tar-minitar-0.5.2
Successfully installed ruby_core_source-0.1.5
Successfully installed linecache19-0.5.12
Successfully installed ruby-debug-base19x-0.11.29
5 gems installed
Installing ri documentation for columnize-0.3.5...
Installing ri documentation for archive-tar-minitar-0.5.2...
Installing ri documentation for ruby_core_source-0.1.5...
Installing ri documentation for linecache19-0.5.12...
Installing ri documentation for ruby-debug-base19x-0.11.29...
Installing RDoc documentation for columnize-0.3.5...
Installing RDoc documentation for archive-tar-minitar-0.5.2...
Installing RDoc documentation for ruby_core_source-0.1.5...
Installing RDoc documentation for linecache19-0.5.12...
Installing RDoc documentation for ruby-debug-base19x-0.11.29...

C: >>gem install ruby-debug-ide
Fetching: ruby-debug-ide-0.4.16.gem (100%)
Building native extensions.  This could take a while...
Successfully installed ruby-debug-ide-0.4.16
1 gem installed
Installing ri documentation for ruby-debug-ide-0.4.16...
Installing RDoc documentation for ruby-debug-ide-0.4.16...

Now you will be able to breakpoint on, and step through code in your favorite IDE that supports Ruby development.  I am currently using RubyMine by JetBrains. A professional license can be obtained at a very reasonable cost and they offer a free license for open source development. A lot of Ruby programmers use vim as their Ruby development IDE.  If you have never used Linux or any Unix like system and don’t know what vim is, then just stick with RubyMine.

There are many more useful gems you will want to install.  It really depends on your goals.  First and foremost I would recommend installing rspec and cucumber.  These are BDD style testing tools and each deserves its own blog post. 

For your next step into Ruby, do yourself a favor and download and complete the Edgecase Ruby Koans.  These koans are an excellent introduction to the capabilities of the Ruby programming language.  You will not need any additional gems to run the koans.

Sunday, May 15, 2011

Reporting only Failed Test in GoogleTest

GoogleTest with GoogleMock (gtest and gmock) is an outstanding platform for testing C++ code.  At work, we use gtest and gmock for unit test as well as integration style test.  On one of out projects, we are up to over 600 unit tests.  This is great for code coverage, but can generate very verbose output.  When adding new tests, and a failure occurs, the explanation of the failed will scroll by to quickly to read and may be lost if your console buffer is not large enough.  It would be really nice to only output a summary of the executed tests and any failed tests. 
The default output printer in gtest is an object named PrettyUnitTestResultPrinter that extends the TestEventListener class.  TestEventListener objects can hook into the test framework and handle various events as tests are executed. You can learn about implementing TestEventListener classes here.  The documentation even shows you how to implement a minimalist printer, but without the functionality of the default printer.  The built in PrettyUnitTestResultPrinter does a nice job of reporting test results.  The results are even colored green for success and red for failure.  Being an open source project, you could just copy the implementation of PrettyUnitTestResultPrinter and modify it to your needs.  I did not want to take this approach as it would be messy to extract this class and all of its dependencies and cause more maintenance. 
Instead I created a TestEventListenerProxy class that implements all of the methods in TestEventListener and forwards the calls to a TestEventListener object.  This class will take ownership of the pointer passed in to the constructor (deleting it in the destructor).  Thus if I can get a pointer to a TestEventListener object, I can wrap it and override methods.  See the code below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23




class TestEventListenerProxy : public TestEventListener 
{
public:
explicit TestEventListenerProxy(TestEventListener* event_listener);
virtual ~TestEventListenerProxy();

virtual void OnTestProgramStart(const UnitTest& unit_test);
virtual void OnTestIterationStart(const UnitTest& unit_test, int iteration);
virtual void OnEnvironmentsSetUpStart(const UnitTest& unit_test);
virtual void OnEnvironmentsSetUpEnd(const UnitTest& unit_test);
virtual void OnTestCaseStart(const TestCase& test_case);
virtual void OnTestStart(const TestInfo& test_info);
virtual void OnTestPartResult(const TestPartResult& result);
virtual void OnTestEnd(const TestInfo& test_info);
virtual void OnTestCaseEnd(const TestCase& test_case);
virtual void OnEnvironmentsTearDownStart(const UnitTest& unit_test);
virtual void OnEnvironmentsTearDownEnd(const UnitTest& unit_test);
virtual void OnTestIterationEnd(const UnitTest& unit_test, int iteration);
virtual void OnTestProgramEnd(const UnitTest& unit_test);

protected:
TestEventListener* listener;
};

Then from this class I created the CaseSummaryAndFailurePrinter class.  This class will override some of the TestEventListener methods with empty implementations to effectively remove the printing functionality of those methods.  Then, in the OnTestEnd method, the default printer is only called in the event that the test failed.  This class will then only print the overall summaries, case summaries and failed tests.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17




class CaseSummaryAndFailurePrinter : public TestEventListenerProxy
{
public:
explicit CaseSummaryAndFailurePrinter(TestEventListener* default_printer)
    : TestEventListenerProxy(default_printer)
{
}

virtual void OnEnvironmentsTearDownStart(const UnitTest& /*unit_test*/) { }
virtual void OnEnvironmentsSetUpStart(const UnitTest& /*unit_test*/) { }
virtual void OnTestStart(const TestInfo& /*test_info*/) { }

virtual void OnTestEnd(const TestInfo& test_info) {
    if (test_info.result()->Failed())
        listener->OnTestEnd(test_info);
    }
};


I use the CaseSummaryAndFailurePrinter for integration style tests.  These tests may take a while to run and having the case summaries printed allows you to see progress.  Test cases are basically the tests defined in one C++ class.

Finally, for unit tests I use the SummaryAndFailurePrinter class.  This class extends CaseSummaryAndFailurePrinter and overrides the OnTestCaseStart and OnTestCaseEnd methods to remove the printing of the test case information.  Thus for my 600+ unit test I see only the summary of the number of tests that were executed, and any that may have failed.  This removes the verbosity of the default printer in gtest while still getting its functionality when needed.



1
2
3
4
5
6
7
8
9
10
11




class SummaryAndFailurePrinter : public CaseSummaryAndFailurePrinter
{
public:
explicit SummaryAndFailurePrinter(TestEventListener* default_printer)
    : CaseSummaryAndFailurePrinter(default_printer)
{
}

virtual void OnTestCaseStart(const TestCase& /*test_case*/) { }
virtual void OnTestCaseEnd(const TestCase& /*test_case*/) { }
};


The only thing left is to replaces the default printer in gtest with out wrapper.  This can be accomplished in your main method with the following code:



1
2
3
4
5
6
7
8
9
10




testing::InitGoogleTest(&argc, argv);
testing::InitGoogleMock(&argc, argv);

testing::TestEventListeners& listeners =
testing::UnitTest::GetInstance()->listeners();

auto default_printer = listeners.Release(listeners.default_result_printer());
listeners.Append(new SummaryAndFailurePrinter(default_printer));

return RUN_ALL_TESTS();


As shown above, we can tell the TestEventListeners container to release ownership of the default printer.  We can then wrap this default printer with a SummaryAndFailerPrinter and add that object to the TestEventListeners container.

The more I use gtest and gmock the more impressed I am.  This is by far the best C++ testing framework that I have ever seen.  Great job google people!

Saturday, May 14, 2011

Going Agile…

I haven’t posted in a while because I started a new job.  The new job is an agile shop with a firm belief in software craftsmanship.  I had never worked on an agile team before so it was all new to me.  We use Test Driven Development (when possible), pair programming, and have a 2 week iteration cycle.  I say when possible about TDD because working in a C++ project with an approximately 3 minute compile time does not allow you to do TDD the way it was meant to be done.  The new gig is mostly a .NET shop but, as mentioned, they still have some C++ applications out there.  The interesting thing is that not many of the team members had any real experience with C++.  It is a youngish team that mostly started with C# and .NET, with a few that have some Java experience.  C++ frustrates these people so much, it is kind of funny to sit back and watch.  I get the feeling that the C++ code is a major reason that I was pulled in to this team.

I have been busy learning the new code base.  Now about 3 iterations in, I believe that I am adding a lot of value to the team.  The one major omission with this job is that they do not use, and don’t plan on using, Python.  I will miss using Python at work.  Any new software is to be done in C# on the .NET platform.  Not that there is any real dependency on Microsoft, this is just team policy. A well respected contractor tried to turn the team on to Ruby, but so far nothing has come of it.

The team has been reading “Clean Code” by Robert C. Martin.  There is a weekly book club meeting where we discuss a chapter of the book.  This is part of the software craftsmanship initiative.  I think this is a great idea and it helps to improve the quality of all of the software developers.  We will be reading a TDD book after we finish the clean code book.

I hope to get back to writing one post a month.  This was my initial goal for this blog.  Time flies when your coding!

Monday, February 21, 2011

Python 3.2 Released

On Sunday February 20th Python 3.2 was released. Normally I wouldn’t mention when a new point release of Python, but this one is special to me.  It is the first release of Python in which I have played a (very small) part. 

A while ago, I was playing around with Python’s multiprocessing module which, at the time, was new to me.  All of the simple examples that I found would use the os.getpid() and os.getppid() functions to show that the spawned process was indeed a child of the initial Python process.  The problem is that on the Windows platform, os.getppid() was not implemented.  I downloaded a zip of the source code and within a hour I had a working implementation of os.getppid() and could run the samples, great.  Time goes by, and I start to think about contributing my implementation to the actual Python source code.  I had no idea how to go about and do such a thing.  Fortunately, the Python developer guide has wonderful documentation on how you can contribute.  So, I created an issue for os.getppid() on Windows, stumbled through the process of creating a patch, submitted the patch, and then waited.  A core Python developer picked up the issue and told me that I would have to provide a unit test before any check-in would happen.  This meant I would have to spelunk through the source tree to find the unit tests and implement a meaningful test for this.  Well, you all know how life happens, time goes by, and I actually forgot that I had even created this issue.  Eventually I received an email from a core Python developer that moved the issues from the 2.x branch of Python to the 3.x branch.  I got back involved, implemented the unit test and the documentation changes, submitted all the patches and before I knew it, the code was checked in and the issue was closed.

About this time Tim Golden posted a call out to Windows developers to help contribute to Python.  I looked around for other issues to work on.  I ended up being involved on two other issues for Python 3.2 (on Windows).  First, I implemented os.getlogin().  It was just another os module function not available on Windows.  Then, I fixed a bug in the generation of .pyc files.  There was a call to _mkdir to create a directory.  The signature of this function on Windows has one less parameter than on *Nix systems.  Thus, every time _mkdir was called it would push an extra parameter on the stack that was not used.  Being that it is using the C calling convention, the caller cleans the stack, so the extra parameter was actually cleaned up off the stack.  Thus, this was a very minor issue that most likely wouldn’t have caused any problems, but I fixed it anyway.

I would like to write a more in-depth entry about my experience contributing to Python.  I would also like to get involved again in more issues.  Most likely one at a time because:  It can be time consuming, no one is going to pay you, and you must work on the issues on your own time.  It is also great fun and a learning experience.  I think every professional developer should be involved in some open source project.  There are many open source projects, even in the Windows and .NET worlds too (NUnit, NHibernate, IronPython, IronRuby, checkout codeplex).  Go find an open source project and grab an issue from the issue tracker and go for it!      

Friday, February 18, 2011

Essential OperatingSystem Extensions

There are times that you may need to determine which version of Windows your program is running upon.  In the .NET Framework there is the System.OperatingSystem class that provides many of these details.  You can acquire an instance of this class via the System.Environment.OSVersion property.  The OperatingSystem class will tell you essential details like the major and minor version numbers, the revision and build numbers, and even the service pack version (if any) installed on the computer.  There is also a PlatformID property that will tell you the platform type.  This is an enumeration of the various supported platforms.  If you look this up in the MSDN Documentation, you may be surprised to find values such as: Win32S, Win32Windows (were talking Windows 95, 98, ME), WinCE, Xbox, MacOSX and even Unix!  Most commonly this value will be Win32NT as this is used for Windows 2000, XP, server 2003, server 2008, and 7.  These are the NT line of the Windows operating system.  Generally this is the information that your program needs for most purposes like a diagnostic report.
Missing from the OperatingSystem class is any method or property that would tell me if I am running on a server operating system.  I recently had a need to know this information, thus I had to resort to using platform invoke to call into some native Windows functions.  I used the extension method mechanism to integrate my methods with the OperatingSystem class.  Here is my IsServer extension method.
 1 public static class OSExtensions
 2 {
 3     public static bool IsServer(this OperatingSystem Os)
 4     {
 5         var osinfo = new OSVERSIONINFOEX();
 6         osinfo.dwOSVersionInfoSize = (UInt32)Marshal.SizeOf(osinfo);
 7 
 8         if (GetVersionEx(ref osinfo))
 9         {
10             return (VER_NT_WORKSTATION != osinfo.wProductType);
11         }
12 
13         throw new Win32Exception();
14    }
15 }

You can see that this is a simple method that calls the Win32 API GetVersionEx to get even more useful version information.  This function provides a wealth of information about the underlying Windows installation.  In this function, I look at the wProductType field of the OSVERSIONINFORMATIONEX structure to see if its value is not VER_NT_WORKSTATION.  The wProductType field can be one of the following values: VER_NT_WORKSTATION, VER_NT_SERVER, or VER_NT_DOMAIN_CONTROLLER.  Thus if the value is not VER_NT_WORKSTATION then I must be on a server (a domain controller is a server).  Below is some of the interop code that makes this work.

 1 [StructLayout(LayoutKind.Sequential, CharSet=CharSet.Unicode)]
 2 private struct OSVERSIONINFOEX
 3 {
 4    public UInt32 dwOSVersionInfoSize;
 5    public UInt32 dwMajorVersion;
 6    public UInt32 dwMinorVersion;
 7    public UInt32 dwBuildNumber;
 8    public UInt32 dwPlatformId;
 9    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 128)]
10    public string szCSDVersion;
11    public Int16 wServicePackMajor;
12    public Int16 wServicePackMinor;
13    public Int16 wSuiteMask;
14    public SByte wProductType;
15    public SByte wReserved;
16 }
17 
18 private const SByte VER_NT_WORKSTATION = 0x01;
19 private const SByte VER_NT_DOMAIN_CONTROLLER = 0x02;
20 private const SByte VER_NT_SERVER = 0x03;
21 
22 [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
23 private static extern bool GetVersionEx(
24    ref OSVERSIONINFOEX lpVersionInfo
25    );

I placed these items in the OSExtensions class as well.  The MSDN documentation for OSVERSIONINFOEX will show you all of the interesting information you can retrieve if needed.

Another piece of information you may want is the friendly name of the operating system.  The OperatingSystem class includes a VersionString property that will give you the information, but not in the format that you may desire.  Yes, telling your users that the OS is “Microsoft Windows NT 6.1.7600.0” is precise, but the end user of your software will tell you, “No, I’m running Windows 7.”  Below is the Name extension method.  It is not exhaustive, but it covers all of the cases in which my software was interested.  If the friendly name is not determined, it returns the same value as the VersionString property.

 1 public static string Name(this OperatingSystem Os)
 2 {
 3     string name = String.Empty;
 4     if (PlatformID.Win32Windows == Os.Platform)
 5     {
 6         name = GetLegacyName(Os);
 7     }
 8     else if (PlatformID.Win32NT == Os.Platform)
 9     {
10         if (Os.IsServer())
11             name = GetServerName(Os);
12         else
13             name = GetWorkstationName(Os);
14     }
15 
16     if (String.IsNullOrEmpty(name))
17         return Os.VersionString;
18     else
19         return String.Format("{0}, {1}", name, Os.Version.ToString());
20 }
21 
22 private static string GetLegacyName(OperatingSystem Os)
23 {
24     string name = String.Empty;
25     switch (Os.Version.Minor)
26     {
27         case 0:
28             name = "Windows 95";
29             break;
30         case 10:
31             name = "Windows 98";
32             break;
33         case 90:
34             name = "Windows ME";
35             break;
36     }
37 
38     return name;
39 }
40 
41 private static string GetServerName(OperatingSystem Os)
42 {
43     string name = String.Empty;
44     switch (Os.Version.Major)
45     {
46         case 3:
47         case 4:
48             name = "Windows NT Server";
49             break;
50         case 5:
51             if (Os.Version.Minor == 0)
52                 name = "Windows 2000 Server";
53             else if (Os.Version.Minor == 2)
54             {
55                 if (0 == GetSystemMetrics(SM_SERVERR2))
56                     name = "Windows Server 2003";
57                 else
58                     name = "Windows Server 2003 R2";
59             }
60             break;
61         case 6:
62             if (Os.Version.Minor == 0)
63                 name = "Windows Server 2008";
64             else if (Os.Version.Minor == 1)
65                 name = "Windows Server 2008 R2";
66             break;
67     }
68 
69     return name;
70 }
71 
72 private static string GetWorkstationName(OperatingSystem Os)
73 {
74     string name = String.Empty;
75     switch (Os.Version.Major)
76     {
77         case 3:
78         case 4:
79             name = "Windows NT";
80             break;
81         case 5:
82             if (Os.Version.Minor == 0)
83                 name = "Windows 2000";
84             else if (Os.Version.Minor == 1)
85                 name = "Windows XP";
86             else if (Os.Version.Minor == 2)
87                 name = "Windows XP x64";
88             break;
89         case 6:
90             if (Os.Version.Minor == 0)
91                 name = "Windows Vista";
92             else if (Os.Version.Minor == 1)
93                 name = "Windows 7";
94             break;
95     }
96 
97     return name;
98 }

There was a little more interop code required to make this work.  This code allows you to retrieve the value of the SM_SERVERR2 flag, which if non-zero means you are running the R2 variant of Windows Server 2003.  Here is the code.

1  private const int SM_SERVERR2 = 89;
2 
3 [DllImport("User32.dll", SetLastError = false)]
4 private static extern int GetSystemMetrics(int nIndex);

Through the use of extension methods and a little p/invoke we are able to determine if a .NET program is running on a server OS and provide a more friendly name that could be displayed to the user.  Until next time.

Saturday, February 5, 2011

Using Encrypted Data Between Python and Silverlight

I had a chance to work on a project in which data was encrypted and shared between a Python program on the server side and a Silverlight .NET Framework application on the client side.  Both programming environments offer a rich set of libraries for doing data encryption.  On the Python side I chose to use the excellent PyCrypto library.  At the time that I wrote the code I was using version 2.1 of PyCrypto and version 2.7 of Python.  PyCrypto is in the public domain from the way I understand the license files.  On the .NET side, the System.Security.Cryptography namespace provides the classes needed.  In a Silverlight application the System.Security.Cryptography namespace is much, much smaller than in the full .NET Framework.  Thus AES would have to be the encryption algorithm used. 
AES is the Rijndael symmetric algorithm with a fixed block size (128 bits) and iteration count. For the Silverlight version of this algorithm the cipher mode and padding mode are CBC and PKCS7 respectively and can not be changed.  Padding is used to ensure that the data to be encrypted is a multiple of the block size. 
Here is an example of encrypting the data in Python. You can find the code in this article on GitHub.  Obviously, you will need to have Python and PyCrypto installed to use it.
 1 from Crypto.Cipher import AES
2 from pkcs7 import PKCS7Encoder
3 import base64
4
5 key = 'your key 16bytes'
6 # 16 byte initialization vector
7 iv = '1234567812345678'
8
9 aes = AES.new(key, AES.MODE_CBC, iv)
10 encoder = PKCS7Encoder()
11
12 text = 'This is my plain text'
13
14 # pad the plain text according to PKCS7
15 pad_text = encoder.encode(text)
16 # encrypt the padding text
17 cipher = aes.encrypt(pad_text)
18 # base64 encode the cipher text for transport
19 enc_cipher = base64.b64encode(cipher)
20
21 print enc_cipher




The data to be encrypted is first run through the padding encoder.  This will ensure that the data is a multiple of the block size (16 bytes in this case).  Note that even if the data is already the correct size it is still padded.  The data is always padded thus the padding must always need to be removed.  This alleviates the programmer from having to know if the data is padded or not. This program will print the following encoded string: ZeYXkFf8wPbvzdC91V4adwx4U56o2zMMOathdDYuBOE=





The PKCS7 padding code is built in to the Silverlight AesManaged class.  On the Python side, I had to write this my self.  PKCS7 is described in RFC 2315 and is actually very simple, here is the code.








 1 import binascii
2 import StringIO
3
4 class PKCS7Encoder(object):
5 '''
6 RFC 2315: PKCS#7 page 21
7 Some content-encryption algorithms assume the
8 input length is a multiple of k octets, where k > 1, and
9 let the application define a method for handling inputs
10 whose lengths are not a multiple of k octets. For such
11 algorithms, the method shall be to pad the input at the
12 trailing end with k - (l mod k) octets all having value k -
13 (l mod k), where l is the length of the input. In other
14 words, the input is padded at the trailing end with one of
15 the following strings:
16
17 01 -- if l mod k = k-1
18 02 02 -- if l mod k = k-2
19 .
20 .
21 .
22 k k ... k k -- if l mod k = 0
23
24 The padding can be removed unambiguously since all input is
25 padded and no padding string is a suffix of another. This
26 padding method is well-defined if and only if k < 256;
27 methods for larger k are an open issue for further study.
28 '''
29 def __init__(self, k=16):
30 self.k = k
31
32 ## @param text The padded text for which the padding is to be removed.
33 # @exception ValueError Raised when the input padding is missing or corrupt.
34 def decode(self, text):
35 '''
36 Remove the PKCS#7 padding from a text string
37 '''
38 nl = len(text)
39 val = int(binascii.hexlify(text[-1]), 16)
40 if val > self.k:
41 raise ValueError('Input is not padded or padding is corrupt')
42
43 l = nl - val
44 return text[:l]
45
46 ## @param text The text to encode.
47 def encode(self, text):
48 '''
49 Pad an input string according to PKCS#7
50 '''
51 l = len(text)
52 output = StringIO.StringIO()
53 val = self.k - (l % self.k)
54 for _ in xrange(val):
55 output.write('%02x' % val)
56 return text + binascii.unhexlify(output.getvalue())




Remember that each time the AES key is used to encrypt/decrypt a block of data, the internal state of the key changes.  Thus each time you are preparing encrypted data to transmit to the server you must create a new AES key object from the original key and iv or that data will not be properly transformed.  The server will also have to create a new AES key with the original data as well. If you do not, you will have a hard time keeping the server and client AES keys in sync and data will not be transformed properly on either side.





Decryption on the Silverlight side is very straight forward.  I am presenting this here as a console based program but there should not be any relevant difference when used in a Silverlight program.








 1 using System;
2 using System.Text;
3 using System.Security.Cryptography;
4
5 namespace AesTest
6 {
7 class Program
8 {
9 static void Main(string[] args)
10 {
11 // This was the output of our Python program.
12 string enc_cipher = "ZeYXkFf8wPbvzdC91V4adwx4U56o2zMMOathdDYuBOE=";
13
14 var textEncoder = new UTF8Encoding();
15
16 // defaults to CBC and PKCS7
17 var aes = new AesManaged();
18 aes.Key = textEncoder.GetBytes("your key 16bytes");
19 aes.IV = textEncoder.GetBytes("1234567812345678");
20
21 var decryptor = aes.CreateDecryptor();
22 var cipher = Convert.FromBase64String(enc_cipher);
23 var text_bytes = decryptor.TransformFinalBlock(cipher, 0, cipher.Length);
24
25 var text = textEncoder.GetString(text_bytes);
26 // Should print 'This is my plain text'
27 Console.WriteLine(text);
28 }
29 }
30 }




It should be easy for any programmer to reverse the process so the the Silverlight client encrypts data that is then decrypted on the python side.  The hard part here is the age old problem of how you share the key to the symmetric encryption algorithm.  For this we used RSA public key cryptography.





PyCrypto once again comes to the rescue as it includes RSA encryption classes, for the most part.  Once again I had to write a padding encoder/decoder to be used along with the RSA classes. RSA can use a variety of padding schemes like PKCS#1 v1.5, and OAEP (optimal asymmetric encryption padding), with OAEP being recommended for new applications.  I wrapped the Crypto.PublicKey.RSA class in my own RSAkey class shown below.  As you can see in the code the data is padded, encrypted, and then base64 encoded.  The data can easily be transmitted via web protocols when it is base64 encoded.








 1 from Crypto.PublicKey import RSA
2 from pkcs1 import OAEPEncoder
3 import base64
4 import binascii
5 import os
6 import pickle
7
8 class RSAKey(object):
9
10 def __init__(self, keybitsize, encoder=OAEPEncoder()):
11 self._encoder = encoder
12 self._keysize = keybitsize
13 self._key = RSA.generate(keybitsize, os.urandom)
14
15 @property
16 def key(self):
17 return self._key
18
19 @property
20 def key_size(self):
21 return self._keysize
22
23 ## Get the public RSA key used to encrypt data as
24 # an XML string.
25 # @param xml_format True if the key should be returned as XML.
26 # If False, the key is returned as a base64 encoded pickled
27 # Python object.
28 # @return: An XML string representation of the
29 # public RSA key. Each node is a base64
30 # encoded string. It has the following
31 # structure.
32 # \<RSAKeyValue\>
33 # \<Exponent\>AQAB\</Exponent\>
34 # \<Modulus\>some data\</Modulus\>
35 # \</RSAKeyValue\>
36 def public_key(self, xml_format):
37 pkey = self._key.publickey()
38
39 if xml_format:
40 # Pads with leading zeros if needed.
41 def ensure_length(hexstr):
42 if len(hexstr) % 2 != 0:
43 return '0' + hexstr
44 else:
45 return hexstr
46 # make an encoded child node
47 def add_child(tag, n):
48 str_n = ensure_length('%x' % n)
49 n_bytes = binascii.unhexlify(str_n)
50
51 sub = et.SubElement(root, tag)
52 sub.text = base64.b64encode(n_bytes)
53
54 root = et.Element('RSAKeyValue')
55 add_child('Exponent', pkey.e)
56 add_child('Modulus', pkey.n)
57 return tostring(root)
58 else:
59 return base64.b64encode(pickle.dumps(pkey))
60
61 ## Encrypt data with the public RSA key.
62 # @param data The data to be encrypted
63 # @return A base64 encoded string that is the encrypted data.
64 def encrypt(self, data):
65 enc_data = self._encoder.encode(data, keybits=self._keysize)
66 cipher = self._key.encrypt(enc_data, '')
67 return base64.b64encode(cipher[0])
68
69 ## Decrypt data with the private RSA key.
70 # @param encoded_cipher A base64 encoded string of encrypted data.
71 # @return The decrypted data as a string.
72 def decrypt(self, encoded_cipher):
73 cipher = base64.b64decode(encoded_cipher)
74 enc_data = self._key.decrypt(cipher)
75 data = self._encoder.decode(enc_data)
76 return data




Following are the classes I wrote that implement two of the padding schemes (OAEP, PKCS#1 v1.5)described in rfc 2437.  Coming from a C background, I find it hard to work with binary data in python. i know that there must be more efficient ways to handle binary data. The python standard library modules cStringIo, binascii and struct come in very handy.





You may notice that some of the variable names in the following code are not ideal (sorry Uncle Bob!).  They actually reflect the names used within the RFC to make it easier for someone to follow along with the RFC document.  Also, for those of you that do not know Python well, an '_' prefix to a field name in a class is the convention for specifying a private class field. When a Python programmer sees a field like this: self._hash_length, it is clear (to Python programmers) that the author intended this to be private.  This is just a convention that you must enforce upon yourself as the Python language will allow you to access the field via an instance of the class.








  1 import binascii
2 import cStringIO
3 import hashlib
4 import os
5 import struct
6
7 class PKCS1Error(RuntimeError):
8 '''
9 Base class for PKCS1 encoding/decoding errors.
10 Error of this or derived classes should be caught
11 by the calling code and then a generic error message
12 should be returned to the caller.
13 '''
14 pass
15
16 class DecoderError(PKCS1Error):
17 '''
18 Raised when a decoding error has been detected.
19 '''
20 pass
21
22 class EncoderError(PKCS1Error):
23 '''
24 Raise when an encoding error has been detected.
25 '''
26 pass
27
28
29 class PKCSAuxiliary(object):
30 '''
31 Auxiliary functions used in RFC 2437
32 '''
33
34 def __init__(self):
35 self._hash_length = None
36
37 @property
38 def hash_length(self):
39 if not self._hash_length:
40 hasher = self.create_hasher()
41 self._hash_length = hasher.digest_size
42
43 return self._hash_length
44
45 @staticmethod
46 def create_hasher():
47 return hashlib.sha1()
48
49 @staticmethod
50 def compute_hash(data, hex_digest=False):
51 hasher = PKCSAuxiliary.create_hasher()
52 hasher.update(data)
53 if hex_digest:
54 return hasher.hex_digest()
55 else:
56 return hasher.digest()
57
58 def mgf(self, seed, length):
59 '''
60 RFC 2437 page 28 MFG1
61 '''
62 counter = 0
63 output = cStringIO.StringIO()
64 try:
65 limit = length / self.hash_length
66 while counter <= limit:
67 C = self.i2osp(counter)
68 output.write(self.compute_hash(seed + C))
69 counter += 1
70
71 raw_mask = output.getvalue()
72 if len(raw_mask) < length:
73 raise PKCS1Error("MGF: mask too long")
74 finally:
75 output.close()
76
77 mask = raw_mask[:length]
78 return mask
79
80 def i2osp(self, x):
81 '''
82 RFC 2437 page 6 I2OSP
83 Special case where length = 4
84 '''
85 if x > 256 ** 4:
86 raise PKCS1Error("I2OSP: integer too large")
87
88 sp = (
89 int((x >> 24) & 0xff),
90 int((x >> 16) & 0xff),
91 int((x >> 8) & 0xff),
92 int((x >> 0) & 0xff)
93 )
94
95 return struct.pack('BBBB', *sp)
96
97 @staticmethod
98 def xor(a, b):
99 '''
100 RFC 2437 bitwise exclusive-or of two octet strings.
101 page 23
102 '''
103 if len(a) != len(b):
104 raise PKCS1Error("XOR: invalid input lengths")
105
106 output = cStringIO.StringIO()
107
108 try:
109 for i in xrange(len(a)):
110 x = int(binascii.hexlify(a[i]), 16)
111 y = int(binascii.hexlify(b[i]), 16)
112 output.write('%02x' % (x ^ y))
113
114 data = output.getvalue()
115
116 finally:
117 output.close()
118
119 return binascii.unhexlify(data)
120
121
122 class OAEPEncoder(PKCSAuxiliary):
123 '''
124 RFC 2437 9.1.1 EME-OAEP PKCS1-v2.0
125 9.1.1.1 EME-OAEP-ENCODE
126 9.1.1.2 EME-OAEP-DECODE
127 '''
128
129 def __init__(self):
130 super(OAEPEncoder, self).__init__()
131
132
133 def encode(self, msg, salt='', keybits=1024):
134 k = keybits / 8
135 if len(msg) > (k - 2 - 2 * self.hash_length):
136 raise EncoderError("EME-OAEP: message too long")
137
138 emLen = k - 1
139 if (emLen < (2 * self.hash_length + 1) or
140 len(msg) > (emLen - 1 - 2 * self.hash_length)):
141 raise EncoderError("EME-OAEP: message too long")
142
143 pslen = emLen - len(msg) - 2 * self.hash_length - 1
144 output = cStringIO.StringIO()
145 try:
146 for _ in xrange(pslen):
147 output.write('%02x' % 0)
148 ps = binascii.unhexlify(output.getvalue())
149 assert len(ps) == pslen, "PS: invalid length"
150 finally:
151 output.close()
152
153 shash = self.compute_hash(salt)
154 dbout = cStringIO.StringIO()
155 try:
156 dbout.write(shash)
157 dbout.write(ps)
158 dbout.write('\x01')
159 dbout.write(msg)
160 db = dbout.getvalue()
161 finally:
162 dbout.close()
163
164 seed = os.urandom(self.hash_length)
165 assert len(seed) == self.hash_length
166
167 dbMask = self.mgf(seed, emLen - self.hash_length)
168 maskedDB = self.xor(db, dbMask)
169 seedMask = self.mgf(maskedDB, self.hash_length)
170 maskedSeed = self.xor(seed, seedMask)
171 emout = cStringIO.StringIO()
172 try:
173 emout.write(maskedSeed)
174 emout.write(maskedDB)
175 emsg = emout.getvalue()
176 finally:
177 emout.close()
178 return emsg
179
180
181 def decode(self, emsg, salt=''):
182 if len(emsg) < (2 * self.hash_length + 1):
183 raise DecoderError("EME-OAEP: decoding error")
184
185 maskedSeed = emsg[:self.hash_length]
186 maskedDB = emsg[self.hash_length:]
187 seedMask = self.mgf(maskedDB, self.hash_length)
188 seed = self.xor(maskedSeed, seedMask)
189 dbMask = self.mgf(seed, len(emsg) - self.hash_length)
190 db = self.xor(maskedDB, dbMask)
191 shash = self.compute_hash(salt)
192
193 db_shash = db[:self.hash_length]
194 if db_shash != shash:
195 raise DecoderError("EME-OAEP: decoding error")
196
197 index = db.find('\x01', self.hash_length)
198 if - 1 == index:
199 raise DecoderError("EME-OAEP: decoding error")
200
201 return db[index + 1:]
202
203
204
205 class PKCS1v1_5Encoder(object):
206 '''
207 RFC 2437 9.1.2 EME-PKCS1-v1_5
208
209 9.1.2.1 EME-PKCS1-v1_5-ENCODE
210 9.1.2.2 EME-PKCS1-v1_5-DECODE
211 '''
212
213 def encode(self, msg, keybits=1024):
214 emLen = keybits / 8 - 1
215 if len(msg) > (emLen - 10):
216 raise EncoderError("PKCS1-V1.5: message too long")
217
218 ps = self.rnd_non_zero(emLen - len(msg) - 2)
219 assert len(ps) >= 8, "PKCS1-V1.5: invalid PS"
220
221 emout = cStringIO.StringIO()
222 try:
223 emout.write('\x02')
224 emout.write(ps)
225 emout.write('\x00')
226 emout.write(msg)
227 emsg = emout.getvalue()
228 finally:
229 emout.close()
230
231 return emsg
232
233
234 def decode(self, emsg):
235 if len(emsg) < 10:
236 raise DecoderError("PKCS1-V1.5: decoding error")
237
238 if '\x02' != emsg[0]:
239 raise DecoderError("PKCS1-V1.5: decoding error")
240
241 index = emsg.find('\x00')
242 if - 1 == index:
243 raise DecoderError("PKCS1-V1.5: decoding error")
244
245 ps = emsg[1:index]
246 if len(ps) < 8:
247 raise DecoderError("PKCS1-V1.5: decoding error")
248
249 return emsg[index + 1:]
250
251
252 @staticmethod
253 def rnd_non_zero(length):
254 rnd = os.urandom(length)
255 while - 1 != rnd.find('\x00'):
256 rnd = rnd.replace('\x00', os.urandom(1))
257 return rnd




Silverlight does not provide any RSA cryptography classes in the its version of the .NET framework.  Fortunately, the Scrypt project exists and provides a nice RSA library for Silverlight (version 3+) and windows phone 7!  the Scrypt project is licensed under the Microsoft public license (ms-pl).  the RSA.RSACrypto class has an interface that is very similar to that of System.Security.Cryptography.RSACryptoServiceProvider in the full .NET framework.





In my project the Silverlight client would obtain the RSA public key in XML format via a web service call.  This would be used to encrypt login credentials.  During that process an AES key would be generated for the login session.  Any sensitive data would then be encrypted by the AES key shared between the Python server and the Silverlight client. Here is an example of the encryption on the Silverlight side.








 1 using System;
2 using System.Text;
3 using RSA;
4
5 // ... other code ...
6
7 // in this example, e is the GetPublicKeyCompletedEventArgs
8 // parameter from an asynchronous web service call
9 var pkey = new RSACrypto();
10 pkey.FromXmlString(e.Result.GetPublicKeyResult);
11
12 StringBuilder output = new StringBuilder();
13 // fill output with some data...
14 Byte[] raw_data = Encoding.UTF8.GetBytes(output.ToString());
15 var cipher = pkey.Encrypt(raw_data);
16 var encodedCipher = Convert.ToBase64String(cipher);
17
18 // now encodedCipher is ready to be transported
19 // to the Python server.




Remember, with RSA public key cryptography you can only encrypt up to keysize / 8 – 1 bytes of data.  Thus if you have a key that is 128 bytes (1024 bits), you can only encrypt up to 127 bytes (1016 bits) of data (depending on the implementation details, maybe less).  The padding schemes used with RSA ensure that the data you are to encrypt is exactly keysize / 8 – 1 bytes in length.  Thus if your data is short, like a 32 byte AES key (16 byte key, 16 byte iv for 256 bits), the padding scheme will pad out the data to keysize / 8 – 1 bytes before the data is encrypted.





This was a fun project to work on.  I learned a great deal and had a blast. I appreciate all of the hard work that others have put in to make the excellent encryption libraries that exist and are freely available.  Maybe some of you will find my padding encoders useful, Cheers!