Introduction

In some cases you need to know what is the best codepage (encoding) to either transfer a text over the internet or store it in a text file. One could argue that Unicode is always does the trick but I needed the most efficient (byte saving) way to transfer data.

Detecting the code page from a given text is a very tricky task. But luckily Microsoft provides the MLang API. In special the IMultiLang3 interface which is for outbound encoding detection.
Analogous the IMultiLang2 interface has a function to detect the encoding of an incoming byte array. This is very handy for codepage detetion of text stored in files or send over the internet.

The EncodingTools class offers some easy to use functions to determine the best encoding for different scenarios.

Background

The problem

I started this along with another component that constructs MIME conformant emails. The body of the email is passed as String. The user had to provide the charset to use for the Transfer-Encoding by hand. This is fine as long as you know the target character set or assume always Unicode. But it definetly is not a good solution if you have an end-user GUI application (most users do not even know what an "encoding" is).

I wondered if it is possible to detect the best encoding from the given text....

The dirty hack attempt

My first attempt was a simple brute-force attack:

• built a list of suitable encodings (only iso-codepages and unicode)
• Iterate over all considered encodings
• encode the text using this encoding
• encode it back to unicode
• compare the results for errors
• if no errors remember the encoding that produced the fewest bytes

This nis not only ugly, it does not even work properly. All single byte encodings are binary equal in their encoding result. The codepage is only used to map the single bytes to the correct character for display.
So this method can only distinguish between ASCII (7bit), single byte (8bit) and the different Unicode flavors (UTF-7, UTF8, Unicode etc.).

Finding somthing better

Then I remembered the IMultiLang2.DetectInputCodepage method that was introduces along with the Internet Explorer 5.0. This method detects the encoding used in a text (used by the Internet Explorer to do automatic codepage detection if the header is missing from a page). Even this was not suitable for my problem I wondered if there might have been some development since version 5.0. A wrapper function to the DetectInputCodepage is provided in the EncodingTools class.

Since Internet Explorer 5.5 there is a new interface exported from the MLang dll: IMultiLang3. This is what MSDN says about this interface:
This interface extends IMultiLanguage2 by adding outbound text detection functionality to it.

Wow! This sounded more than promising! The interface has only two methods:

1. DetectOutboundCodePage (for strings)
2. DetectOutboundCodePageInIStream (for streams)

I chose to use the first one.

Using MLang

The MLang.dll is in the Windows\system32 directory. Along some exported functions it provides some COM classes but does not contain a typelibrary. So the easy way (Add Reference... in Visual Studio) did not work.

The MLang.idl is part of the Platform SDK and can be found in the include directory.
To create an assembly from the idl file use the following commands from the Visual Studio Command Prompt:

c:\temp\>midl MLang.idl
C:\temp>midl MLang.idl > null
Microsoft (R) 32b/64b MIDL Compiler Version 6.00.0366
Copyright (c) Microsoft Corporation 1991-2002. All rights reserved.
MLang.idl
unknwn.idl
wtypes.idl
basetsd.h
guiddef.h
oaidl.idl
objidl.idl
oaidl.acf

C:\temp>tlbimp mlang.tlb /silent

The result of those two commands is a brand new Assembly named MultiLanguage.dll.

Using Lutz Roeder's and Reflector I had a look at the signature:

MethodImpl(MethodImplOptions.InternalCall, MethodCodeType=MethodCodeType.Runtime)]
void DetectOutboundCodePage([In] uint dwFlags, 
    [In, MarshalAs(UnmanagedType.LPWStr)] string lpWideCharStr, 
    [In] uint cchWideChar, 
    [In] ref uint puiPreferredCodePages, 
    [In] uint nPreferredCodePages, 
    [In] ref uint puiDetectedCodePages, 
    [In, Out] ref uint pnDetectedCodePages, 
    [In] ref ushort lpSpecialChar);

I was not so happy with the ref uint for the puiPreferredCodePages and puiDetectedCodePages parameters. Also a typed enum for the dwFlags was missing.
So I first exported the generated assembly to c# source code and then changed it a litle:  

[Flags]
public enum MLCPF
{
    // Not currently supported.
    MLDETECTF_MAILNEWS = 0x0001,

    // Not currently supported.
    MLDETECTF_BROWSER = 0x0002,
    
    // Detection result must be valid for conversion and text rendering.
    MLDETECTF_VALID = 0x0004,
    
    // Detection result must be valid for conversion.
    MLDETECTF_VALID_NLS = 0x0008,

    // Preserve preferred code page order. 
    // This is meaningful only if you have set the puiPreferredCodePages parameter
    // in IMultiLanguage3::DetectOutboundCodePage 
    // or IMultiLanguage3::DetectOutboundCodePageInIStream.
    MLDETECTF_PRESERVE_ORDER = 0x0010,

    // Only return one of the preferred code pages as the detection result. 
    // This is meaningful only if you have set the puiPreferredCodePages parameter 
    // in IMultiLanguage3::DetectOutboundCodePage 
    // or IMultiLanguage3::DetectOutboundCodePageInIStream.
    MLDETECTF_PREFERRED_ONLY = 0x0020,

    // Filter out graphical symbols and punctuation.
    MLDETECTF_FILTER_SPECIALCHAR = 0x0040,
    
    // Return only Unicode codepages if the euro character is detected. 
    MLDETECTF_EURO_UTF8 = 0x0080
}             
        
[MethodImpl(MethodImplOptions.InternalCall, MethodCodeType=MethodCodeType.Runtime)]
void DetectOutboundCodePage([In] MLCPF dwFlags, 
[In, MarshalAs(UnmanagedType.LPWStr)] string lpWideCharStr, 
[In] uint cchWideChar,
[In] IntPtr puiPreferredCodePages, 
[In] uint nPreferredCodePages, 
[In] IntPtr puiDetectedCodePages, 
[In, Out] ref uint pnDetectedCodePages, 
[In] ref ushort lpSpecialChar);

Then I added the source files to my project (no more MultiLanguage.dll assembly needed).

Using IMultiLanguage3::DetectOutboundCodePage

Geting an instance of COM class implementing the IMultiLanguage3 is straight forward:

// get the IMultiLanguage3 interface
MultiLanguage.IMultiLanguage3 multilang3 = 
	new MultiLanguage.CMultiLanguageClass();
if (multilang3 == null)
    throw new System.Runtime.InteropServices.COMException("Failed to get IMultilang3");

The next thing is to fill the parameters.
The first parameter dwFlags is a combination of the tagMLCPF flags. I chose always to set the MLDETECTF_VALID_NLS because the result will be used for conversion.
The MLDETECTF_PRESERVE_ORDER and MLDETECTF_PREFERRED_ONLY are used depending on the parameters passed to my detection method.

The next to parameters (lpWideCharStr and cchWideChar) are simply the sting passed for detection and its length  

With the next two parameters (puiPreferredCodePages and nPreferredCodePages) the detection can be limited to a subset of all codepages. This is very usefully if you only want to return a certain subset of codepages.

The last three parameters contain the result of the detection after the method has completed successfully.

So the actual call looks like this:

uint[] preferedEncodings; // array of uint passed as parameter to the function
int[] resultCodePages = new int[preferedEncodings.Length]; // result array

// ... call the function
multilang2.DetectInputCodepage(options,0, ref input[0], ref srcLen, 
	ref detectedEncdings[0], ref scores);

// evaluate the result
if (scores > 0)
{
    for (int i = 0; i < scores; i++)
    {
        // add the result
        result.Add(Encoding.GetEncoding((int)detectedEncdings[i].nCodePage));
    }
}

Finally the COM object should be freed.

Marshal.FinalReleaseComObject(multilang3);

Using IMultiLanguage2::DetectInputCodepage

After being able to choose the best encoding to save send a text over the internet or save it to a stream the next task was to detect the best encoding for incoming text if the sender (or storer) did not chose the best encoding.

The DetectInputCodepage has (at least) two practical uses. By default Windows stores text files in the current default (UI) Encoding. For example on my system this is "Windows-1252". A user from russia will write its text using "Windows-1251". Both codepages are singlebyte and do not have any preamble. So a text file will not contain any information about the used codepage.

So if you open a text file containing text created with codepage diffrent to the current UI code page a StreamReader will read the text as if it was stored in the UI's current codepage. (The encoding detection of the StreamReader is mostly a preamble check. So it wil fail for almost any non Unicode files (or those Unicode files without BOM.)
Most characters outside of the common ASCII charset will be displayed incorrectly.

This is where the DetectInputCodepage comes in handy. Its accurance is not 100% but it is definetively better than the on from the StreamReader.

In the demo application you can double click on a encoding to test which method has the better result (see "Testing the DetectInputCodepage perfomance" below).

The other practical use is, to detect the encoding of emails from badly implemented mime mailers. Some wired mailers send emails in 8bit encoding without specifying any characterset in the header. In this case DetectInputCodepage can help a lot.

As for the DetectOutboundCodePage method I change the method signature a little and added the MLDETECTCP enumeration. The resulting code looks like this:

public enum MLDETECTCP {
    // Default setting will be used. 
    MLDETECTCP_NONE = 0,

    // Input stream consists of 7-bit data. 
    MLDETECTCP_7BIT = 1,

    // Input stream consists of 8-bit data. 
    MLDETECTCP_8BIT = 2,

    // Input stream consists of double-byte data. 
    MLDETECTCP_DBCS = 4,

    // Input stream is an HTML page. 
    MLDETECTCP_HTML = 8,

    //Not currently supported. 
    MLDETECTCP_NUMBER = 16
}

[MethodImpl(MethodImplOptions.InternalCall, MethodCodeType=MethodCodeType.Runtime)]
void DetectInputCodepage([In] MLDETECTCP flags, [In] uint dwPrefWinCodePage,
    [In] ref byte pSrcStr, [In, Out] ref int pcSrcSize, 
    [In, Out] ref DetectEncodingInfo lpEncoding, 
    [In, Out] ref int pnScores);
 

The usage of the function is almost identical to the DetectOutboundCodePage described earlier.

int maxEncodings; // parameter specifying how many encodings to return

int srcLen = input.Length; // lengt of the input
int scores = detectedEncdings.Length; // the number of detected scores

// setup options (none)
MultiLanguage.MLDETECTCP options = MultiLanguage.MLDETECTCP.MLDETECTCP_NONE; 

// finally... call to DetectInputCodepage 
multilang2.DetectInputCodepage(options,0, ref input[0], ref srcLen,
	ref detectedEncdings[0], ref scores);

// get result
if (scores > 0)
{
    for (int i = 0; i < scores; i++)
    {
        // add the result
        result.Add(Encoding.GetEncoding((int)detectedEncdings[i].nCodePage));
    }
}

My first tests were not that promissing. I always had a COMExcpetion with E_FAIL thrown when i tried to detect a codepage.

The DetectInputCodepage will fail on too short texts with not BOM (Byte Order Mask / Encoding Preamble) prefixed data. There are two kinds of faliure. If the input data is very short (less than 60 bytes) there is a good chance that the wrong codepage will be detected. Below 200 bytes there is a good chance that DetectInputCodepage will return E_FAIL, because it could not finally decide which codepage to use. For the later problem I implemented a nasty workaround. I simply multiplied the input data up to 256 bytes. This seems to return reasonable results even for short strings.

// expand the string to be at least 256 bytes
if (input.Length < 256)
{
    byte[] newInput = new byte[256];
    int steps = 256 / input.Length;
    for (int i = 0; i < steps; i++)
        Array.Copy(input, 0, newInput, input.Length * i, input.Length);

    int rest = 256 % input.Length;
    if (rest > 0)
        Array.Copy(input, 0, newInput, steps * input.Length, rest);
    input = newInput;
}

Wrapping it all up

I decided to create a static class to provide access to the DetectOutboundCodePage and DetectInputCodepage methods. It has some public methods that offer different levels of abstraction.

Those are the six high level Methods that are should cover most of the usage scenarios:

• GetMostEfficientEncoding
• GetMostEfficientEncodingForStream
• DetectInputCodepage
• ReadTextFile
• OpenTextFile
• OpenTextStrem

It also has three public static arrays of predefined codpage sets:

• PreferedEncodings
• PreferedEncodingsForStream
• AllEncodings

Those arrays contain the codepages not in the natural sort order, but in the order that return the best result.

Testing the DetectInputCodepage performance

The the screenshot below shows a comparison of the StreamReader encoding detection and the EncodingTools detection. The sample texts come from Unciode.org.

Actually all the samples were detected correctly.

Using the EncodingTools class

The folowing code snippes show how to use the EncodingTools class.

Outgoing Encoding

Detect best encoding for a Stream

 // save the given text using the optimal encoding
private void SaveToStream(string text, string path)
{
    // this is all... detect the encoding
    Encoding enc = EncodingTools.GetMostEfficientEncodingForStream(text);
    // then safe
    using (StreamWriter sw = new StreamWriter(path, false, enc))
        sw.Write(text);
}

Detect best encoding for an email body

 // save the given text using the optimal encoding
private void SaveToAsEmail(string text, string path)
{
    // this is all... detect the encoding
    Encoding enc = EncodingTools.GetMostEfficientEncoding(text);
    // then safe
    using (StreamWriter sw = new StreamWriter(path, false, Encoding.ASCII))
    {
        sw.WriteLine("Subject: test");
        sw.WriteLine("Transfer-Encoding: 7bit");
        sw.WriteLine("Content-Type: text/plain;\r\n\tcharset=\"{0}\"", enc.BodyName);
        sw.WriteLine("Content-Transfer-Encoding: base64"); // should be QP
        sw.WriteLine();
        sw.Write(Convert.ToBase64String(enc.GetBytes(text),Base64FormattingOptions.InsertLineBreaks));
    }
}

Incoming Encoding

Open a Text File

private void OpenTextFileTest()
{
    // read the complete file into a string
    string content = EncodingTools.ReadTextFile(@"C:\test\txt");

    // create a StreamReader with the guessed best encoding
    using (StreamReader sr = EncodingTools.OpenTextFile(@"C:\test\txt"))
    {
        string fileContent = sr.ReadToEnd();
    }
}

Reading from a Stream

private void ReadStreamTest()
{
    // create a streamReader from a stream
    using (MemoryStream ms = new MemoryStream(
        Encoding.GetEncoding("windows-1252").GetBytes("Some umlauts: öäüß")))
    {
        using (StreamReader sr = EncodingTools.OpenTextStream(ms))
        {
            string fileContent = sr.ReadToEnd();
        }
    }
}

• Download demo project - 19 Kb
• Download source - 57 Kb

References

• MLang documentation on MSDN

History

• 17/01/2007: initial release

Introduction

The PropertyGridEx control shows how to add a new tab to the standard System.Winows.Forms.PropertyGrid. In this sample a custom page that shows all instance fields of the selected object. 

Additionally it shows how to implement and use the IPropertyValueUIService to show additional icons in the grid rows behind the property name.
I saw first this when I started using the .NET 3.0 Workflow classes and saw this little blue icon for the DependencyProperties. In this sample the icons will show an icon if the member is serializable and a second icon if the member implements ISerializable. A double click on the icon will open a (very raw - and probably erroneous) assumption of the resulting serialization graph.

For the sake of brevity and readability I omitted most of the source code from this article. I tried to focus on the approach not on the implementation details. The interested should read the source code.

Background

I am currently developing a pretty large application that uses some serialization. When I started to optimize the serialization of my objects I found it hard to follow the serialization graph and to see what is actually serialized. Since I use the BinaryFormatter I thought it would be nice to utilize the PropertyGrid (which I already use in my project to show the properties of my objects) to show me the members and their serialization attribute.

Adding a new PropertyTab

Implementing the PropertyTab

First I created my own RawMemberTab by deriving it from the abstract System.Windows.Forms.Design.PropertyTab class.
A valid PropertyTab must return a valid Bitmap and a valid Name. And since I wanted my tab to work with any object I implemented the CanExtend to always return true.

The tricky part was implementing the GetProperties method. It returns a PropertyDescriptorCollection containing a PropertyDescriptor for each property in the grid. In my example I chose to return not the properties but the fields of the selected object. To get a list of all instances (not static) fields I used reflection on the object's type:

// get all instance FieldInfos
FieldInfo[] fieldInfos = type.GetFields( BindingFlags.Instance | 
    BindingFlags.Public | 
    BindingFlags.NonPublic);

Next thing to do was to wrap the returned array of FieldInfo in to a collection of PropertyDescript objects.
The System.ComponentModel.PropertyDescriptor is an abstract class and cannot be used directly. All derived classes that Microsoft uses in the PropertyGrid are internal. So I had to write my own.

Implementing a custom PropertyDescriptor (FieldMemberDescriptor)

A PropertyDescriptor is a wrapper class to allow generalized access to (virtual) properties. It does not only describe the property by providing a name and the associated attributes it also provides access to the value and the child properties.
I created aPropertyDescriptor called FieldMemberDescriptor to wrap the FieldInfo return via reflection. The FieldInfo is passed to the FieldMemberDescriptor's constructor. (Additionally the owning object's type is passed to construct a name for the PropertyDescriptor)

Most members of the FieldMemberDescriptor are straight forward (see code for details). Worth mentioning is the Attributes property.
The Attributes property returns a list of Attributes that are attached to the underlying type. The nice thing about the PropertyDescriptor is that you are allowed to return whatever attributes you like.
There are some attributes that have a strong relation to the PropertyGrid:

Attribute	PropertyGrid usage
System.ComponentModel.CategoryAttribute	used to group properties by category
System.ComponentModel.DescriptionAttribute	text displayed in the help pane
System.ComponentModel.TypeConverterAttribute	Used to determine the TypeConverter. The TypeConverter is also used to determine if a property is expandable.

Knowing this enables the FieldMemberDescriptor not only to provide a meaningful category and description but also to ensure that the object will always be expandable in the grid (if there is no TypeConverterAttribute attribute is provided or the provided TypeConverter does not derive from ExpandableObjectConverter simply override it with an ExpandableObjectConverter).

Implementing a two more custom TypeDescriptors

After having the FieldMemberDescriptor implemented and tested I was still missing one feature in my grid.
Even though I had all types tweaked to be expandable I had still no convenient way to inspect the items of collection (especially of Hashtables having no member containing an array of the items nor for the keys).
I needed two more TypeDescriptors to cope with the elements of lists and collections: The ListItemMemberDescriptor deals with classes implementing IList and the DictionaryItemMemberDescriptor with those implementing the IDicionary interface.

Enabling the new PropertyTab

The PropertyGrid holds a collection of PropertyTabs that has the public method AddTabType to add new tabs.
The first parameter is the Type of the PropertyTab the second is the scope. I chose the make the RawMemberTab static i.e. it will be always available. It is added in the constructor of the PropertyGridEx.
If the tab should be displayed only for certain object types simply override the OnSelectedObjectsChanged method and add the tab with a different scope.

Step2: Adding the Icons

IServiceProvider Background

The designer infrastructure of .NET uses IServiceProvider pattern in many places.
An IServiceProvider is a great way to offer lots of different services to components in very versatile way.
Any component that has access to an IServiceProvider can query it for a certain type (interface) of service and use it without knowing anything about the actual implementation.
Some common services are:

Service	used for
System.ComponentModel.Design.ISelectionService	access to the current selection and nofiication about selection changes
System.ComponentModel.Design.IComponentChangeService	notifications on component changes (i.e. rename, remove)
System.Windows.Forms.Design.IUIService	provide access to GUI functions (like show a dialog)
System.ComponentModel.Design.IDesignerEventService	tracking of the active IDesignerHost
System.ComponentModel.Design.IDesignerHost	access to the currently designed component and its designer, this one is a service provider by itself
System.ComponentModel.Design.IMenuCommandService	provide global menu command handling
System.Drawing.Design.IToolboxService	Toolbox management
System.Drawing.Design.IToolboxUser	client service for toolbox users
System.ComponentModel.Design.IPropertyValueUIService	PropertyGrid ValueUIHandlers

A component can access an IServiceProvider through its Site property.
One thing to always keep in mind is, that the no IServiceProvider guarantees to implement a certain service. So, before using any service you have to check if the IServiceProvider actually provides it.

For example a ListView control set the globally selected component to the Tag of the current selected item:

private void listView1_SelectedIndexChanged(object sender, EventArgs e)
{
    // has a site?
    if (this.Site != null)
    {
        // site provides ISelectionService?
        System.ComponentModel.Design.ISelectionService selectionService = 
          this.Site.GetService(
            typeof(System.ComponentModel.Design.ISelectionService)) 
          as System.ComponentModel.Design.ISelectionService;
        if (selectionService != null)
        {
            if (this.listView1.SelectedIndices.Count == 1)
            {
                // set the current selection the current items tag
                selectionService.SetSelectedComponents(new object[] 
                  {this.listView1.Items[this.listView1.SelectedIndices[0]].Tag});
            }
            else
            {
                // multi selection is no supported
                selectionService.SetSelectedComponents(new object[] { null });
            }
        }
    }
}

The IPropertyValueUIService

The PropertyGrid uses the IPropertyValueUIService to allow service consumers to add type or value specific extensions to the PropertyGrid. The extensions are displayed as 9x9 images with a tooltip that can react to a double click.

The IPropertyValueUIService has to aspects:

• For the PropertyGrid it returns an array of PropertyValueUIItem that should be added to the value.
• For the client that wants to add PropertyValueUIItem to a PropertyGrid it offers a methods to (un-)register itself

The .NET framework does not come along with a ready to use implementation of the IPropertyValueUIService. So I had to implement one. The interesting thing implementing this service was the necessity to implement a delegate that is assigned through a method (AddPropertyValueUIHandler and RemovePropertyValueUIHandler) and not simply by having a public event.
My first approach was a little crude by having a list of all delegates that were invoked via an iterator.
After a little research I came across the Delegate.Combine method.

Make the service available

The implementation of the service alone does not yet allow the PropertyGrid to use it.
My straight forward approach (having a ServiceContainer as private member in my PropertyGridEx, adding my IPropertyValueUIService implementation to it and overiding the GetService method did - surprisingly - NOT work.
But why? It looked so simple. The PropertyGrid has a public and virtual method named "GetService". Why was it not called with a request for an IPropertyValueUIService?

An yet another moment to bow down before Lutz Roeder and his brillant Reflector tool!

After digging through the classes used by the PropertyGrids I finally found the location where the IPropertyValueUIService is queried. In the PainLabel method in the System.Windows.Forms.PropertyGridInternal.PropertyDescriptorGridEntry class call to the PropertyValueUIService property.
Walking up the call tree that this property issues I ended at the System.Windows.Forms.PropertyGridInternal.SingleSelectRootGridEntry class and its GetService implementation. This method first checks if it has an active IDesignerHost (which I did not provide) and then queries its "baseProvider" for the service in question. This "baseProvider" was passed to the constructor of the SingleSelectRootGridEntry. After locating the call to the constructor if found out this mysterious base provider is the PropertyGrid's Site!
So I created a DummySite that is only used to publish the private ServiceContainer. This DummySite is only used if no other valid site is set.

Utilize the IPropertyValueUIService

After having made the IPropertyValueUIService available the usage of the service is quite simple. As soon as a new ServiceProvider is applied to the PropertyGridEx (via constructor, the Site property or the SetServiceProvider method) any handlers on the old IPropertyValueUIService (if any) are deregistered (RemovePropUIHandler) and if the IServiceProvider provides an IPropertyValueUIService a new handler is added (AddPropUIHandler).

The handler itself is a PropertyValueUIHandler delegate. It is implemented in the PropertyValueUIHandler method in the PropertyGridEx control. The handler has two branches: One for FieldMemberDescriptor and one for other descriptors. If the field in a FieldMemberDescriptor is marked as serializable (not having the NotSerialized attribute) a blue disk icon is added. If the value type of the field implements ISerialzable a second icon (three blue disc - squeezed from 16x16 to 9x9 pixels). A double click on the icon opens an experimental serialization graph viewer (not in the scope of this article, so please no comments on this. It is only in to have some meaningful action behind the icon)

Using the control

The sample control is used just like any other control. 

If you already have an IServiceProvider that you use in your project you might want to use this for the control too. There are two ways to use an exsisting IServiceProvider:

Use the PropertyGridEx constructor that takes an IServiceProvider as parameter

ServiceContainer globalServiceContainer = new ServiceContainer();
// ... add some services 

// Instatiate a new PropertyGridEx
PropertyGridEx propGrid = new PropertyGridEx(globalServiceContainer);
this.Controls.Add(propGrid);

Or assign the IServiceProvider anytime you like:

private void Form1_Load(object sender, EventArgs e)
{
     // assign the global ServiceProvider
     this.propertyGridEx1.SetServiceProvider(this.GlobalServiceProvider);
}

• Download demo project - 17 Kb
• Download source - 34 Kb

History

• 15/01/2007: initial release