Archive

Archive for August, 2013

Simple string compression/decompression

August 18, 2013 Leave a comment

There have been times where there is a need to compress a string before sending it to a server. However during a string compression, there are cases where the compressed string is longer than the original string.

 

So I’ve written a quick string extension that will cater to such conditions. During compression it encodes it to base64 for transmission and appends a header before the string, since base64 does not contain such characters, there is no worry about corruption

 

The Compress function does a few checks before it really returns a compressed string

  1. Original string must be at least 256 in length, catering for double byte characters, this would mean no encryption will be done for < 512 bytes
  2. Between the original string and compressed string, there needs to be at least a 10% saving in compression, anything lesser will return the original string. This is to reduce the overheads of decompression
  3. To determine whether a compressed string is returned, a special header {*} is appended in front and at the end of the string. this is used by the decompress function to determine if the input string is compressed or uncompressed.

 

Codes as follows:

public static string Compress(this string s)
        {
            string Delimiter = "{*|0}";
            //if lesser than 256 characters (512bytes due to possible double byte encoding), dont bother compressing
            if (s.Length < 256)
                return s;

            //Use the maximum unicode to get the bytes
            var bytes = Encoding.UTF32.GetBytes(s);
            using (var ms0 = new MemoryStream(bytes))
            {
                using (var ms1 = new MemoryStream())
                {
                    using (var gs = new GZipStream(ms1, CompressionMode.Compress))
                    {
                        ms0.CopyTo(gs);
                    }
                    var ec = Delimiter + Convert.ToBase64String(ms1.ToArray()) + Delimiter;

                    //compare source and compressed string, only return compressed string if there is at least 10% savings
                    if ((s.Length – ec.Length)*1.0/s.Length > 0.1)
                        return ec;
                    else
                        return s;
                }
            }
        }

        public static string Decompress(this string s)
        {
            string Delimiter = "{*|0}";
            if (s.IndexOf(Delimiter) == 0 && s.Substring(s.Length – Delimiter.Length, Delimiter.Length) == Delimiter)
            {
                //takes out the front and ending delimiter

                try
                {
                    var bytes = Convert.FromBase64String(s.Substring(Delimiter.Length, s.Length – Delimiter.Length * 2));
                    using (var ms0 = new MemoryStream(bytes))
                    {
                        using (var ms1 = new MemoryStream())
                        {
                            using (var gs = new GZipStream(ms0, CompressionMode.Decompress))
                            {
                                gs.CopyTo(ms1);
                            }
                            return Encoding.UTF32.GetString(ms1.ToArray());
                        }
                    }
                }
                catch (FormatException ex)
                {
                    //if for some reason, base64 decryption fail, we could be looking at a real string
                    if (ex.Message.Contains("Invalid length for a Base-64 char array or string"))
                        return s;
                    else
                        throw ex;
                }
                catch (Exception ex)
                {
                    throw ex;
                }
            }
            else
                return s;
        }

Advertisements
Categories: ASP.NET

data.gov.sg Focus Group

August 14, 2013 Leave a comment

As part of data.gov.sg efforts, the Infocomm Development Authority (IDA) has embarked on a pilot project with the National Library Board (NLB). The pilot project revolves around the use of linked data technology to better manage and share data. We are looking for developers to test our pilot project, and would like to invite you to participate in a focus group session for this.

If you are interested and are familiar with APIs and app development, do read on below for further details and if you are keen to be part of this pilot

1. Background:
The objective of the pilot is to enable better sharing of government data with linked data. (If you are interested in linked data, read more about it here.)

For the pilot project, we have created a mobile app, LinkMe and a web browser app, Pathfinder with government linked data. The types of government data used in the pilot include land and routes data (SLA), traffic data (LTA), weather data (NEA), parks data (NParks), etc. As part of Pathfinder, we have also made available some data for developers to test. These datasets are shared via APIs and SPARQL queries.

As a participant, you will explore the above and provide feedback on your experiences at a focus group session. You will also be entitled to receive incentives amounting to $50.

2. What will be expected of you:

Step one :  Attend a focus group session (at IDA HQ, 10 Pasir Panjang Road on 17th Aug from 9am to 2pm)
Step two : Use or explore the apps/datasets/APIs
Step three : Provide feedback via survey after using the apps/datasets

3.  If you accept our invitation and would like to participate in our focus group sessions:

Please provide the following details and email it to liyana_muhammad_fauzi [at] ida.gov.sg :

Details Required

Please indicate below

Name (as per NRIC):

 

NRIC Number:

 

Age:

 

Gender:

 

Mobile Number:

 

What smartphone device are you using, and what is the version currently
(E.g.: iPhone 4/5, Android phone, Windows phone, etc.)?:

 

If you are using an iPhone 4S or later, please provide your UDID:
Here’s a URL showing how to locate the UDIDs – http://www.innerfence.com/howto/find-iphone-unique-device-identifier-udid.

 

 

IDA seeks your cooperation not to divulge any information or data related to this pilot project.

(Note: All focus group participants are required to sign an undertaking for your participation in this.)

Categories: General