Symmetri Developer Blog

The Random class in the .NET framework is used to generate “random” numbers. I use quotations around the word “random” because the output of the function Random.Next() is actually what is known as “pseudo-random”; it is the output of a deterministic mathematical algorithm that “mimics” randomness. Specifically, it is an implementation of Knuth’s subtractive random number generator, which is an algorithm that performs incremental lookups on a large table of values and outputs the difference between selected numbers. The output is close enough to random for most practical purposes encountered in software development, with the exception of cryptography (for which purpose the .NET framework also includes the System.Security. Cryptography.RandomNumberGenerator class).

Recently I was confronted with an interesting question in the line of work: just how “random” is the Random class? Is there a way to quantify the “randomness” of a so-called random number generator (RNG)? One can imagine a linear scale of RNGs ranked by “randomness” - at one end would be entirely predictable, non-random generators (such as a routine that returns a constant value); and at the other end what we might call “true” random processes that occur in nature such as atomic decay¹. In between, all other RNGs that one can think of would lie somewhere along that scale. But how exactly can they be ranked and “randomness” quantified?

It turns out that there are several suggested ways of measuring randomness. They all start by taking a sample output from the RNG in question, and one that is of sufficiently large size (bit streams of about 12MB have been mentioned in the literature); and then subjecting that output file to tests. There are many different testing processes that have been proposed for the task. Below are a selected few:

1) compression - a simple test is to compress the sample output using commercial compression formats such as ZIP and then to compute the ratio of compressed to uncompressed size. This test is really only good at finding large biases in your RNG; i.e. if the compression is able to siginificantly reduce filesize, you should assume that the RNG is producing output with definitive patterns

2) a visual test of Monte-Carlo plot - by plotting the values of RNG output onto a two-dimensional array, any large scale patterns become apparent to the human eye. A great example can be found at RANDOM.ORG²

3) DIEHARD test - the DIEHARD series of tests was developed by Prof. George Marsaglia at Florida State University. It applies a number of statistical tests to the sample RNG output; the composite of their results can be used to make an overall pass/fail judgement. It is a well-regarded process that is used, for example, to certify the fairness of the operation of online gambling sites

4) test for FIPS 140-2 - FIPS 140-2 is a standard maintained by the National Institute of Science and Technology in the United States that concerns itself with the security of cryptographic providers. Section 4.7.1 outlines requirements for approved RNGs; they are based mainly around the idea of entropy, which very loosely speaking is a measure of the information content of a sequence

Exactly how far one wants to go in testing a RNG for randomness depends on the context. For commercial non-cryptographic applications, it is probably good enough to use the built in RNGs that come with software development platforms - such as the .NET Random class (a caveat is that the seeding process for the RNGs should use a source that incorporates some natural uncertainty - such as a combination of time, local machine state and user input). For stronger “randomness”, there are a number of implementations of RNGs floating around in the public domain that can be used (e.g. a popular one is the Mersenne Twister). RNGs with a very high “randomness” have to rely on external, physical sources of entropy such as atmospheric data or atomic decay, and are probably only warranted in very specialized scientific applications.

¹There remains the philosophical question of what is “truly” random; for example, do the quantum mechanical laws that govern atomic decay generate “output” that is the result of some as-yet undiscovered deterministic process? The answer is probably no, but it’s a question that exercises the minds of some of the world’s best physicists and philosophers.

²The demonstration shows the alarming large scale patterns inherent in the RNG that is the PHP rand() function - alarming because most developers think of functions like that as “true” random number generators

This is an age-old problem, and the latest version of .NET 3.5/WPF has not solved it. I have been working with the WPF DataGrid that comes with the WPF toolkit, and I have found that refreshing the DataGrid’s DataContext was *not* updating the DataGrid’s display in the UI. I had to add a second line of code to call DataGrid.Items.Refresh() to get the DataGrid to update its display:

dg.DataContext = _items;
dg.Items.Refresh();

Simple really, and a problem that is common to other similar components on other platforms too. However, I am not sure I see the justification in not re-rendering the DataGrid when changing the data context automatically. I suppose there might be an argument for separating out the rendering process if it is a high cost operation and giving the developer control over it (which is what the DataGrid.Items.Refresh() method does).

However, I do not see that behaviour as having been properly documented; and in the absence of documentation, I think it is very reasonable for the developer to expect that the grid display is automatically refreshed when the underlying data is changed. The philosophy behind WPF seems to be that data binding should be more flexible and seamless; overall, it is, but I think having to do this kind of old-school “refresh”-ing detracts from that goal.

Following on from the previous post about anonymous types in C#; how do we access the properties of an anonymous type? The answer is to use reflection. The code below shows it all in a nutshell:

importState = new {recordCount = recordCount, skipCount = skipCount};
...
string recordCount = importState.GetType().GetProperty(“recordCount“).GetValue(importState e,null).ToString();
string skipCount = importState.GetType().GetProperty(“skipCount“).GetValue(importState, null).ToString();

It’s a bit verbose, but it works! It seems logical to encapsulate the reflection process into a helper function:

public string GetPropertyFromObject(object o, string property)
{
    return o.GetType().GetProperty(property).GetValue(o,null).ToString();
}

An even better version would be templated to work for any property type, not just string.

C# is moving further and further away from its old C/C++ancestry (or should it be Java?). I like some of the innovations in C#3.0, although most of them have been added in response to what other OO languages have been supporting for some time. One philosophical shift seems to be that C# is now less strict with type safety. You can create new objects and leave it to the compiler to create a temporary type at runtime:

backgroundWorker.ReportProgress(recordCount/10000, new {recordCount = recordCount, skipCount = skipCount});

That second parameter in the method call is an anonymous object; kind of like just a container to hold a couple of properties. It’s only needed for this specific call, so we can create it as an anonymous type and use it only in this very limited context. Behind the scenes, I suspect the compiler is create a temporary class { ... } definition.

Developers like to compare IDEs. I use both Adobe Flex Builder 3 and Microsoft Visual Studio regularly. I think both are pretty good; each has its quirks and problems, perhaps Visual Studio marginally more than Flex Builder. One little thing I noticed while developing my Flex project today; Flex Builder compiler generates a warning when a single equal sign (=) is used in place of a double (==) for a equality comparison within a conditional statement. For example, the following statement:

if (x = y) {// do something }

will generate an compiler warning “Assignment within conditional statement (did you mean == instead of =?)”. I appreciated that. This is one of the most common syntax errors for programming newbies, and even us old hands fall prey to it on those late nights when the code is being churned out on autopilot. I think VS2008 would let this statement through the compile without batting an eyelid. Are there any other IDEs which pick up this very common syntax error?

Everything you need for a webpage in one file, including external links to images, Flash files etc? Sounds like a good idea I guess; it would mean that you could easily email a single webpage to someone for offline viewing, for example. This was, I presume, the idea behind the MHTML format, which MIME-encodes a HTML webpage and associated resources into one long binary file. However, the problem is that very few browsers support the display of MHTML files natively. Internet Explorer does, but not Firefox, Safari or Google Chrome.

In my particular case, I have a bunch of reports generated by a third-party which are saved as MHTML files which need to be displayed within a Flex iFrame component (which is basically a floating browser container). I could display directly in the browser frame, in which case only users on IE will see the display. I could get the client to use a MHTML to HTML converter software. Or I could write my own server-side code to do the conversion on-the-fly, maybe using this code as a starting point.

This is an issue which caused me much grief, and the solution took quite a lot of research to uncover; it is worth putting it on the record for the benefit of others. As part of my last ASP.NET project, we provided users with a payment page that allowed them to take payments from credit cards using a payment gateway. The code (in C#) was using the WebRequest object to make a HTTP POST request to the gateway URL, and a WebResponse object was used to get the returning XML response from the gateway:

objWebRequest = (HttpWebRequest)System.Net.HttpWebRequest.Create(m_sSecurePayURL);
objWebRequest.Method = “POST“;
byte[] objByteArray = Encoding.UTF8.GetBytes(sURLData);
objWebRequest.ContentType = “application/x-www-form-urlencoded“;
objDataStream = objWebRequest.GetRequestStream();
objDataStream.Write(objByteArray, 0, objByteArray.Length);
objDataStream.Close();
objWebResponse = objWebRequest.GetResponse();

That’s all fairly standard functionality, and everything worked nicely on both development and testing servers. After going live with beta-testing, however, I got a panicked call from the client, saying that there were payments being made multiple times on the same credit card; customers were ringing up to complain about being double-charged. I checked the logs, and what I found was that the WebRequest.GetResponse() was throwing the following exception:

WebException: The server committed a protocol violation. Section=ResponseStatusLine - System.Net.HttpWebRequest.GetResponse()

The exception was only being thrown intermittently; the same payment would work only a few seconds later. Worst of all, despite the exception, sometimes the request was still being sent successfully to the payment gateway server. The user, of course, would see an error and retry the payment, unaware that the first payment had gone through. At the point of time that the exception was being thrown, the state of the request was completely opaque to my application; I had no way of knowing if the payment had gone through or not.

Things were looking pretty bad: I was faced with a bug that was intermittent, poorly understood, environment-dependent and generated deep within the bowels of an atomic .NET fx function over which I had no control; not to mention a bug that had unpleasant real-world implications for the client’s business.

I did my research, and after much searching and reading I found only one real solution on the Web, which was to add the following to the web.config file:

<system.net>
<settings>
<httpWebRequest useUnsafeHeaderParsing=“true“ />
</settings>
</system.net>

(See for example, https://channel9.msdn.com/forums/TechOff/257803-HttpWebRequestResponse-The-server-committed-a-protocol-violation-SectionResponseStatusLine/)

That looked promising, but it didn’t work, even when applying the setting in code using reflection. It wasn’t the header of the request that was a problem, it seemed, but something about the request itself that was causing the server to intermittently reject the request or response on the grounds that it did not conform to the HTTP protocol. Why exactly the same request should be allowed to go through just seconds later was a mystery.

As is often the case when developing commercial software, sometimes we have be more interested in solutions that explanations. Tinkering with the properties of the HttpWebRequest object, I came across a property called ProtocolVersion. I realized that with the .NET 2.0 version of WebRequest, I was by default using the HTTP/1.1 version of the HTTP protocol. It turns out that there are some key differences between HTTP/1.0 and HTTP/1.1; generally speaking, the older version is looser in its specification. So, finally, I was at a solution to my problem; force my request and response cycle to use the HTTP/1.0 protocol instead of HTTP/1.1:

objWebRequest = (HttpWebRequest)System.Net.HttpWebRequest.Create(m_sSecurePayURL);
objWebRequest.Method = “POST“;
byte[] objByteArray = Encoding.UTF8.GetBytes(sURLData);
objWebRequest.ContentType = “application/x-www-form-urlencoded“;
// set HTTP/1.0 as the protocol version to stop intermittent protocol violations
objWebRequest.ProtocolVersion = System.Net.HttpVersion.Version10;
objDataStream = objWebRequest.GetRequestStream();
objDataStream.Write(objByteArray, 0, objByteArray.Length);
objDataStream.Close();
objWebResponse = objWebRequest.GetResponse();

It worked like a charm, and all of us could finally heave a sigh of relief. Both the problem and the solution are quite obscure. What exactly caused the exception is still not clear to me, and I’ll never have the time to investigate and find out. Hopefully, though, this description might help someone who is stuck with this same issue.

This one caused me a lot of grief over the past week or two. A page in my latest ASP.NET project had begun to intermittently throw the following error: “Thread was being aborted”. I tracked the error down to a part of the code where I was doing a Response.Redirect(), which looked something like this:

try
{
// do some stuff
...
Response.Redirect(url);
}
catch()
{
// handle exceptions
...
}

Well, it turns out that the Response.Redirect call within the try .. catch block will throw the “Thread not aborted exception”, and according to Microsoft this behaviour is by design - see http://support.microsoft.com/default.aspx?scid=kb;en-us;312629.

I must admit, their explanation makes no sense to me. Why ending a request in the middle of the try.. catch block should be a problem is not clear - I can certainly see that it is inelegant, but I am not sure where the exception is coming from. Also, the weird thing is that in my case the exception was only thrown intermittently, about 10% of the time that the code ran. Surely, if it was a violation of the ASP.NET request/response cycle in some way, it should happen consistently?

Anyway, when developing code to timelines and budgets, you sometimes have to accept that you won’t have time to 100% understand the fix that you have found, even though you need to use it. In this case, there are two things I needed to do to fix this. Firstly, I moved the Response.Redirect out of the try .. catch block. Secondly, just to make sure, I used the Response.Redirect(url, false) overload, which does not abort the running thread when it launches the new one.

The term “Web 2.0″ can mean a lot of things to a lot of people, but one thing that most would agree on is that one part of the “Web 2.0″ revolution is the development of a new generation of applications known as “Rich Internet Applications” (RIAs). Loosely speaking, RIAs are web applications that do not behave like traditional web applications; they offer a user experience that is fast and interactive and hides the server/client post-and-response cycle from the user. The earliest prototypes of RIAs used client-side Javascript to manage change in parts of a traditional webpages without having to refresh the entire page; in time, this led to the development of what is now known as Ajax. From a different direction, many developers (including myself) began using Flash to build user interfaces that were visually rich but difficult to create and manage; as Flash did not natively support many of the activities needed for building interfaces, we were forced to build our own code to do even simple tasks such as a building forms and managing common components.

The past year and a half have seen a quantum leap in RIA development. Adobe has released Flex, a new development environment that synthesizes Flash/AS3 with HTML-like front-end code to build RIAs that run within Flash Player 9. Microsoft has released Silverlight, a RIA platform based on .NET which allows applications to run within the browser using the Silverlight plugin. Both of these seek to leverage off existing technologies: Actionscript 3 for Adobe, C#/.NET framework for Microsoft; both allow the developer to quickly create and deploy visually impressive and functional web applications that would have taken much longer the old way.

So which way to go - Flex or Silverlight? I am facing that decision for a new project that involves developing two web applications: a smart client application that needs to run as either a disconnected or connected client, store data locally and upload to a server; and a server-based web application that creates reports based on that aggregated data and makes them accessible on the Web. I want to build an RIA for both - I have ruled out doing in the traditional way (desktop app/web 1.0 application).

There are plenty of forums and articles out there that compare Flex and Silverlight; as usual when developers discuss their choice of technology, much of the discussion can get quite heated and parochial. It is hard, sometimes, to judge things on an objective basis when people’s opinions are so often coloured by their prejudices about Adobe and Microsoft. As a developer I can honestly say that I do not share those prejudices; I have spent almost exactly equal time developing in Flash/AS3 and .NET over the past two or three years, and I am equally comfortable with the technology and support tools for both.

Here is my attempt at weighing up the pros and cons for each technology (although I am thinking within the context of my new project, I believe the issues are general enough):

Adobe Flex:

Pros: more mature technology with a larger developer community (as of now) which means better support; leverages off existing AS3 knowledge; can use Flash animations; Adobe AIR makes it easy to build RIAs that live both on and off the web; Flash Player 9 available on 97%+ client machines; both development process and the deployed application run truly platform-independently; better charting capability for reports?

Cons: Flex Builder 3 IDE costs about $500; Flash Player 9 still has some quirks (!); harder to talk to existing server apps in ASP.NET

Microsoft Silverlight:

Pros: leverage off existing C#/.NET domain knowledge; VS2008 is a superior IDE (IMO) compared to Eclipse based IDE of Flex Builder; integrate seamlessly with traditional .NET 3.5 web apps; can launch from local filesystem and run as a desktop app; Silverlight plugin exists for all browsers; C# is more strongly typed and has some features (e.g. generics) that AS3 still lacks

Cons: need VS2008 license $800+; cannot develop on Mac OSX (as far as I can see); Silverlight plugin is not as widely installed as Flash Player 9; newer technology that is just out of Beta and therefore has less of a developer community and possibly more bugs and unresolved issues

I have decided to use Flex. I am more comfortable with the Flash Player as a platform right now than I am with the Silverlight plugin, and I am also conscious of how hard it might be to find good examples and support when developing with Silverlight; in the context of a commercial project, those inevitable bottlenecks can make or break timelines and budgets. Also, I like the flexibility of developing in Mac OSX as well as Windows, and I cannot do that with Silverlight; and I think that the Adobe AIR provides as excellent solution to my problem of creating a disconnected desktop client that can also run within a browser (although I know that Silverlight can do that as well, deployment seems trickier and it is not as well documented at this stage as Adobe AIR).

These are exciting times for web developers. Both of these new technologies are impressive in their own right, and there isn’t that much between them. I suspect the market will decide on a combination of both, and neither Adobe nor Microsoft will find it easy to dominate the RIA space. Ultimately the competition will hopefully benefit all of us. I am looking forward to getting stuck into proper RIA development with Flex.

Microsoft Silverlight showcase: http://silverlight.net/Showcase/
Adobe Flex showcase: http://flex.org/showcase/

Session variables are the easiest way to persist data for a client across pages requests in an ASP.NET application, but they also have their drawbacks. One is that sessions expire, and expired sessions must be handled gracefully. Another is that a session can be lost on the server side under some load-balancing scenarios. We have a deployment on our customer’s web farm that illustrates the point nicely. Our ASP.NET application has a number of pages that use Session variables to persist data across postbacks when doing multi-part user operations - the result of each operation is kept in memory as a session variable and the whole lot is committed to the database only once all the steps are complete. We’ve found that these pages are losing session state intermittently on the live environment, any time that the load balancer decides to switch servers within the Web Farm.

ASP.NET 2.0+ has an elegant solution to this problem - you can use different Session State Modes to store session data in places other than the web server process, such as in a SQL database or even with a custom storage provider. The StateServer mode is especially interesting, as it allows you to store data on a particular server in a particular process, regardless of which Web Server is handling page requests.

In our case, since we are persisting data on the same page across postbacks (and not between pages), we also had the option of using the ViewState instead of the Session. Since our data was small enough not to bloat the page, we decided on that as the quickest and easiest solution.