(Yes, I know I’m behind the times – “LINQ to SQL? Who needs it when there’s the newest preview/alpha/beta of the Entity Framework?" Well, I did start this application in EF v1 and ran away when I saw “unsupported” stickers plastered all over it. So, no thanks, I’m waiting for the proverbial “Microsoft v3.11” (or 3.51, whatever they call it)).

Looking superficially, one would say that all ORMs are alike. Moreover, as one of the newest to come into the world, LINQ to SQL would be expected to have it’s philosophy and design done according to previously accumulated knowledge. Erm, yes, it’s a polite way of saying that I expected it to be a rip-off of NHibernate…

This similarity may exist in general, but there are some areas in which the two are completely separate worlds. The example that I encountered is performance optimization. Coming from the NHibernate background I was surprised to discover that there are not much optimization topics in common with the two. In some aspects, NHibernate has already solved (at least for me) issues that LINQ to SQL has not yet stumbled upon, but in others, LINQ to SQL focuses on performance issues that don’t even exist as topics in NHibernate.

The search started with the (for an NHibernate guy like me common) “N+1” record problem. Well, either I don’t know how to search for “N+1” in Google or this subject is non-existent in the LINQ world. The “N+1” problem happens if you need to read additional data (say, related records) for each record fetched and do this by executing one SQL command for each record. You fetch 100 records in one query and then execute additional 100 queries for 100 records. So, in total you have 101 SQL commands executed for 100 records read.

Of course, this happens only if you want to load joined data – that is, read multiple joined records and the ORM doesn’t know how to do this in one go. NHibernate sometimes join-loads data by itself but LINQ to SQL doesn’t, so you have to explicitly instruct it to do this. For this, it you can use Eager loading – you instruct the ORM (using DataLoadOptions.LoadWith() method) to fetch the related record along with the original one in a single query. The trouble with this is that if you specify too complicated a structure for eager fetching (for ex. multiple eager-fetch collections on a single record), the ORM won’t be able to read everything in a single query and will execute N+1 SQL commands by itself. Both NHibernate and LINQ to SQL actually do this in a similar fashion. But in NHibernate, you can solve this by pre-fetching additional records in a separate query: you separate the data structure in portions and join-fetch each portion in a single query (this way, each of the multiple mentioned collections gets a separate query and a separate join-fetch). All records get cached in the NHibernate session and each query adds a portion of the missing structure. At the end, you have, say, 3 queries executed to get the full structure.

In LINQ to SQL this cannot be done because you must use DataLoadingOptions to say what gets loaded with what, and once you set the DataLoadingOptions, they cannot be changed. Because of this, you cannot load one set of records at a time: either set DataLoadingOptions to load all or you’re left with partial data. Also, you cannot explicitly load joined objects like in NHibernate (e.g. the statement “load Client along with its associated Orders” would in NHibernate sound something like “from Client c join fetch c.Orders”), here joins are used solely for filtering data.

Obviously, if we want to load two different child collections on a single object we cannot do this in one go. But we also cannot do two consecutive queries with different fetching strategies because we are obliged to use DataLoadingOptions which we cannot change. There may be some secret handshake that can allow me to pre-fetch data separately, but I don’t know about it. Probably the EF will come up with a wider and more mature feature set. As I mentioned, last time I checked EF it was pretty much infantile. But testing it paid off because I learned one important thing: using LINQ for all queries allowed me to rapidly move to LINQ to SQL, and this is where I am now with my N+1 problem. How do you think I’m going to solve it? Yes! LINQ to NHibernate, here we come.

To conclude on the note with which we started, the difference between the two engines… To me it seems that nobody even thought of the N+1 problem in LINQ to SQL: here the biggest performance optimization seems to be on the client side as everyone mentioning query optimization talks about compiling queries… I find it strange that optimizing parsing of a LINQ query could give a performance gain bigger than reducing the number of SQL commands executed, but I never checked so I’ll leave it at that. In NHibernate, on the other hand, there are such things as cached and named queries, but if their impact on performance is proportional to the amount of space given to them on NHibernate blogs and in docs, they are not on top of the list. Could it be that the two technologies are so much different that they have to be optimized in different ways? I don’t think so. It is evident that NHibernate has significantly more mileage than LINQ, but NHibernate tech guys are notorious for not documenting basic stuff, so if something is not mentioned in the docs it doesn’t mean it’s unimportant (ah, the irony of it…) On the other hand, Microsoft technologies receive the most attention in the pre-release stages, when most of the information available is extremely superficial (and I must add that the bloggers behave like a flock of parrots, mindlessly replicating the content from Microsoft CTP announcements ad infinitum and flooding search engines). Because of this, most of the high-tech LINQ information has yet to come. Of course, after LINQ to SQL was discontinued and they started to build yet another ORM – the Entity Framework – from scratch, the wait is naturally going to get even longer. I don’t expect that EF will have a radically different philosophy in the end, but it must at one point implement a way to solve the N+1 problem. Maybe it will do this by itself? It could, and so could NHibernate…

Or: how to solve the setup project message “ERROR: An error occurred while validating.  HRESULT = ’80004005′”

It seems that for a large part of features in Visual Studio .Net, the development stops at the point where they are mostly usable and effectively demo-able. The Microsoft people are very eager to show you how easy it is to solve a trivial problem with a couple of clicks, but they are very reserved once something really serious has to be done.

A case in point: the Setup/Deployment projects in Visual Studio. (Yeah, the ones, Zero Click Deployment – they make it sound like it reads your thoughts – and the like). If you have a missing file in your solution, and if the file is of a non-critical type (e.g. a resource) so that the solution compiles, you have a big problem because the setup won’t. It will fail with a moronic message “ERROR: An error occurred while validating.  HRESULT = ’80004005′”. Which file is missing? Well, if you really really care – go through all files in your projects and check (don’t forget to expand the controls, some RESX child file may be the culprit!)

Another thing that can happen is that a reference in one of the projects is bad. For example, you had the project reference another project and you moved the other project out of the solution… The first project doesn’t use anything from it so that the build passes but the setup is not as forgiving. In this case, you need to check all references.

If, like me, you have a solution that consists of thirty project and thousands of files, this would mean a major headache. One way to solve it would be to create a new, temporary setup project and add to it one by one all outputs from the original setup. Build after each step and when the error appears you’ll know which project is at fault.

Now, if the first is the case (a missing source file), one possible solution is to automate the manual search by using a macro. Below you’ll find one, modified from the example on the excellent MZ Tools site. As for the missing reference, it is somewhat easier to find since there are considerably less references in a solution than project files (you can detect these using the one-by-one method described above). I’m leaving it to you as a TODO: improve this script to detect missing references, post it on your blog and let me know so I can add a link to it.

Option Strict Off
Option Explicit Off
Imports System
Imports EnvDTE
Imports EnvDTE80
Imports EnvDTE90
Imports System.Diagnostics
Imports System.Windows.Forms

Public Module Module2

    Sub FindMissingFiles()

        Dim objProject As EnvDTE.Project

        Try
            If Not DTE.Solution.IsOpen Then
                MessageBox.Show("Please load or create a solution")
            Else
                For Each objProject In DTE.Solution.Projects
                    NavigateProject(objProject)
                Next
            End If
        Catch objException As System.Exception
            MessageBox.Show(objException.ToString)
        End Try

    End Sub

    Private Sub NavigateProject(ByVal objProject As Project)

        Dim objParentProjectItem As ProjectItem

        Try
            objParentProjectItem = objProject.ParentProjectItem
        Catch
        End Try

        NavigateProjectItems(objProject.ProjectItems)

    End Sub

    Private Sub NavigateProjectItems(ByVal colProjectItems As ProjectItems)

        Dim objProjectItem As EnvDTE.ProjectItem

        If Not (colProjectItems Is Nothing) Then
            For Each objProjectItem In colProjectItems
                For i As Integer = 1 To objProjectItem.FileCount
                    Dim fileName As String = objProjectItem.FileNames(i)

                    If fileName <> "" And Not System.IO.File.Exists(fileName) _
                            And Not System.IO.Directory.Exists(fileName) Then
                        MessageBox.Show("File missing: " + fileName)
                    End If
                Next

                If Not (objProjectItem.SubProject Is Nothing) Then
                    ' We navigate recursively because it can be:
                    ' - An Enterprise project in Visual Studio .NET 2002/2003
                    ' - A solution folder in VS 2005
                    NavigateProject(objProjectItem.SubProject)
                Else
                    ' We navigate recursively because it can be:
                    ' - An folder inside a project
                    ' - A project item with nested project items (code-behind files, etc.)
                    NavigateProjectItems(objProjectItem.ProjectItems)
                End If
            Next
        End If

    End Sub

End Module

I remember that when the first version of Java was released one of the frequent questions was: why doesn’t it have multiple inheritance? This also happened later with C#. The answer in both cases was: you can use interfaces instead. It is not the same, you don’t inherit the logic, but you can at least simulate it by implementing multiple interfaces.

Well, this was obviously forgotten by the people designing C# classes: even worse, they seem not to have been aware of it because most of the classes in v1.0 onward don’t have equivalent interfaces that would allow this.

There’s a real-life example (or should I say everyday example? Because I’ve encountered it a million times, you probably did too) that perfectly illustrates the point: why isn’t there an IControl interface that represents a (windows, web, whichever) control type? The logical answer (and it was probably the real reason the framework designers came up with) would be: well, it would be too difficult to implement. Anything that is supposed to work as a control should really have to be derived from Control. True: but the purpose of interfaces is not only to implement them.

Let’s say I want to create my control type that implements some interface, IMyInterface. How do I reference this component, using a variable of Control type or IMyInterface? Whichever I use it isn’t complete, I have to cast it into the other one and I lose compile-time type safety. I could create a ControlWithMyInterface base class that implements the interface and use this instead of both, but then I’d have to inherit everything from it, which in most cases would not be possible. No, the only solution would be to support the “interfaces instead of multiple inheritance” principle and derive IMyInterface from IControl, but IControl doesn’t exist. Although it wouldn’t be too difficult to implement, one could just put all of Control’s public members into the interface.

There could really be a programming rule enforcing the existence of an equivalent interface for every class, even something that generates them on the fly from its public members. I wonder if there is a pre-processor that can help with this?

It seems that the Microsoft Sync Framework is being developed in a hurry. It is a quite a big task – or at least it should become big if we’re to have a serious data distribution framework – therefore it probably merits some patience, but the first thing that is sacrificed in similar cases is documentation. So what we now have is a piece of software for which it is not easy to figure out how it works, and once you do, there are cases when you’re left to your own power of deduction to figure out how it’s supposed to work.

I haven’t dug into the framework deeply enough, but I’m digging. And I have the intention to document the findings, even if it means just sketching everything in short sentences. There are many things not obvious until you start disassembling (and Reflector-ing, of course) the innards of the dlls. And even in that case, you have to keep notes because it’s not a simple system. I intend to come back to this subject in the posts to come (and get way more technical), be sure to check back if you’re interested.

I cannot say for sure, but given the complexity of problem the Sync Framework set out to solve, it is commendably (somewhat bravely, even) comprehensive, well thought out – and quite stable for a Microsoft V1. There’s a V2 "on the air" right now, but it’s a technology preview and it mostly contains a more mature version of the stuff we’ve already seen before. But even V2 or V3 would only be the small first step: what it currently does, copying database rows back and forth between PCs is not a mechanism that will allow us to one day easily build distributed systems. Even Sync Framework guys themselves acknowledge that the biggest obstacle is replication conflicts – irregularities that occur when the same piece of data is changed in multiple locations at the same time. Microsoft cannot help but give us a simplified solution in the form of record-by-record detection and resolution, and this is because the framework is in its very early stages: I don’t know even if (or when) it will grow smart enough to handle more serious conflict resolution.

The thing is, record-by-record resolution cannot help you enforce business rules: for example, if your business logic depends on an invoice not containing the same product multiple times, how do you prevent this from happening in a distributed system? Two users working with two different databases can each add a record for a single item, but when the records replicate you get two of them. This really needs to be detected, and not in such way that would require programming a separate copy of validation logic for synchronization issues (which, when you think of it, should validate data that was already succesfully written to the database… I shudder to think of it). The synchronization framework would really need to somehow integrate with validation logic: in this aspect (and this is probably the biggest issue but only one of the issues present), Microsoft Sync Framework is much closer to the start line than to the finish. But at least it’s moving…

The IT industry has so far moved on mostly in a step by step fashion, by implementing better solutions than the existing ones. This is where the Sync Framework will be of most use, to finally help us start thinking in terms of distributed data. Also, once there’s a working system for data distribution, most will be interested in having it. And once they do, it will be much easier to persuade them that they need to structure their data and/or applications differently. Hopefully we’ll be moving onto an application design philosophy in which it is a "good thing" to have distributed data just like it currently is a "good thing" to have object-orientedness, layered structure etc. There’s a good chance CRUD will be the one of the things that we’ll start getting rid of. Because, once you look at it, storing the current state of data (which is the essence of CRUD – Create, Read, Update, Delete – philosophy) is the major factor in causing replication conflicts. If the databases stored operations – that is, changes to the data – besides data itself, it would be much easier to resolve conflicts, many of them automatically. The logic would know what the two mentioned users did – added the same product to the invoice – and act with this knowledge. In this concrete example there would be a much clearer situation for conflict resolution, the system could replay the operations so that the second one gets a chance to detect there already is a record present and act accordingly – be it to add the second quantity to the first or raise an error. Note that now a common validation logic for the operation could be employed… This is light years away from getting "fait accompli" duplicated rows and having to do a Sherlock Holmes to discover what has happened. Of course, this is also light years from where we currently are, but when you think of it, the database servers are way overdue for serious feature upgrades – and in any case, they already store something that resembles this in transactional logs.

So, it seems we’re making the first step in the right general direction, even if we’re not sure what precise direction we should move in. Trying to wrap our heads around distributed data philosphy is good – and seeing this practice widely deployed will be even better.