Menu

Tech blog (43)

How and why to take the long way for installing ADFS

I use several applications in my LAN - a bug/issue/ticket tracker, a knowledge base, Git and Mercurial servers, a continuous integration server… Administering users across each of them can be a lot of work, especially if it's unnecessary: and I do have a Windows domain and an Active Directory server which is usually supposed to be the central point for user authentication. If your applications can authenticate against an Active Directory, user centralization is just a matter of configuring them - well, each of them, which could be easy but also it could not be. The Orchard Core software I intend to use outright doesn't support AD authentication other than Azure AD, but it does support OpenID, so one way to integrate it would be to install the AD Federation Services. This then opens up several authentication possibilities, from OpenID through OAuth to SAML and I don't know what else… It raises its compatibility, and if you ask me it should come preinstalled with AD.

 

But not only that it doesn't, it requires you to take five steps back in order just to start, and first couple of these steps deal with SSL certificates: if your service is public you probably have no choice but to buy a commercial certificate and be done with it. But for an intranet - especially with several web applications - it may be worth it to implement your own local certification authority. This is not as hard as it seems if you do it the right way, and as a result you will be able to create your own certificates easily for any purpose. (If you don't, it will be hard).

 

And this was the main purpose of this post, to record the dos and don'ts of such an endeavor. I won't go into much details here, instead I'll provide links to pages with instructions on how things are done with comments and guidance as to what worked for me and what didn't and especially for the things that you must do and those that aren't mandatory, since much of the documentation deals with all the available options, not stating exactly what the procedure would be for an "ordinary" basic setup. I did this with a Windows 2019 server, but I think that for anything involving Windows 2012 or newer the procedures should be similar.

 

Certificate Authority

 

Installing a certificate authority (CA), I think, was quite uneventful as I don't remember what I did: this probably means I had no big problems. I think the official Microsoft guide would suffice for this. One detail you may want to pay attention to is the validity of your root certificate, I set it to hundreds of years… Which might be bad practice, but for an intranet I don't think it's too big of a problem: if someone manages to steal your root certificate, they will probably use it immediately and having it expire after five years won't matter much. On the other hand, when it expires after five years, you'll have a lot of work renewing all the certificates you issued and reconfiguring apps to use them (which, admittedly, may not be a lot of work if you have a couple of intranet apps).

 

The root certificate's private key should be exportable (it may well be so by default?) and you should probably export it for backup. Also, make it SHA2 (or SHA256, whatever) if you can - this also should be the default on newer Windows. Newer software like the Chrome browser complain (with NET::ERR_CERT_WEAK_SIGNATURE_ALGORITHM) if you use a SHA1 certificate for SSL - not that it would matter in this case since this is a root certificate and you can produce an SHA2 SSL certificate off an SHA1 root - but it's better to be up to date from the start.

 

 

Since this is a self-signed root certificate that nobody knows of, you should accept it as a trusted root on all of the computers that will access your intranet sites - that way, when you generate SSL certificates for your sites, they will be automatically accepted because they come from a trusted root. For this, you need to export the root certificate without the private key, distribute it to your intranet users and make them install it into their Trusted Root Certification Authorities store (or, of course, do it for them).

 

Some of the guides say that you have to get the certificates right the first time since it isn't possible to change anything afterwards. From what I've seen, this is not true and I think you can generate a new root certificate and delete the old one: but, if you generated SSL certificates using the old root, you'll need to regenerate all of them.

 

Generating SSL certificates

 

I found this guide that describes the full procedure including setting up the certificate template: https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn781428(v=ws.11)

 

You need to create the template only once and with that, creating a SSL certificate is trivial, just as described on the page: none of that IIS nonsense with creating a request and then accepting it.

 

If you want your certificates to last longer than the default two years, read the solution here: https://social.technet.microsoft.com/Forums/en-US/4bff59d2-925a-4b28-a982-c1a0246f9bee/issued-certificates-not-following-template-validity-period?forum=winserversecurity

 

With this, to create a certificate you just need to fill in its attributes. For SSL certificate, this means setting the common name and one or more DNS alternative names and not much else. When creating a certificate for a single web app, just use its domain qualified name (for example, myapp.mydomain.local) for both and you're done. One interesting thing that you could also do is issue a wildcard certificate (like *.mydomain.local) which can then be used for each of the sites. You do this in exactly the same way, just use the same wildcard name for common and alternative name.

 

Creating a certificate for the AD FS service is similarly straightforward, just issue a certificate for your federation service address (for example, fs.mydomain.local). You may also want to add "certauth.fs.mydomain.local" to the alternative names, I'm not sure when one needs it (I don't think I did - it's for certificate authentication) but one of the final steps in ADFS installation displays a warning if you don't. Depending on further requirements - if you need one for token signing etc. - you might need to issue additional certificates with different characteristics. The service certificate seems to be the only one required for AD FS installation.

 

Installing Active Directory Federation Services

 

On Windows 2016 and newer, ADFS does not work through IIS but cooperates with it. This means that some ADFS paths (like, fs.yourdomain.local/adfs/fs/federationserverservice.asmx) will be reachable through any of the IIS virtual sites… I don't know if that would work for ADFS: Since I have several web applications, I assigned multiple IP addresses to the server and limited IIS to use only a couple of them (one should suffice). So it might be a good idea to add a separate IP for ADFS and add this to your DNS. But don't let IIS bind to it. Also note that, in order to use an SSL certificate for a IIS site, you need to check "Require Server Name Indication" in the site's binding dialog. Otherwise, the selected certificate will be used across all sites in the IIS, including ADFS which will present a wrong certificate when accessing the FS site.

 

When you have the SSL certificate for your ADFS service, you can proceed to install the ADFS. I followed this guide (more or less, I skipped the powershell gMSA part since it didn't work and I didn't even know what it was): http://arnaudpain.com/2019/08/05/windows-server-2019-adfs-step-by-step/

 

Here's how you change the FS service certificate: https://www.petenetlive.com/kb/article/0001634

 

A neat trick - to display all the URLs that ADFS registered, run this from the command prompt:

 

netsh http show urlacl

 

To check if ADFS is running: https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/deployment/verify-that-a-federation-server-is-operational

 

And that's it… The ADFS is only installed and it isn't configured for anything, we covered the configuration for Orchard Core in another post. But now you have a way to integrate your AD authentication with a greater number of applications and do all kinds of wonderful (and overtly complicated) authentication schemes until your brain melts.

Finding ASP.Net Core Error Logs on IIS

I don't think there are many IT tasks more frustrating than troubleshooting Microsoft technologies on Windows. These things are simply not made so that anyone knows what's going on, and the really brutal stuff happens when you combine Windows, IIS and Asp.Net Core. Correct me if I'm wrong, but by default it's not configured to report any errors anywhere, unless you run it in a console. And if you want to troubleshoot a deployment that isn't working, be prepared to spend hours finding out what the hell has happened. NOT fixing the error, just identifying it.

There's a myriad of places where logged messages could appear, and this seems to depend on the error type: if you knew what's going on you'd probably be able to guess where the message is, but in that case you wouldn't need the error logs in the first place. And if you ask Google, you'll find sites that list these options for you. I'll try to list them here:

  • IIS access log - useless for this as it lists successful requests
  • HTTPERR - for errors, but in my IIS it's empty even if my web app falls apart. It's in C:\Windows\System32\LogFiles\HTTPERR
  • Event Log - they say that Asp.Net errors appear in the Application section here. But Asp.Net Core doesn't seem to respect that.
  • Detailed error page - this doesn't seem to exist anymore (even though it can be turned on in IIS configuration). Some sites suggest that the only solution is to implement our own!
  • Adding a custom logger - serilog works great, but I need to change the source code to use it. Why would that be necessary? I was unable to find a way to just configure logging to save messages to a file - it can do this to console, but to a file, it seems, not. There isn't even a standard config entry for the log file name, you have to add your own configuration logic.
  • Failed request tracing - this, in fact, does produce some results but again it also hides a lot. It seems to be good for errors in the IIS pipeline, but when my Asp.Net Core application broke, the trace showed a warning (!) that a component in the pipeline returned a 500 internal server error. But what error exactly, it didn't say.
  • Stdout error loging in web.config - finally something useful, it can be turned on without modifying the code and it helped me see the actual error, after 4-5 hours of fighting the damn thing (note that you need to create the log folder yourself and possibly give the web app permissions to write to it). But this is seemingly just part of the solution, I'm guessing it works only if Asp.Net Core works. If not, see above.

Configuring Orchard Core for AD (AD FS) Authentication

How to set the Orchard Core CMS to use a local domain / Active Directory for authentication? The current Ochard version (1.1) doesn't seem to support it, at least not that I know... It does support Azure AD but that is a different beast. One way to make it work is to install Active Directory Federation Services (ADFS) on your server, which would broaden the AD's compatibility by making it support several additional authentication protocols, amongst which is OpenID which is supported by Orchard.

This post will be focused on how to make Orchard and ADFS work together, not how to install ADFS - but I intend to write another post (effectively, a prequel!) that will provide guidance on how to do it. There's a myriad of configuration options and many choices to be made in the process - most of which, in fact, you don't have to make because they don't apply to your particular problem, but there's noone to tell you that. I'll try to fill the gap by providing information on which steps led me to the solution and which can be ignored... But more on that next time, for now I'll say this: setting up ADFS is not that hard, setting up a Certificate Authority to work with it seeems nightmarishly complicated but isn't, and I think both are worth the effort if you have applications that can authenticate through ADFS... If you centralize the users, they will have to remember - and have the chance to forget - only a single password.

Ok, so how to do this for Orchard? I'm writing this as I have done it on Windows 2019, other versions should be similar and post-2012 versions almost identical, from what I've seen. Note that I'm by no means an expert on authentication protocols, I'm just documenting the setup that worked for me in order to help others trying to do the same thing.

 

AD FS Application Group

Configuring ADFS for this is dead simple when you know where to go: type "AD FS" in the start menu to filter down to "AD FS Manager" and then start it (alternatively, it is accessible from the Tools menu of the Server Manager app). Right-click Application Groups in the left list and click Add Application Group. Select the Web Browser Accessing a Web Application option and give it a name. Click Next to go to native application setup: here you get a generated Client ID (you'll need that for Orchard configuration) and the option to add one or more redirect URLs. From what I've seen, Orchard always uses its root as a redirect URL so add that. This URL must coincide with Orchard configuration, otherwise the authentication will throw an error. Multiple URLs can be added which is handy because you can configure a single application group to work with a development Orchard instance you run in Visual Studio (something like https://localhost:5001 - yes, it will work with localhost as well) and a production server (https://orchard.mydomain.local). Click Next - I left this setting at "Permit everyone" - and then two Next's and one Close to finish. And now you have a configuration that can work with Orchard. It can be improved, but more on that later.

One detail that might be of interest is that the relying party identifier in the web application properties is the same as the native app's client ID. This is obviously right and it works that way, but the form explicitly gives "https://fs.contoso.com/adfs/services/trust" as an example for a relying party identifier - so, a URL and not an ID: a false lead.

 

Orchard OpenID Authentication Client

Orchard Authentication Client Settings for ADFSNow log in to Orchard's dashboard, go to Configuration - Features, filter the list for "OpenID" to find the OpenID Client and click the Enable button next to it. While you're at it, find the Users Registration feature and enable it if it isn't already.

Now move on to Security - OpenID Connect - Settings - Authentication Client. Set the display name to something that your users can recognise, because it will be displayed on an alternative login button - for example, "[mydomain] AD FS" (it would probably be less scary for them without "AD FS" but I kept that part so that I myself know what it is). The Authority field is the base URL where your Federation Service server accepts OpenID requests. It's something like https://fs.yourdomain.local/adfs, and you can check this from your AD FS Manager app: expand the Service node on the left and then select Endpoints. It will list a bunch of available URLs for different protocols and uses, none of them will be exactly /adfs/ but all will start with it, and several will be explicitly tagged as OpenID. I suppose Orchard knows how to add the rest of the URL by itself.

Next in the Orchard settings is the ClientID - you should copy it from the application group in AD FS. (If you forgot to do it, open AD FS Manager, select Application Groups on the left, then double click your application in the list and double click the native app in the dialog that opens... You will see the client ID, copy it and paste it back into Orchard). For basic configuration, all that is left is to select the authentication flow: I myself got it to work by checking "Use id_token response type" in the Implicit Authentication Flow section. The various callback and redirect paths I left at default (blank) as well as the scope (blank) and response mode (form_post).

A word of caution is in order here: if you're experimenting with settings, be careful around the Client Secret field (it isn't visible for implicit authentication flow). I'm not sure if it's a bug in Orchard but once set, the field cannot be reset directly as it doesn't show the saved value: there is nothing to delete. (Looking at the code, I'm under the impression that selecting a different flow type could reset it, though, but I was unable to find the right combination, possibly because the Edge browser joined in on the game, see below). Why is this important? The newer versions of AD FS report an error if you send a client secret where it is not needed, so nothing works because of it. It's quite a puzzle, you'll see the client secret field empty but AD FS will complain that it receives one. To make things worse, I allowed the Edge browser to remember my login password and for some reason it seemed to fill the Client Secret field with it, making it non-empty even if I didn't set it and making it look like Orchard did in fact show the saved value... So just be careful with it, don't say I didn't warn you.

 

User Registration

Now go to Security - Settings - User Registration, make sure that in the top combo box you have AllowRegistration or AllowOnlyExternalUsers selected. The first will allow anyone to register in Orchard (locally or through external OpenID authentication) and the second will allow only externally authenticated users to register. There seems to also be an option that the user log in with a local name and then link an external login to it - something like this can be done by logging in as a local user, then opening the user menu (usually top right and showing the username), clicking on External Logins and then logging in with an additional external user. But I haven't experimented much with this, my goal for Orchard was - as stated above - to use external users only.

At this point, you should be able to log in with the external user: log out, go to the login page and you should see a "Use another service to log in" label and a button with the AD FS name as configured previously. But when you do, Orchard will ask you to type in your local username (also provide you with an ugly generated default) and an e-mail.

 

Retrieving additional claims

To make this process smoother, we can configure it to retrieve the username and e-mail from AD. First we need to tell the AD FS to send additional stuff: go to AD FS Manager, select Application Groups, double click the group created previously, and double click its Web application. Go to the Instance Transform Rules tab. Click Add Rule: it will select "Send LDAP Attributes as Claims" by default for rule template, so click Next and add rules for different attributes. I added these pairs (LDAP attribute => Outgoing claim type):

  • SAM-Account-Name => "SAMAccountName" (I typed the claim name in myself, it isn't present in the dropdown... It doesn't matter what name you give it as long as it doesn't clash with another one and you use the same name in the Orchard registration script afterwards)
  • E-Mail-Addresses => E-Mail Address

For experimentation, I added various other attributes (given name, surname) to get the user's name - but haven't found a place to store them. It doesn't seem that Orchard cares for having anything more than a username and e-mail for a user.

Click OK to save this rule. Some sites mention that additional scopes have to be added here to be able to retrieve all the claims (specifficaly the allatclaims scope): they are in the Client Permissions tab and can be checked but everything seems to work for me with only the default - openid - scope checked. By default, Orchard uses the "openid" and "profile" scopes. (I suppose you could restrict your rules to a different scope and add that scope on the Orchard's Authentication client settings page).

We now need to modify Orchard's user registration script to use this data when creating a user. It's in Security - Settings - User registration, you need to check "Use a script to generate username based on etc." to make it editable. My script looks like this:

if(context.loginProvider == "OpenIdConnect")
{
  var unameClaim = context.externalClaims.find(c => c.type == "SAMAccountName");
 
  if(unameClaim != null)
  {
      context.userName = unameClaim.value;
  }
}

You can also uncomment the already provided line -

log("warning", JSON.stringify(context));

- to see the full structure of the returned claims in the log. This can help troubleshoot problems - for example, I experimented with code authentication flow but couldn't get ADFS to send any of the additional claims whatever I did, although authentication worked.

(By the way, the language used in the script is JavaScript, and a rather recent version it seems if it can handle these LINQ-like expressions... Haven't been able to find this information anywhere, but luckily I made a couple of syntax errors and got yelled at by some kind of JavaScript engine, which made it clear).

Having set this, you can now also tick the checkboxes below the script - "Do not ask username", "Do not ask email address" and "Do not create local password for external users" because everything can now be retrieved from AD.

We can even go a step further and tie a domain group to a local Orchard role. This is done using the same Issuance Transform Rules tab as before in the Web application properties: click Add Rule as before but this time select "Send Group Membership as a Claim". Here you can select a domain group and have the ADFS send a claim if the user is a member - there's a predefined "Group" claim type (I'm not sure if that's what it is meant for, looks like it is) and then type in a claim value - which I set to the group name.

But for my configuration I opted for a simpler solution: if the user logs in as the domain administrator, I'll assign it Orchard's administrator role. I think it's cleaner that way: ordinary users are just business users (they deal with content) and an administrator logs in rarely just to administer stuff. I used the following script (it can be set through Security -> Settings -> User Login, check "Use a script to set roles based on etc" to make the field editable):

if (context.loginProvider == "OpenIdConnect")
{
  // if it's the domain admin, accept it as the local administrator
  var unameClaim = context.externalClaims.find(claim => claim.type == "SAMAccountName");
  if(unameClaim != null && unameClaim.value == "Administrator")
  {
    context.rolesToAdd.push("Administrator");
  }
}

And, by and large, that's all it takes... Admittedly, it requires some time to get done but it's not very hard to do - and wouldn't be too hard to configure differently, I suppose. The bigger part of work lies, of course, in installing the ADFS and the CA (if needed) for it, but that is a different story which I hope to get to tell soon.

Repairing the WS-Management service on the Windows server

Some features of the Windows 2019 server can easily and inadvertently be incapacitated and many seem to get screwed up on their own. And it is quite possible that one error triggers another and getting out of that tangled mess is very hard and time consuming. It takes a lot of time just to restore the basic functionality of the Windows server.

It took me months of on-and-off investigation to figure out how to persuade the DNS not to add an entry for each and every one of the server's IP addresses, because when it does so, the DNS queries return these different IPs in a round-robin fashion and this causes errors. And there are no good solutions on the internet, they all say that "Register this connection's addresses in DNS" should be unchecked, but that's not enough: you also need to unbind the DNS server from all IP addresses but one (it's in Properties-Interfeaces in the DNS app) and also removing the IPs from the Name Servers tab in your DNS zone's properties.

After I did this - and it may or may not be related - the Server Manager started reporting errors like "The WS-Management service is configured to not accept any remote shell requests." and because of this I was now unable to install or uninstall anything. I solved the by using PowerShell equivalent commands, but it worked for some tasks and for some it didn't. The predominant solution suggested on the net was to turn on AllowRemoteShellAccess using gpedit.msc, but it required me to do gpupdate /force and this in turn didn't work because of some policy replicating problem on Windows. And all the time, the stuff that worked in Windows still worked but it was a gamble trying to change anything on the server. There was a bunch of solutions on the internet for this but none of them worked: just using the File Explorer to browse through the problemmatic SYSVOL share, I got the impression that the domain shortcut that was there pointed to the wrong server - it actually pointed to the domain but that resolved to the wrong server's IP... I believe I did some tinkering in the DNS that may have helped but wasn't the final answer. In the end, a brute force solution was required, I just copied SYSVOL contents from the other domain server. (Having two domain servers could have been the root of the problem anyway, and possibly the DNS changes).

Back to the original WinRM problem, I was now able to turn AllowRemoteShellAccess on, to no effect. So I tried other suggestions - and once again, most solutions revolved around the same repeated stuff, none of which worked. Running winrm quickconfig returned "WsManFault" with error number -2144108526 0x80338012, saying "The client cannot connect to the destination specified in the request" and that I should run winrm quickconfig to configure the WinRM service. Issuing dir wsman:\localhost\Shell to check the active configuration said that the path does not exist, even if doing dir wsman:\ listed localhost as a result. A StackOverflow post suggested this bit of PowerShell scripting that finally shed some light on this:

$c = Get-Credential
New-CimSession -ComputerName localhost -Credential $c

Running this for localhost reported an error but did work if I used the name of my server. So WinRM worked, but not on localhost?!

I found the final bit of information on Robin CM's IT Blog, here somebody finally approached the problem instead of repeating the same mantras already present on the internet. Running netstat -aon | find "5985" to find what IP address is the WinRM port (5985) bound to (and note that this is command prompt, not Powershell) returned:

TCP    192.168.24.82:5985      0.0.0.0:0              LISTENING       4

So the service really is bound to the external IP and not localhost!

The blog post suggested I do netsh http delete iplisten 192.168.24.82, but it seemed too drastic to me - from what I understand, this is actually the IIS binding (I could be wrong) and I didn't want IIS listening on all IP addresses. So I simply added localhost to the list (this was OK for IIS) and it worked:

netsh http add ipaddress=127.0.0.1

(note that I also did iisrestart, which may or may not be necessary). The Server Manager now works and winrm quickconfig stopped erroring out... In the end, I don't have AllowRemoteShellAccess configured in the GPO at all, it allows the connections by default, and dir wsman:\localhost\Shell now shows this with no complaints.

Now let's see what else is faulty... How about "Windows cannot access the specified device, path, or file." when the settings app tries to run control.exe? No - maybe next year.

Short tips to getting Excel working with ASP.Net Core OData service

I've been tinkering with an Asp.Net Core OData service designed to be consumed by the Excel client... There are a lot of details that will screw your brains if you don't get them right, and it doesn't help that ASP.Net is a fidgety beast in that bits move around in each version and there are more examples on the net that don't build than the ones that do. I could do a separate post for each of them since it's very hard to get relevant information, but for the time being I'll just list them here.

For one, older versions of Excel (pre-2016) have some OData support but it doesn't seem to be adequate, it's best to install the Power Query plug-in and use that for OData. In 2016 and later, be careful as there are two OData options - under Get External Data and under New Query... You want the second one.

If you need authentication, don't count on using OAuth (which was what I would have expected - OData, OAuth, right?). Excel supports basic, windows, web api (i.e. some kind of a security key) or organizational account (which I suppose is AD). My users are stored in my application database, so the only way was to use the basic authentication... Which is so deprecated that Asp.Net guys refuse to support it and I had to roll my own. (Not too difficult, there are examples on the net, but additional work nevertheless).

There's a ton of examples on how to implement a trivial Asp.Net OData service that returns data from a single Entity Framework-mapped table. I haven't found (m)any that show how to use more complex SQL queries. Because, if you need this data in Excel, you'd like to have complex query - or not? Apparently nobody thought about that. There's an open source component called DynamicODataToSql that understands the OData data-shaping commands and can convert them to SQL. Well - only if the SQL you start with contains a single table name... Uhm, at least it could be a view in the database, I guess EF can do that as well. But with a couple of modifications, this component can be persuaded to at least treat what you give it as a nested SQL query (turning it effectively into a table) and add its magic on top of that.

Also, Asp.Net doesn't know how to return appropriate errors through OData, it spits out an HTML error page whereas a Web API is supposed to return XML or JSON. So, more manual labour: solutions exist online on how to add a filter that does this, but it's not supported out of the box. Still, even with this, Excel seems to sometimes ignore some errors. Especially in the query editor window, the preview that it shows isn't necessarily the data that it pulled from the server at that precise moment (as in: you see your code throwing an exception on the server side but Excel pays no attention and still shows data). The best way to check if the service works seems to be to load the data into a sheet and refresh it from there.

And the final headbanger - for now, at least - was how to get Excel to do server-side data shaping. Because, for unknown reasons, it sometimes decides to load everything from the database and then filter data by itself... Which is insane. One important bit I found that makes or breaks this is the URL you give Excel to access the data. If you target your OData controller directly (e.g. http://localhost/odata/MyData), everything will seemingly work but the data will be filtered on the client. If the URL points to the base OData directory (e.g. http://localhost/odata), Excel's editor will add a Navigation step to the query to select the controller - and with this the server-side filtering will work. Now, I'm talking only about filtering as I'm not sure if Excel supports other OData stuff like grouping: filtering is fundamental and I'm OK with counting the rest - especially given all the problems listed above - as a bonus.

VisualStudio install cache folder is important

I tried to find information about the "VisualStudio install cache" folder that Visual Studio installer creates, but was unable to find much. The folder takes a healthy couple of gigabytes of disk storage and, having the "cache" keyword in it, seems to be safe to delete. But deleting it will cripple Visual Studio updates. My Visual Studio 2017 started saying that its version is 15.0.0.0 and that everything is up to date, while for Visual Studio 2019 the update tool reported missing arguments. (Also, trying to start the same tool from command prompt with different arguments resulted in the maddening "Sorry, something went wrong" error message).

The main reason for this is the most important part of that folder, and that is Packages/Instances. This is where the installer seems to keep the information about your Visual Studio's current state. There's a folder for each installed Visual Studio instance - and the number seems to be random which makes sense because you may have multiple instances of the same (or same-ish) version of Visual Studio: you can have, for example, a release and a preview version of VS 2019. If this folder is lost, the installer won't know which versions you're on, your updates won't work, and the current installer refuses to install everything anew on top of an existing installation, so there's no easy way out. You will be forced to use install cleaner tools (probably without success), then delete the whole Visual Studio folder and install everything from scratch.

But Packages/Instances is not the only problem: when starting, the installer also seems to check for installation packages of installed extensions and won't work without them. These have semi-intelligible names but there's more than one folder for each extension, and some depend on external packages which have their own folders... Therefore, it's not easy to figure out which of the packages are important and which aren't. The unimportant ones can be deleted, the installer will complain about some but will know how to download and restore them so that everything is back to normal.

If you want to get rid of the excess packages, the regular way seems to be to run the installer with "--nocache" argument. (Or, use a more elaborate script from Microsoft). If that fails, you can make a backup of the existing content, remove everything you think you can do without (leaving the Instances folder is a must!) and then run the installer. It will complain that there was an error, and when you click Retry it will give the exact folder name for the first package it was unable to find so that you can restore it by copying the folder back. But you must repeat this for all missing packages - and I had more than ten, I think. At one point, clicking Retry will stop giving specific error messages (even if the installer reported an error), but the "install" button will become available and you will be able to click it to automatically download the rest of the missing stuff. I managed to shave off two thirds of my cache folder, reducing it from 3-4 gigs to slightly over 1 GB this way.

How to set a category for Visual Studio new item templates

A .vstemplate file defines a template for an item that appears in Visual Studio's "Add New Item" dialog. But it doesn't have an option to choose which of the categories (shown in the left part of the dialog) it will appear in.

Add New Item Category

I know of two ways to use the .vstemplate file and there are two different solutions to specify the category.

If you are simply copying the template to your Documents\Visual Studio XXXX\Templates\ItemTemplates\Visual C# or similar folder, just create a subfolder named the same as the category you want. It seems that Visual Studio processes these folders once when the first Add New dialog is opened and ignores further changes so it needs to be restarted in order to refresh.

If you have a VSIX add-in with a VS project containing item templates, you need to right click on an vsitemtemplate file in the Visual Studio solution explorer, choose Properties, and in the displayed properties window set the Category property to your desired category name. Note that the Category property is invisible unless the Build Action for the file is set to VSTemplate - but if it isn't, then the template won't work anyway.

Visual Studio property window for the vstemplate file

Implementing Equals() and GetHashCode() in ORM classes with autoincrement keys

The requirements for implementing Equals() and GetHashCode() in .Net are very hard to satisfy and in some areas nearly impossible. There are some situations where an object's identity inevitably changes, and saving an ORM-mapped object with autogenerated key is a case in point. Making its hash code conform with requirements would require an inordinately large quantity of code.

The plot goes something like this: loading ORM objects on the same session (in NHibernate speak) or context (Entity Framework) guarantees that for one record only one object instance will be created. But, if you use multiple sessions/context, no such guarantee exists. (And you want to do this if objects have different lifespans: for example, you fill a combo box once and then bind it to different records... Obviously, I'm talking about WinForms here, but the principle applies to server-side logic although it's probably not as frequent). In case of multiple instances, .Net components don't know it's the same record unless you override Equals() and get them compared by primary key values. In WinForms, for example, this means that a combo box won't know which record in the dropdown is equal to the one in the bound property, and won't select it.

Ok, so we override Equals(): usually, the record has an autoincrement key called, say, ID. We implement it so that it compares ID's of different objects (and type, obviously)... And now we run into the .Net requirement which says that two objects that are equal must have the same hash code, and that the hash code must never change for an object. We can override GetHashCode() to return the hash of the ID, but if the object gets saved to the database, the ID - and therefore the hash - will change.

Here's an example of how it would work: create a new ORM object instance, it's ID is NULL or zero. Use it as index in a dictionary, the dictionary retrieves the index's hash code and stores data in a special bucket for this hash. Save the record - the ID changes. If the hash code changes now, you won't be able to retrieve your data from the dictionary anymore. But if you load this record on a different session/context, it will have a different hash code unless we somehow notify it to use the already generated one... Which would probably mean using a static instance of a component that tracks all objects. Way too much work to get a hash code right, isn't it...

A couple of details that could lead us closer to a solution:

  • On a new object instance (unsaved - with ID null or zero or whatever), we cannot use the ID in our overrides. Two objects with the empty ID are not equal nor will they ever be: if they both get saved into the database, it will create two separate records, and their IDs will acquire new values that were never used before... An unsaved object is only equal to itself. We could generate same hash codes for unsaved objects, but this wouldn't resolve our problem if a saved object gets its hash from the ID - it would still be different.
  • While we're at it, it's useful knowing how the Dictionary works: it calls GetHashCode() on the given index and stores the entry in a bucket tagged with this hash code. Of course, there may be multiple objects with the same hash, and a bucket may contain multiple entries. Therefore, when retrieving data, the dictionary also calls Equals() on indexes in the bucket to see which of the entries is the right one. This means we have to get both Equals() and GetHashCode() right for the dictionary to work: Equals() should be OK in its simplest form if we always use the same instance for the index - basically, Equals() must be able to recognise the object as equal to itself.
  • Other components, like grids and combo boxes, also use hash codes to efficiently identify instances, so a dictionary isn't the only thing we're supporting with this.

One part of the solution seems mandatory: we need to remember the generated hash on the object. This is mandatory only on an object whose ID may change in the future: if a saved object gets a permanent ID (as they usually do), caching is not necessary. If an unsaved object gets saved, we still use the hash we generated using the empty ID. We do this on-demand: when GetHashCode() is called, the hash is generated and remembered. This is probably the only meaningful way to do this, but it's worth pointing out one detail: if the object isn't used in a dictionary, it's hash won't be generated and won't change when it's saved. Thus, we narrowed down our problem to where this feature is actually used.

But there's still the possibility to have two objects that are equal (same record loaded on different sessions) but have different hash codes (the first one saved into the database and then the same record loaded afterwards). I'm not sure what problems this would create, but one is obvious: we wouldn't be able to use both of them interchangeably in a dictionary (or, possibly, a grid). This is not entirely unnatural, at least to me: if I used an object as an index in a dictionary, I'm regarding the stored entry as related to the object instance and not the record that sits behind it. I'm unaware of other consequences - please comment if you know any.

Note that we can also avoid the dictionary problem by using the IDs themselves instead of objects... But still, it would remain for grids and elsewhere. Also I'm not sure if it could be resolved by not using the data layer (ORM) objects in the dictionaries and grids but having data copied into business layer object instances: if we did this, we'd still need a component that tracks duplicates, only it would track business objects instead of data objects.

Can we narrow this further down? A rather important point is that the ID gets changed only when saving data - and ORMs usually save the whole session as one transaction. If we discarded the saved session and started afresh, we'd get correct hash codes and unchanged IDs. We'd only have a brief period of possible irregularity in the time after the old data is saved and before new data is loaded, and only if we load data on different sessions and use it in a dictionary or some other similar component. In client-side applications, this is a risky period anyway because different components get the new data at different times, and care must be taken not to mix it. At least some kind of freeze should be imposed on the components - suspending layout, disabling databinding etc. Also, reloading data is natural if you have logic that runs in the database (usually triggers) that may perform additional changes to data after we saved ours (and it may do this on related records, not just the ones we saved)... But that is a different, as they say, can of worms: it's just that these worms often link up to better plot our ruin.

Subscribe to this RSS feed