Canopy Coding

The Mysterious Keyboard Remapping Gnome

Mon, 15 Oct 2007 22:10:44 GMT

I work, as do most of the people at Quercus, on a Dell Latitude D820 laptop. They are great machines and I wouldn't hesitate to recommend the entire line to anyone who is looking for a good desktop replacement. It's only real drawback is that the internal HDD is a little on the small size. We also run most of our applications in a virtual machine or through a citrix server. And this is where it sometimes gets goofy.

Every once in a while, and for no apparent reason, the keyboard would remap itself. The basic keys (a-z & 0-9) would remain the same, but everything else would shuffle around. The ' key would suddenly turn into the ` key, and the " would become the ~. The \ became an E with an accent on top, and the # would become a /. All in all, it made it very difficult to cut code. Even typing a basic letter was difficult. Rebooting the machine would sometimes fix the problem, but not always. A cold boot would usually fix the problem, but again, not always. To make it even odder, the keyboard would be fine if you left the virtual machine environment.

Well, this happened to me just the other day while I was writing some code. One second my keyboard was fine, and the next it was confused. I had always figured that there was a key combination that was causing this, but I could never find it, even after looking on Dell's forums and trying Google. What do you search for? I tried "keyboard remapped" and "keys change position", but they didn't get any useful hits.

So this last time it happened I started playing with the keys that my fingers were close to when the keyboard shifted. And lo-and-behold, this is what I found.

Shift + Ctrl + Fn

Hold down the three keys together for a second, and release them simultaneously (that seems to be important), and the keyboard will go back to normal. Or at least, it's worked everytime since I first discovered it.

Incidently, if you ever get a Logitec keyboard that has the extended (non-english) characters on the sides of the keys (I have one at work), you will notice that the keys don't randomly remap, but they instead change to the non-US keyboard. I'm guessing Dell just puts the right keys on top of the keyboard for the layout you order, but all of the machines are capable of handling any of their keyboard layouts. I just wish that it actually changed the keyboard in Windows (it still shows US standard) or gave you a popup, or at least a help file somewhere that told you how to un-do it. Why it sometimes survives a reboot and sometimes not is beyond me, you'd think it would either be written to permanent storage somewhere or not...

Checking SQL security from your code

Wed, 10 Oct 2007 23:11:39 GMT

The client I'm working with at the moment has decided to push their security settings into the database. By that I don't mean that they are setting up a special table for permissions or anything, they are actually using the built-in security in the database to control access to the data. At first I wasn't convinced this was a good idea, since the more normal N-Tier methodology was to simply lock everyone except the data layer out of the database, and then let either the data layer or the business logic layer handle who could do what.

The problem with relying on the DAL or the BLL to secure the database is that Microsoft has been making it easier and easier for non-technical people to access databases. Where you used to have to enter a long arcane connection string using a specialized program to access a SQL database, now it is as easy as 3 clicks of the mouse in Excel to connect to a remote database. If the users ever find a valid username and password to get into the database, they can play at will and you won't even know they are in there. So what do you do if you have users who like to play? You lock them out using their Active Directory login.

The client has configured their databases to only use integrated authentication, there are no SQL logins (except SA, which is password protected by the DBA). So anything you want to do has to be granted to you by your AD group membership. Therefore it doesn't matter how you get into the database, be it using the application that we wrote or one that they hacked together on their own, they still can't do anymore then then they can from inside the application. So one problem solved.

The problem that using SQL security causes is that it is not immediately accessible by the UI. So that leaves the UI developer with the option of leaving the Save button enabled and then telling the user that they can't save the record after they try clicking on it, or else finding some way to query the database to see if the update or insert is going to succeed when the user opens the form, thereby allowing the UI to disable buttons that are not going to work anyway. We decided to see if we could convince SQL to give up its secrets.

Luckily Microsoft anticipated this requirement and provided a stored procedure which makes it fairly painless to do this. While the security is stored in several of the system tables which could be queried directly, since MS spent the time to make a nice neat stored proc, I figure we might as well make use of it. It is called sp_helprotect. (Note that there is only one "p" in that word...) If you just call the stored procedure with no parameters it will return a list of all of the objects that you have rights to. However, if you pass in a table name, or a view name, or a stored procedure in as the first parameter, you can get just a list of the rights (Insert, Update, Delete & Select) that pertain to that object. Note that the Select and Update may make use of an additional column in the resultset which is "Column", since you may have select rights to all of the columns, but you can only update a few of them. If column-level security is used, you will get a row for each column for both update and select permission types.

There is also another stored procedure that is called sp_helprolemember which will tell you if the user belongs to a specific role in SQL (not Active Directory). This is useful if you have non-data related security that you want to create, for instance a CanPrint role which allows users to print a page. This stored procedure also accepts the role you are looking for as the first parameter, but I never use it since instead of coming back with zero rows if you don't belong in the role, you instead get a SQL error thrown. It's cleaner to get a list of all the roles (using no parameters) and then step over them to find the role you are looking for.

Grr, stupid arrays. Next time use an ArrayList...

Thu, 06 Sep 2007 13:47:26 GMT

Ok, after a brief hiatus over the summer where time to blog just didn't seem to exist, I figure it's time to get back to it.

So, for the last week or so I've been converting some custom controls that I originally wrote in VB.NET into C#, since our client has decided to switch their standard. This isn't terribly difficult, mostly just tedious, which makes it all the more frustrating when the 3 lines of code that were working in the VB version don't work in the C#, for no apparent reason. Once I finally figured it out, all I can say is that it would have been a lot easier if the debugger would have broke on the line that actually threw the error, instead of on the next line...

So, here is the VB code. This code is trying to find the name of an object used to track transactions in a DAL generated using LLBL Gen Pro, but the actual name of the DAL is not known at compile time, so it uses reflection to find the correct assembly and load the transaction. The transaction object does not have a default constructor, the simplist constructor takes 2 parameters.

'Get a transaction object from the DAL

'Get the assembly that is loaded that contains the type of the DataSource (in the DAL).

Dim a As System.Reflection.Assembly = System.Reflection.Assembly.GetAssembly(CType(DataSource, Object).GetType)

'Get the root namespace from the assembly

Dim RootNamespace = a.FullName.Substring(0, a.FullName.IndexOf(","))

'Convert the entity name into the complete name of a transaction object

Dim FullName As String = RootNamespace & ".HelperClasses.Transaction"

'Setup the params for the constructor

Dim params(1) As Object

params(0) = System.Data.IsolationLevel.RepeatableRead

params(1) = "del" 'Name of transaction

'Get the transaction object

Dim tran As ORMSupportClasses.ITransaction = a.CreateInstance(FullName, True, Reflection.BindingFlags.CreateInstance, Nothing, params, _ System.Globalization.CultureInfo.CurrentCulture, Nothing)

And here is the same code in C#. When this is run, it throws an error or "Index out of bounds" and stops on the line that uses the assembly to create the transaction.

//Get a transaction object from the DAL

//Get the assembly that is loaded that contains the type of the

//DataSource (in the DAL).

System.Reflection.Assembly a = System.Reflection.Assembly.GetAssembly(((object)DataSource).GetType());

//Get the root namespace from the assembly

string RootNamespace = a.FullName.Substring(0, a.FullName.IndexOf(","));

//Convert the entity name into the complete name of a transaction object

string FullName = RootNamespace + ".HelperClasses.Transaction";

//Setup the params for the constructor.

object[] p = new object[1];

p[0] = System.Data.IsolationLevel.RepeatableRead;

p[1] = "del";

//Get the transaction object from the DAL assembly

SD.LLBLGen.Pro.ORMSupportClasses.ITransaction tran = (SD.LLBLGen.Pro.ORMSupportClasses.ITransaction)a.CreateInstance(FullName, true, System.Reflection.BindingFlags.CreateInstance, null, p, System.Globalization.CultureInfo.CurrentCulture, null);

Anone spot the problem? I spent hours trying to figure out why the constructor was failing. The truth is, it's not the constructor at all. The compiler is stopping on the wrong line! It should have stopped on the next line up, where p[1] is assigned.

You see, when you define an array in VB.NET, the number that you put in the brackets is the maximum index that is allowed in the array, which makes every VB.NET array one element larger then the number specified (since arrays are zero based), so it is 2 elements big in this case. However, in C# the number you enter in the definition is the actual element count that the array should have, so in this example, I only get one element, which is p[0]. I changed the array declaration to allow for 2 elements and presto, everything works.

Now this would have been way easier to find if Visual Studio hadn't lead me astray. Instead of spending hours trying to figure out why the constructor was working in VB and not in C#, which was a reasonable assumption since the constructor was consuming the array and there are other constructors which require more parameters, I could have immediately figured out that I'd simply declared my array too small if it would have stopped on the line where p[1] was actually set.

So, what I learned today is...

If you ever get an Index Out Of Range error, check your array declarations first, regardless of which line Visual Studio reports the error from. This is especially true if you accessed an array on the previous line of code from where Visual Studio stops.

Default View oddities

Wed, 05 Sep 2007 21:04:21 GMT

For those of you who don't use LLBL Gen Pro, it generates Entity objects for you, one for each of the tables in your database. You can have it do more then that, but it can do that much simply by reverse engineering your database. It also creates strongly typed collections for each of these entities.

On the surface the collections work about as you expect. You can get any of the elements in the collection by index, you can iterate over them using For/Each, you can Add and Remove entities, you can load them using a filter, etc. They add another nifty feature not normally found on collections, and that is a .Sort method.

It isn't terribly well documented that access to the collection is actually granted through a DefaultView property on the collection, you don't have direct access to the collection. LLBL creates views that are very similar in function to the DataViews that Microsoft implements for their DataTables. The DefaultView is simply a view that is always present on the collection. This view starts with a blank filter (show everything in the collection) and a blank sort (show the entities in the order they were put into the collection). By simply changing either the Filter or the Sort properties on the DefaultView though, you can get very different behavior from your collections. You can also define additional view objects and attach them to the collection so you can have multiple views of the same list at the same time.

Of course I don't bother writing about this kind of stuff unless I got tripped and thrown against the wall at some point while using it. So here's the WTF for today.

I built a subroutine that accepted an IEntityView (generic interface for views) as a parameter. This subroutine was responsible for taking the view, which contained a list of properties that needed to be displayed on the form, and converting that list to actual controls and adding them to the page. A sort was applied to this view so that the controls showed up in the right order, and then the view was passed to the routine for processing.

SetupItemEnt.SetupEntity.SetupEntityDetail.DefaultView.Sorter = MySortExpression
'Sort is correct here...
BuildTableRowsForEntity(SetupItemEnt.SetupEntity.DefaultView)

The really weird part is that the sort didn't survive the passoff. It was there right before I called the subroutine, and on the first line of the subroutine it wasn't anymore.

Protected Sub BuildTableRowsForEntity(ByVal SetupEnt As ORMSupportClasses.IEntityView)
'Sort is gone, using database order now...
End Sub

When I switched the code to pass the entire entity collection (instead of the default view on that collection), everything started working.

SetupItemEnt.SetupEntity.SetupEntityDetail.DefaultView.Sorter = MySortExpression
'Sort is correct here...
BuildTableRowsForEntity(SetupItemEnt.SetupEntity)

Protected Sub BuildTableRowsForEntity(ByVal SetupEnt As LWS_DAL.EntityClasses.SetupEntityEntity)
'Sort is here now...
End Sub

I tried changing the parameter to passing by reference instead of by value, it didn't make a difference. The only thing I can think of is that the .DefaultView property must be somehow passing a different instance of itself to the subroutine then the one that I applied the sort to, but for the life of me I can't see why it would. I've passed entity views around before and it seems to work fine. This is the first (and last) time I ever tried passing the .DefaultView as a parameter though.

So, what I learned today is...

Don't pass a entitycollection.DefaultView as a parameter if it is sorted, the sort gets lost

Update: The catch with DefaultViews is that they are not persistently conected to the collection, but are instead generated at request. This means that every call to the .DefaultView property returns a new view, not the one that you got last time. Incidently this is also why a 2 column sort never works when using the DefaultView; It gets an instance of the view, sorts the view by the first column, throws away the first view and gets a new instance, then sorts by the second column.

Loading Collections (or any class) by Name

Wed, 05 Sep 2007 20:58:06 GMT

The application I've been working on for the past couple of months has a very generic UI on it. It uses a database table which configures which tables a user can see, how they see them, and what data they contain. This is all based on the entities that are available to the system through the LLBL generated data layer. Basically anything in the DAL can be configured, and anything that is configured can be modified by the user.

The UI literally consists of 1 form and 2 web controls, which configure themselves to suit whatever entity we are working with at the time. While this is all working fairly well at this point, some of the code that we needed was a little different. We are using reflection in lots of places to access properties on objects because we can't tell at design time which type of object we will be working with. LLBL defines a bunch of interfaces for their entities and collections which made this all possible. But one thing caused me some grief.

When we first load the form we need to get a list from the table based on some criteria and display all of the applicable records. Again, at design time we didn't know what type of collection we would need, so we couldn't use

Dim c As new LWS_DAL.CollectionClasses.ClientCollection

for instance. We instead had to use

Dim c As ORMSupportClasses.IEntityCollection = new LWS_DAL.CollectionClasses.ClientCollection

But how to load a collection at runtime when you know the name of it, but not the type? LLBL provided a factory that would return a collection of the correct type if you already had an entity, and the collection would return an entity of the correct type, but getting in the door the first time was a pain.

At first we had a long Case statement which would accept the entity name as a string and return the new collection, but it was a pain to maintain, and was not my intention to keep anyway, it was just all that I had at the time. We replaced it with this:

Public Shared Function GetCollectionByName(ByVal EntityName As String) As Object
    Dim retval As Object = Nothing

    'Get the assembly that is loaded that contains the type of AddressBookCollection (the DAL).
    'Doesn't matter what type we use, as long as it is in the LWS_DAL namespace
    Dim a As System.Reflection.Assembly = _
                  System.Reflection.Assembly.GetAssembly( _
                              GetType(LWS_DAL.CollectionClasses.AddressBookCollection))

    'Convert the entity name into the complete name of a collection of the appropriate type
    Dim FullName As String = "LWS_DAL.CollectionClasses." & EntityName & "Collection"

    'Use the referenced assembly to get the specified collection
    retval = a.CreateInstance(FullName, True)

    'Return the object.
    Return retval

End Function

This code gets the assembly that contains a well-known type at design time. Which well-known type isn't important, as long as we know that it is in the DAL assembly. Because this assembly is already loaded by the application when it starts up, we can find the assembly from any of the types in it. Once we have the assembly we build a string that contains the complete name of the object we are looking for, and then tell the assembly to give us one of those. If you wanted you can also load the assembly directly from the path, but they you'd need to know where it was.

Reflection is one of those things that made .NET famous, and in my experience very few people actually have ever needed to use. This project is the first one that I've used a lot of reflection in. Most of the time it's just not necessary, you always know the types you are working with when you build the page or control.

Displaying LLBL Validation Errors

Wed, 05 Sep 2007 20:55:49 GMT

For the last few days I’ve been adding validators to my DAL for the project I’ve been working on. This project is again using the LLBL Gen Pro generated datalayer. Last time I had the validators working so they would catch my custom validation errors and throw them to the application as exceptions. Today I had to setup something to catch these errors and display them to the screen.

Of course, as always, things are never as simple as they first appear. One of the things that's changed in the most recent releases of LLBL Gen Pro is that they have implemented IDataErrorInfo on the entities, which is a standard interface provided in the System.ComponentModel namespace. This allows the developer to attach an error message directly to a field on an entity, so when we go displaying the errors we know who to highlight. Unfortunately there seem to be some issues with the way M$ defined this interface.

One of the odd things with this interface is that it implements a .Item property. Now that doesn't seem odd until you realize that the .Item property is extremely common in .NET classes, and it is usually the indexer to boot. By declaring the interface to have such a common name, M$ made it virtually impossible to ever access the list directly from the class the interface is defined on. Instead you have to cast your object to an IDataErrorInfo object so that you can get to the .Item property. This is a little odd, but workable once you figure it out. It did have me scratching my head for a few minutes trying to figure out how to access the errors though. I was expecting a DataError property or something like that, not a .Items property.

Ok, so we write our DisplayErrors routine so that the first thing it does is cast the object-with-errors to an IDataErrorInfo object. Now you might think that we can access the errors using something like this

Dim dei As ComponentModel.IDataErrorInfo = ObjectWithErrors
For Each err As String In dei.Item
'Handle error
Next

Well, guess what. Yup, that'd be too easy. I'm not sure what exactly the IDataErrorInfo.Item is defined as, but it is not a standard collection or hashtable. It doesn't support an iterator, you can't even define one for it. It doesn't have a Count property available either, so you can't even tell if anything has errors on it before you start trying to display them. You have to know exactly what key was used to put the error into the collection or you won't be able to get it back out again. Luckily my custom validator routine put the errors into the collection, so I can figure out what all the possible fields are that might have caused an error. So, the code has to look like this instead

For Each f As ORMSupportClasses.IEntityField In ObjectWithErrors.Fields

   'Check this column for an error (either nothing or zero length string)
   If dei(f.Name) IsNot Nothing AndAlso dei(f.Name).Length > 0 Then

      'Add the error to the list
      ErrList.Add(dei(f.Name))

   End If
Next

Here I'm looping over all of the fields on the entity to see if any of them have errors. Any errors that I find I add to a simple arraylist. I'll convert this arraylist to an HTML bullet list for display later. If the arraylist comes out the other side of the loop with nothing in it, I assume that there were no errors and I hide the error listing control.

Oh, and I have to check for zero length strings since the first time I run this (before a postback) there are errors only on the fields that have errors and the rest are nothing. But after the postback the errors are all cleared out for me, but they (meaning the guys as LLBL) did that by setting each of the possible errors to a zero length string. So either condition can mean no error...

This is definately one of those days that seems like it was way harder then it had to be...

So, what I learned today is...

the IDataErrorInfo interface is only useable when you cast your object to it...
And even then it's hard to figure out who has errors and who doesn't

LLBL Gen Pro Validator blues

Wed, 05 Sep 2007 20:54:00 GMT

For the last few days I’ve been adding validators to my DAL for the project I’ve been working on. This project is again using the LLBL Gen Pro generated datalayer.

So the validators are actually really easy to build. I want my business rules separated from the DAL, so I create a validator by inheriting from the LLBL ValidatorBase class, override the ValidateFieldValue method, check if the new value for the field is going to be “” and return false if it is, and then attach an instance of the validator to the entity when the entity is created. Presto, like magic the method fires when I try to assign a blank string to the field, the error message for that field gets set, and the SetNewValue method returns false saying it didn’t set the value.

Here’s where things start to fall apart…

The first thing I found is that SetNewValue returns false whenever it didn’t set the value of the field, not when it got an error. This could mean that the validation failed, but it also returns false when you are going to set the value of the field to the same value that is already there. So false may mean there is a problem, or it may mean that the data simply hasn’t changed. Well, I guess that makes that flag kinda useless for checking for errors.

But wait. If I break any of the basic rules that the entity enforces on its base type (data type, data length, not null, etc), it doesn’t return false, it throws an Exception instead. Now maybe I’m just weird, but I expect my validation routines to simply check the data and then return IsValid (true/false). Then I will do whatever I want to do about invalid data. To me this is a bad use of an exception. It interrupts the flow of the code not because something catastrophic or unexpected has occurred, but simply because the user entered bad data. That’s hardly an unexpected event. Worse, if you are scraping many new values off of the form and updating the entity, the exception is going to cause it to stop as soon as the first validation error occurs, so you don’t get any feedback on the rest of the fields. Ick. Click Submit, get error, fix error, click submit, get next error… To stop this you would either have to do the updates in a loop so you can catch each error as they happen (which thankfully I am), or else build code that looks something like this…

Try {
   retval = SetNewFieldValue (Company.Name, “McDonalds”)
}
catch(ORMEntityValidationException ex) {
   //Handle error
}
Try {
   retval = SetNewFieldValue (Company.ID, 1)
}
catch(ORMEntityValidationException ex) {
   //Handle error
}

Now that’s just plain ugly. Each line it it’s own try/catch?

The author of LLBL does make one comment that caused me to put the same exceptions in my own validator though (since I had to handle the exceptions anyway).

“The exception makes sure any transaction currently in progress will be terminated.”

Ok, that is a good thing. I still think an exception is the wrong way to handle it though.

So, what I learned today is...

LLBL Validators are easy to write and use
LLBL Base validation errors throw exceptions when you try to set the new value
Apparently Exceptions aren’t always bad. Sometimes they’re just misunderstood.

LLBL Gen Pro rocks! (usually...)

Wed, 05 Sep 2007 20:51:07 GMT

The current client I am working with is using the LLBLGen Pro code generator by Solutions Design, and it rocks! After seeing this product in action I'm not sure why everyone isn't using it. If you are not you really owe it to yourself to check out the demo. (Thanks Robbie for showing me the light)

So far I have started one new application using LLBLGen, one of the other developers has started a new application using it, and we just finished converting an existing app from using datasets to using a generated DAL. I'd estimate that we removed 60-75% of the code that was in the application while we were converting it. Whole subroutines were nuked because they were no longer needed. Plus we got to redesign and improve the front end since it was so much easier to find related entities and use them, so even the business users are going to be impressed with the changes. The lazy-loading feature also provided a noticable performance boost.

I'd guess that we deleted at least a month's worth of work that was setting up the dataset and datatables that the application used. And it took us only an hour or so to setup the replacement! Modifying the rest of the code to use the entities and collections instead of dataset and datatables took about a week for this application (a fairly simple windows application with a couple of main data forms and a few maintenance forms). I think we could have been done faster if we hadn't taken the opportunity to clean up the database at the same time (adding foreign keys, removing unused columns, etc).

But all is not roses. Now that we have a few applications that are using this thing over here at Ledcor the "can we do this?" questions are starting to pop up. Most of them are pretty simple, but every once in a while something isn't so easy. Or it is easy and the documentation isn't terribly clear. Or it's really easy and I'm just having a slow day. Still, I figure any day that you learn something is a good day, so...

Today I learned:

The guy building the other new application using a LLBL data layer was having a problem getting one of his entities to update. He setup his entity, loaded it from the database, modified a couple of values and then fired the .Save method. Seems pretty simple, right?

aEntity.Save()

However, the changes were not getting written back to the database.

We added some debug.prints to his code to check the IsDirty flag on the entity before and after the Save method was called. Sure enough, it was dirty before we told it to save the changes, and it came out the other side claiming to be completely up to date. Hmm...

System.Diagnostics.Debug.Print("IsDirty flag before save: {0}", aEntity.IsDirty)
aEntity.Save()
System.Diagnostics.Debug.Print("IsDirty flag after save: {0}", aEntity.IsDirty)

IsDirty flag before save: true
IsDirty flag after save: false

So we checked the return value of the .Save method. Since LLBL is so nice to pass one back surely it would tell us if something went wrong during the update. WRONG. It returned true. We checked the documentation to make sure it wasn't supposed to be returning false for some reason, but nope. The entity was convinced it was clean now.

After a bit more headbashing and an uncountable number of mutterings and un-printable rants, we finally figured out what the problem was. We had gotten this message when we refreshed the LLBL project from the database, but hadn't paid it too much attention. Oops.

“WARNING! There are one or more tables which don’t have a primary key set and when used as an entity will probably not result in the code you want.”

"probably not result in the code you want" is a serious understatement. This causes the LLBL Gen code to completely skip the updates on tables that don’t have Primary Keys. Instead of comparing all of the database fields to their original values (a la Access), it just refuses to even try. You can configure LLBL to output its SQL scripts as it runs them, and the .Save method didn't result in any kind of update statement being run at all.

Ok, I can understand why it doesn't work, and I agree with adding primary keys to the table so you can conclusively determine which row you are trying to update or delete. However, the definition of "probably not result in the code you want" could have been a little clearer. How about "These entities will not be able to update or delete their data"?

In yet another odd turn of events, the Insert statements fire properly. Obviously you don't need a Primary Key in order to insert a new row, but why make the entity able to insert when it can’t update or delete? Wouldn't it be more consistent to just make the entity read-only? At least then it just reads data, and you know it just reads data. Having it able to insert new rows but not push update makes it look broken. And why, oh why, would you have the .Save method return true when it can't run the update? That sounds like a pretty big frickin error to me! Well, maybe not an error since it is expected behavior, but some warning would have been nice. And even if you argue that it shouldn’t return false, why the heck would it flip its dirty flag when it is incapable of updating the database? Isn't that the definition of dirty for an entity, when the data in the entity doesn't match the data in the database? An entity that is changed, but which cannot push those changes to the database, can only become clean by re-reading the database, not by calling an .Save method that does nothing.

So, what I learned today is...

Always have a primary key on your tables (duh)
If you see this message when refreshing your catalogue in LLBL Gen Pro, see #1
A return value of true doesn't always mean all is ok, sometimes it just means "don't you worry your pretty little head about it"