Code generation and architectures – for MV developers

I thought I’d take another shot at this topic of code generation, to define some terms and set the informational base that may help more people to understand and get excited about my other blog entry on this topic.

Pick guys who move from BASIC to .NET first struggle with OOP and the concept of a framework. With mv.NET for some there is a bit of a struggle to translate READ to .Read(), etc. At some point developers move beyond syntax and start thinking about structure, and it’s neat that a number of us are on this path together. A recent posting to the mv.NET forum shows how some developers approach common questions in isolation. Well, you’re not alone.

A lot of people have been down the exact same path, abstracting data access from file classes and UI. There are books and websites dedicated to the topic. Code architecture is a whole little world unto itself. The problem of "how do I structure my code" gets a bit more complicated when you realize that there is no one way that everyone writes code. There are many theories about how to make consumer / client code completely isolated from the database that provides the data. The question become more "which way do I go". Some people will pick a framework and run with it, helping to develop it, or at some point jumping to another one as they realize that it doesn’t really suit their needs or personality. Code architecture is a very personal topic but of course it dove-tails with more pragmatic points like "it may not be perfect for me, but is this framework suitable for our development needs?"

The people who devise these architectures base a great deal of their work on Design Patterns, originally documented by the so called Gang of Four (GoF). There are many books and websites dedicated to design patterns and some people can really go overboard with them. Moderation is key. Terms associated with design patterns include Singleton, Factory, and Adapter. These are the basis underlying architectures where developers decide what objects should be singletons and how factories should be implemented. If you’d like a very friendly start to Design Patterns, I highly recommend the book from the Head First series at O’Reilly. It was written for Java but you won’t be able to tell the difference if you’re working with C#. Another more sterile book on the topic is "Design Patterns in C#" by Steven Metsker. I think the Metsker book is more for school classes and it’s definitely heavy on the geek-speak, when the Head First book is fun as well as informative all the way through. Between these extremes you’ll find a lot of material in books and websites.

Three common architectures are .netTiers, CSLA, and NHibernate. Each of these has strengths and weaknesses and I encourage you to spend a little time understanding each. I admit my understanding in this area is limited – there’s just too much information to sift through, and like most developers, I don’t have time to become an expert on architectures, I need to write real application code. So as mentioned above, my research has allowed me to get just enough information about each of these to make a decision and run with it, and more pragmatic issues may compel me to make changes. CSLA is fully defined in a book by its developer, Rocky Lhotka, which you can purchase from his website. CSLA is well thought out and always evolving. There are also books on NHibernate, which is defined as an Object Persistence Framework (OPF), and puts more focus on how data is persisted and retrieved, in addition to the architectures used to operate on the data.

An architecture is generally implemented through code generators. Sure you can hand-write code according to established best practices but you’ll get tired of this when working on your first class definition. Implementing an architecture in your code is a massive undertaking because it requires many classes inter-relating in precise ways to hand off responsibility (Responsibility is another type of design pattern BTW) to other special purpose classes. And when you’re off to your second file class you’ll realize – wait, I’ve done all of this before but the names are just different… So when looking to implement a specific architecture, you’ll find there are code generator products seeking your attention and perhaps some part of your development budget.

But it’s not the code generator that’s key here, it’s the templates. Generating code is like cooking where you have a recipe, a cookie cutter or mould, ingredients, and a final product. The template is the recipe and the application code is like the final product – the cookie. The code generator simply follows the instructions in the template, stamping out new instances of your application code. For ingredients a code generator uses database schema. It pulls in filenames like Customers and Orders, fieldnames like Name and Date, and using the templates it shapes that schema information into business classes. The effectiveness of the final output, the final taste if you will, is dependent on good ingredients (like a well defined schema), a good recipe (the templates), and the skills of a code generating "chef" that brings it all together.

The .netTiers architecture is implemented as a series of scripts and code specifically for use in the CodeSmith Tools code generator. If you like the .netTiers theory of data handling then you must make use of the templates and related code created for CodeSmith. This is really a great combination, and the one I happen to be using now. If you don’t agree with how .netTiers does something, all of the source is there for you to change – but I don’t advise doing so until you’re very familiar with the big picture.

CSLA is also implemented as templates, and there are CSLA templates for CodeSmith, but they are very old and not well supported. There is no ‘authority’ that is paid to maintain templates for us, this is an open source community endeavor. On the other hand, the most recent defintions of CSLA are implemented in templates for the CodeBreeze code generator. My personal opinion is that CodeBreeze is not as mature an offering as CodeSmith, so while the CSLA templates are much better, and CSLA might be preferred to .netTiers by some, unfortunately I felt a need to move toward stability.

What we see here is that an architecture might be well defined and even quite popular, but without templates to implement it, we’re sort of stuck with a chef and ingredients, but no recipe. If we want cookies now, we need to keep moving and find a recipe and a chef that can make it into something nice.

The other factor here is the "ingredients" – the database schema. A chef can’t make use of the best recipe without ingredients. To get at a database and pull information about files/tables and fields, the world relies on adapters and providers. These are bridges between more agnostic clients that don’t really care what your data looks like, and data sources that have all sorts of nuances which make them different from the others. An adapter or provider exposes a well defined API to a client, and does whatever it needs to internally to satisfy client requests against a specific data source. The data source can be anything from Oracle to a text file. As long as there is an OracleSchemaProvider or a TextFileSchemaProvider, someone can point a code generator at it, and the chef now has the ingredients for applying the recipe template, and cranking out massive numbers of cookies – aka code modules.

In the MV world we have the issue that our database of choice isn’t relational. It’s somewhere between the highly relational Oracle environment and the text file. So to allow CodeSmith to make use of .netTiers templates and generate code for an MV data source, I had to create an mvSchemaProvider. This is what my other blog post was all about.

Following through, after nice classes are generated off of the MV DBMS, we’re going to want to use them. Buried in the depths of polymorphism and encapsulation, all of the framework templates at some point need to implement a Data Access Layer. At run-time the code needs to access a database. This is completely separate from the data acces made possible by a provider for the code generator. So the next step in this process is to start modifying the templates to execute BASIC programs rather than Stored Procedures with SQL, and to use mv.NET to read records as mvItems rather than executing SQL Select queries to return a DataRow.

Remember, those changes aren’t in the final code – you can’t go and change the taste of all of the cookies after they’re made (sprinkles and frosting don’t count). The data access code needs to be implemented into the templates – that is, the recipes need to be changed so that the cookies taste the way we want when they come out of the oven. In practical terms that means we should be able to generate code that runs against a database immediately. Obviously since the template code currently includes a lot of SQL handling, it’s a daunting task to retrofit that with mv.NET calls.

Another aspect of modifying the templates is that these are provided as open source and modifications to the templates are expected to go back to the community. I have full intention to contribute template updates back to the .netTiers code base, but no one outside of MV developers will be interested in those changes. So one challenge now is whether to make changes to the templates which are harmonious with existing SQL code, or to simply remove and replace the SQL code with MV code. Not only does that fork the code and create a maintenance issue but it does a dis-service for all of us when the templates are fixed or enhanced by someone else and the changes constantly need to be retrofit back in with the MV-specific templates. No, I think the right way to do this is to use a flag set in project configurations, and when the developer says the code is targeting MV then the template should simply invoke MV-specific code generation routines. Getting this to work at all is a major undertaking. Getting it to work without breaking anything in the core templates adds to the adventure.

Thinking aloud (…errr, even more so) (huh?) (nevermind)

Considering the relatively few .NET developers we have in the MV market, there is a very small pool of people who would be interested in contributing to this project, though Dick Thiot did offer to assist with code. Without funding for this I’m stuck trying to split my time between a project that will save massive amounts of time later, and projects that consume time now because I don’t have those tools. And there is at least one major project that I have in the queue now that I don’t dare even start to hardcode. There are hundreds of files and as many screens that need to be fit into a GUI. This would take a year to write by hand but only a couple months with the proper tools. And yet without the tools, the project never gets started – it’s very "chicken and egg", and something has to budge on this soon.

So the development is moving slowly. I would be very happy if someone offered to pay for my time to work on this open source project, rather than paying me to hand-code a major app that will be more subject to human error, will continue to require more manual fine tuning in the future. Unfortunately, that sort of sponsorship is unheard of in our community.

I do want to come back to something, having mentioned open source. CodeSmith is a for-fee product – only $100 for basic tools and $400 for the the whole package. (Amazingly low cost considering the value.) My mvSchemaProvider is not open source and would be required for purchase by anyone who wants to generate code. The free and open source templates provide the recipe, but that’s unrelated to access to the data ingredients. As an alternative to funding for working on the templates themselves, I would be happy to continue leading development on the templates if there were enough purchases of the mvSchemaProvider. Considering the significant long-term and short-term benefits of all of this (essentially the same freedom enjoyed by developers for relational platforms who also pay for their tools) I’m hoping people will understand and help with this project.

Regardless of this project and financing issues, I hope you’ve found this all to be informative enough to help you with some of your own development challenges and plans.