ASP.NET -- How to effectively use design patterns without over-engineering!

Setting aside the possibility that these levels of abstraction are required by the organization (unlikely, but it does happen), it is very common for developers - especially in non-agile, corporate/enterprise environments - to add too many layers of abstraction. It happened with classic ASP, .NET just makes it a lot easier. Most of us in this profession have a natural tendency to overcomplicate things, and it takes discipline and an active mind to overcome that.

Also, many mediocre developers wrongly believe that maximizing layers of abstraction and pattern use makes them better developers.

Finally, many mediocre developers have been taught that they should use layers, abstractions and patterns, but have not been taught well enough to know how, when or why. So they go into a project thinking "well, I know I should have some layers here..." and you can imagine what comes out.

Ultimately, the "best practice" of best practices is to code as simply as possible UNTIL you run into a real problem that is preventing you from moving forward. Many times, you should have a pattern or best practice in your toolbox that is right for the task, just like you have the right wrench for a nut. Best practices and patterns are tools - you don't get out a hammer and start swinging until you have a nail that needs to be pounded. Likewise with software development.

Some people are architecture astronauts, and they will insist that these layers are all necessary and the design is absolute garbage unless you include them.

On the other end of the spectrum are people who push this all aside and just have the code all munged together: it's simple, right, so why not just have the SQL query right there in the code-behind populating to a listbox control?

You've identified the problem with the astronaut version already, it's just so overly complex to do simple tasks.

The problem with the second method is that while it works fine for small, tiny apps, those same apps have a tendency to grow into large, big apps, and then it all breaks down. You spend tons of time fixing the same bugs over and over, because that same query or piece of code has been copied/pasted as needed, and slightly modified, and spread everywhere.

There is, of course, a middle ground. Where exactly that lies is going to be different depending on who you ask.. but it's there.

Building a sane business layer is a good step to getting there. This should be a layer that presents the simplest view of the objects in your system, and contains no public-facing database-related stuff (aside from maybe a Connection object that tells the business layer how to get to the database). This consolidates all your logic into one place, so the ASP part of the project will just be wiring up the abstract "User.LoadAll()" method to the table that displays a list of users. The ASP app should have no clue if it's reading from a database, web service, or just a bunch of made up stuff.. it simply talks to the business layer.

Under the business layer, you can have queries directly, you can use ORM, you can build a deeper data access layer that does these things .. the nice thing is this decision is entirely contained by the business layer (and you can change it later, if you need to, without affecting the public API of the business layer).

Now any queries dealing with users are contained within the User object.. and if you have a bug with the way you load permissions, it can be fixed in one place.

I just did a migration like I described above (ASP with SQL in code-behind, to a distinct presentation - business - datasource (ORM) layer), and it is great. Though it does add a few layers to getting at actual information, once the business methods work, the rest of the code works. Fixing a bug is simple, and fixes it everywhere. Near the end, it quickly became the case that when you needed to get a list of reports that a user could run, someone had already written a business method for that (User.GetReports()), where before, you'd probably never find it if it had been done - if you were lucky, it was written as a function and not just inline code.

I've worked on projects that are badly done in the opposite direction (200+ MB source trees with no design patterns). Believe me, that's not the right solution either. The code gets unmaintainable, there's old code laying around that developers trip over, etc.

On the other hand, to answer your questions about design patterns:

No this is not progress. Yes, some people are going overboard. There probably is a happy medium, but it requires the design pattern wonks of the world to calm down a little bit, I think.

I try to make my solutions as simple as possible. This may involve a design pattern... or it may not; it really doesn't matter, as long as it works, and it's easy to maintain. The important thing is... design patterns are not a panacea for bad code. Just because one uses design patterns doesn't mean one's code is well-written. Unfortunately, too many developers are taught (or believe on their own) exactly that: that if they use design patterns, their code quality is higher; that's not necessarily true (not even true on a regular basis, I think).

As an aside: If you want an excellent example of overdesign, look at the source code for the CruiseControl.NET tray application. It's an excellent demonstration of design patterns run amok (honestly, Thoughtworks as a whole is an excellent demonstration of that).