Re^3: Perl to protect database field name info

[punch_card_don]

OK, then could someone please write something more concrete on this infamous abstraction?!?!

I mean, just how abstract can you get in receiving "first_name = bob", cleaning it, and writing it to users.fname?

Thanks.

[dragonchild]

In your case where you have a firstname field that is going to be stored completely within the firstname column, the abstraction doesn't make any sense. And, if you never get any more complicated than that, then the abstraction layer I'm talking about is counter-productive. Seeing as about 50% of all programmers never work on applications that need this kind of abstraction, 50% of all programmers would be ill-served in using it.

Let's say, however, that you're in the other 50% - the one that does need the abstraction. But, you don't understand why. The reason why is very simple - because your requirements will change. When requirements change, you will receive requirements that are from the user's perspective and within the user's domain of understanding. If you have designed your application correctly, you will be able to change the user-visible form WITHOUT touching the underlying database tables. This is not possible if your database table is used to generate your form.

For example, let's say that your company is so successful that you expand into Indonesia. Your application was a big part of that success, so you're leading the charge. And you run into someone without a first-name (many Indonesians only have one name instead of the two that we have in the US or Europe). Now, because your database table generates your form, you cannot abstract away the lack of a firstname within your code - your users have to do it for you.

Rule #1: Users will always make mistakes. Your application should make it hard for them to make common mistakes. It should never require the user to account for your mistakes.

Let's say that you've managed to account for the issues in Indonesia. Now, you're moving along and your application is being sold to the Australian aborigines. You recently had space issues, so you contracted all those VARCHAR(255)'s down to MAX(colname) + 10. In the case of firstname, that was 43, rounded up to 50. Whoops! Aborigines can have extremely long firstnames. Now, while this would require a database change anyways to account for them, instead of having constraints in the application as well as the database, the users see the database error.

Rule #2: Users should always be presented with useful errors that they understand. You should never present a user with an error from anywhere other than the immediate layer that they are interacting with. An error from the database is good only if the user is knowingly interacting with the database. Otherwise, you should be providing the error they see.

Rule #3: Processing should stop as soon as possible. Errors should be detected as soon as possible and handled as close to the source as possible. So, if there is a length problem with a field, that length should be encoded in the user interface. If the UI is a web browser, the length should be checked in Javascript, Perl, and the database.

Let's talk about #3 some more. Once you have separated out what's acceptable for the UI from what's acceptable for the database, you have some really neat possibilities. For example, you can have different restrictions based on different requirements, all going to the same database. So, if firstnames can only be 30 characters in Finland, but they have to be 45 characters in Denmark, you can easily handle that. Alternately, your database can now be used for an application you write and a completely separate application that your friend writes. Both of these situations are very common in large businesses. (I have maintained more of these than I can count.)

Does this help or would you like more discussion on the topic? It might help if you looked up MVC on the wikipedia.

[punch_card_don]

OK, we're getting to the nub of the matter. Sometimes, some fields are best abstracted, either so that the user interface can be optimized for users, or in case the use of the site changes.

That sounds very reasonable.

Thanks for the discussion.

[dragonchild]

Close. The nub is that software is the only thing humans make that is going to change repeatedly and radically over its expected lifecycle. The only way to properly design software is to build into the very core a capability to change safely.

Every change will always bring a risk that a bug has been introduced. (I once fixed a bug in an application and another bug appeared. After I traced down the second bug, it turned out that the first bug had been compensating for it and my correct fix for it broke the application.) So, the proper discussion to have about bug-minimization is minimizing the chance of introducing bugs when making the inevitable changes.

When we discuss this in terms of structuring subroutines, we end up discussing the promotion of internal cohesion and avoidance of external coupling. Internal cohesion is summed up as "A subroutine should do one thing, one thing only, and that one thing well." External coupling is summed up as "A subroutine should only depend on the parameters it was passed in and the values that it and it alone controls." If you think about it in terms of your subroutines, those two things make perfect sense for improving maintainability.

Now, take the external coupling concept and extend it to the components of your application. We have, at a minimum, the user interface (UI), data storage (DS), and the stuff in the middle (APP). Your proposal would couple the UI to the DS. That's about as large an external coupling as you can get. It's not something you would do with your subroutines. Why would you do it with the basic chunks of your application? That is only guaranteeing that you will run into maintainability issues, as ptum has already discovered.

[ptum]

I'll take a stab at it, although you might want to take what I say with several pinches of salt, since I haven't done very well at this lately.

Suppose I have a simple web application that gathers information about a customer -- their name, address, billing preferences, etc. Suppose I also want to store that information in a relational database. I think what dragonchild is saying is that how you gather that information from the user ought to be the way that best fits their domain -- that is, you might gather some or all of that information in a way that is user-friendly to them, whether squeezing it all on a single screen so that they don't have to keep hitting 'submit' all the time, or maybe you will slowly gather the information from them, snippets at a time, so as not to annoy your user. However you do this, it ought not to be constrained by your database structure or how you implement it on the back end -- in fact, you ought to 'abstract' the two domains (maybe by building classes that reflect the user's domain but which 'save' themselves to the database whenever they have enough information to do so). To make use of your example, it may not make sense for the user to separate his first and last name just because your database has a field for fname -- he may prefer to enter his full name in and let your application figure it out.

One way to think of it is as though two people (maybe they don't like each other very well) are writing the application -- one is a front-end developer and the other works on the database side. Your class modules will provide an API for the front-end, and another for the back-end, so that either can change independently of the other, and not break anything. So you might have a Customer class, which has methods for setting its private data elements (like name) and which knows how to save itself to one or more database tables. If the front-end guy decides to accept the customer's name in four fields (first, middle, last, suffix) your class just needs a method that can handle all four of those arguments and make sense of them. Similarly, if the back-end guy decides to munge all four fields into a single database field and moves that field to another table while he's at it, your class needs to know how to save the customer name, but the front-end guy shouldn't have to worry about it.

I fear I may not be doing a very good job at describing this concept, and I suspect it is something of which I have an imperfect understanding. Can anyone help me out with this?

Update: This thread: Design help: OO Perl and db tables dealt with some of the issues you raise, a month ago or so.