MV to the web – Part 3

In part 1 of his series I described how to modify your existing code so that it can be re-purposed for a GUI. In part 2 I described a general coding structure that will allow you to properly execute any program from “outside”. In this final part of this series I will discuss various connectivity methods that get you into those programs. (EDIT: There’s really a part 4 with a different title.)

For a quick summary to this point, you have a subroutine that identifies a request using a variable like OP for OPERATION, a CASE statement that checks for various operations supported by the current code item, and various internal or external subroutines which process specific requests. A sample subroutine, minus some of the constructs mentioned in part 2, might look like this:
SUBROUTINE WEB.CUST.INFO( INVAL, OUTVAL, MISC, ERR )
ARG1 = CHANGE(INVAL,CHAR(9),@AM) ; * convert tab-delimited input to atbs
OP = ARG1<1>
KEY = ARG<2> ; * expected from calling code
INVAL = "" ; * no need to pass full string back up to client
ERR = "" ; * initialize to no-error
OUTVAL = "" ; * initialize just in case
GOSUB INIT ; * open files, etc
READ CUST FROM F.CUST,KEY ELSE CUST = ""
BEGIN CASE
CASE OP="GET CSZ"
GOSUB GET.CSZ
CASE OP="GET PHONES"
GOSUB GET.PHONES
CASE 1
ERR = "Unrecognized command"
END CASE
RETURN
*
GET.CSZ:
OUTVAL = CUST<6>:", ":CUST<7>:" ":CUST<8>
RETURN
*
GET.PHONES:
PHONES = "" ; * we'll load atb1 with numbers and 2 with descriptions
CALL WEB.CUST.GET.PHONES(KEY,PHONES) ; * more complex operation
IF PHONES<1> = "" THEN PHONES = "" ; * final housekeeping
OUTVAL = PHONES
RETURN

The code that calls that from “outside” is called the Client. The Client can be a local program that invokes this directly:
P1 = "GET CSZ" : CHAR(9) : "J43L2" ; * OP and Key
P2 = ""
P3 = ""
ERR = ""
CALL WEB.CUST.INFO(P1,P2,P3,ERR)
IF ERR # "" THEN
GOSUB SHOW.ERROR
RETURN
END
CSZ = P2
GOSUB SHOW.CSZ

That exact same code could be written in any programming language, and invoked from any external Client using free or commercial connectivity tools. Rather than using a specific product as an example, I’ll provide some pseudo-code here which will convey the point but this specific code won’t compile with any specific language compiler.
object conn = new MVConnection("myserver")
object account = conn.Login("testacct")
string p1 = "GET CSZ" & "\t" & "J43L2"
string p2 = ""
string p3 = ""
string err = ""
account.CallProgram("WEB.CUST.INFO",p1,p2,p3,err)
account.Logout
conn.Disconnect
if (err != "") { HandleError(err) }
else { ShowCSZ(p2) }

As the MV developer, your task is simply to ensure that your code can be called from another BASIC subroutine. Then you need to convey the specs to whomever is coding a remote Client. Such specs would be something like this:

To get the customer Zip, call WEB.CUST.INFO with 4 parameters:
1- In: Instruction “GET CSZ” and the cust# delimited with a tab
2- Out: data or null on error
3- Out: not used for this interface
4- Out: not used for this interface
5- Out: error if any, or empty string

That’s what we call an API, the Application Program Interface. Using that spec, anyone can connect into your system from any language, using any connectivity method, and they’ll get the info they need. Notice that your code does not need to know what the external pipe or language is. It’s all about data.

To make that magic happen, someone needs to ensure that there is a “listener” setup which will accept inbound communication requests. That listener then needs to connect into the MV DBMS. Various mechanisms are used to do this and each one just has a different setup. Some connectors (pipes as I call them) go “directly” into the DBMS. Others come in via telnet or another path. The client must be able to connect to the server on the listener, the listener must forward the request to the DBMS, and the DBMS must execute the named subroutine. Your code then takes control, uses the input data to generate output, and when you RETURN, the data gets pushed all the way back out to the client.

What’s important in that process is that your code does not have INPUT or PRINT statements, as discussed in part 1 of this series. Your code cannot STOP or ABORT. You must do your best to completely avoid any output from being displayed. The intent of any such output is to inform a CUI user that something has happened, but there is no CUI here and every pipe mechanism behaves differently with “random” output thrown into the data stream.

Persistence

When the connectivity mechanism attaches to the DBMS, there can be a long delay if the port is not already logged-in. To avoid this “pain”, you will want to create a process which remains logged-in between transactions. This is called a persistent connection. The first connection through the pipe will need to login, but each subsequent connection should find the port already logged-in and ready to execute the next subroutine. Notice the Logout and Disconnect functions in the example above. That doesn’t need to be a physical logout. It can be a virtual logout. While the connection is being used, no one else can put data on the wire to get into the same DBMS port. But after a virtual logout, the port is still logged-in, and another user can then execute a subroutine. The Disconnect seen above disconnects the remote client from the server. It does not disconnect the DBMS process from the connectivity interface.

In the Relational world, this is done almost exactly the same way, a connection is established, authentication is validated, transactions are performed, and the connection is disconnected by the client.

The trick to this whole process is to keep the connection to the DBMS as short as possible. You do not want to tie up a DBMS license for more than a few seconds because other people will be waiting for that access into the environment. In other words, you want your client code to be as non-persistent as possible – make a quick request and get out of there. The trade-off here is that the shorter the connection, the less data you move on each connection, and that may require more requests. With fewer requests, your connection will need to be persistent for longer than you may want. And the longer a process takes to get a response, the longer users wait – and that’s not good for many reasons.

Session pooling

To avoid having people wait for a connection, session pooling can be employed. You setup some number of persistent connections, each with their own dedicated connection to the DBMS. Let’s say we have 3 such connections. With a LISTU or similar command you’ll see 3 DBMS users logged-in, consuming 3 licenses. A single connecting from outside uses the first available port. Let’s say that process takes 10 seconds to complete its task. 2 seconds after the first request begins another request comes in. Rather than waiting for the first process to complete, a second port is used for this new request. If another request comes in within 8 seconds, it may get the third port. After that it may get any of the other ports. In this scenario clients connect into a single server and a single listener, but the request is satisfied by the first available port. The user has no idea how many ports have been allocated for their access, all they know is how satisfied they are with the timeliness of the response.

As far as I am aware, none of the free connectivity tools provided by the MV DBMS vendors supports connection pooling. This includes MVSP (D3 and mvBase), UniObjects (Universe and Unidata), jRCS (jBase), and QMClient (QM). Let’s face it, their goal is to make you buy as many DBMS licenses as possible, even if some of them aren’t being used effectively or at all. mv.NET on the other hand does support session pooling and can thus process a lot of requests very quickly. mv.NET is actually a super-set of the other options, using platform-specific connectivity to get into the DBMS, but exposing a consistent client API for all MV platforms.

[AD] You can try to create your own session pooling mechanism but I guarantee you that it will take a long time and you’re not going to get the same high-quality feature set available in mv.NET. It’s much more cost-effective to simply purchase a low-cost license for mv.NET, rather than going through the agony of trying to create this one component yourself.

One shortcoming of mv.NET is that it requires a Windows client/middle-tier, so clients connect to Windows which then connects to a local or remote MV DBMS. Compare this to UniObjects for Java (UOJ) or QMClient where a connection on a Linux box can connect to a local DBMS.
[/AD]

In full disclosure: Another product, formerly called PDP.NET, is (I believe) the only product in the MV market that allows *nix-based session pooling for Java applications.

Each one of those options exposes a different API to client code. One might use account.Connect() and another might use account.Login(), but they do exactly the same thing, while each one adds some amount of product-specific value-add. Some options might require you to use product-specific Common in your code, but it’s best to avoid that where possible.

The bottom line on pooling is that it helps you to use the fewest number of ports, while processing requests in parallel which is faster than serial processing.

Asynchronous processing

Another method of processing requests quickly is to handle the requests asynchronously. Consider this: A user executes your subroutine which has an EXECUTE statement to Select a file with some millions of records. You know that processing this data is going to take anywhere from 30 seconds to 30 minutes. You can’t ask a browser user to wait that long for a response.

Analogy: I’m sure you’ve been walked into a restaurant, ordered food, and accepted a ticket with a number on it. You stand around waiting for your number to be called while people behind the counter are busy cooking and bagging your food. Your number is called, you grab the food, and leave.

Your code should follow the exact same model. The solution here is for your code to accept the request, and then kick off a phantom to actually do the processing. Give the phantom a unique ID to identify the request, and pass this ID back to the client. That only takes a few seconds and allows your called subroutine to return in a reasonable time frame. When the phantom is done it writes response data into some location, identified by the unique ID. Meanwhile, the client should be coded to accept just the ID, not the actual response. The client should then call another routine periodically to ask “has this ID been processed yet?”. When the answer is yes, a request can be made to get the response.

From the above, we see that your server code must be aware of roughly about how long it might take to process specific kinds of jobs, and you must provide 1) the request-to-run interface, 2) the polling interface, and 3) the request-for-response interface. All of this can actually be done in the same subroutine with just a few extra lines of code.

Callbacks

Someone reading this might say “hey, in a restaurant they call your number but in this scenario, the customer keeps asking about their order – can we get the server to call the client instead of having the client constantly calling the server?” The answer is yes, but it depends on the technology being used. In short, this involves push technology – the server must somehow have a connection to the client in order to force up a completion notice. For an MV app, the phantom or some other process wouldn’t just write the response data to a file, it would do that and then execute some communications code to send a message to the client. Most push technology is actually based on polling anyway. I hope I’ve answered that question for our purposes, but the whole concept of having a server pushing a message to a client requires an article of its own, which you can find elsewhere. If you really need this sort pro-active client notification, I can create it for you.

Web Services

The concept of web/browser interfaces and Web Services might seem foreign to most people, but the MV code behind them is exactly the same. Your BASIC code should have no idea about whether the client is a browser or “headless” Web Service.

Uh, so what exactly is a Web Service? A call to a Web Service can be made a few different ways depending on the methods used, but the result is just a subroutine call exactly like the code seen above. That code can be used to return a web page to a browser, or pure data to some other server. A Web Service returns data independent of the UI which is different from a service that returns a web page. Web Services are called with an interface similar to your BASIC subroutine: GetCustInfo(p1,p2,p3,p4,err). That call is just made from some other server which is oblivious to how the request will be processed.

To see how to wire your BASIC code to a Web Service, see the video on this page. The example uses mv.NET as the connectivity pipe, but that’s irrelevant. You could use QMClient.NET, UO.NET, MVSP, or some other pipe in the same place. What’s important is that the client doesn’t know what technology is being used on the back-end, including your MV BASIC code.

Summary

I’ve shown you how to modify your code to extract out the current character user interface (CUI). Then I showed you how to structure your subroutines so that any client could call them. Then I discussed how various clients connect into the server, then the DBMS, to execute your subroutines. With this knowledge you should now feel empowered to modify some code and expose it for access by some new client. As a BASIC developer you might get stuck there. Which language? Which pipe? Decisions like this need to be made based on how the system will be accessed. Do you want an all Java solution via Linux/Apache/Tomcat? An all Windows solution via IIS/ASP.NET? Do you already have someone in-house who can write Visual Basic code? Can they extend a bit further to VB.NET? Do you know any PHP developers? Is your company eager to start using Ruby and the Rails framework? Do you like Visual Studio? Eclipse? NetBeans? Notepad? Do you prefer to learn everything about as much as you can, or are you looking to do as much as possible while knowing as little as possible? Are you going to do this yourself? Hire? Contract? Answers to these questions will help you to determine the client and middle-tier tools used.

But for your purposes right now, you can start to re-fit your application to prepare for a GUI or Web Services – without yet deciding what external languages or tools will be used. Start that process now because there’s nothing to stop you (but time, life, maybe money to pay employees, and other silly mundane things). The investment will pay off when you start connecting in later.

In fact, depending on your audience, refitting your application now can be a huge feature enhancement of its own. Your sales and marketing can include phrases like “Web Services ready” or “comes with an API for custom user interfaces”. You don’t need to actually write external connectivity yourself. Just the fact that someone can do this can help to sway a decision in your favor. Look around – most applications out there don’t have an API, but the newer and/or more successful ones do.

If you need any help with this or you want more ideas about what to do with your application, you know where to find me. 🙂

After I published links to this series, a discussion brought out a number of excellent tips for this kind of development. I especially encourage you to see Albert Kallal’s comments in this thread.