Web Services for MV – The basics

As discussed in another blog, I recently created a new product which will make it a LOT easier for MV developers to make their applications accessible via Web Services. This blog/article goes deeper into explaining what that product does and the thought process in creating it.

Before I describe this further, let’s all get on the same page:

What is a Web Service?

An end-user company running MV-based software is a vendor to other companies who may want to make electronic queries for product availability or order status. The MV-based company is also a customer to vendors who may provide catalog updates or shipment confirmations. Web Services are standard interfaces for these kinds of queries.

A Web Service is essentially a web page viewed over the internet by a program rather than a person at a browser. Web pages have links that allow viewers to jump to content. Similarly, Web Services include functions which are executed by Web Service clients.

A Web Service is exposed by a company via a URL just like any web page. A Web Service client (written by someone interested in “consuming” that service) connects to that URL and then executes one of many functions that are available at that address. These functions (also called Methods or Operations) have names like GetPrice, GetAvailableQuantity, or GetProductList.

If you’d like to know more about Web Services, I wrote a series of articles for Spectrum Magazine some years ago. They’re a little dated but still a good primer on the topic.

We return you now to our regularly scheduled program…

Those functions might seem familiar: Hey, that sounds just like a BASIC subroutine! Yup. Many developers write BASIC subroutines that perform a simple service, completely unaware of the programs that will Call them. Other developers often write client programs to Call a subroutine, without any knowledge of how that subroutine (a service) actually works.

The problem most developers face these days is how to make a subroutine available for someone on the internet to call. If only there were an easy way to publish specific subroutines so that trading partners could use them for querying data and sending updates… Well, that’s what I’ve created here.

Let’s identify the components

• A Pick BASIC subroutine accepts data as parameters, processes the data, and returns results through any of the parameters.
• A Web Service is “somehow” made available on a local middle-tier server, connecting to the subroutines to do the transaction.
• The local server is made accessible to trading partners, including other IT people in the same company who work with different platforms.
• These trading partners then create Web Service client interfaces based on the exposed services.

I do this kind of thing all the time. I’ve even published a video on our website that shows people how to create and consume Web Services in just a few minutes – all free. Most DIY developers spend weeks on this stuff, only to get stuck on the communications, XML, and protocols (usually SOAP). They also struggle with DBMS-specific tools created to make tasks like this “easy”. But those tools themselves are complex and have their own issues – just have a look around in our community forums for people looking for help with these tools and you’ll see what I mean. Why do people spend so much time on this? Probably because they haven’t watched my video. *grin*

Seriously though, I won’t trivialize what can be a complex topic. On one hand I will say that carefully crafting a Web Service can be a time consuming task. On the other hand most companies have a simple goal – to create and expose basic services for trading partners to consume. That basic task only takes a few minutes … if you know how (or if you’ve seen my video) … but we’ll fix that soon so that you don’t need to know “how” it works. As with anything, complexity introduces all kinds of concerns, but for a company just getting started, there is no need to start a project off with complexities. Just create your services, offer them to your partners, and get the project off the ground. Most partners will be happy with what you have. For the remainder, take on the bigger tasks later.

Developing a general purpose solution

OK, my goal here was to create something that would make it easy for MV BASIC developers to deploy Web Services. The first version of my new solution had a lot of hardcoded functions, just to kickstart the process so that I could see where the pain points were. It’s easy (yeah, if you know how) to create a basic service and then populate it with a lot of operations. I actually had four services and about 5 operations on each. For example:

• service Products had operations GetAvailableQuantity and GetProductDecription
• service Customers had GetOpenOrderList
• service Company had GetSalesRepForTerritory

I then linked those functions to BASIC code, and voilà, they were fully functional! But every time a new subroutine was created on the back-end, I needed code changes on the middle-tier. And not every company would want to expose an operation to retrieving sales reps. I couldn’t very well provide a utility that requires my manual effort every time someone needs customizations. Remember, my goal wasn’t to offer this initial version but to slap something together to see where improvements were required.

Dynamic code generation

It was obvious that the first task was to add a lot of flexibility. For technical reasons, it can be difficult to expose ad-hoc Web Services – the details must be defined in code so that they can be exposed at runtime. What was needed was a code generator which looked at the available BASIC subroutines, and then dynamically created the middle-tier code. The next time the Web Services are turn on, the new functionality would be available. Done it.

Strongly typed data

Next, let’s talk about data types. In MV, arguably, the only real data types we have are String, Integer, and Date/Time. Sure, we can operate on floating-point data, but it’s really just string data that happens to be treated like a number whenever we use a numeric operation. In the rest of the world there are different types for Integer, depending on the size of the data (16, 32, or 64 bit), Long numbers (overlapping with Integers), Decimal and Double types for floating-point, Boolean true/false, DateTime types combining Date and Time, and many others. People using web services expect data to be strongly “typed”, not with everything defined as a string. So in the next version I needed to make provision for data in and out of BASIC subroutines to be defined with a data type, so that generated Web Service operations will be exposed to trading partners with the right type of data. Done it.

Complex business structures

Web Services can take several values as input, but unlike BASIC subroutines where parameters are usually bi-directional, a Web Service operation is like a mathematical function where there is only one value on the left side of the “=” operator. A Web Service only returns one result per query. So how is complex data returned? As a single business structure, or object, object composed of many simple types. It seems like this is where most people get stuck – with Web Services that involve the transport of business objects, because the business objects are organized as XML documents.

Almost everyone in IT knows that a Web Service transports XML, and that XML contains nested structures. Here is an example of data represented as XML:

283748347
SomeCorp Widgets

1 N Main
Suite 343
Yourtown

2010-Mar-14 16:45


That data closely resembles a dynamic array where, perhaps the address attribute is multi-valued. But how do we get from the dynamic array to XML? Usually people hardcode it:

XML = XML : "" : REC<14> : ""

That’s tedious and error-prone. In another enhancement I would need to allow the developer to define a business object which can be returned as the result of an operation. In a Manager GUI the developer should then be able to check the fields and make sure the data types are proper, before actually generating the code. You know what’s coming… Done it.

If you’re not familiar with XML I don’t want to trouble you further with details – that’s what I’m managing inside my code so that you don’t have to. If you are familiar with XML, the above makes obvious a number of other enhancements that must be made:

• What about XML nodes using Attributes rather than inline data? As in:
  
• What about empty nodes? The data typing mechanism already allows for null/empty data, so you can have . But what about or simply leaving that node out when there is no data?
• What about N instances of any node? In the above example street1 and street2 are individually defined, but in XML it’s also valid to have zero-or-more instances of .
• And what about namespaces and other nuances?

I’d like to stamp those and many other details with a “Done it” but those features require a good amount of code – and they’re not absolutely required for a v1.0 offering. Coming back to notes above, if you’re just getting started with Web Services, then none of those items are an immediate concern. To get started, just create services, document how they work for your trading partners, and invite them to use the functions that exist. You can make changes later in your own “v2”, “v3”, etc. (I’ll blog soon about getting trading partners to use your services.)

Another requirement for enhancement will be the ability to define input fields as business object types. So for example, where an end-user company wants to allow their clients to send in purchase orders, we’ll need a function like:

CreatePurchaseOrder( OrderData )

That OrderData will need to be defined as having a Header, multiple Detail lines, and with a collection of fields for each.

What about file dictionaries to define business structures?

Many people would say “for business structures, why not use a dictionary from a file?” I actually already added an enhancement which scans a provided dict for definitions, and then the developer can go through and work out which fields to use, data types, etc. But despite how elegant that sounds, most communications do not use business object definitions which match local files so closely. You might have a hundred attributes in your Orders file, or you will have two files for OrderHeader and OrderDetail. It doesn’t make sense to try to bully that structure which is good for BASIC code into a new web services interface for use by trading partners, especially when you only need to expose a few fields.

And you will have different objects for different types of trading partners. Some partners will ask you to accept and return some custom field for their own accounting – a field which you do not have in your application, and there’s no reason to change your file layouts for a single partner. So basing a business object on file dictionaries sounds like a nice idea to start but it quickly becomes an apparent mis-match. Think about this from a code perspective – when you create a BASIC subroutine that accepts a multi-attribute dynamic array, do you create a file to define that parameter? No. Are all parameters passed between subroutines items from a file defined with dict items? No. This is no different.

We’re passing data between programs, not reading/writing data to files. For all of these reasons, we can define business objects based on a dictionary to start, but we shouldn’t try to link a business object to a dictionary for its entire lifetime. It’s not required and not even what we normally do. When data gets passed into your subroutine, you will know what it looks like because you provided the definition. How closely it matches your internal file structures is up to you and should be a matter of convenience rather than necessity.

A new product will obviously have lots of room to grow.

• For those who wish to base a business object on a file, it will need to more intelligence when extracting dict items to minimize post-extraction modifications.
• The business objects and other generated code may need to be configured differently to support different clients.
• The Manager GUI will need many enhancements to allow developers to customize the environment.
• The services may need to be versioned so that trading partners using version 1 don’t get errors when version 2 is installed.
• Enhancements will need to be made to support various types of security and performance improvements.
• I might try to dynamically generate client interfaces which can be given to trading partners.
• A pre-defined library of functions and subroutines will need to be created and included in the package for samples. These will serve as a guide to those working with their own subroutines. End-users can completely customize or discard these as desired.

So an ongoing effort will be required to ensure that this product provides continuing value to users. But even in it’s 0.x Alpha state now, I think this is a pretty cool utility.

(The following down to the Summary is copied from one of my recent CDP postings, feel free to skip if you’ve read it already.)

The goal now is to work with developers to identify what’s required for a first release. This is being developed for VARs and end-users who have real needs, right now. I’d like to ensure their requirements are covered for early and successful adoption. When their needs are satisfied, then the product will be a better tool for everyone, so my current focus is on discussion with people in this group.

That is a different group from those who want to find an application for a new tool, those who want to add “something” to their software but they’re not really sure what that should be. I will work with this group separately.

For both groups, now is a good time to talk with end-users about data exchanges with their trading partners, about the next step for their IT development. Don’t talk “technology”, talk about how current data exchange processes work, what data you need to exchange now and in the near future, how much time and money it all costs, and about whether you’d like to reduce those costs. I will be happy to partner with VARs and IT management to discuss these topics.

Summary

Before offering a production release, I will be looking for Beta sites to implement and run in live environments with active trading partners. If you’d like to discuss this, and maybe you’d like some help in coordinating these discussions with your partners, please feel free to post a brief comment here, or email me for more extended exchanges.