D3 FlashBASIC Part 2

A while back I wrote a blog entry on how D3 FlashBASIC works. That blog/article was recently re-published in International Spectrum magazine. Yay! In the QM forum a question came up as to how D3 caches BASIC object code. I thought the little I knew about the topic would serve as a supplemental update to that other entry – frankly I’m not too pleased with this particular posting, it seems to be missing some "sizzle" but it’s something for people who are interested.

The question is this: when we already have a program running, and we recompile it, when do users see the new object code? Assume for the following that you’re testing code on port 1 and making changes to it simultaneously on port 2, maybe with port 1 at an INPUT statement while you’re making changes.

With D3, non-Flashed BASIC object: The cache only exists for the current return stack level. So let’s say you call from S1 to S2 and S2 to S3. If you return back from S3, recompile S3, then go back to it, you are still using the old object.

So try this: Return back from S3 to S2, and S2 to S1, then change S2 and S3. When S1 calls back to S2 it’s still using the old object but if you then go from S2 to S3 again, this new instance of S2 will get the new instance of S3. S1 is still holding its initial instance to S2 and will not release it until you go back to TCL.

With D3 FlashBASIC, the object is cached for all instances until all have released back to TCL, so even if you change S3 and go back to S1, it will still be the old S3 when you come back to it through S2.

But I want the latest version no matter where I call from!

Well, if your end-user is running code that has already called to a subroutine, it’s usually unlikely that you’ll want to call to the same program and get back potentially different results.

However, for those of us who work with communication tools like mv.NET or DesignBais, we often need to be able to re-compile code and then execute that code. If these programs have a mechanism to call from their main entry point to a secondary tier before they call subroutines, then you can recompile your code with impunity. If they call directly from the main entry point to subroutines, then you’ll need to bounce these main programs to TCL and restart them in order for them to make use of new object code. If you have a situation like this, talk to your vendor.

For more information about re-compiling FlashBASIC code, see another blog entry.

With D3 FlashBASIC you also get the K option to core-lock object modules – these modules are also called Shared Memory Segments. This leaves object code in memory even when there are zero references. So when you request a subroutine the runtime first checks to see if the object is already in memory before it reads the binary object code and pushes it out to memory. You can see the core-locked memory in D3Linux with shpstat. In D3NT shpstat isn’t available but you can look at shared memory segments from the D3 File Manager under Tools>Flash Shared Code.

By the way, everyone knows that the Catalog verb allows D3 to run a program from TCL without specifying RUN PROG-FILE PROG.NAME. And people familiar with non-D3 platforms generally know about local catalogs versus global catalogs. Did you know that D3 also has global catalogs? Yup, but they call them Account catalogs and Domain catalogs. I won’t get into the details here, there is enough info in the D3 Reference Manual, so I’ll refer you there for more info. For now just be aware that if you’re using BASIC file triggers or calling BASIC from the D3 Class Library that you’ll need to understand domain cataloging.

If I recompiled my code, how is it still executing the old object?

When you write any item to disk most of us are familiar with how frames get shuffled. In D3, frames are updated in-place if the item size is the same. So if you have linked frames it doesn’t need to delete, shift the group up, then append the new item to the bottom. Similarly, when you compile object code that is still in use, the new object code doesn’t substitute the old. The new object (or a pointer to it anyway) is added to the bottom of the group and the original item is flagged for deletion. This process is called Basic Protection and can be toggled off with the basic-prot command, though I wouldn’t recommend that, as swiping object code out from under live users is not a good idea. That said, for products like mv.NET and DesignBais where we are actively developing code and testing it in a browser on a separate, persistent process, I’ve found it helpful to add basic-prot-off to my user item, so that any updates are available immediately.

I don’t claim to understand why (and I asked TigerLogic about this but after two weeks and a couple reminders got no response), but when you recompile code, the original object isn’t deleted from disk when all users are done using it. Pointers to the object code are moved from the dict of your program file to a file in dm called object.queue. This is a "dy" file that isn’t saved in backups, and a clear-file is done on the file at coldstart time. So it really is "out of sight and out of mind." I only mention this for developers who do a lot of code compilation but don’t reboot their systems very often. Over time you may start to see a few frames being consumed in this non-descript file. It can’t hurt in this sort of environment to use basic-prot-clear which is just a macro that re-initializes basic protection and does a clear-file on object.queue. I believe the Save process also removes old object code, so regular backups should make this entirely a non-issue.

Since I’m discussing some esoteric areas of the system here, perhaps it’s worth mentioning that D3 also has an H option so that you can actually hide object code from other users. I admit that I’ve never used this, so I asked TL about this too but got no response, sorry. It’s my understanding that object code generated with the K option remains in shared memory for everyone to use, but the H option is completely the opposite and hides one user’s object module from everyone else. When would you use this? An example might be when you want users on the system to be running a consistent version of the object code while you’re still doing development. Sure, there are better ways to do this, but it’s good to have options. I’ll post an addendum here when I learn more about this.

Outside looking in

I won’t provide too much detail here but there is another interesting facet of D3 FlashBASIC., the ability of D3NT (not *nix) to execute BASIC modules from the OS, just like other common commands like “dir” or “ipconfig”. And I don’t mean wrapping “d3tcl” either. jBase users do almost everything from the OS command line – to them a BASIC executable is an EXE file and subroutines are DLLs that get called from the EXEs. To see how this works in D3, check the reference manual, Chapter 5 BASIC, section FlashBASIC Modules. You’ll want to learn about the G option on the compile verb and the .d3f file extension. As I understand this functionality, it is limited with regard to file access and using VME functions like Execute, OConv, etc. So for some purposes you may want to look into that idea of wrapping “d3tcl”. I do this all the time, for example, to invoke D3 functions just by clicking icons on my desktop.

Inside looking out

Finally, with D3 you can call external functions written in C/C++ from BASIC using %function calls. For example, many people like to use the popular communications library, cURL, but they don’t want to shell out using execute “!curl …” because of the performance overhead. So you import any library you want and extend D3 to do all sorts of functions that most people think are impossible. Sourceforge.net and other freeware sites are full of useful C/C++ utilities for graphing, communications, email, reporting, complex math, OS file manipulation, network management, and anything else you want.

There are two kinds of %functions, built-in ones and custom functions that you can import. I have another blog entry on the topic of %functions, and that one is about the built-in functions. Those are documented in the D3 ref and include things like %memcpy, %ioctl, and other “wrapper” functions that map directly to their OS-level counterparts. Here, I’m just talking about functionality that you can add to D3, making these external functions as much a part of the platform as IConv, DCount, System, or any other BASIC function.

Using custom %functions requires linking the external C modules with the D3 kernel – it’s a non-trivial operation, and not well advertised, but fully supported by TigerLogic. I mentioned this feature in the U2 forum recently and someone said they are using the Universe GCI (General Calling Interface) to do exactly this – and they’ve imported over 300 useful utilities to extend Universe. TigerLogic already built the cURL library into D3 v7.5. I don’t know if they documented it but you should be able to make use of it immediately if you have a current release. Maybe that’s another blog… It’s my hope that TigerLogic will add more of these helpful functions. Better yet, I hope that someday they re-visit the documentation, write some tech notes or tutorials, and maybe even write some wrapper code to make it easy for developers to extend D3 with their own C modules (and C#, and Java…). I think that would open a whole new world for many of us.

As always, please feel free to comment here in the blog, or for an extended discussion you’re welcome to start a thread in our forum.