donderdag 12 juli 2012

Indy 10 Websockets, RemObjects and realtime duplex events!

In a previous blog, I demonstrated a proof of concept of Websockets for RemObjects. The downside of the used implementation is that it is "websocket only", whereas websockets actually starts as a http connection. So it should be possible to have a webserver (in Delphi ofcourse :) ) which supports both normal http and websockets.
I also played with websockets for duplex communication (bi-directional), so a server can send something to a client at any time the server wants, without some kind polling needed by the client. I wanted to use this for RemObjects events too, which use normally polling (the normal tcp RO channels and the javascript http channel; RO also has "super" channels which are duplex too but also much "heavier" and complicated). Websockets are designed for "real-time" duplex communication, so it would be very nice to use it with RemObjects!

Result

Well, it took some time, but I made it! Because Indy has already a nice http server + client implementation (that is used by RO too) and because I could not find a good http implementation for synapse, I decided to make it with Indy 10. Unfortunately there was no code yet of websockets for Indy 10, so I took the official specification (protocol version 13) and started "hacking" :).
In short the result is:
  • an Indy IOHandler (TIdIOHandlerWebsocket), which reads and writes websocket frames (used by both server and client)
  • a Indy http + ws client (TIdHTTPWebsocketClient), which has a "TryUpgradeToWebsocket" function that can be used to try to upgrade to a websocket connection when the webserver supports it.
  • a RemObjects http + websocket channel (TROIndyHTTPWebsocketChannel), which uses the above http + ws channel.
  • a RemObjects http + websocket server (TROIndyHTTPWebsocketServer) (unfortunately "TROIndyHTTPServer" creates directly a "TROIdHTTPServer" without a virtual function, so I could not make a special Indy http + ws server), which support both http and websocket clients (!).
  • support for duplex RO events, both in Delphi and in Javascript(!)

Implementation

The IOHandler is not very complicated: it reads the websocket header and concatenates several frames (if it is fragmented) to a single message. And it writes the ws header and the raw data after it. One note: clients must send all data with a mask. This is needed because otherwise the http cache can be poisoned by a hacker via a special composed websocket message (all ws data is send via a http port), or at least something like this is stated in the specification :).

The Indy http client contains the logic for a special upgrade request, and the handling and checks for the response of the server. The upgrade request is a set of extra http headers, together with a SHA1 key. In theory the websocket connection can support some extensions (like gzip compression) and subprotocols (like SIP for VOIP) but this is not implemented yet.

The RemObjects channel tries to upgrade (once) to a websocket connection, but can also operate as an "old" http channel. The special addition in this channel is a lightweight background read thread. This in contrast to the RO super tcp channel, which has a dedicated read thread per connection. Because my customer has Windows RO services with connection/channel pools (multi-threaded, to support multiple concurrent client calls), I did not want to double the already high count of threads! However some kind of continuous reading of the client connections is needed because of the duplex character of websockets, like sending ping/pong frames by the server, sending/pushing data from server to a client at any time, etc. 
After some research I made a single background thread, which uses the tcp "select" winsock api call, which can wait for incoming data for up to 64 connection! This thread only handles unexpected data from the server (ping + close frames and RO events), normal RO dispatching of calls is done by the caller thread itself. This is enough for my customer because of the (probably) low traffic of RO events, but additional or dedicated threads can be made of course in case of higher loads in the future. 
A nice trick I have to mention is: I use dummy connect to 0.0.0.0 to stop the "select" wait in case a new connection is added or when the wait thread has to terminate. I don't know if there is a better way, but at least it works :). By the way: I could not use "WSAAsyncSelect" because Indy needs blocking sockets.

The Indy + RemObjects http server contains both the logic for (possible) upgrade requests and some extra RO specific handling. It has a seperate "execute" function for websocket contexts, otherwise the normal Indy http handling would give errors :).  

To support duplex RemObjects events (without polling!), I had to add a special message number at the end of each message. Positive numbers mean normal RO RPC calls, negative numbers mean RO events send by the server to the client. I did not "invented" this myself, but something similar uses RO in their super tcp channels too :). I also made a small javascript extension for RemObjects, to be able to use websockets channels and duplex events. Note: only binary messages can be used for RO events!

Quality

I already spent some fair amount of time to make it stable (like killing client + server while sending and receiving etc), so it should be stable enough to use. We are busy with a pilot project to see how this stuff in combination with Smart Mobile Studio will work (backend RO services, frontend html5 clients) and if it can be used in real-life production situations. Anyway, my customer allowed me to make it open source (hopefully it will be integrated in Indy 10 itself), so please test it, fix some bugs, add new features, etc!  


Demo

All right, enough text and theory, time for a demo :). For the demo I made a simple RO function, which will send a lot of "progress" events (during the call!) back to the client each millisecond (to stress test the communication :) ). I used the progressbar example of Lennart to demonstrate the high update speed of websockets in javascript using SmartMS. This works in both the html and the Delphi client:


For the Delphi client, you need to connect to the server by clicking for example the "Sum" button. If you have done this for both Delphi clients, you can press the "Progress" button in the web page (served by the same server!). Then you will see all 3 progress bars running at the same speed :).

Note: even Android 4.0 does not support websockets! You need Opera Mobile (and enable websockets) although my demo still did not work with it (haven't time yet to debug it). Or you need Chrome for Android (Android 4.0 and higher) and then my demo works fine. Probably I have to use long polling over http if I want to use it with older Android mobile phones... :(

Note 2: an RO event can also be send at any other time, for example when an other user logs in etc. To test this I added the button "Send event" on the server form (note: only for this demo :), normally you will make a Windows service for the server instead of a GUI app). Press the button and in both clients a popup will be shown:


(note: I only added a handler for this event in Delphi but it also works in javascript)


Download source and binaries

The source code of the Indy 10 websocket implementation, as well for the RemObjects extensions for Delphi and Javascript can be downloaded here (thanks to my customer: www.rbk.nl!). The source code of the demo can be downloaded here.


Conclusion

With this websocket for Indy 10 implementation, you have to use only 1 port for "everything": normal http file handling (via TROHTTPFileDispatcher), RemObjects communication via http (for webclients with no websocket support like Android :( ), tcp-like low overhead communication via websockets and realtime RO events! And you can of course also use it for normal communication between Delphi RO servers and Delphi RO clients, it is not only for web clients.

donderdag 21 juni 2012

RemObjects integration in Smart Mobile Studio (part 2)

In part 1, I explained how to create a very basic RemObjects service. In this post (part 2), I will show how easy it is to create a fancy looking mobile and web front-end for this service.
In the mean time, Smart Mobile Studio is updated(!), so get this new version to use the nice RO integration :).

First, we create a new project in Smart Mobile Studio (SMS):
Just choose a nice project name and press OK.

Now we can use the brand new RemObjects wizard to import our previous created RO service file:
(Smart Mobile Studio -> Tools -> Import Remobject SDK library)

In the wizard that pops up, we select our .rodl file (created by the "RemObjects Service Builder" tool):

We press the "import" button, and then we have a generated unit for our service!

As you see, it has a "{$R 'file:TestLibrary_intf.js'}" line in it. This javascript file contains the real client implementation, generated by the integrated RemObjects template engine in the wizard. This file is the same as you can generate with the "RemObjects Service Builder" tool. The magic is in some lines below. First we have our callback definitions (remember: everything is async! so no functions with a result, but you get the result async when the http call returns the value via the callback):
  TBasicService_Sum_Result = procedure(aResult: Integer);
  TBasicService_GetServerTime_Result = procedure(aResult: DateTime);
 Then the service class definition:

  { Service definition }
  TBasicService = class external 'BasicService'
    constructor Create(aChannel: TROHTTPClientChannel;
                                 aMessage: TROMessage;
                                 aServiceName: string = "BasicService");
    procedure Sum(
               A: Integer;
               B: Integer;
               aCallback: TBasicService_Sum_Result;
               aOnError: TErrorResult);
    procedure GetServerTime(aCallback: TBasicService_GetServerTime_Result;
                                            aOnError: TErrorResult);
  end;

You see the "external" keyword? This means our "BasicService" class maps directly to the javascript "BasicService" object (defined in the included $R TestLibrary_intf.js). We have a constructor with 3 params, which maps to the JS constructor. And we have 2 procedures (not functions!): Sum and GetServerTime. These are the same as defined in the .rodl file and implemented on the server side. Both procedures have a least a result callback and an error callback (in case the connection gets lost etc).

Before we can call these functions, we will make a simple form with some components on it. We place some labels and textboxes and a button on the form:

We can preview how it looks like using the preview button:

Looks nice isn't it?


Code

Now some real code and fun :). We go to the source ("Source" tab or press F12), create a button click event, and attach it to the button OnClick:
type
  TForm1=class(TW3form)
  private
    { Private methods }
    {$I 'Form1:intf'}
  protected
    { Protected methods }
    Procedure InitializeObject;override;
    Procedure FinalizeObject;override;
    Procedure StyleTagObject;reintroduce;virtual;
    Procedure Resize;override; 
    procedure btnSumClick(Sender: TObject);
  end;
Procedure TForm1.InitializeObject;
Begin
  inherited;
  {$I 'Form1:impl'}
  btnSum.OnClick := btnSumClick;
End;

procedure TForm1.btnSumClick(Sender: TObject);
begin
end;
To be able to call our RO service, we have to add the following uses clause:
uses
  RemObjectsSDK,
  TestLibrary_intf;
Now we can create a http channel, a JSON message, attach this to our client side TBasicService object and then call the Sum function:
procedure TForm1.btnSumClick(Sender: TObject);
begin
  var chn := TROHTTPClientChannel.Create("http://localhost:8099/JSON");
  var msg := TROJSONMessage.Create; 
  var srv := TBasicService.Create(chn, msg);
  srv.Sum(
    edtValue1.Text.ToInteger, edtValue2.Text.ToInteger,
    procedure(aResult: Integer)
    begin
      edtResult.Text := aResult.ToString;
    end,
    ROErrorCallback);
end;
We supply the 2 values of the 2 editboxes, and when calculation is done (on the server), we put it in the 3rd editbox (using an inline anonymous method).

Run

Now compile this and run it! ("Execute" or F9 in SMS). Oh, and don't forget to run the server we made in part 1!
We click on the "Sum" button and.... hmmm, not exactly what I wanted...

The reason is, I forgot something in the server project (in part 1): I must set the property "WrapResult" of the TROJSONMessage to True (see "Tips & Tricks" in this RemObjects wiki item). Now we recompile our server, run it, and click on the "Sum" button again:

That's better :).
And this is how it looks on an android smartphone (Samsung Mini, Android 2.3):

Now, is this very cool or not!? :)
It is of course a very simple example, I hope to blog more about this in the future.

Demo

If you want to run this demo yourself:

Note: string/integer helper classes

Note: I added 2 helper classes for "string" and "integer" to be able to use ToInteger and ToString on these basic types (like "edtValue1.Text.ToInteger"):
type
  TString_helper = helper for string
    function  ToInteger: Integer;
  end;
  TInteger_helper = helper for integer
    function  ToString: string;
  end;
function TString_helper.ToInteger: Integer;
begin
  Result := StrToInt(Self);
end;
function TInteger_helper.ToString: string;
begin
  Result := IntToStr(Self);
end;




vrijdag 15 juni 2012

RemObjects integration in Smart Mobile Studio (part 1)

In the next update (planned the 18th) of Smart Mobile Studio (SMS), full support for RemObjects SDK services will be added.
With RemObjects (RO) it is very easy to make multi tier applications. For example, you can create one or more RO services (as middle tier) to centralize your main business logic. And with the next SMS update, it is very easy to create nice looking mobile and web front ends (presentation tier) for these remote services!

In this blog, I will try to explain the basis steps to create a RemObjects service. And in the next blog I will explain how to use this RO service in (then updated) Smart Mobile Studio.

Under the hood

Under the hood, the SMS RO integration is actually a typed wrapper around "RemObjects for Javascript" (SMS generates javascript (JS) code after all :) ), and not a complete new implementation. This way, the high RO quality is maintained and all RO features are supported. Also future updates of "RO for JS" can be applied. However, SMS cannot apply smart linking and obfuscation to this external JS code.

Basic RemObjects service

First, we will create a very basic RemObjects service in Delphi, using the nice and easy RO wizard:
(Delphi -> File -> New -> Other -> RemObjects SDK)

We choose for "VCL Standalone", because this is the easiest to start with (but if you plan to create a Windows service (for Intranet deployment), I can recommend to use the "Combo Service/Standalone" option). The first page of the RO wizard will be shown:

We choose a project name and folder, and we click on the "Advanced Project Options" for some more settings:

Here you can name the first "RO service" within the "RO library" (normally you will create a number of services, e.g. an OrderService for all order functions, and one library per Windows service). Most important is the HTTP "Server class" (in this case based on Indy, but you can choose other implementations), because this is the only option supported by "RO for JS" to communicate with the server (a browser cannot use plain TCP or Windows messages, however I already made a websocket server class). Next, we can choose between JSON and Binary message format. JSON is supported by all browsers, but binary is not fully supported by Internet Explorer (IE), so we choose for JSON to be on the safe side.

When we finish the wizard, RO will generate a server project for us:

The first thing we do is setting the "SendCrossOriginHeader" to true! This is needed, because our Smart app will run standalone and seperate from the http server. However modern web clients do not allow to call other foreign web servers (for safety reasons). There are other ways to solve this, but for internal use this single setting is the easiest :).

The next step is less obvious for first time RO users: we start the seperate "RemObjects Service Builder" tool  (via Delphi -> RemObjects SDK -> Edit service library). The RO wizard has created our BasicService with default 2 functions: "Sum" and "GetServerTime":

We can add more functions, structures, arrays, etc but we will leave this for now. We just close this tool and compile our server project in Delphi. Every time we create a new RO service within the RO Service Builder tool, the RO integration in Delphi will ask once at the next compile to create a implementation unit for the defined service:

We now have a "BasicService_Impl.pas" datamodule, which contains our 2 functions:
    function Sum(const A: Integer; const B: Integer): Integer;
    function GetServerTime: DateTime;
We will implement the simple :) functionality as follows:
    function TBasicService.Sum(const A: Integer; const B: Integer): Integer;
    begin
      Result := A + B;
    end; 
    function TBasicService.GetServerTime: DateTime;
    begin
      Result := Now;
    end;
This is all we have to do for our first RemObjects service, easy isn't it? :)
In the next blog post we will create a Smart client to use this remote service.

vrijdag 1 juni 2012

Remobjects via Websockets and in Node.js

After playing with Smart Mobile Studio (SMS) and RemObjects for Javascript, I thought it would be cool to have a HTML5 websocket communication channel (instead of only http). Note: websockets are some kind of "TCP over HTTP", so very efficient and fast for e.g. binary data.
At my current customer we use TROIndyTCPChannel and TROBinMessage internally, because this is the fastest option.

Websockets in Delphi

Because RO does not have websocket support (yet), I tried to make a Proof of Concept (PoC) to see if it is possible. I searched and tried some Delphi implementations, but one was old (Indy9, old websocket spec), otherone had no sourcecode yet (Indy10, no  response on my mail so far), etc. I found one which is based on "synapse", which is also included in RO, namely "bauglir-websocket".
I looked at how RO made their TRO***Server and TRO***Channel and made a simple implementation for websockets: uROWebsocketChannel.pas and uROWebsocketServer.pas. This was almost too easy :).
I tested this with a Delphi RO demo server and client (using the RO wizard) and: tada, it works! :).

RO websocket channel

Next I had to make a websocket channel in "RemObjects for Javascript", and after some copy and paste I added this to the end of the RemObjectsSDK.js file. I made a simple html test file and this one works too!

Node.js

Okay, what can we do next? Make a websocket server in html? Hmm, websockets (in html) can only make client connections and cannot act as a server...
Maybe I can use it in Node.js? Note: Node.js is a server side javascript implementation, based on the Google V8 Javascript engine.
I found a websocket chat demo in Node.js, so it should be possible... Because it is not easy to debug in Node.js, I tried to make it work in Google Chrome first, by using the "websocket.onmessage" event and by sending a fetched JSON string (using the network debug tab in Chrome) when the html page connects to my  Delphi server. After some hacking I could read the RO message, do the "sum" and send the data back to the client. Sweet!
After some little more hacking I also had it working in Node.js! So both my html page and my Delphi client can connect to the Node.js server. Ain't that cool! :)

Download and test

If you want to test it yourself, you can download the zip file, and do the following:
- start "ROWebSocketsServer.exe"
- start "ROWebSocketsClient.exe" and open "\client\client.html"
- push the sum buttons, both should work

- stop "ROWebSocketsServer.exe"
- start the node.js server via "\node\start RO.node.bat"
- push the sum buttons in the clients (maybe restart them?), both should work too :)


woensdag 9 mei 2012

I made a Proof of Concept debugger for "DWS to Javascript" (so it's also possible with Smart Mobile Studio :) ), using the WebKit Remote Debugging Protocol of Google Chrome. The protocol is JSON based over WebSocket. The specification of these communication JSON objects is nicely documented here, for example the supported debugger commands.

My PoC is a modified version of the MandelbrotJS demo of DWS. I used the websocket implementation of "Delphi on Rails" and for JSON handling I used "superobject". This made the PoC very easy to make!

Some screenshots of the PoC in action:
First, when it's started:

Next, after the "Compile and Run" button is pressed:
(note: chrome.exe is started with "--app" and embedded afterwards in the tab, so no TChromium but the latest version of Google Chrome! I also added a "setInterval" so we can add breakpoints to executing JS code)

Now we can click the "Remote Connect" button to connect to the remote Chrome.exe instance via the remote debugging protocol. When this button is pressed, the debugger is activated and gets (async) a list of loaded script files (in our case only 1 html file). We click on the "Load debug script" button to load the latest script in our debugger:

The debugger is running (we can only "pause" now), and we can add a breakpoint by double clicking on a line, for example line 14:

Because of the "setInterval" this code is executed every second, so we get soon a real breakpoint! Because of my "/*@15*/" code added to the generate JS, the debugger can determine the original Delphi/Pascal source line (with extra offset of "<%pas2js"), which is highlighted and selected in SynEdit (see above).
Also the current stack and the "scope" (local variables) and the properties/values of the local variables are loaded and shown at the left side (note: double clicking on the stack of scope also works: source code position is updated, local variables etc). And hoovering above a variable shows the current value (and the JSON object is also shown for debug purpose).

Pressing the "Step over" button continues the execution to the next line. Because DWS has no line numbering info for each symbol and line, I could not add the magic "/*@" to all the code, so the debugger now shows the current line in the Javascript code:
(note: I stepped a bit further than line 16)
The values of variables are updated, so it gets refreshed every time. The debugger really works!

And to show the stack and scope can be inspected too (double click on the item):
(note: line 61 is selected, is not shown here. And yes it should be line 60, nice off by one error...)

So go ahead, grab the code and play with it!

Conclusion: it it possible the debug Delphi/Pascal code in "DWS to Javascript", but DWS should be modified a bit more, so debug info is generated for each line (in a different debug symbol file?).

donderdag 12 januari 2012

"Smart" business (OP4JS, Delphi/Pascal for Javascript)


After reading various blog posts about OP4JS (now named “Smart Mobile Studio”), I was very curious if I could use it for my current customer. We have a couple of RemObjects services, which contain our business logic (multi tier, SOA (service oriented architecture)), and we would like to create a HTML client for it (for iPad and Android).
I did a simple test with the latest alpha release, and it turned out to be relative easy! (it is an alpha, so no code completion, internal debugging, delphi shortcuts etc yet, so I had to do everything by hand :-) ).

First I made a simple RemObjects server using the Delphi Wizard:
Delphi -> New -> Other -> RemObjects SDK -> VCL Standalone:
I selected HTTP server + JSON message format (needed for Javascript communication).

Default it already contains 2 functions: “Sum(A,B: Integer): Integer” and “GetServerTime: TDatetime” (see below for screenshot of the RemObjects Service Builder). When compiling the server for first time, it will generate a datamodule for the server side implementation for these 2 functions (I won’t post the implemention: a “sum” function is too simple :-) ). So we have a RemObjects HTTP server now, which listens to the default port 8099 and uses the JSON data format (normally we use TCP + Binary messages for fast and low overhead messaging).

Next, we need to generate some code for the client side. In the RemObjects Service Builder we can generate various client implementations: C++, PHP, Delphi (also automatic done in Delphi IDE) and also Javascript. In the latest RemObjects release they added a completely new Javascript implementation but I could not get it working yet, so I have taken the "old" JSON RPC implementation:

You need to download some extra Javascript files, because it is dependent on the Yahoo YUI toolkit (I have tested the RemObjects Javascript implementation before, so I already had these files).

After that, I started “Smart Mobile Studio” and played with the delivered demos. I created a new “visual project”, and added 2 edits, a button, and another edit to the form (all typing by hand, no designer yet):

Procedure TForm1.InitializeObject;
Begin
 inherited;

 FEdit1:=TW3EditBox.Create(self);
 FEdit1.setBounds(10,10,100,32);
 FEdit1.text:='2';

 FEdit2:=TW3EditBox.Create(self);
 FEdit2.setBounds(10,50,100,32);
 FEdit2.text:='3';

 FBackBtn:=TW3Button.Create(self);
 FBackBtn.setBounds(10,95,100,32);
 FBackBtn.Caption:='Sum';
 FBackBtn.Visible:=true;
 FBackBtn.OnClick := ButtonClicked;

 FEdit3:=TW3EditBox.Create(self);
 FEdit3.setBounds(10,140,100,32);
 FEdit3.text:='';
End;
I created a “button click” procedure, which calls my RemObjects server. Because I have to call some pure Javascript code, I used the “asm” keyword for my “low level” code:
Procedure TForm1.ButtonClicked(Sender:TObject);
var
 s1, s2, s3: string;
Begin
 s1 := Self.FEdit1.text;
 s2 := Self.FEdit2.text;
 asm
   var service = new NewService("http://localhost:8099/JSON");
   service.Sum(@s1,
               @s2,
               function(sumresult)
               {
                  TW3EditBox.setText(Self.FEdit3,sumresult);
               });
 end;
End;
Note: I had to put the values of the 2 edits in seperate variables, because I could not directly use “@FEdit1.text” in the “asm” block? For the result callback (async!) I had to do something different, so I used “TW3EditBox.setText(FEdit3,result)” because this code is generated by SmartMobileStudio too when you type “FEdit3.text := ‘value’.

This code almost works: I have to include the client Javascript files somewhere. You can include Javascript in the project, but also in the main index.html (like you normally do with html).   
 script type="text/javascript" src="lib/yahoo.js">
 script type="text/javascript" src="lib/event.js">
 script type="text/javascript" src="lib/json.js">
 script type="text/javascript" src="lib/connection.js">
However, SmartMobileStudio uses a temp dir when running from the IDE (so it misses my extra .js files), so I just compiled the project, and ran the generated index.html directly from the project dir in Google Chrome.

So it’s done now? No, I got an error in Chrome, because I call an other HTTP server from my (local) index.html:
I had to enable cross site calls in myRemObjects HTTP server:
 ROServer.SendCrossOriginHeader := True;
But then Chrome complains about “Access-Control-Allow-Headers”, so I manually changed this by overriding the RemObjects implementation of the internal Indy “Other” function:

ROHttpServerCommandOther           := ROServer.IndyServer.OnCommandOther; ROServer.IndyServer.OnCommandOther := InternalServerCommandOther;

procedure TServerForm.InternalServerCommandOther(AContext: TIdContext;
 ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
begin
 //original RO handling
 ROHttpServerCommandOther(AContext, ARequestInfo, AResponseInfo);

 //need to set extra allow-header
 if ROServer.SendCrossOriginHeader then
  AResponseInfo.CustomHeaders.Values['Access-Control-Allow-Headers'] := 'Origin,   X-Requested-With, Content-Type';
end;

And then it works! After the initial problems, it should be easy now to extend the functionality further. For example, I would like to create a "Javascript to Delphi wrapper class" generator based on the .rodl interface file, so I have type safety again (no asm sections in code, only in wrapper class).
I hope to blog about this more in the future...

Note: demo code + binaries can be found here. First start "\ROSmartTest\NewProject.exe" and then "\Project1\Project1\index.html"