Mouse cursor

[MMSoft 0]

Let me introduce myself:
I program as a hobby, and have been doing it for several years.
Because my English is very bad I used a Dutch Forum, but unfortunately
that is used very little these days, so hopefully there are still active Delphi users here.

 

(Delphi 11.3, FMX, Android)
I would like to add a mouse cursor in my Android(TV) App, and be able to move it and click on an Item.
How can I do that?
(Sorry for my bad English)
 

[MMSoft 0]

I do come across some things:
https://stackoverflow.com/questions/64350673/how-to-programmatically-include-the-mouse-cursor-in-android-tv

But unfortunately nothing for Delphi, and I just understand how I can make this code suitable for Delphi.

Edited September 12 by MMSoft

[Sherlock 663]

As a starting point (and not elaborating on your SO find) I would try to add a shape and react to dragging and clicking actions. One would have to reinvent the mouse cursor with this "solution". But it might turn out to be not that much work.

[MMSoft 0]

As a start, I use a TWebBrowser and try to put something on it as a corsor.

But I can't get both an Image and a Rectangle visible on top of this TWebBrowser.
So as 1st how do I get something visible on top of this TWebBrowser ? 

[Dave Nottage 557]

6 hours ago, MMSoft said:

But I can't get both an Image and a Rectangle visible on top of this TWebBrowser.

You need to use a control that supports proper "z-order" with platform-derived controls such as TWebBrowser. Kastri has such controls (TNativeImage, TNativeRectangle etc), and I've now created a demo that could be used as a starting point to achieve what you want. Please note that you would need to first install the KastriFMX package, which contains the native controls.

[MMSoft 0]

Perfect, I now have a mouse icon (NativeImage) that is displayed on top of the TWebBrowser.

I'm now going to try to apply this "NativeImage" in my own App.

 

Edit:

It doesn't matter where I put a "NativeImage" on the form, it comes up  
always to be at the top left, it is of course important that I also do this somewhere
Otherwise, insert pan.

The only difference I can find is that with "LiveBindings Designer" > Visible Element this 
is set to True and mine is set to False.
And that under Element Layers it says "NativeImage1"
Unfortunately, I don't get to stand there.
Could that be the problem, and how should I do it?

 

Edited September 26 by MMSoft

[MMSoft 0]

I've gone further, and can now move the mouse cursor in all directions across the entire TWebBrowser, and can catch the OK/Enter key.
The next problem is, how can I click something on the TWebBrowser so that it runs?

[MMSoft 0]

There is still a difference between the Demo and my Project and I can't seem to find what I'm doing wrong, so I started a new project, but this time with a 
"NativeRectangle".
If I put in the Demo: \Kastri-master\Demos\NativeControls\WebBrowserOverlay\WBOverlay.dproj
a "NativeRectangle" place then it works (only the "NativeRectangle" is shown at the top left, so that's already strange), but in my Project the Project crashes. 

I've started a new project:
-Style: Android
-A "NativeRectangle" placed on the form
-Fil > Color: Red
-Project > Options > Delphi Compiler > Search path:
 ....\Kastri-master\API
 ....\Kastri-master\Controls
 ....\Kastri-master\Core

And when I start the Project, I get this error message:
Project Test_1.apk raised exception class EJNI with message 'Java type JDWRectangleDrawable could not be found'.

What am I doing wrong?

 

On 9/26/2024 at 2:48 PM, MMSoft said:

It doesn't matter where I put a "NativeImage" on the form, it comes up  
always to be at the top left, it is of course important that I also do this somewhere
Otherwise, insert pan.

If the height or width is slightly icy, it doesn't happen

 

Edited September 28 by MMSoft

[Dave Nottage 557]

11 hours ago, MMSoft said:

Project Test_1.apk raised exception class EJNI with message 'Java type JDWRectangleDrawable could not be found'.

What am I doing wrong?

You need to add dw-kastri-base-3.0.0.jar to the Libraries node of the Android targets. If you want to target both 32-bit and 64-bit, unfortunately because of a bug in Delphi 11.3,  you'll need to make a copy of the jar file in another folder in order to be able to add it to both targets.

11 hours ago, MMSoft said:

It doesn't matter where I put a "NativeImage" on the form, it comes up  
always to be at the top left

I've been having that trouble, too. It works OK for me in some apps, but not in that demo. Still looking into why.

 

11 hours ago, MMSoft said:

If the height or width is slightly icy, it doesn't happen

"Slightly icy"?

[MMSoft 0]

11 hours ago, Dave Nottage said:

"Slightly icy"?

I switched to Delphi 12 (didn't test this again in 11.3)

If I change the height and/or width of the "NativeImage" it works fine.

[Dave Nottage 557]

13 hours ago, MMSoft said:

If I change the height and/or width of the "NativeImage" it works fine.

Thanks.. I'll look into it

[MMSoft 0]

As for the next problem (starting something on the site that is on the TWebBrowser).
Unfortunately, it is very complicated for me, but can we do something with this?
(see Rik's post 30 Sep. 10:44)
https://www.nldelphi.com/forum/vraagbaak/firemonkey/44452-/page2

 

Edit:

As I understand I need to figure out the "ElementId":  

//Run JavaScript code to simulate a click on TWebBrowser
procedure SimulateClick(WebBrowser: TWebBrowser; ElementId: string);
var
  Script: string;
begin
  Script := Format('document.getElementById('%s').click();', [ElementId]);
  WebBrowser.ExecuteScript(Script);
end;

 
Who has experience with this?

Edited September 30 by MMSoft

[MMSoft 0]

To be able to start something on a site that is on the TWebBrowser, I am looking at the Demo "WebBrowserExt"
Do you think this demo can be used for this?

Unfortunately I get this error message there:
[DCC Fatal Error] DW.JavaScript.WebView2.pas(6): F2613 Unit 'Winapi.WebView2' not found.

 

I have the search path at: Project > Options > Delphi Compiler > Search path 
Set to the file: DW.JavaScript.WebView2.pas

But unfortunately, the error message is still there.

How can I get this to work ?

Edited October 2 by MMSoft

[Dave Nottage 557]

34 minutes ago, MMSoft said:

Unfortunately I get this error message there:
[DCC Fatal Error] DW.JavaScript.WebView2.pas(6): F2613 Unit 'Winapi.WebView2' not found.

Presumably you are attempting to use this unit on a non-Windows platform, which it is not intended for. Please give a complete example of what you are trying to do.

 

[MMSoft 0]

22 hours ago, Dave Nottage said:

Presumably you are attempting to use this unit on a non-Windows platform, which it is not intended for. Please give a complete example of what you are trying to do.

 

I'm trying to create an Android project, where a Web browser shows a Site.
With the Cursor I then point to an Item, and with a click I want to run that item.
I have the 1st part working, now I'm still trying to get the 2nd part working.

[Olli73 4]

I would try another way:

 

Make a dervied class from TWebRowser and make MouseClick (or MouseDown / MouseUp) function(s) public there.

 

Now call this function(s) with your mouse cursor coordinates.

 

[Dave Nottage 557]

12 hours ago, MMSoft said:

I'm trying to create an Android project, where a Web browser shows a Site.

That's not a complete example, and obviously not Windows, so I'm not sure why you felt the need to add a unit that requires Windows.

[MMSoft 0]

This is the code I have now. 
The cursor works, but now I'm trying to be able to click something on the displayed site so that it executes.

 

unit Unit1;

interface

uses
  System.SysUtils, System.Types, System.UITypes, System.Classes, System.Variants,
  FMX.Types, FMX.Controls, FMX.Forms, FMX.Graphics, FMX.Dialogs, FMX.StdCtrls,
  FMX.ScrollBox, FMX.Memo, FMX.Controls.Presentation, IdBaseComponent,
  IdComponent, IdCustomTCPServer, IdTCPServer, IdContext,
  Androidapi.JNIBridge, Androidapi.JNI.GraphicsContentViewText,
  Androidapi.JNI.JavaTypes, Androidapi.Helpers, Androidapi.JNI.NET,
  System.IOUtils, FMX.Edit, FMX.Objects, FMX.Memo.Types,
  System.Permissions,
  Androidapi.Jni.Os,
  Androidapi.JNI.Webkit, FireDAC.UI.Intf, FireDAC.FMXUI.Wait, FireDAC.Stan.Intf,
  FireDAC.Comp.UI, FMX.WebBrowser, FMX.Ani, DW.NativeImage, DW.NativeShape;


type
  TForm1 = class(TForm)
    WebBrowser1: TWebBrowser;
    NativeImage1: TNativeImage;
    procedure FormCreate(Sender: TObject);
    procedure FormActivate(Sender: TObject);
    procedure FormKeyUp(Sender: TObject; var Key: Word; var KeyChar: Char;
      Shift: TShiftState);
    procedure FormKeyDown(Sender: TObject; var Key: Word; var KeyChar: Char;
      Shift: TShiftState);
  private
    { Private declarations }
  public
    { Public declarations }
  end;


var
  Form1: TForm1;
  Muis_Cursor_Max_Horz: Integer;
  Muis_Cursor_Max_Vert: Integer;
  Muis_Cursor_Stap_Groote: Integer;
  Muis_Cursor_Horz: Integer;
  Muis_Cursor_Vert: Integer;
  Site: String;


implementation

{$R *.fmx}


//------------------------------------------------------------------------------

procedure TForm1.FormCreate(Sender: TObject);
begin
 //Form1
 Form1.ClientHeight:= 536;  //Scherm hoogte
 Form1.ClientWidth:= 955;   //Scherm Breedte

 //Instel waardes:
 Muis_Cursor_Max_Horz := 940;
 Muis_Cursor_Max_Vert := 520;
 Muis_Cursor_Stap_Groote := 5;

 //Begin Waardes:
 Muis_Cursor_Horz := 470;
 Muis_Cursor_Vert := 260;

 //De Moude Cursor op het begin punt zetten:
 NativeImage1.Position.X := Muis_Cursor_Horz;
 NativeImage1.Position.Y := Muis_Cursor_Vert;
end;

//------------------------------------------------------------------------------

procedure TForm1.FormActivate(Sender: TObject);
begin
 Site:= 'https://en.delphipraxis.net/topic/12144-mouse-cursor/';
 WebBrowser1.Navigate ( Site );
end;

//------------------------------------------------------------------------------

procedure TForm1.FormKeyDown(Sender: TObject; var Key: Word; var KeyChar: Char;
  Shift: TShiftState);
begin

 if Key = vkDown then
  begin

   Muis_Cursor_Vert := Muis_Cursor_Vert + Muis_Cursor_Stap_Groote;
   if Muis_Cursor_Vert > Muis_Cursor_Max_Vert then Muis_Cursor_Vert := 0;
   NativeImage1.Position.Y := Muis_Cursor_Vert;
  end;


 if Key = vkUp then
  begin
   if Muis_Cursor_Vert > -1 then Muis_Cursor_Vert := Muis_Cursor_Vert - Muis_Cursor_Stap_Groote;
   if Muis_Cursor_Vert < 0 then Muis_Cursor_Vert := Muis_Cursor_Max_Vert;
   NativeImage1.Position.Y := Muis_Cursor_Vert;
  end;


 if Key = vkRight then
  begin
   Muis_Cursor_Horz := Muis_Cursor_Horz + Muis_Cursor_Stap_Groote;
   if Muis_Cursor_Horz > Muis_Cursor_Max_Horz then Muis_Cursor_Horz := 0;
   NativeImage1.Position.X := Muis_Cursor_Horz;
  end;


 if Key = vkLeft then
  begin
   if Muis_Cursor_Horz > -1 then Muis_Cursor_Horz := Muis_Cursor_Horz - Muis_Cursor_Stap_Groote;
   if Muis_Cursor_Horz < 0 then Muis_Cursor_Horz := Muis_Cursor_Max_Horz;
   NativeImage1.Position.X := Muis_Cursor_Horz;
  end;

end;

//------------------------------------------------------------------------------

procedure TForm1.FormKeyUp(Sender: TObject; var Key: Word; var KeyChar: Char;
  Shift: TShiftState);
var
 Cursor_Position_X: Integer;
 Cursor_Position_Y: Integer;

begin

 if ((vartostr(Key) = '0') and (KeyChar = '')) then
  begin
   //Op OK gedrukt
   Cursor_Position_X := Round( NativeImage1.Position.X );
   Cursor_Position_Y := Round( NativeImage1.Position.Y );
  end;

end;

//------------------------------------------------------------------------------


end.

 

Edited October 3 by MMSoft

[Dave Nottage 557]

4 hours ago, MMSoft said:

I'm trying to be able to click something on the displayed site so that it executes.

const
  cJavaScriptClickAtXY = '(function() { '#13#10 +
    'var windowX = %d, windowY = %d'#13#10 +
    'let x = windowX - (window.scrollX || window.pageXOffset)'#13#10 +
    'let y = windowY - (window.scrollY || window.pageYOffset);'#13#10 +
    'var element = document.elementFromPoint(x, y);'#13#10 +
    'if (element) {'#13#10 +  
    '  var clickEvent = document.createEvent("MouseEvents");'#13#10 +
    '  clickEvent.initMouseEvent("click", true, true, window, 0, 0, 0, x, y, false, false, false, false, 0, null);'#13#10 +
    '  element.dispatchEvent(clickEvent);'#13#10 +
    '} else'#13#10 +
    '  console.log("No element found at coordinates:", x, y);'#13#10 +
    '})()';

// Example call:
WebBrowser1.EvaluateJavaScript(Format(cJavaScriptClickAtXY, [Cursor_Position_X, Cursor_Position_Y]));

The code above should do what you want - it has had very limited testing in my environment

[MMSoft 0]

The code has been added.
I place the cursor at the top left of the site on the Item "FMX" and give a click.
But unfortunately, nothing visible happens.

[Dave Nottage 557]

32 minutes ago, MMSoft said:

I place the cursor at the top left of the site on the Item "FMX" and give a click.

I think it may be that the click is being invoked on the span, or the image (i):

<li>
  <a href="https://en.delphipraxis.net/forum/28-fmx/">
	<span>FMX <i class="fa fa-angle-right"></i></span>
  </a>
</li>

When it needs to be on the anchor (a) element, so I'll need to tweak the JavaScript a bit.

[Dave Nottage 557]

37 minutes ago, Dave Nottage said:

When it needs to be on the anchor (a) element, so I'll need to tweak the JavaScript a bit.

Updated JavaScript:

  cJavaScriptClickAtXY = '(function() {'#13#10 +
    'var windowX = %d, windowY = %d'#13#10 +
    'let x = windowX - (window.scrollX || window.pageXOffset)'#13#10 +
    'let y = windowY - (window.scrollY || window.pageYOffset);'#13#10 +
    'var element = document.elementFromPoint(x, y);'#13#10 +
    'while (element) {'#13#10 +
    '  if (element.tagName !== "A" && !element.hasAttribute("onclick"))'#13#10 +
    '    element = element.parentElement;'#13#10 +
    '  else'#13#10 +
    '    break;'#13#10 +
    '}'#13#10 +
    'if (element) {'#13#10 +  
    '  var clickEvent = document.createEvent("MouseEvents");'#13#10 +
    '  clickEvent.initMouseEvent("click", true, true, window, 0, 0, 0, x, y, false, false, false, false, 0, null);'#13#10 +
    '  element.dispatchEvent(clickEvent);'#13#10 +
    '} else'#13#10 +
    '  console.log("No clickable element found at coordinates:", x, y);'#13#10 +
    '})()';

 

[MMSoft 0]

Unfortunately, nothing is happening yet

[Dave Nottage 557]

8 minutes ago, MMSoft said:

Unfortunately, nothing is happening yet

Not sure why it's not working for you - I'm using the same URL as in your example. I could put together a minimal test case to demonstrate it.

[Dave Nottage 557]

31 minutes ago, Dave Nottage said:

Not sure why it's not working for you

It appears Android's WebView does not like part of the JavaScript (it works on macOS and iOS). I'm looking into it