GIGA Display Shield: How to capture touch events?

[KurtE]

I am trying to understand how to have an Arduino sketch be able to use the touch events generated
by the GIGA Display shield.

If I look at the overlay file, that is defined for the giga in zephyr:
D:\github\zephyr\boards\shields\giga_display_shield\boards\arduino_giga_r1_m7.overlay

It has:

&i2c4 {
	pinctrl-0 = <&i2c4_scl_pb6 &i2c4_sda_ph12>;
	pinctrl-names = "default";
	clock-frequency = <I2C_BITRATE_FAST>;
	status = "okay";

	gt911: gt911@5d {
		status = "okay";
		compatible = "goodix,gt911";
		reg = <0x5d>;
		alt-addr = <0x14>;
		reset-gpios = <&gpioi 2 GPIO_ACTIVE_LOW>;
		irq-gpios = <&gpioi 1 GPIO_ACTIVE_HIGH>;
	};
};

So it is using the gt911 input ... It has status of okay so it starts up. with IRQ on GPIO I1...
And it is probably using the code in zephyr\drivers\input\input_gt911.c

I found the zephyr example: samples\subsys\input\draw_touch_events, which I was curious if it would
build and run... So far did not build.

D:/zephyrproject/zephyr/drivers/display/display_stm32_ltdc.c:422:25: error: 'CONFIG_VIDEO_BUFFER_SMH_ATTRIBUTE' undeclared (first use in this function)
  422 |                         CONFIG_VIDEO_BUFFER_SMH_ATTRIBUTE,
      |                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
D:/zephyrproject/zephyr/drivers/display/display_stm32_ltdc.c:422:25: note: each undeclared identifier is reported only once for each function it appears in
[41/48] Linking C static library zephyr\kernel\libkernel.a

Thought I would try doing the same thing in a sketch, however I don't think this will work.

static void touch_event_callback(struct input_event *evt, void *user_data)
{
	if (evt->code == INPUT_ABS_X) {
		touch_point.x = evt->value;
	}
	if (evt->code == INPUT_ABS_Y) {
		touch_point.y = evt->value;
	}
	if (evt->code == INPUT_BTN_TOUCH) {
		touch_point.pressed = evt->value;
	}
	if (evt->sync) {
		k_sem_give(&sync);
	}
}
INPUT_CALLBACK_DEFINE(touch_dev, touch_event_callback, NULL);

As I don't believe the INPUT_CALLBACK_DEFINE will work in a sketch?

Wondering best way to handle this?

Potentially, maybe define some callback within the ArduinoCore-zephyr code space.
Like maybe in loader\fixups.c ?

Turn off the interrupt? And try to poll it?

Suggestions?

[KurtE]

Quick note:
The example sketch: ~/zephyrproject/zephyr/samples/subsys/input/input_dump
does build and run.
From the monitor window:

I: input event: dev=gt911@5d             type= 3 code=  0 value=373
I: input event: dev=gt911@5d             type= 3 code=  1 value=444
I: input event: dev=gt911@5d         SYN type= 1 code=330 value=1
I: input event: dev=gt911@5d             type= 3 code=  0 value=373
I: input event: dev=gt911@5d             type= 3 code=  1 value=444
I: input event: dev=gt911@5d         SYN type= 1 code=330 value=1
I: input event: dev=gt911@5d             type= 3 code=  0 value=373
I: input event: dev=gt911@5d             type= 3 code=  1 value=444
I: input event: dev=gt911@5d         SYN type= 1 code=330 value=1
I: input event: dev=gt911@5d             type= 3 code=  0 value=373
I: input event: dev=gt911@5d             type= 3 code=  1 value=444

Note: if you hold your finger down at all, you will see a lot of messages like:

I: input event: dev=gt911@5d             type= 3 code=  0 value=368
I: input event: dev=gt911@5d             type= 3 code=  1 value=424
I: input event: dev=gt911@5d         SYN type= 1 code=330 value=1
I: input event: dev=gt911@5d             type= 3 coW: Event dropped, queue full, not blocking in syswq.
de=  0 value=368
I: input event: dev=gt911@5d             type= 3 codW: Event dropped, queue full, not blocking in syswq.
W: Event dropped, queue full, not blocking in syswq.
e=  1 value=424W: Event dropped, queue full, not blocking in syswq.
W: Event dropped, queue full, not blocking in syswq.
W: Event dropped, queue full, not blocking in syswq.

[KurtE]

Edit: meant to add it to this issue instead of display library issue.

Quick notes: It was unclear to me how I might adapt the Arduino_GigaDisplayTouch library to zephyr. There is a PT911 object
defined in the overlay for the board GIGA under the Shield (GIGA Display Shield), that I know is defined and running.

As as soon as I touch the display I get lots of these messages in the monitor window:

[00:00:25.648,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:25.658,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:25.669,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:25.679,000] <wrn> input: Event dropped, queue full, not blocking in syswq.

I know that the PT911 class is looking for me to hook up a callback function. like some of the input samples show:
INPUT_CALLBACK_DEFINE(touch_dev, touch_event_callback, NULL);
But I know that adding this define to a library that gets loaded in our LLTEXT code won't work. Was not sure if there are any
Dynamic ways to setup the callback. So I asked up on Zephyr Discussion:
zephyrproject-rtos/zephyr#89415

The response I received:
zephyrproject-rtos/zephyr#89415 (comment)
From fabiobaltieri,

Can't be done at the moment, but it'd be very easy to a static callback that goes through a dynamic list. Or just implement the >whole thing in the subsystem and make it Kconfig optional, don't see a reason not to accept such a contribution.

My read of this is, that I would need to add the callback code into some place that is built into the loader, like maybe fixups.c
and maybe that code would receive the callbacks and save the touch information and maybe set a semaphore or the like
and then the library code could detect the semaphore and retrieve the last callback information.
Probably beyond my pay grade 😆, especially if your plans are to always use LVGL.

[KurtE]

Experiments update:

Trying directly on zephyr:
I generated a slightly modified version of one of the zephyr samples, sort of a cross between the two in
samples/subsys/input

I put a copy of it up at: https://github.com/KurtE/zephyr_test_sketches/tree/master/print_touch_events
Nothing much to it:

/*
 * Copyright (c) 2024 Antmicro <www.antmicro.com>
 *
 * SPDX-License-Identifier: Apache-2.0
 */

#include <zephyr/logging/log.h>
#include <zephyr/kernel.h>
#include <zephyr/device.h>
#include <zephyr/input/input.h>
#include <zephyr/drivers/display.h>
#include <zephyr/sys/util.h>

static const struct device *const touch_dev = DEVICE_DT_GET(DT_CHOSEN(zephyr_touch));

static struct k_sem sync;

static struct {
	size_t x;
	size_t y;
	bool pressed;
} touch_point;

static void touch_event_callback(struct input_event *evt, void *user_data)
{
	if (evt->code == INPUT_ABS_X) {
		touch_point.x = evt->value;
	}
	if (evt->code == INPUT_ABS_Y) {
		touch_point.y = evt->value;
	}
	if (evt->code == INPUT_BTN_TOUCH) {
		touch_point.pressed = evt->value;
	}
	if (evt->sync) {
		k_sem_give(&sync);
	}
}
INPUT_CALLBACK_DEFINE(touch_dev, touch_event_callback, NULL);

int main(void)
{

	if (!device_is_ready(touch_dev)) {
		//LOG_ERR("Device %s not found. Aborting sample.", touch_dev->name);
		return 0;
	}

	k_sem_init(&sync, 0, 1);

	while (1) {
		//k_msleep(REFRESH_RATE);
		k_sem_take(&sync, K_FOREVER);
		printk("TOUCH %s X, Y: (%d, %d)\n", touch_point.pressed ? "PRESS" : "RELEASE",
			touch_point.x, touch_point.y);
	}
	return 0;
}

Which when I touch the display I get messages like:

TOUCH PRESS X, Y: (466, 281)
TOUCH PRESS X, Y: (466, 281)
TOUCH PRESS X, Y: (466, 281)
TOUCH PRESS X, Y: (466, 282)
TOUCH RELEASE X, Y: (466, 282)

With a reasonable response, although there is a delay for the release...

Built on Ubuntu and Windows using west build...
Note: on windows west flash fails on WIndows: I created an issue on this:
zephyrproject-rtos/zephyr#89434
Can flash on windows using arduino dfu command:
"C:\Users\kurte\AppData\Local\Arduino15\packages\arduino\tools\dfu-util\0.10.0-arduino1/dfu-util" --device 0x2341:0x0366 -D "build\zephyr\zephyr.bin" -a0 --dfuse-address=0x08040000:leave

ArduinoCore-zephyr:

I added similar code to the above into fixups.c and added entry to llext_exports.c
In fixups, I extended the code under GIGA/VIDEO #if

Note: currently just an experiment and if actually works out, maybe should be conditional on if LVGL is defined for the build.

Currently:

#if defined(CONFIG_BOARD_ARDUINO_GIGA_R1) && defined(CONFIG_VIDEO)
#include <zephyr/kernel.h>
#include <zephyr/device.h>
#include <zephyr/drivers/clock_control.h>
#include <zephyr/logging/log.h>

#include <zephyr/input/input.h>

// experiment to try to capture touch screen events
typedef struct {
	size_t x;
	size_t y;
	bool pressed;
} touch_point_t;

touch_point_t last_touch_point;

static struct k_sem touch_event_sync;

bool getVideoTouchEvent(touch_point_t *tp, k_timeout_t timeout) {
	if (k_sem_take(&touch_event_sync, timeout) != 0) return false;
	// BUGBUG: should probably put stuff in to return only
	// data from whole event, but first see if anything works
	memcpy(tp, &last_touch_point, sizeof(touch_point_t));
	return true;
}


static void touch_event_callback(struct input_event *evt, void *user_data)
{
    printk("touch_event_callback(%p %p): %p %u %u %u %d\n", evt, user_data,
            evt->dev, evt->sync, evt->type, evt->code, evt->value);
	if (evt->code == INPUT_ABS_X) {
		last_touch_point.x = evt->value;
	}
	if (evt->code == INPUT_ABS_Y) {
		last_touch_point.y = evt->value;
	}
	if (evt->code == INPUT_BTN_TOUCH) {
		last_touch_point.pressed = evt->value;
	}
	if (evt->sync) {
		k_sem_give(&touch_event_sync);
	}
}
static const struct device *const touch_dev = DEVICE_DT_GET(DT_CHOSEN(zephyr_touch));
INPUT_CALLBACK_DEFINE(touch_dev, touch_event_callback, NULL);


int camera_ext_clock_enable(void)
{
	int ret;
	uint32_t rate;

	// Hack in init semaphore for touch events
	k_sem_init(&touch_event_sync, 0, 1);


	const struct device *cam_ext_clk_dev = DEVICE_DT_GET(DT_NODELABEL(pwmclock));

	if (!device_is_ready(cam_ext_clk_dev)) {
		return -ENODEV;
	}

	ret = clock_control_on(cam_ext_clk_dev, (clock_control_subsys_t)0);
	if (ret < 0) {
		return ret;
	}

	ret = clock_control_get_rate(cam_ext_clk_dev, (clock_control_subsys_t)0, &rate);
	if (ret < 0) {
		return ret;
	}

	return 0;
}

SYS_INIT(camera_ext_clock_enable, POST_KERNEL, CONFIG_CLOCK_CONTROL_PWM_INIT_PRIORITY);

#endif

Added the callback, added semaphore, and had it init in the camera_ext_clock_enable.

In the exports I added the callback in the section:

#if defined(CONFIG_VIDEO)
FORCE_EXPORT_SYM(video_buffer_aligned_alloc);
FORCE_EXPORT_SYM(video_buffer_alloc);
FORCE_EXPORT_SYM(video_buffer_release);
FORCE_EXPORT_SYM(getVideoTouchEvent)
#endif

So far really really bad performance/latency!!!
The first touch sort of goes through then get lots of syswq full messages, then delay then
some events processed, then full then messages... And these happen long after my
finger was removed...

[00:00:01.605,000] <inf> usb_cdc_acm: Device configured
touch_event_callback(0x2400e1f4 0): 0x8062e9c 0 3 0 303
touch_event_callback(0x2400e1f4 0): 0x8062e9c 0 3 1 203
touch_event_callback(0x2400e1f4 0): 0x8062e9c 1 1 330 1
touch_event_callback(0x2400e1f4 0): 0x8062e9c 0 3 0 303
X = 303 Y= 203
[00:00:09.887,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:09.899,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:09.909,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:09.919,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:09.930,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:09.940,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:09.950,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:09.961,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:09.971,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:09.982,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:09.992,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:10.002,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:10.013,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
touch_event_callback(0x2400e1f4 0): 0x8062e9c 0 3 1 203
touch_event_callback(0x2400e1f4 0): 0x8062e9c 1 1 330 1
touch_event_callback(0x2400e1f4 0): 0x8062e9c 0 3 0 303
X = 303 Y= 203
touch_event_callback(0x2400e1f4 0): 0x8062e9c 0 3 1 203
touch_event_callback(0x2400e1f4 0): 0x8062e9c 1 1 330 1
touch_event_callback(0x2400e1f4 0): 0x8062e9c 0 3 0 303
X = 303 Y= 203
touch_event_callback(0x2400e1f4 0): 0x8062e9c 0 3 1 203
touch_event_callback(0x2400e1f4 0): 0x8062e9c 1 1 330 1
touch_event_callback(0x2400e1f4 0): 0x8062e9c 0 3 0 303
X = 303 Y= 203
[00:00:21.087,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:21.097,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:21.107,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
...

Edit: test sketch with the touch code in it:
https://github.com/KurtE/Arduino_GIGA-stuff/tree/main/sketches/zephyr/zephyr_GIGA_shield_touchpaint

Not sure if something blocking? threads? priorities? ...

[KurtE]

@iabdalkader @pillo79 @facchinm @mjs513 (and all)

Quick update:
Current potential theory: The Display output code interferes with the input thread: zephyr/subsys/input/input.c
We get a lot of Event dropped message from the same file (input_report).

One issue I had with my code was to have printk calls in the callback function mentioned in previous mentioned so I removed it.

It still was still not working right.

So I created another test sketch: up on my github.... zephyr_GIGA_shield_touch_only, where I started from the
touchpaint sketch, but removed everything associated with the display (did not start it, ...)
It appears like it is properly getting the callbacks from the PT911 input object....

So wondering if the display update code is running long periods of time and that thread is being starved?

Wondered if maybe increasing the size of the QUEUE might help, so tried:

CONFIG_INPUT_QUEUE_MAX_MSGS=10
Did not help:
Tried a few different priorities (2 4 -1) for this thread:
CONFIG_INPUT_THREAD_PRIORITY=-1
Did not help.

Wondering what else to try? Like maybe have PT911 not be interrupt? in the config for GIGA under the shield it has
CONFIG_INPUT_GT911_INTERRUPT=y
Maybe try NO, I think it then uses Timer interrupt?

Side note: I was curious about threads in sketches. So I hacked up our test sketch: GIGA_Display_first:

I added a thread to the sketch, that blinks a pin and every n
times blinks LED_BUILTIN... The interesting parts:

K_THREAD_STACK_DEFINE(blink_pins_thread_stack, STACKSIZE);
static struct k_thread blink_pins_thread_data;

void blink_pins_thread(void *dummy1, void *dummy2, void *dummy3)
{
	ARG_UNUSED(dummy1);
	ARG_UNUSED(dummy2);
	ARG_UNUSED(dummy3);

	Serial.print("thread_a: thread started \n");

	while (1)
	{
    static uint8_t pin_state = 0;
    static uint8_t loop_count = 0;
    pin_state ^=1;
    digitalWrite(BLINK_PIN, pin_state);
    loop_count++;
    if ((loop_count & (LED_BLINK_COUNT - 1)) == 0) digitalWrite(LED_BUILTIN, (loop_count  & LED_BLINK_COUNT)? HIGH : LOW);
  	k_msleep(SLEEPTIME);
  }
}
...
void setup() {
  while (!Serial && millis() < 5000)
    ;
  Serial.begin(115200);

  if (!display.begin()) {
    Serial.println("Failed to start display");
  };

  Serial.println("Display configured!!");
  pinMode(BLINK_PIN, OUTPUT);
  pinMode(LED_BUILTIN, OUTPUT);

  // Start a thread:
    k_thread_create(&blink_pins_thread_data, blink_pins_thread_stack,
      K_THREAD_STACK_SIZEOF(blink_pins_thread_stack),
      blink_pins_thread, NULL, NULL, NULL,
      PRIORITY, 0, K_FOREVER);
  k_thread_name_set(&blink_pins_thread_data, "blink pins");

	k_thread_start(&blink_pins_thread_data);



  void* FB =  display.getFrameBuffer();
  if (FB == nullptr){
...
  delay(2000);
  display.startFrameBuffering();
  fillScreen(RGB565_BLACK);
  display.endFrameBuffering();
}

void loop() {
}

And the sketch runs and when it reaches the end of main, the thread never runs again.

However if I change loop to:

void loop() {
  // put your main code here, to run repeatedly:
  delay(500);
}

The issue is in cores\arduino\main.cpp:

int main(void) {
#if (DT_NODE_HAS_PROP(DT_PATH(zephyr_user), cdc_acm) && CONFIG_USB_CDC_ACM)
  Serial.begin(115200);
#endif

  initVariant();

#ifdef CONFIG_MULTITHREADING
  start_static_threads();
#endif

  setup();

  for (;;) {
    loop();
  #if 0 //(DT_NODE_HAS_PROP(DT_PATH(zephyr_user), cdc_acm) && CONFIG_USB_CDC_ACM) || DT_NODE_HAS_PROP(DT_PATH(zephyr_user), serials)
    if (arduino::serialEventRun) arduino::serialEventRun();
  #endif
  }

  return 0;
}

There is nothing in the for loop calling loop() that yields or delays... So this logical thread never
yields back to the task manager.