add timestamp to global state table for lease table cleanup (#861)

* add LastAccessTime column to GlobalState

* update docs with clean up scripts

* update createtable

* add comments

* refine comments

* fix script error

* add print

Author: MaddyDev

docs/BindingsOverview.md

@@ -22,6 +22,7 @@
       - [Startup retries](#startup-retries)
       - [Broken connection retries](#broken-connection-retries)
       - [Function exception retries](#function-exception-retries)
+      - [Lease Tables clean up](#lease-tables-clean-up)
 
 ## Input Binding
 
@@ -147,7 +148,7 @@ If an exception occurs in the user function when processing changes then the bat
 
 If the function execution fails 5 times in a row for a given row then that row is completely ignored for all future changes. Because the rows in a batch are not deterministic, rows in a failed batch may end up in different batches in subsequent invocations. This means that not all rows in the failed batch will necessarily be ignored. If other rows in the batch were the ones causing the exception, the "good" rows may end up in a different batch that doesn't fail in future invocations.
 
-You can run this query to see what rows have failed 5 times and are currently ignored, see [Leases table](#az_funcleases_) documentation for how to get the correct Leases table to query for your function.
+You can run this query to see what rows have failed 5 times and are currently ignored, see [Leases table](./TriggerBinding.md#az_funcleases_) documentation for how to get the correct Leases table to query for your function.
 
 ```sql
 SELECT * FROM [az_func].[Leases_<FunctionId>_<TableId>] WHERE _az_func_AttemptCount = 5
@@ -168,3 +169,101 @@ e.g.
 ```sql
 UPDATE [Products].[az_func].[Leases_<FunctionId>_<TableId>] SET _az_func_AttemptCount = 0 WHERE ProductId = 123
 ```
+
+#### Lease Tables clean up
+
+Before clean up, please see [Leases table](./TriggerBinding.md#az_funcleases_) documentation for understanding how they are created and used.
+
+Why clean up?
+1. You renamed your function/class/method name, which causes a new lease table to be created and the old one to be obsolete.
+2. You created a trigger function that you no longer need and wish to clean up its associated data.
+3. You want to reset your environment.
+The Azure SQL Trigger does not currently handle automatically cleaning up any leftover objects, and so we have provided the below scripts to help guide you through doing that.
+
+- Delete all the lease tables that haven't been accessed in `@CleanupAgeDays` days:
+
+```sql
+-- Deletes all the lease tables that haven't been accessed in @CleanupAgeDays days (set below)
+-- and removes them from the GlobalState table.
+USE [<Insert DATABASE name here>]
+GO
+DECLARE @TableName NVARCHAR(MAX);
+DECLARE @UserFunctionId char(16);
+DECLARE @UserTableId int;
+DECLARE @CleanupAgeDays int = <Insert desired cleanup age in days here>;
+DECLARE LeaseTable_Cursor CURSOR FOR
+
+SELECT 'az_func.Leases_'+UserFunctionId+'_'+convert(varchar(100),UserTableID) as TABLE_NAME, UserFunctionID, UserTableID
+FROM az_func.GlobalState
+WHERE DATEDIFF(day, LastAccessTime, GETDATE()) > @CleanupAgeDays
+
+OPEN LeaseTable_Cursor;
+
+FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId;
+
+WHILE @@FETCH_STATUS = 0
+BEGIN
+    PRINT N'Dropping table ' + @TableName;
+    EXEC ('DROP TABLE IF EXISTS ' + @TableName);
+    PRINT 'Removing row from GlobalState for UserFunctionID = ' + RTRIM(CAST(@UserFunctionId AS NVARCHAR(30))) + ' and UserTableID = ' + RTRIM(CAST(@UserTableId AS NVARCHAR(30)));
+    DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId
+    FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId;
+END;
+
+CLOSE LeaseTable_Cursor;
+
+DEALLOCATE LeaseTable_Cursor;
+```
+
+- Clean up a specific lease table:
+
+To find the name of the lease table associated with your function, look in the log output for a line such as this which is emitted when the trigger is started.
+
+`SQL trigger Leases table: [az_func].[Leases_84d975fca0f7441a_901578250]`
+
+This log message is at the `Information` level, so make sure your log level is set correctly.
+
+```sql
+-- Deletes the specified lease table and removes it from GlobalState table.
+USE [<Insert DATABASE name here>]
+GO
+DECLARE @TableName NVARCHAR(MAX) = <Insert lease table name here>; -- e.g. '[az_func].[Leases_84d975fca0f7441a_901578250]
+DECLARE @UserFunctionId char(16) = <Insert function ID here>; -- e.g. '84d975fca0f7441a' the first section of the lease table name [Leases_84d975fca0f7441a_901578250].
+DECLARE @UserTableId int = <Insert table ID here>; -- e.g. '901578250' the second section of the lease table name [Leases_84d975fca0f7441a_901578250].
+PRINT N'Dropping table ' + @TableName;
+EXEC ('DROP TABLE IF EXISTS ' + @TableName);
+PRINT 'Removing row from GlobalState for UserFunctionID = ' + RTRIM(CAST(@UserFunctionId AS NVARCHAR(30))) + ' and UserTableID = ' + RTRIM(CAST(@UserTableId AS NVARCHAR(30)));
+DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId
+```
+
+- Clear all trigger related data for a reset:
+
+```sql
+-- Deletes all the lease tables and clears them from the GlobalState table.
+USE [<Insert DATABASE name here>]
+GO
+DECLARE @TableName NVARCHAR(MAX);
+DECLARE @UserFunctionId char(16);
+DECLARE @UserTableId int;
+DECLARE LeaseTable_Cursor CURSOR FOR
+
+SELECT 'az_func.Leases_'+UserFunctionId+'_'+convert(varchar(100),UserTableID) as TABLE_NAME, UserFunctionID, UserTableID
+FROM az_func.GlobalState
+
+OPEN LeaseTable_Cursor;
+
+FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId;
+
+WHILE @@FETCH_STATUS = 0
+BEGIN
+    PRINT N'Dropping table ' + @TableName;
+    EXEC ('DROP TABLE IF EXISTS ' + @TableName);
+    PRINT 'Removing row from GlobalState for UserFunctionID = ' + RTRIM(CAST(@UserFunctionId AS NVARCHAR(30))) + ' and UserTableID = ' + RTRIM(CAST(@UserTableId AS NVARCHAR(30)));
+    DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId
+    FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId;
+END;
+
+CLOSE LeaseTable_Cursor;
+
+DEALLOCATE LeaseTable_Cursor;
+```

src/TriggerBinding/SqlTableChangeMonitor.cs

@@ -955,7 +955,7 @@ LEFT OUTER JOIN {this._leasesTableName} AS l ON {leasesTableJoinCondition}
                 IF @unprocessed_changes = 0 AND @current_last_sync_version < {newLastSyncVersion}
                 BEGIN
                     UPDATE {GlobalStateTableName}
-                    SET LastSyncVersion = {newLastSyncVersion}
+                    SET LastSyncVersion = {newLastSyncVersion}, LastAccessTime = GETUTCDATE()
                     WHERE UserFunctionID = '{this._userFunctionId}' AND UserTableID = {this._userTableId};
 
                     DELETE FROM {this._leasesTableName} WHERE {LeasesTableChangeVersionColumnName} <= {newLastSyncVersion};

src/TriggerBinding/SqlTriggerListener.cs

@@ -321,8 +321,12 @@ IF OBJECT_ID(N'{GlobalStateTableName}', 'U') IS NULL
                         UserFunctionID char(16) NOT NULL,
                         UserTableID int NOT NULL,
                         LastSyncVersion bigint NOT NULL,
+                        LastAccessTime Datetime NOT NULL DEFAULT GETUTCDATE(),
                         PRIMARY KEY (UserFunctionID, UserTableID)
                     );
+                ELSE IF NOT EXISTS(SELECT 1 FROM sys.columns WHERE Name = N'LastAccessTime'
+                    AND Object_ID = Object_ID(N'{GlobalStateTableName}'))
+                        ALTER TABLE {GlobalStateTableName} ADD LastAccessTime Datetime NOT NULL DEFAULT GETUTCDATE();
             ";
 
             using (var createGlobalStateTableCommand = new SqlCommand(createGlobalStateTableQuery, connection, transaction))
@@ -390,7 +394,7 @@ IF NOT EXISTS (
                     WHERE UserFunctionID = '{this._userFunctionId}' AND UserTableID = {userTableId}
                 )
                     INSERT INTO {GlobalStateTableName}
-                    VALUES ('{this._userFunctionId}', {userTableId}, {(long)minValidVersion});
+                    VALUES ('{this._userFunctionId}', {userTableId}, {(long)minValidVersion}, GETUTCDATE());
             ";
 
             using (var insertRowGlobalStateTableCommand = new SqlCommand(insertRowGlobalStateTableQuery, connection, transaction))