Friday, October 23, 2009

Accessing Invisible Tables

In SQL Server, there are two kinds of tables are invisible. One is system protected tables, for instance, sys.sysrscols. You could access them by using Dedicated Administrator Connection (DAC) to connection to the server.

Enable DAC:
exec sp_configure 'remote admin connections', 1 
reconfigure
Connect using DAC: specify ADMIN:ServerName in connection string, then you can run select * from sys.sysrscols to get result of the table. This type of tables can only be seen by admin connection.
Another type of tables is temporary tables, including local temporary table and table variable; they are only accessible by owner scope and there is no "official" way to access them out of scope. For instance, if a table variable is created in a function or procedure or a batch, it can only be accessed by them but not other sessions, and procedures and functions called by them, including dynamic SQL. If a local temporary table is created in a procedure or batch (you can't create local temporary table within a function), it cannot be accessed by non-owner session too, but they can be accessed by the procedure and function invoked by them, including dynamic SQL.
How can you access a temporary table/table variable created by other session and how can you access system protected table without connecting with DAC? The answer is directly access data page in the data page. By knowing the name of the table, we could find out the object id in sys.objects table and then use DBCC IND lists all index pages, then use DBCC PAGE to show what is on the page and then PIVOT it to the structure like the original table structure. Please see the code below.
use master
go
create procedure dbo.sp_ReadTable
(
@ObjectID int
)
as
begin
dbcc traceon(3604)
declare @SQL nvarchar(max)
create table #ind(id int identity(1,1) primary key, PageFID smallint, PagePID int, IAMFID smallint, IAMPID int, ObjectID int, IndexID int, PartitionNumber bigint, PartitionID bigint, iam_chain_type varchar(50), PageType int, IndexLevel int, NexPageFID int, NextPagePID int, PrevPageFID int, PrevPagePID int)
create table #page(id int identity(1,1) primary key,ParentObject varchar(300), Object varchar(300), Field varchar(300), Value varchar(6000))
create table #res(id int primary key, Page varchar(100), Slot varchar(300), Object varchar(300), FieldName varchar(300), Value varchar(6000))
create table #ColumnName(ColumnID int primary key, Name sysname)
insert into #ColumnName
select column_id, name
from sys.columns
where object_id = @ObjectID
select @SQL = 'dbcc ind(' + quotename(db_name())+', ' + cast(@ObjectID as varchar(20)) + ', '+cast(b.indid as varchar(20))+')'
from sysobjects a
inner join sysindexes b on a.id = b.id
where b.indid in (0,1) and a.id = @ObjectID
insert into #ind
exec sp_executesql @SQL
declare @PageFID smallint, @PagePID int
declare c cursor for
select top 3 PageFID, PagePID from #ind where PageType = 1
open c
fetch next from c into @PageFID, @PagePID
while @@fetch_status = 0
begin
delete #page
select @SQL = 'dbcc page (' + quotename(db_name()) + ',' + cast(@PageFID as varchar(20)) + ',' + cast(@PagePID as varchar(20)) + ', 3) with tableresults'
insert into #page
exec sp_executesql @SQL
delete from #page where Object not like 'Slot %' or Field = '' or Field in ('Record Type', 'Record Attributes') or ParentObject in ('PAGE HEADER:')
insert into #res
select id, cast(@PageFID as varchar(20)) + ':' + CAST(@PagePID as varchar(20)), ParentObject, Object, Field, Value from #page
fetch next from c into @PageFID, @PagePID
end
close c
deallocate c
--select *, @ObjectID ObjectID ,object_name(@ObjectID) ObjectName from #res order by id
select @SQL = '
select __HashedRecID__'+cast((select ',[' + Name + ']' from #ColumnName order by ColumnID for xml path(''), root('root')) as xml).value('/root[1]', 'nvarchar(max)')+'
from (select HASHBYTES(''SHA1'', slot) __HashedRecID__, FieldName, Value from #res) p
PIVOT
(
max(Value)
FOR FieldName IN
( ' + stuff(cast((select ',[' + Name + ']' from #ColumnName order by ColumnID for xml path(''), root('root')) as xml).value('/root[1]', 'nvarchar(max)'), 1,1,'') + ' )
) AS pvt

'
--print @SQL
exec(@SQL)
end
go
exec sp_MS_marksystemobject 'dbo.sp_ReadTable'
Now, let's do a test. First of all, let's see how it works with system protected tables
use tempdb
exec sp_ReadTable 5 --sysrowsets

Wow, that's cool. __HashRecID__ is an internal ID that I created for pivoting records generated by DBCC PAGE.

Now let's see if this procedure could help see records in the temporary table that created by other session.

First of all, start one window in Management Studio and create a temporary table

Let's see the result:

From this screen, we could tell, we are able to display the content of temporary table created in session 60 in session 58! By using the same way, you can also see the content of table variables created by any session. For instance, in one window, you can create a table variable and have an indefinite loop to insert one record per second. Open another window and change the database to tempdb then guess which line is the table variable is created by that session, like the screen shot above, you could not tell which table variable line number 4 is. And then pass the object id to sp_ReadTable procedure.

Here is the result:

In SQL Server, there is almost no secret. All tables can be accessed by DBCC PAGE. When a database is created with Transparent Database Encryption, the data in temporary table will be encrypted as well, nevertheless, those encrypted temporary table can also be read in this way as I mentioned in my first blog that encrypted data gets decrypted before entering buffer pool. DBCC PAGE is reading decrypted from buffer pool.

Friday, September 18, 2009

Implement Sequence in SQL Server (3)

In my previous post, I introduced supporting procedures and functions of Sequence. In this article, I will post one example for performance testing and another two for generating set of numbers.

print '------use temporary table------'
if OBJECT_ID('tempdb..#1') is not null
drop table #1
go
create table #1(id int identity(1,1) primary key)
set nocount on
declare @i int, @d datetime2(3)
select @i = 0, @d = GETDATE()
while @i < 100000
begin
begin transaction
insert into #1 default values
select @i = SCOPE_IDENTITY()
rollback
end
select @i NumberOfSequence, DATEDIFF(millisecond, @d, getdate()) Milliseconds
go
print '------use permanent table------'
if OBJECT_ID('test1') is not null
drop table test1
go
create table test1(id int identity(1,1) primary key)
set nocount on
declare @i int, @d datetime2(3)
select @i = 0, @d = GETDATE()
while @i < 100000
begin
begin transaction
insert into test1 default values
select @i = SCOPE_IDENTITY()
rollback
end
select @i NumberOfSequence, DATEDIFF(millisecond, @d, getdate()) Milliseconds
drop table test1
go
print '------use permanent Sequence------'
if exists(select * from Sequence.Sequence where Name = 'Test')
exec Sequence.Remove 'Test'
exec Sequence.CreateOrAlter @Name = 'Test', @Cache = 100
set nocount on
declare @i int, @d datetime2(3)
select @i = 0, @d = GETDATE()
while @i < 100000
begin
select @i = Sequence.NextVal('Test')
end
select @i NumberOfSequence, DATEDIFF(millisecond, @d, getdate()) Milliseconds
go
print '------use session Sequence------'
exec Sequence.SessionReset
set nocount on
declare @i int, @d datetime2(3)
select @i = 0, @d = GETDATE()
while @i < 100000
begin
select @i = Sequence.SessionNextVal('#1', @@SPID)
end
select @i NumberOfSequence, DATEDIFF(millisecond, @d, getdate()) Milliseconds
------use temporary table------
NumberOfSequence Milliseconds
---------------- ------------
100000 5767

------use permanent table------
NumberOfSequence Milliseconds
---------------- ------------
100000 5730

------use permanent Sequence------
NumberOfSequence Milliseconds
---------------- ------------
100000 3050

------use session Sequence------
NumberOfSequence Milliseconds
---------------- ------------
100000 1734

From this example you could see, for generating 100,000 numbers, temporary table needs 5.767 seconds, permanent table needs 5.73 seconds, and permanent sequence needs 3.05 seconds and session sequence only needs 1.734 seconds

In this example, I will use ListVal method to generate a set of sequential numbers by using session dependent Sequence. This is extremely useful when you want to have a sequential value set to join tables.

select * from Sequence.ListVal('#'+cast(newid() as varchar(50)), 5)
--select Sequence.SessionReset()
exec Sequence.SessionReset -- free-up resource.
N
--------------------
1
2
3
4
5

(5 row(s) affected)

Another example

set nocount on 
declare @t table (id int, v int)
insert into @t(id) select N from Sequence.ListVal('#TestA', 5)
select * from @t
update @t set v = Sequence.NextVal('#TestA')
select * from @t
exec Sequence.SessionReset -- free-up resource.
id          v
----------- -----------
1 NULL
2 NULL
3 NULL
4 NULL
5 NULL

id v
----------- -----------
1 6
2 7
3 8
4 9
5 10

Tuesday, September 15, 2009

Setup Procedure to Run Automatically When SQL Server Starts

That's simple, setup a job and run the procedure and schedule it runs when SQL Server agent started. See the problem here? My title is "when SQL Server Start", not "When SQL Server Agent". SQL Server can start without agent. Procedure in SQL Server can be started automatically when SQL Server starts. In order to do that, you have to set "scan for startup procs" through sp_configure function, and then use sys.sp_procoption to set a procedure to be launched as server started. See example below.

exec sp_configure 'show advanced options', 1
reconfigure
exec sp_configure 'scan for startup procs', 1
reconfigure with override
exec sp_configure 'scan for startup procs'
go
alter procedure simpletest
as
begin
waitfor delay '01:00:00'
return
end
go
exec sys.sp_procoption 'simpletest','startup', 1

Once a procedure marked as start up automatically, it will be launched by server as it starts in a user session (session id may be less than 50). You are able to kill that session while it's running. The login name is sa. When the procedure is started while SQL Server Starting, it will also be logged

2009-09-15 00:50:09.01 spid7s      Launched startup procedure 'simpletest'.

There are some restrictions for creating auto-start procedure:

  • You have to be a sysadmin
  • The owner of procedure must be dbo
  • The procedure must be in master database
  • Procedure must not have a parameter

There are two ways you can determine if a procedure can be started automatically or not.

  • ObjectProperty(object_id,'ExecIsStartup') = 1
  • select * from sys.procedures where is_auto_executed = 1

Check Microsoft website for more information.

Implement Sequence in SQL Server (2)

In previous article, I explained the way using identity to implement Sequence object. I also gave the source code of my CLR Sequence which can also be downloaded here. As a continuation of this series, in this article, I will explain how to install Sequence object in SQL Server and how to use it.

  1. Download the source code and decompress it.
  2. Run <<Sequence\SQL Code\Create Tables.sql>> to create tables.
    1. Specify the database you want to deploy
    2. Ensure your database having trustworthy turned on.
  3. Open <<Sequence.csproj>> project, and compile it. (don't deploy it.)
    1. Make the binary visible by SQL Server after compile
  4. Run <<Sequence\SQL Code\Deploy CLR.sql>> to deploy CLR and the wrap of CLR
    1. Specify the database you want to deploy
    2. Specify the path of the assembly.
  5. Done and ready to use Sequence.

Sequence objects are all created under Sequence schema. This code is tested on SQL Server 2008. If you need it on SQL Server 2005, you need to modify Sequnce.InternalUPdate procedure.

Sequence.Sequence

This is a table to host all session independent sequences. After CLR is deployed, this table is read-only to user but writable to CLR. When you modify this table, you will see error below

Table could only be changed by Sequence internal process.

Msg 3991, Level 16, State 1, Procedure SequenceMonitorDataModification, Line 1

The context transaction which was active before entering user defined routine, trigger or aggregate "SequenceMonitorDataModification" has been ended inside of it, which is not allowed. Change application logic to enforce strict transaction nesting.

The statement has been terminated.

This is because I called Transaction.Current.Rollback. No matter what, I could not prevent this error message from showing up to client. A sequence has following properties:

Name: Name of the sequence, nvarchar(128), should be unique

CurrentValue: Current Value of the sequence, bigint, not fully utilized yet, will be more accurate in future version.

StartValue: Starting Value of the sequence, bigint, default(0), not nullable

EndValue: End Value of the sequence, bigint, not nullable, default is the maximum of bigint value 922,337,203,654,775,807. How big is this value? If you have a computer which consume 300,000 numbers per second without any bread, the sequence can provide 100 years of service! If you starting value is -922,377,203,775,808, the sequence can serve 200 years at volume of 300k per second.

Step: the step from one number to next number. Default is one. This value cannot be zero but can be a negative number. When it is a negative number, ending value must be smaller than Starting value.

Cache: this value must be greater than zero. Sequence object will preserve some values in memory as it needs when new value being assigned. The default value is 100. Bigger cash can give better performance.

CachedValue: internal use only. When the values are cached, the cached value will be written ahead before the cached values to be used. This means that if the server get restarted before the cached values being consumed, there is a potential that there are hole(s) in the assigned sequence number.

Error: Record internal error.

Sequence.Service

This is the service procedure. This procedure provides service to running procedures for function. For information of running SQL within function, please see my prior post here. This procedure has no parameters. It is good to have this procedure to be started when SQL Server start by setting procedure options. For setting up procedure option, please read article here. Please be careful that the CLR might be unloaded when the server is under memory pressure. The monitor process of this process has not been developed. End user should know this defect.

Sequence.TerminateService

Running this procedure will terminate the Sequence.Service process.

Sequence.ListSequence(@type)

This function returns a table list. This function returns relatively accurate CurrentValue of Sequence. The @type can be

  • All: all sequences
  • AllStatic: all global sequences
  • AllSession: all sequences that created by different sessions
  • ThisSession: all sequences created by this session

Sequence.ListVal(@Name, @N)

This is the quickest way to get @N numbers from Sequence @Name. For instance, select * from Sequence.ListVal('#TempSequence', 100)will give you 100 unique numbers. But it is not guaranteed there is no gap in the list.

Sequence.CreateOrAlter(@Name sysname, @StartValue [bigint] = null, @EndValue [bigint] = null, @Step [bigint] = null, @Cycle [bit] = null, @Cache [int] = null, @CurrentValue [bigint] = null)

This function is to create or alter a Sequence. For instance

declare @name sysname

exec @name = Sequence.CreateOrAlter @Name = '#test'

select @name

Sequence.Remove(@Name)

This is the function to remove a sequence.

Sequence.NextVal(@Name)

This function returns the next value of specified sequence. When you use this to access a session dependent sequence, you don't have to create it first. The sequence will be created automatically with default settings. For instance, select Sequence.NextVal('#unknown name')

Sequence.SessionNextVal(@Name, @SPID)

This function returns value of a session dependent sequence. The meta-data of session dependent sequence is save in the memory, it can provide the most efficient way to get a number. You can also use Sequence.NextVal to get a value from a session dependent sequence, but it is not as fast as this function. The only difference is NextVal function issues context connection to SQL run select @@SPID to get current session id. If you know how to get session id in CLR without running that command via a context connection, please let me know.

Sequence.NextVal1(@Name)

Same as Sequence.SessionNextVal, but it does not need to pass spid to the function. the function essentially calls SessionNextVal function. This function is faster than NextVal but slower than SessionNextVal.

Sequence.SessionReset()

This function removes all session dependent sequence in current session.

Sequence.SequenceMonitorDataModification

This is the CLR trigger behind Sequence.Sequence table. Please don't remove this trigger.

Sequence.InternalRead, Sequence.InternalUpdateError, Sequence.InternalDelete, and Sequence.InternalUpdate

These procedures are called internally by CLR.

Now I have explained the procedure and functions that sequence object provide. Some examples will be given in future post as well as explanation of the CLR.

Saturday, September 12, 2009

Implement Sequence in SQL Server (1)

Sequence is a type of object in Oracle for generating sequential number. A sequence can be seen from all sessions. Sequence is not transactional. Calling NextVal function to a sequence will always give you a new number. It is a very common use in Oracle to create one sequence object for one table object and use the value coming from it as table's primary key. The counterpart of Oracle sequence in SQL Server is Identity. Identity can only be associated to a table object, including permanent table, global/local temporary table, table variable and table type variables. A table can and only can have one identity column. Once a column is defined as an identity column, the column becomes read-only. The values of the field are assigned automatically when new records entered. Identity field in SQL Server is transaction independent – rolling back transactions will not cause generated identity value to be rolled back. By using this feature, we can mimic Oracle Sequence. Please see code below.

if OBJECT_ID('GetNextVal') is not null
drop procedure GetNextVal
go
create procedure GetNextVal(@Name sysname, @value bigint output)
as
begin
set nocount on
declare @sql nvarchar(1000)
select @sql = 'delete '+@Name+' ;insert into '+@Name+' default values; select @value = scope_identity()'
exec sp_executesql @sql, N'@value bigint output', @value output
end
go

We are using dynamic SQL to generate a value and pass that value back to an output variable. Now let's do a test to see how it performs.

print '-----Test: use permanent table----'
if OBJECT_ID('dbo.Identity1') is not null
drop table dbo.Identity1
go
create table dbo.Identity1(ID bigint identity(1,1) primary key)
go
declare @i bigint, @j bigint, @d datetime2(3)
print 'get one value to let the system to generate IAM chain to the object'
exec GetNextVal 'Identity1', @i output
select @d = GETDATE()
while @i < 100000
begin
exec GetNextVal 'Identity1', @i output
end
select @i CurrentVal, DATEDIFF(MILLISECOND, @d, GETDATE()) TotalMilliseconds
go
print '-----Test: use permanent table----'
go
if OBJECT_ID('tempdb..#Identity1') is not null
drop table #Identity1
go
create table #Identity1(ID bigint identity(1,1) primary key)
go
declare @i bigint, @j bigint, @d datetime2(3)
print '-----get one value to let the system to generate IAM chain to the object-----'
exec GetNextVal '#Identity1', @i output
select @d = GETDATE()
while @i < 100000
begin
exec GetNextVal '#Identity1', @i output
end
select @i CurrentVal, DATEDIFF(MILLISECOND, @d, GETDATE()) TotalMilliseconds
go
-----Test: use permanent table----
get one value to let the system to generate IAM chain to the object
CurrentVal TotalMilliseconds
-------------------- -----------------
100000 86550

(1 row(s) affected)

-----Test: use permanent table----
-----get one value to let the system to generate IAM chain to the object-----
CurrentVal TotalMilliseconds
-------------------- -----------------
100000 9600

(1 row(s) affected)

By using permanent table, generating 100k sequential numbers used 86 seconds whereas generating the same count of sequential number by a temporary table only need 10 seconds. Both tests are running on the same virtual machine with tempdb and test db on the same hard drive. (From this test, we could also make another conclusion that the performance of manipulating temporary B-Tree table in SQL Server is faster than regular B-Tree table because the data manipulations on temporary tables are not fully logged!)

Using this approach to generate sequential number has few issues:

  • Performance does not sound promising.
  • We have to create one table object for one sequence we need.
  • By using temporary table, the sequence could only be visible only to the session. we have to manually create temporary table out of the procedure.( if we create it in one procedure, the temporary table could not be accessed by other procedures)
  • We have to invoke GetNextVal in a procedure, trigger or a SQL batch but not for a function even though you call procedure within function through CLR, see my post explained here.
  • If you use my theory here, you are able to run procedure in a function, but you will not be able to have session dependent sequence.
  • The first record of the sequence table will become to "hot spot" as we could see any procedure call make one delete and insert. Under high volume procedure calls from many sessions will cause excessive lock contention on row and page.

Is this insuperable? Answer is NO. Recently I wrote sequence component for SQL Server by using C#. You could see the performance below. I will describe it in next posts tomorrow. The code can be downloaded from here.

Monday, August 31, 2009

Run Dynamic T-SQL from a Function with Data Change

This is today's topic – Run Dynamic T-SQL from a function which allows data modification…wait a second…in this post, a point has been laid out that "You can NOT do any data modifications within a function". Is the rule changed? Answer is no. Microsoft hasn't made any promise yet to make a function to be able to update tables. How can it be implemented? In my experiment, I use SQL CLR to make different thread (spid) to be able to talk to each other so that a function which has an arbitrary SQL passed in sends the SQL(s) to a procedure running at the backend and let the store procedure to run the SQL Statement. At the end, the procedure returns the result to the function and the function returns it back to user session. Before continuing this article, I disclaim responsibilities for any potential corruptions and/or damage to your SQL environment and/or business logics by using following code. Use of following code in your production environment is on your own risk.

The Service process is a CLR procedure running at backend to receive a request from any function calls. Session A, as illustrate below, puts the SQL to a internal structure, sends this structure to the queue and then signals the Service Process and then wait until service queue signal back Session A. Once the Service Process is signaled, it will be aware of the SQL in the queue and hence dequeues the structure, runs SQL, writes result to the structure and then signal Session A and continue to wait for next signal. Once Session A is signaled, it accesses the structure that has been enqueued and compose the result to the client.

I am using ManualResetEvent class to tell when a thread should run or wait. Please see the code below.

using System;
using System.Collections;
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using System.IO;
using Microsoft.SqlServer.Server;
using System.Threading;

public class RunDynamcSQLInFunctionWithUpdateAndResultSet
{
[Microsoft.SqlServer.Server.SqlFunction
(
FillRowMethodName = "RunSQLFillRow",
IsDeterministic=false,
TableDefinition = "ID int, ResultSetSchema xml, ResultSet xml, Error nvarchar(max)"
)
]
public static IEnumerable RunSQL(SqlString SQL)
{
SQLProcess req = new SQLProcess();
if (SQLProcess.QueueSQL == null)
{
req.Error = new Exception("Service Process is not running.");
}
else
{
req.SQL = SQL.Value;
if (req.SQL == null)
req.SQL = "";
lock (SQLProcess.QueueSQL)
{
SQLProcess.QueueSQL.Enqueue(req);
}
SQLProcess.ServiceEvent.Set();
req.ProcessEvent.WaitOne();
}
//Now the result is in req structure
List ret = new List();
ret.Add(req);
return ret;
}
public static void RunSQLFillRow(object obj, out int ID, out SqlXml ResultSetSchema, out SqlXml ResultSet, out SqlString Error)
{
SQLProcess req = (SQLProcess)obj;
if (req.Results.Tables.Count == 0)
{
ResultSet = SqlXml.Null;
ResultSetSchema = SqlXml.Null;
}
else
{
MemoryStream schema = new MemoryStream();
MemoryStream data = new MemoryStream();
req.Results.WriteXmlSchema(schema);
req.Results.WriteXml(data);
ResultSet = new SqlXml(data);
ResultSetSchema = new SqlXml(schema);
}
if (req.Error == null)
Error = SqlString.Null;
else
Error = req.Error.ToString();
ID = 1;
}
[Microsoft.SqlServer.Server.SqlProcedure]
public static void ServiceProcess()
{
SQLProcess req = null;
SQLProcess.ServiceEvent = new ManualResetEvent(false);
SQLProcess.QueueSQL = new Queue();
//infinite loop as a service.
while (true)
{
//wait for client request(s)
SQLProcess.ServiceEvent.WaitOne();
lock (SQLProcess.QueueSQL)
{
while (SQLProcess.QueueSQL.Count > 0)
{
req = null;
req = SQLProcess.QueueSQL.Dequeue();
if (req.TerminateServiceProcess)
{
req.ProcessEvent.Set();
SQLProcess.ServiceEvent.Reset();
SqlContext.Pipe.Send("Terminate signal received, terminating...");
while (SQLProcess.QueueSQL.Count > 0)
{
req = SQLProcess.QueueSQL.Dequeue();
req.ProcessEvent.Set();
}
SQLProcess.ServiceEvent = null;
SQLProcess.QueueSQL = null;
SqlContext.Pipe.Send("Done.");
return;
}
else
{
if (req != null)
{
try
{
using (SqlConnection connection = new SqlConnection("context connection=true"))
{
try
{
connection.Open();
SqlCommand cmd = connection.CreateCommand();
cmd.CommandText = req.SQL;
cmd.CommandType = CommandType.Text;
cmd.CommandTimeout = 0;
(new SqlDataAdapter(cmd)).Fill(req.Results);
}
catch (Exception e)
{
req.Error = e;
}
finally
{
connection.Close();
}

}
}
finally
{
//tell req that process finished.
req.ProcessEvent.Set();
}
}
}
}
//Back to original status
SQLProcess.ServiceEvent.Reset();
}
}
}
[Microsoft.SqlServer.Server.SqlProcedure]
public static void TerminateServiceProcess()
{
SQLProcess req = new SQLProcess();
req.TerminateServiceProcess = true;
lock (SQLProcess.QueueSQL)
{
SQLProcess.QueueSQL.Enqueue(req);
}
SQLProcess.ServiceEvent.Set();
req.ProcessEvent.WaitOne();
}

};
internal class SQLProcess
{
public static ManualResetEvent ServiceEvent = null;
public static Queue QueueSQL = null;

public ManualResetEvent ProcessEvent = new ManualResetEvent(false);
public Exception Error = null;
public string SQL = "";
public DataSet Results = new DataSet();
public bool TerminateServiceProcess = false;
};

I created internal class with static member, " publicstaticManualResetEvent ServiceEvent = null;", for Service Process. The Service process initializes it and calls "SQLProcess.ServiceEvent.WaitOne();" to wait for a request. As this is static public variable, it could be access from other process which is the function with SQL code passed in. The SQL function calls "SQLProcess.ServiceEvent.Set();" to signal Service Process to process the queue after it puts the structure in the queue. Within the structure, there is a member called ProcessEvent, which is used for Service Process to signal back the function process, "publicManualResetEvent ProcessEvent = newManualResetEvent(false);". After enqueuing the structure, the function calls "req.ProcessEvent.WaitOne();" to wait a signal to be sent by service process. Once the Service Process receives the request from the function, it dequeues the structure, run the SQL and save the result to the structure and then call ProcessEvent.set in the structure to tell the function that the SQL is done then it calls "SQLProcess.ServiceEvent.Reset();" to back to original status then wait for next request. Function gets signaled after Service Process finishes the task then wakes up and then return the result to client.

Let's deploy the code and do a test. From one window, run "exec dbo.ServiceProcess". Open another window, run following code

select * from dbo.RunSQL('select * into #1 from sys.objects')
select * from dbo.RunSQL('select * from #1')
select * from dbo.RunSQL('drop table #1')
exec dbo.TerminateServiceProcess

Here is the result

See, you are free to create and drop table as well as data modification within a function. The last statement, "exec dbo.TerminateServiceProcess" tells the Service process to quit. You will see following result from the first window.

Terminate signal received, terminating...
Done.

Now it seems like I break the rule but actually not. You may already be aware of

  • The SQL statement is actually NOT running under its current security context, instead, it runs under the session of the service procedure. This will be a huge security hole as anyone who has access to the function will automatically have the same permission as the caller of service procedure.
  • The temp table created by the session could only be accessed by the function through dynamic SQL. The temporary table could also be access by other session calling this function.
  • If the service process is killed for any reason, the running functions will never return as they will have no chance to be signaled by service process.
  • The service procedure can only have one instance so that all the request from function will run sequentially

This is just an experiment of thread communication. I hope you enjoy it, have fun with it but not use it in your production.

Thursday, August 27, 2009

Restrict Connections for a Process

Application lock in SQL Server is very powerful as I mentioned in my previous post here and here. If you have a process, many clients are calling it, and you want to restrict only 10 requests can be processed simultaneously, other requests wait until any of the running requests finish. How do you do it? – haha, you got it, use application lock!

I will have an example to explain who it works. First of all, you define a process called ProcessA. You want this process to have maximum 2 concurrencies. You should have you application to acquire an eXclusive lock on ProcessA1 first. If it could not acquired, then try resource ProcessA2. If both of them could not acquire, then wait and repeat previous processes. Once you get one, let say ProcessA2, you can start your real work and then at the end you release the lock you acquired. Just that easy! Please see the code I wrote below.

use master
if object_id('sp_AcquireSlotLock') is not null
drop procedure sp_AcquireSlotLock
go
create procedure sp_AcquireSlotLock
(
@process_name sysname,
@max_concurrent int,
@queue_timeout int = 0,-- 0, means wait indefinitely, milliseconds
@return_immediately bit = 0 -- if set to 1, take a peek at the server and return immediately.
)
as
begin
declare @entry_date datetime
declare @slot_name nvarchar(150) -- OUTPUT - must give back slot acquired to caller so caller can later release.
declare @basetime datetime
declare @delaytime int -- in milliseconds
declare @retcode int
declare @i int
if @queue_timeout<0
begin
raiserror(21344, 16, -1, '@queue_timeout')
return (1)
end
if @max_concurrent<=0
begin
raiserror(21344, 16, -1, '@max_concurrent')
return (1)
end
select @entry_date=getdate(),
@delaytime = case when @return_immediately = 1 then 0 else 100 end, --polling interval is defaulted to 0.1 seconds
@slot_name = NULL, -- If terminate anywhere unexpectedly, dont want to give caller a lock they didnt really get.
@i = 1
while 1 = 1
begin
if @queue_timeout>0 and dateadd(millisecond, @queue_timeout, @entry_date) < getdate()
begin
select @slot_name = NULL -- waited but failed to get one
break
end
if @i > @max_concurrent
select @i = 1
select @slot_name = @process_name + convert(varchar,@i)
if @i <> @max_concurrent
exec @retcode=sp_getapplock @Resource=@slot_name,@LockMode=N'Exclusive',@LockOwner='Session',@LockTimeout = 0 --the call is not blocking, return immediately having acquired the lock or not
else
exec @retcode=sp_getapplock @Resource=@slot_name,@LockMode=N'Exclusive',@LockOwner='Session',@LockTimeout = @delaytime --if acquiring the last slot, then wait for 100 milliseconds
if (@retcode not in (0, 1, -1))
begin
raiserror(21414, 16, -1) -- Unexpected failure acquiring application lock.
return @retcode
end
if (@retcode in(0, 1)) -- got lock for that slot
break
else
begin
if @i = @max_concurrent and @return_immediately = 1
begin
select @slot_name = null
break
end
end
select @i = @i + 1
end
select @slot_name
RETURN 0
end
go
exec dbo.sp_MS_marksystemobject sp_AcquireSlotLock
go
use master
if object_id('sp_ReleaseSlotLock') is not null
drop procedure sp_ReleaseSlotLock
go
create procedure sp_ReleaseSlotLock
(
@slot_name nvarchar(150)
)
as
begin
declare @retcode int
exec @retcode = sp_releaseapplock @slot_name, @LockOwner=N'Session'
if @retcode <> 0
begin
raiserror(21415, 16, -1)
return 1
END
return @retcode
end
go
exec dbo.sp_MS_marksystemobject sp_ReleaseSlotLock
go

Let's have a test. Open 4 windows in Query Analyzer, and run "exec sp_AcquireSlotLock 'test', 3". 3 of them will return slot name, test1, test2 and test3, but the last one will wait infinitely until you run exec sp_ReleaseSlotLock slot_name in a window that has slot name returned.

SQL Server merge replication uses the same concept to limit the maximum concurrent merge agents. There are two undocumented procedures, sp_MSacquireSlotLock and sp_MSreleaseSlotLock, do almost the same thing as the procedures I wrote. One difference is MS procedure check locks availability once every 2 seconds when all slots are used while mine checks every 100 milliseconds when slots are full without using "wait for" command. (Tip, in this sense, you are able to have a procedure to do "wait for" command at lower granularity – milliseconds without introducing any CLR, extended procedure, or any other COM calls.)