Does sqlsrv_query limit the number of statements that can be exectuted in one query?

I am generating a SQL insert statement within a PHP for loop.

The SQL string generated is a large number of individual SQL statements like this:

INSERT INTO tbl VALUES(1,2,3);
INSERT INTO tbl VALUES(4,5,6);
INSERT INTO tbl VALUES(7,8,9);

etc…

Then I execute with:

$InsertResult = sqlsrv_query($conn, $InsertSQL);

The problem is that only the first 312 statements get executed instead of the full 2082 lines (only 312 rows are inserted into the table).

When I output the $InsertSQL variable to the JavaScript console and then execute it manually in SSMS it works perfectly and inserts all 2082 lines. Only when I run the $InsertSQL variable through sqlsrv_query does it not go to completion.

I also don’t get any errors and the query result comes back true as tested in this line:

if(!$InsertResult) die('Problem with Insert query: ' . $InsertSQL);

When I searched for a solution to this problem I saw that (although it’s not mentioned in the PHP manual site) sqlsrv_query apparently has a string character limit on the $SQL variable (around 65k characters).

See the other StackOverflow article here: length restriction for the sql variable on sqlsrv_query?

I figured this was the problem and so created a shorter version of the string (by only adding in the column values that I actually wanted to import). This much shorter version however, still only Inserts the first 312 lines! So now it seems this is NOT related to the max string length. In fact, if it was, I should only get around 250 lines (after 250 statements I’m at about 65k characters).

I can also execute each insert statement individually but of course this takes much longer. In my testing, it takes 90s or so to do it this way where as running the combined statement manually in SMSS takes only around 40s.

Note that I’ve also looked into SQL Server’s Bulk Insert however I won’t be able to send the file to the machine where SQL Server is installed (SQL Server and Web servers are on separate computers). From my understanding this eliminates this possibility.

Any help is much appreciated as I can’t even figure out what it is that is limiting me, never mind fix it and I’d hate to have to execute one line at a time.

Answer

Explanations:

There is a known issue with this driver, posted on GitHub, about executing large SQL statements. One part of the provided solution are the following explanations:

Seems like when executing a large batch of SQL statements, Microsoft SQL Server may stop processing the batch before all statements in the batch are executed. When processing the results of a batch, SQL Server fills the output buffer of the connection with the result sets that are created by the batch. These result sets must be processed by the client application. If you are executing a large batch with multiple result sets, SQL Server fills that output buffer until it hits an internal limit and cannot continue to process more result sets. At that point, control returns to the client. This behavior is by design. Client app should flush all the pending result sets. As soon as all pending result sets are consumed by the client, SQL Server completes executing the batch. Client app can call sqlsrv_next_result() until it returns NULL.

So, I don’t think that there is a limit for the SQL statement length, only the size of a PHP string variable ($InsertSQL in your case) is limited to the maximum allowed PHP memory limit. The actual reason for this unexpected behaviour is the fact, that with SET NOCOUNT OFF (this is by default) and a large number of single INSERT statements, the SQL Server returns the count of the affected rows as a result set (e.g. (1 row affected)).

Solution:

I’m able to reprodiuce this issue (using SQL Server 2012, PHP 7.1.12 and PHP Driver for SQL Server 4.3.0+9904) and you have the following options to solve this problem:

• Flush the pending result sets using sqlsrv_next_result().
• Execute SET NOCOUNT ON as first line in your complex T-SQL statement to stop SQL Server to return the count of the affected rows as a resultset.
• Use parameterized statement using sqlsrv_prepare()sqlsrv_execute()

Table:

CREATE TABLE MyTable (
    Column1 int,
    Column2 int,
    Column3 int
)

One complex statement (using sqlsrv_query() and sqlsrv_next_result()):

<?php 

// Connection info
$server = 'serverinstance';
$database = 'database';
$username = 'username';
$password = 'password';
$cinfo = array(
    "Database" => $database,
    "UID" => $username,
    "PWD" => $password
);

// Statement with sqlsrv_query
$sql = "";
for ($i = 1; $i <= 1000; $i++) {
    $sql .= "INSERT INTO MyTable (Column1, Column2, Column3) VALUES (".$i.", 0, 0);";
}
$stmt = sqlsrv_query($con, $sql);
if ($stmt === false) {
    echo "Error (sqlsrv_query): ".print_r(sqlsrv_errors(), true);
    exit;
}

// Clean the buffer
while (sqlsrv_next_result($stmt) != null){};

// End
sqlsrv_free_stmt($stmt);
sqlsrv_close($con);
echo "OK";
?>

One complex statement (using sqlsrv_query() and SET NOCOUNT ON):

<?php 

// Connection info
$server = 'serverinstance';
$database = 'database';
$username = 'username';
$password = 'password';
$cinfo = array(
    "Database" => $database,
    "UID" => $username,
    "PWD" => $password
);

// Connection
$con = sqlsrv_connect($server, $cinfo);
if ($con === false) {
    echo "Error (sqlsrv_connect): ".print_r(sqlsrv_errors(), true);
    exit;
}

// Statement with sqlsrv_query
$sql = "SET NOCOUNT ON;";
for ($i = 1; $i <= 1000; $i++) {
    $sql .= "INSERT INTO MyTable (Column1, Column2, Column3) VALUES (".$i.", 0, 0);";
}
$stmt = sqlsrv_query($con, $sql);
if ($stmt === false) {
    echo "Error (sqlsrv_query): ".print_r(sqlsrv_errors(), true);
    exit;
}

// End
sqlsrv_free_stmt($stmt);
sqlsrv_close($con);
echo "OK";
?>

Parameterized statement (using sqlsrv_prepare() and sqlsrv_execute()):

<?php 

// Connection info
$server = 'serverinstance';
$database = 'database';
$username = 'username';
$password = 'password';
$cinfo = array(
    "Database" => $database,
    "UID" => $username,
    "PWD" => $password
);

// Connection
$con = sqlsrv_connect($server, $cinfo);
if ($con === false) {
    echo "Error (sqlsrv_connect): ".print_r(sqlsrv_errors(), true);
    exit;
}

$sql = "INSERT INTO MyTable (Column1, Column2, Column3) VALUES (?, ?, ?);";
$value1 = 0;  
$value2 = 0;  
$value3 = 0;  
$params = array(&$value1, &$value2, &$value3);
$stmt = sqlsrv_prepare($con, $sql, $params);
if ($stmt === false ) {
    echo "Error (sqlsrv_prepare): ".print_r(sqlsrv_errors(), true);
    exit;
}
for ($i = 1; $i <= 1000; $i++) {
    $value1 = $i;  
    $value2 = 0;  
    $value3 = 0;  
    $result = sqlsrv_execute($stmt);
    if ($result === false) {
        echo "Error (sqlsrv_execute): ".print_r(sqlsrv_errors(), true);
        exit;
    }
}

// End
sqlsrv_free_stmt($stmt);
sqlsrv_close($con);
echo "OK";
?>