I have wondered for a while if there is a way to use SQL Server’s BULK INSERT command using BLOB or CLOB data Instead of a local file.

The reason for this is that when writing imports, using both Coldfusion and PHP, parsing and importing the data into the database is the weakest link.  It is relatively simple to write dynamic SQL to import data into my existing schema when copying from data already present in the database.  It is quite another matter to attempt to run thousands of INSERT statements from my web application.

I can’t ask a client to use SQL Management Studio to import their data into a table and then use a web app to run the import.  I don’t want to expose the database to the user and vice-versa.

At present I can only think of one good way to implement an import process and that is to allow the user to

1. Upload the file.
2. Create some sort of import plan (associate the content).
3. Save that plan in some way.
4. Run a scheduled task to run the import in batches.

I am dealing with imports of possibly hundreds of thousands of records.  Also note that I have limited access to the database as it is possible that the app will reside in a shared hosting environment.