c# - Migrating big data to new database -

i'd transfer large amount of data sql server mongodb (around 80 million records) using solution wrote in c#. want transfer 200 000 records @ time, problem keeping track of has been transferred. i'd follows:

gather ids destination exclude source scope read source (excluding ids in destination) write destination repeat 

the problem build string in c# containing ids exist in destination, purpose of excluding source selection, eg.

select * source_table id not in (<my large list of ids>) 

now can imagine happens here when have inserted 600 000+ records , build string ids, gets large , slows things down more, i'm looking way iterate through 200 000 records @ time, cursor, have never done , here, looking advice.

just reference, reads follows

    sqlconnection conn = new sqlconnection(myconnstr);     conn.open();     sqlcommand cmd = new sqlcommand("select * mytable id not in ("+biglistofids+")", conn);     sqldatareader reader = cmd.executereader();     if (reader.hasrows)     {         while (reader.read())         {             //populate objects insertion mongodb         }     } 

so basically, want know how iterate through large amounts of data without selecting data in 1 go, or having filter data using large strings. appreciated.

there many different ways of doing this, suggest first don't try reinvent wheel @ existing programs. there many programs designed export , import data between different databases, flexible , expensive, others come free options , dbms programs include something.

option 1:

use sql server management studio (ssms) export wizards.

this allows export different sources. can write complex queries if required. more information here:

https://www.mssqltips.com/sqlservertutorial/202/simple-way-to-export-data-from-sql-server/

option 2:

export data in ascending id order. store last exported id in table.

export next set of data id > lastexportedid

option 3:

create copy of data in back-up table. export table, , delete records export them.