.net - Batch data processing in real time -

i tasked optimizing performance of linear data processing routine. here's overview of what's in place:

data comes in on udp ports, have multiple listeners listening on different port , writing raw data sql server database (lets call table rawdata). have multiple instances of single threaded linear application grabbing raw data rawdata table , processing individual datarows. processing means raw data compared received data given entity, calculations done calculate number of different readings, couple of web services called each individual data row , new record added each data row in processeddata table. corresponding entity record updated in other table.

the way see problem, can broken down smaller parts , utilize producer/consumer pattern data processing: 1 thread of producer populates shared (blocking) queue, multiple consumers grab data rows queue , parallel processing of them. after consumers done put processed data shared queue, accessed yet consumer thread (single) sqlbulkcopy insert new records. along process there other shared queue store entity info updates , yet consumer grabbing updated information entities , performing updates.

question is, though seems straight forward, looks me cumbersome approach. feel there's better way of doing i'm looking for. suggestions on implementing above producer/consumer pattern? or should different design pattern problem?

thanks in advance

your proposed solution sounds reasonable, , don't view cumbersome @ all. it's simple understand, simple implement, effective, , efficient. allows tune number of producers , consumers achieve best performance. decomposition smaller parts limited communication among parts thing.

so have multiple threads (producers) reading data udp , storing items in shared queue. call rawdata queue. multiple consumers read queue, process items, , place results shared queue. call processeddata queue. finally, have single thread reads processeddata queue , stores items in database.

the .net blockingcollection perfect this.

this might of help: question on c# threading rfid