IveelySE.Dream Design (Computing Part)

 Iveely.Dream is one of the most important subprojects in Iveely Search Engine, why it names 'dream', as we most hope it can take a fly for Iveely Search Engine, it is a distributed computing and storage system, it most like a dream , but it is true for us. this part we will talk about computing.

        The basic data flow as follow:

 from left to right, for user ,we only open a interface named 'Input' , he can tell us where the files are,base on this ,we set up multiple readers, when reader get the command, send the data to a processor ,how does a reader know which processor to send? there is a information adapter to help them to chose processor, the processor get the data ,process it and send to the result to the writer, then, the writer write the data to a file or  distributed file system. there is no detail here? no ,let talk some details.

What the interface IInput has?

Yes, There is only an Read(string[] filePathes),user should told us those files.for multiple files ,read them by many process of readers, when read a file, multithreading a same file.

What reader is?

reader is a application for read data,Perform steps:

0. When read configuration information , especially adapter server name and port. 

1. Told adapter, I am come, adapter will record its IP and Port. 

2. Wait for supervisor give him the file name.

...

Last edited Mar 28, 2013 at 1:34 PM by iveely, version 5

Comments

No comments yet.