Information Technologies in Borok Geophysical Observatory

[40]  The data logging system is the basic unit to any geophysical observations. It is an allocated computer to carrying out logging and storing of the data from the sensors of geophysical fields.

[41]  The modern data-acquisition equipments, as a rule, have hardware and software interface of measuring devices with a personal computer. The measurement data can be transformed to the digital format directly in the measuring device as well as in the special box of analog-to-digital converter (ADC). Then measurement data come to a personal computer already in a digital format through RS-232 or USB ports. The measuring equipment software provides the data visualization and storage to the hard disk. The wide spectrum of the specialized data logging systems, including systems designed to work in field, have applications in the industry as well as in scientific researches.

[42]  So, the automatic INTERMAGNET magnetic observatory uses the specialized personal computer ENO II (manufactured by IPGP) with the 1386/40 MHz processor as the data logging system. ENO II device is intended for digital registration of geomagnetic data, coming from scalar and vector magnetometers. The sampling rate of the output data is 1 minute and will be increased to 1 second within the next year. The synchronization with the universal time (UT) is carried out by the built-in global position system (GPS).

[43]  The meteorological station WS-2500 is connected to a personal computer via RS-232 socket. The meteorological station software works under the Windows operating system (OS), allowing the operator to get the data from meteorological sensors and to display the current information. The current meteorological data are stored on the hard disk in an internal digital format.

[44]  The complex "Meteo-2'' provides the measurement data to the personal computer via COM-port. Then data processing is curried out by the corresponding software. The data logging software works under the OS Windows in an asynchronous mode, using interruptions of the COM-port adapter. The background mode running provides the measurement data obtaining, processing and storing on the hard disk. The measurement processes are visualized on the display.

[45]  However, the specialized data logging system software very often is badly adapted to tasks of the continuous observation data digitizing. These tasks assume working in continuous mode, the data storing on the replaceable media or the data receiving via the local access network, continuous synchronization of the registration with universal time. Besides the data logging systems have to provide digital data from the already working analog recording equipment. The inexpensive built-in ADC allows easy transforming the common personal computer to the data logging system.

[46]  The choice of ADC characteristics is caused, first of all, by characteristics of the analog data-acquisition equipment. In particular, the number of measured parameters determines the minimum number of ADC input channels, a dynamic range and accuracy of measurement sensors - the ADC word length, frequency characteristics - the sampling frequency on analog inputs. If the signals with various characteristics have to be recorded on the same computer, then the ADC characteristics should get out from the most sensitive sensor and widest frequency band. The requirements to the personal computer on which the ADC is installed, depend from the ADC type. Modern ADC has the built-in timer, the big input buffer to data accumulation and fast co-processor. So this ADC can be installed even on the ordinary personal computers to use them as enough fast and effectively working data logging systems.

[47]  A number of specific problems appear in designing the data logging system software for continuous geophysical observations. The main task is exact synchronization of registered data with universal time. Usually precision GPS serve this purpose in the modern data logging systems. In particular, the UT synchronization via GPS is used by the equipments, operated in networks INTERMAGNET and SAMNET.

[48]  The main data logging system of measuring complex of Borok Geophysical Observatory, logging the data from all recording analog equipment, is synchronized with universal time by signals from precision quartz clocks AKV-2M. The quartz clocks provide a relative error pm 3cdot 10-9 and an opportunity to correct the current time by a radio signal with error 3 cdot 10-3 s. Because the sampling rate of the main data logging system is 10 Hz, this accuracy is quite enough. The program of registration uses time code signals generated by quartz clocks, coming to digital inputs of ADC. The durations of pulses, following with frequency 1 Hz, (the pulse front is corresponded to the beginning of the second), contains the information about the current date, hour and minute. The logging software decodes the time code and exposes sets the calculated current time to the computer clocks, providing the synchronization of the registered data with the universal time.

[49]  The method of interrogation of ADC input channels depends on the kind of ADC synchronization. The command on data reading can be generated by the logging program, then the logging program should trace the time, coming from computer clocks, continuously. Sometimes it can result in data loss because hardware delays. If ADC has the built-in timer, the data buffer and the opportunity to programming the mode of interrogation of input channels, then the command to interrogation of ADC input channels can be generated by the ADC timer pulses. Because the data collect in ADC buffer, the possible data loss is prevented even for enough long delays of data logging software. So, the main data logging system of measuring complex of Borok Geophysical Observatory is equipped with 16-digit and 16-channel ADC, having the built-in data buffer, provides the data preservation in the buffer during 1.6 s for sampling frequency 10 Hz. If there is a precision generator of time marks, a quartz clocks, for example, these marks can be used as pulses triggering the interrogation of ADC input channels. Such algorithm to interrogation of ADC channels is used in the main data logging system of measuring complex of Borok Geophysical Observatory.

[50]  Other specific task is the supporting of durable continuous working of data logging systems with an opportunity of data access. Thus there is necessary to transfer the raw data files from the data logging computer hard disk to another computer, making the data processing, storage and archiving. The raw data from the main data logging system are transferred to the data processing computer via the local network. If there is no opportunity to connect data logging system to the processing computer via a local access network, then the raw data is rewritten periodically on the replaceable media with enough volume. So, the data logging software of INTERMAGNET magnetic observatory writes the raw data on a diskette each hour. The data transfer from the magnetic observatory to the processing computer is made manually by replacement of a diskette in the disk drive (the diskette will be replaced by a compact flash memory stick within the next year, thus making this manipulation unnecessary). Thus data logging does not stop.

Figure 4
[51]  The simplified scheme of main units of the typical data logging program is presented in Figure 4. In the Core unit the current ADC data and the virtual timer are described as global variables. In the Store unit the procedures of initialization and data recording to the media or to the buffer directory, accessible via a local access network, is determined. The ADC unit contains procedures of clock start, and ADC data reading initialization. In the Display unit procedures of initialization and display of the current data plots, as well as procedure of input and processing of operator commands, are realized. In the Main unit the data logging algorithm is realized. The data logging program structure provides easy design and compilation of the program for the any data logging system. It is enough to modify the ADC unit to determine the ADC and synchronization kind, and Store unit to determine the data recording format. Thus the program logic and the operator interface, programmed in Display unit, are kept. The data logging program is written in Pascal language for free distributed interplatform FreePascal compiler that allows using the same source code to compile the program under DOS, Windows and Linux OS.

[52]  Each minute the data logging program writes all input data with the information about current time to the buffer directory on the hard disk of the data logging computer. The storage of the minute data files on the hard disk prevents the data loss caused by possible failures of a local access network or a diskette driver. The transfer of data files from the hard disk buffer directory to the other media or to the data processing computer via the data logging network is carried out each hour or each day. The buffer directory of data logging computer is accessible for reading only to guarantee the raw data safety.

Figure 5
[53]  All data logging systems, the data processing computers and the database server, served the geophysical database, are connected via the local access network. Thus the local data logging network, providing continuous registration and processing of geophysical data, is created (Figure 5). All computers, included in the data logging network of Borok Geophysical Observatory, work under the Linux OS (, communication between them is carried out by Linux software via the switchboard, functioning independently of the local access network and server of Borok Geophysical Observatory.

[54]  The Linux OS installed on the data logging network computers has the following features:

[55]  Continuous entering of the results of regular observations of middle-latitude geophysical fields to the geophysical database is provided by the main data logging system collected the data from all sensors of geophysical fields. The data logging system carries out the data transformation into the digital format, their initial accumulation and transfer to the data processing computer, logged also the data of SAMNET and INTERMAGNET magnetometers.

[56]  Processing the raw data coming from the data logging computers, including their formatting, averaging and transformation to the physical scale, is carried out on the data processing computer. Two main objectives of the processing are to transform the raw data to the digital format, convenient for data storage and access, and to obtain the data images, convenient for data viewing and analysis by users. So, there are two groups of programs. Formatting programs translate the data received from different sources, into the uniform storage format and store them in the database directories. Visualizing programs produce data plots and spectral-temporal diagrams, storing them as graphic files to allocate on the database web site.

[57]  Note, that Linux has the built-in script language Shell and the convenient task manager Cron, allowing to execute during the set moments the data processing scripts, describing all necessary steps to data processing, including running of corresponding programs and commands. Besides it is easy to create "pipeline'' of the programs, allows the data from the program output without storing in file go directly to the other program input. Thus, small programs, each aimed to the certain task, are executed sequentially, consistently transforming the data. Final files are created on the hard disk only in the end of the "pipeline'' of processing programs. Besides economy of resources, such technology allows to create the effective scripts to process the concrete raw data, based on the small programs, solving the concrete problem of data transformation.

[58]  The data processing computer, installed in the data logging network of Borok Geophysical Observatory, executes special programs for the processing of data, coming from the main data logging system, magnetic station SAMNET, automatic magnetic observatory INTERMAGNET and the digital meteorological station. All programs have the features. So, the program for the processing of main logging system data controls the time code, presented in data, finds skipping in the data series and fills them with the data missing indicator.

[59]  After the data pre-processing the programs to transform the data to the physical scales are executed. They transform the hour data files, making simultaneously data averaging and decimation, and store the text data files in database server directories, accessible via Internet.

[60]  The programs, providing the data visualization, read the data base files, process them and create corresponding plots and diagrams. As the database files do not contain time marks, the time is defined by the file name, describing the moment of registration beginning. The sample rate is calculated by the processing program trough the number of records in the hour data files.

[61]  The visualization of the data of registration of slowly varying parameters, such as geomagnetic field or air electric field, proceeds by the plots of their average values. This problem is solved by the set of programs transmitting the data to each other by the "pipeline''. The running average of the plotted data is calculated before the visualization. The observed data sets are processed by the plotting program, producing the amplitude-temporal plots.

[62]  The visualization of the data of registration of rapidly varying parameters, such as ultra low frequency geomagnetic pulsations, proceeds by the plots of the spectral-temporal diagrams. Thus the processing program gets the data sets and makes their transformation by the fast Fourier transform algorithm instead of the averaging. The visualizing program gets transformed data and makes the spectral-temporal diagrams. All graphic data received during data processing is stored as gif files. The file names have format YYYY_MM_DD_HH_CC.gif, where YYYY - year, MM - month, DD - day, HH - hour, CC - the channel number.

[63]  The database server provides access to the data via the Internet by the local access network of Borok Geophysical Observatory. The database web site, located on the server, contains database manuals, graphic files, data files, interface forms for data requests by database users.

Figure 6
[64]  The structure of the data storage directories of the data base server is presented on Figure 6. The data files contain the data from one channel, received during one hour. Names of hour data files have the format YYYY_MM_DD_ HH.CC, where YYYY is year (since 1997), MM is month number (01-12), DD is day of month (01-31), HH is hour (00-23), CC is channel number, which is the unique identifier for each registered parameter. The data file contains the sequence of two-byte records with registered data.

[65]  The descriptions of parameters of recorded fields and recording devices are stored in the data base configuration files, including phase-frequency and amplitude-frequency characteristics of input channels, dynamic ranges of the registered fields in ADC bits, coefficients to transform the data to physical scales. The binary format of the data storage (without transformation to physical scales) has a number of advantages. This is unification of the data storage format, the economy of hard disk volume, prevention of loss of the information during data transformations. Thus the archiving and processing programs, making the final data files for the database users, are special for each geophysical field.

[66]  The program, processing the data from the magnetic station SAMNET, is executed each day, transforming the data from SAMNET format to the database format. For the data received from SAMNET station, the channels with 16, 17 and 18 numbers are reserved in database. The sample rate in the hour files, corresponding to these channels, is 1 s, like in SAMNET data logging system. The programs, processing meteorological data operate similarly.

[67]  Each month the collected geophysical data, stored in the database, are copied on CD-ROM, acting like the basic media to archive database. Copying is made on the archiving computer, having the CD-ROM recorder.


Powered by TeXWeb (Win32, v.2.0).