Python Based Connector for Chunking Your File

Starting from version 2021, Origin has been introducing multiple enhanced integrations with the Python programming language.

One of the major difficulties that researchers meet regularly is that the raw data would come in from different sources/instruments with different formats/structures. And in order to perform the desired analysis routine, the data file needs to be imported and cleaned correctly to ensure all the following up processes.

We’ve already talked about how to set up an import filter that can be used in the classic Import Wizard in this post. Now let’s move one step further to load the filter by a Data Connector so that the datasheet will be dynamically linked with the file and will be automatically updated when the operator launches the project and will get all the latest updates.

The following example can be downloaded here.

Let’s say we have an oversized file that contains several data blocks, either defined by a certain tag line or fixed size or any other structures which will be handled by the Python code correspondingly. In order to process the data correctly, the operator need to split the file so that each block be in its own datasheet. Here we’ll use a simple example (body.dat) with 40 rows and split it by a fixed row size of 10, which will give us 4 sheets as result.

So first of all, we need to create such a filter. You will still need to do that through the import wizard the same way as the previous post. But this time let’s make it simper.

import originpro as op
import pandas as pd
    
    
if __name__ == "__main__":

    # The file chosen by Origin import filter is placed into the fname$
    # LabTalk variable. Must bring it into Python. Don't specify $ in name!
    fname = op.get_lt_str('fname')

    chunksize = 10    # we want each chunk to be size 10
    df = pd.read_table(fname, chunksize=chunksize)    # Read the file into a DataFrame.

    idx = 0
    for chunk in df:
        wks = op.imp.get_sheet(idx)    # This method tells the software which sheets are used for the importing and will be add connectors to them
        wks.from_df(chunk)
        idx = idx+1

In general, you can use your customized python file reading code to break the file down as pandas dataframes, and use the get_import_sheet function to create datasheets with the connector on them.

Once you got the filter file (*.oif), you can use it to set up the connector.

After picking the data file, if there be only one filter available in the same path, the software will be smart enough to automatically loading it. If not, it will prompt you to specify one.

You will have a four sheets book with connector icon setup on each of them.

Now the databook is dynamically linked with the outside file. Once the data be updated, those green icons on the upper-left corner will turn yellow and you can click it to refresh the sheet. You can also turn on the Auto Import to let the software do it for you.

That’s all I have for you guys today. Thanks for taking the time to read my post.