Creating a Python-based Import Filter with Origin 2021

Introduction

While it has always been possible to create a user defined Import Filter with Origin C, starting with Origin 2021 Beta 6, you can also create user defined import filters using Python.

In this blog post, I will show you how to do it based on a request from a user. It imports date, time, and message data from simulated log files. The files have the same structure as the user’s data but they aren’t the actual user files.

Details

Note: I have made all files used in this blog available in a downloadable ZIP file. Please decompress it to the location of your choosing

You use the Import Wizard to create a user defined Import Filter. It is not difficult to do, though it requires some different steps than those typically selected when using the Import Wizard.

Launch Origin and then open the Import Wizard and follow these steps.

Step 1 (Source page)

Per the illustration below, do these things on the Source page (first page) of the Import Wizard:

1. Select an example file to import.
2. Set Data Type to: User-Defined.
3. Set Target Window to: Worksheet.
4. Select Import Mode. For this example, select Start New Sheets.
5. Click Next button.

Step 2 (User Defined Filters page)

Per the illustration below, do these things on the User Defined Filters page (second page) of the Import Wizard:

1. Specify Python as the type of code used.
2. Decide if you want to specify your Python Code directly in the filter or in a Python File. In this case, it will be in the filter.
3. Enter your Python script. To view the actual script, scroll down after illustration.
4. Click Next button.

Note: if you decide to place the Python code in an external file, you can either choose:

• The absolute path to the file by clicking the … button.
• Enter the file name only to use a file in your User Files folder.
• Enter a path to a file relative to your User Files folder. For example: My Scripts/Import Filter.py.

Here is the annotated Python script (Note: I have supplied the annotated script in the downloadable ZIP file). Please read through it completely as it contains important information! Note: it uses pandas which should be installed beforehand.

import originpro as op
import pandas as pd
import re

def read_file(file):
    # Regex for splitting each row into 3 match groups (strings).
    p = re.compile(r'(\S+)\s+(\S+)\s+(.*)')
    rows = []
    with open(file, 'r') as f:
        line = f.readline()
        while line:
            m = p.match(line)
            if m:
                rows.append(m.groups())
            line = f.readline()
    df = pd.DataFrame(data=rows, columns=['Date', 'Time', 'Message'])
    df['Date'] = pd.to_datetime(df['Date'], format='%m/%d/%y')
    df['Time'] = pd.to_datetime(df['Time'], format='%H:%M:%S').dt.time
    return df


# The file chosen by Origin import filter is placed into the fname$
# LabTalk variable. Must bring it into Python. Don't specify $ in name!
fname = op.get_lt_str('fname')

# Read the file into a pandas DataFrame.
data = read_file(fname)

# Returns the active sheet.
wks = op.find_sheet()

# Add DataFrame to sheet starting at first column.
# DataFrame column labels will become worksheet column long names.
# Origin automagically sets column format based on DataFrame data types.
wks.from_df(data)

I have tried to make the what the code is doing as as clear as possible. But you will notice that I do use a Regular Expression and you are likely to use them yourself when creating a Python-based Import Filter . I’m not going to explain regular expressions- that is something you will have to learn for yourself.

For suggestions about developing an import script prior to creating an Import Filter (it’s much easier that way), see the addendum to this post.

Step 3 (Save Filters page)

Per the illustration below, do these things on the Save Filters page (third page) of the Import Wizard:

1. Check Save Filter.
2. Save the Filter in your User Files folder.
3. Give the filter a meaningful Filter file name. in this case “log_file_filter”.
4. Specify the data file name(s) if not already set.
5. Click Finish button.

That’s It- All Done!

That’s all it takes to create a Python-based user defined Import Filter.

I have included two data (log) files in the downloadable ZIP file for you to play with. If you have followed the steps above correctly, you should be able to drag and drop the files into Origin, and they will utilize the Import Filter to import the files.

Thanks for taking the time to read this post.

Addendum: Developing your Import Script

It is a good idea to develop and test your import script prior to trying to create your Import Filter. It will make your life easier!

You can do this by using Code Builder. Simply create a new Python file from the File menu, enter your script into the code editor window, and hit F5 to run the script.

You may need to make temporary modifications to emulate how the Import Filter will select a file. I adapted the script from the blog post by adding a few lines of code to perform the emulation. If you do it this way, simply remove these lines prior to creating the Import Filter.

import originpro as op
import pandas as pd
import re

def read_file(file):
    # Regex for splitting each row into 3 match groups (strings).
    p = re.compile(r'(\S+)\s+(\S+)\s+(.*)')
    rows = []
    with open(file, 'r') as f:
        line = f.readline()
        while line:
            m = p.match(line)
            if m:
                rows.append(m.groups())
            line = f.readline()
    df = pd.DataFrame(data=rows, columns=['Date', 'Time', 'Message'])
    df['Date'] = pd.to_datetime(df['Date'], format='%m/%d/%y')
    df['Time'] = pd.to_datetime(df['Time'], format='%H:%M:%S').dt.time
    return df


# These two lines will emulate the Import Filter file selection.
# Be sure to remove them prior to creating the Import Filter from this script.
tmp = op.file_dialog('*.log')
op.set_lt_str('fname', tmp)


# The file chosen by Origin import filter is placed into the fname$
# LabTalk variable. Must bring it into Python. Don't specify $ in name!
fname = op.get_lt_str('fname')

# Read the file into a pandas DataFrame.
data = read_file(fname)

# Returns the active sheet.
wks = op.find_sheet()

# Add DataFrame to sheet starting at first column.
# DataFrame column labels will become worksheet column long names.
# Origin automagically sets column format based on DataFrame data types.
wks.from_df(data)