Exclude imported data when saving project

With ever increasing data size, users often face the challenge on how to keep their Origin Project File (OPJU) size smaller. If you opt to import and keep all of the data inside your project, then of course the project file size becomes large if your data size is large.

If the data file is available for future use at a reliable location, one option is to NOT save the data with the project, and simply re-import the data when needed, to perform any future analysis.

With this in mind, in Origin version 2019b we have introduced an option to NOT save imported data in the workbook when saving the project (OPJU), or saving the workbook by itself (OGWU). This option is available in the Workbook Properties dialog:

It is important to note that all outputs from analysis (such as calculating values in new columns, regression, statistics etc) performed on the imported data will be saved with the project, and will be available for further graphing and analysis even though the imported raw data is not saved.  This allows you to re-open the project and work with the analysis results. You only need to re-import the data if you want to change existing analysis results, or perform new analysis on the raw data. Of course if you had created any graphs directly from the raw data, such graphs will be empty until you re-import the data.

In Origin 2019b, we have also introduced the concept of a “data connector” for maintaining a connection with either a file on disk or a web file. Data Connectors are available for file formats such as JSON and CSV. When a data connector is used and the file size is larger than 500kB in size, the check box to exclude the imported data when saving project is checked by default.

Modify default behaviors using these LabTalk system variables:

• @DNS: Controls whether data are excluded.
  • 0 = For data files, check @FSE; for web data check @FSEW
  • 1 = Always save imported data with the project
  • 2 = For data files, check @FSE; for web data always save with project
• @FSE: Controls threshold file size in kilobytes, for excluding data when saving the project.
• @FSEW: Controls threshold size in number of Origin worksheet cells x 100 for excluding Web, JSON, Origin, MATLAB data upon save.

Here is an example of using the CSV Data Connector to fetch data from the web, perform some analysis, and then save the project by excluding the imported data:

1) Start with an empty worksheet, select menu Data:Connect to Web…. Copy the URL below to the clipboard. And in the popped-up dialog, paste the  URL into the URL field, click OK.

https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv

2) In the CSV Import Options dialog, make the following settings, click OK.

3) A smart hint message will show reminding you that by default the imported data will be excluded when saving project.

4)  After the data is imported, you should be able to see a data connector icon at top-left of the worksheet. We will use that to re-import data later.

5) We now perform some analysis on the imported data: Highlight column D 、 E, select from menu Statistics: Descriptive Statistics: Statistics on Columns.. Set Group column to Type in the dialog. Click OK to generate the result sheet.

6) Now we save the project file and notice the file size is much smaller.  Open the saved project file, the worksheet with data is empty now for sure. While the statistics result worksheet still contains the previous analysis outputs.  Click the data connector icon of the data worksheet and select Import, the data will get re-imported.

More complex and larger  CSV data files can be found from links below:

https://vincentarelbundock.github.io/Rdatasets/csv/carData/MplsStops.csv

https://vincentarelbundock.github.io/Rdatasets/csv/gamclass/FARS.csv