File Uploads are used to submit activity data into Quarantine in CSR. This functionality allows data to be viewed as it will appear in the system and ensures that the activity data file contains no errors.

Users involved in the file upload process are assigned one or both of the following roles:

  • Data Source Owner (DSO): This is the person (user) responsible for collecting the activity data to upload to the system.  This user may also upload the activity data file to the system.
  • Data Submitter (DS): This is the person (user) responsible for uploading the activity data file to CSR, when this user is not the same as the DSO.

This article contains information on the following:

  • Uploading a File
  • Troubleshooting
  • File Upload Data Flow

Uploading a File

  1. Select File Uploads in the Data Collection menu. 
    • The File Upload page is shown below.
  2. Click the Select button.
    • This brings up a file explorer window to search for and select the activity data file.
    • Please note: the current file upload limit is 60,000 rows of data or 15MB.
  3. Select the configuration from the Choose a data file configuration dropdown (this tells the system how to read the file).
    • For help selecting the correct configuration:
      • Check the content of the task notification email received.
      • Contact the system administrator.
  4. Select the related task from the Choose the related data collection task dropdown.  This will be specified in the content of task notification email received.
  5. Add a reference to the Include a file upload reference? if required (this is not mandatory). If a reference is required this will be specified in the task notification email received.
  6. Select the No, this is not a test option in the Is this a test upload? dropdown.
  7. Click the Upload the file button.
      • While the activity data is being processes a processing message will display.
      • If the file is large the system will work on this in the background to allow you to continue with other work while the data is processed. See the Viewing Results of a File Upload section for more information.
    • A summary of the data you have uploaded, and a Comments and Approval section are displayed.
  8. Review the data summary and ensure that this matches the data in the data file.
    • If the data summary differs from the file click the Cancel This Uploaded Data button and refer to the Troubleshooting section below.
  9. Add a comment about the activity data uploaded and include the following;
    • Any checks performed, e.g., activity data matches file.
    • Explanations of any anomalies in the data, e.g., activity data is high due to a new site added.
    • Other information that may be useful for the person reviewing the activity data you submit. This will also assist with auditing.
  10. Click the Save Data to Quarantine button.

File Upload Summary Actions

If the activity data uploaded already has corresponding information in the system, prompts will be displayed with actions for each line item:

  • Overwrite existing data: On quarantine approval this removes existing information and replaces it with the new activity data being uploaded.
  • Add it to the existing data: On quarantine approval this adds the new activity data to the existing information in the system.
  • Ignore the new data: On quarantine approval this prevents the new activity data from being added to the system.

The default action is set when the task is created. In most instances this will be the action chosen, however this can be overridden by selecting another option from the dropdowns.

As part of the data summary review, it is essential to read the Comments and Existing Comments columns to determine which action is appropriate for each line item. 


Viewing the Results of a File Upload (coming soon)

The Data Source Owner, Data Submitter and Data Overseer can view the result of a file upload on the Upload Status page.

To view this page, go to Data Collection > Upload Status. This page contains the File Upload Status Table.

To view the uploaded file, select the Filename.

  1. This will display the File Upload Page with the Status and Method of file upload. There are three upload statuses:
    • Success: The file upload was successful and the data is waiting for DSO commentary and addition to Quarantine.
    • In Progress: Large files can take some time to upload.
      The system will work on this in the background to allow you to continue with other work while the data is processed.
      This status will change to 'Success' or 'Failed' once it has been read.
    • Failed: The file uploaded had some errors associated with it.
  2. The Method column displays how the file was uploaded; either using SFTP or File Uploads.
  3. Select the Filename to review the uploaded file on the File Uploads page.
    • If the 'Should data go into the Upload Status queue for DSO review before Quarantine?' is selected in the SFTP File Upload Configuration this will allow the DSO/DS to review the data before Quarantine.

Troubleshooting

No Task Listed in the Task Selection Dropdown

Tasks will only appear in the dropdown list if:

  • The task is assigned to the logged in user.
  • The task status is active.

If a task should appear and doesn't, contact the system administrator.

Errors on File Upload

If there are errors in the activity data file, an error message will be displayed at the top of the data summary. If this happens, cancel the upload and fix the error in the file. Common errors may include:

  • Activity data file format has changed and no longer matches the Data Configuration.  
  • Activity data file has hidden tabs and the Data Configuration is set to read the hidden tab.  To check this, in Excel right click on the tab name and choose View Code.  Check the position of the tab that contains the activity data.
  • Date(s) or other data in incorrect format, e.g., text in the column containing activity data units.
  • Spaces at the start or end of identifying information used by the Data Configuration and Look-up Lists.
  • Locations and/or inventory naming in the file does not match the values in the Look-up Lists being used by the Data Configuration.
  • Locations and/or inventory in the activity data file do not matching existing items in the system.  Naming conventions in the activity data file have changed or the location/inventory is 'new'.
  • Where an ICP or meter number in the Connection Point Identifier inventory field is used as the location and/or inventory identifier (normally for electricity, gas or water), the number does not match an existing item.

To get more information about the error, test uploads with varying levels of detail can be performed.

To do a test upload:

  1. Follow the Uploading a File steps 1 - 5 (refer above).
  2. Select one of the test options in the Is this a test upload? dropdown.
    • Yes, with no analysis: This will display the same error received on the original upload. This message contains basic information about the error.
    • Yes, with minimal analysis: This will display the same information that is displayed in the ‘Yes, with no analysis’ option as well as a line by line analysis of each row processed in the file.
    • Yes, with maximum analysis: This will display the same information that is displayed in the ‘Yes, with minimal analysis’ option as well as full row-by-row data extraction.

Depending upon the type of error, updates may be needed to the Data ConfigurationLook-up ListsInventory or Locations.  This knowledge base contains articles to help with this.  If there are still issues, please email [email protected].

Data Declined from Quarantine

If the data you submit is not accepted by the data overseer (DO), you will receive an email notification. The email will contain commentary from the overseer as to why the data was declined. You must amend the file as per the comments and reupload it. 

If you have any questions, please contact the data overseer to discuss the file. You can find the DOs email address in the original task notification email.


File Upload Data Flow

The purpose of data collection task functionality is to:

  • Manage the associated workflow, and
  • Facilitate the notification, collection and subsequent review of a file's content.

This functionality is also utilised for Quarantine, data sources and Data Reviews.

When an activity data file is uploaded it is placed in Quarantine where it is reviewed and approved.  Until approval is completed the activity data will not be added to the system.  Remember that activity data can also be declined during this step.

Full audit trails of this process are generated automatically.

Refer to the related articles for help.