Skip to main content

File from URL datasource

The File from URL datasource allows you to fetch and import data from files hosted on remote servers. This datasource is ideal for accessing CSV, JSON, XML, or text files that are regularly updated on external websites or services.

Prerequisites​

Before creating a File from URL datasource:

  • Ensure the target URL is publicly accessible or properly authenticated
  • Verify the file format is supported (CSV, JSON, XML, TXT, TSV)
  • Confirm network connectivity to the remote server
  • Check that the file size is reasonable for regular updates

Supported URL Types​

  • HTTP/HTTPS URLs: Standard web-accessible files
  • File formats: CSV, JSON, XML, TXT, TSV
  • Authentication: Basic HTTP authentication supported
  • File size: Recommended maximum 50MB per file

Create datasource​

Go to Datasources > Click Add new in the External datasource tab > Choose the File from URL type.

Basic Configuration​

1. Update Schedule: Choose how your datasource refreshes from the source:

  • Refresh Frequency: Set regular intervals (every 5 minutes, hourly, daily)
  • Cron Expression: Specify custom scheduling using cron syntax

2. URL Field: Enter the complete URL to your file

Examples:
  • https://example.com/data/sales.csv
  • https://api.service.com/export/users.json
  • https://feeds.example.org/data.xml

3. File Format Detection: The system automatically detects file format based on:

  • File extension in the URL
  • Content-Type headers from the server
  • File content analysis

Advanced Settings​

4. Datasource Options:

  • Active: Enable/disable the datasource
  • Ignore Error counter: Continue operation despite occasional fetch failures

5. Processing Options:

  • Cache external resources: Store fetched files locally for faster access
  • Remove broken external resource references: Clean up invalid file references
  • Rotate cache on every update: Clear cache with each refresh to ensure fresh data
  • Exchange internal resource references: Update internal file links automatically

6. Request Settings: Configure HTTP request parameters including:

  • Custom headers for authentication
  • User agent strings
  • Timeout settings
  • SSL verification options

7. Data Processing:

  • Randomize arrays: Shuffle array data on each update (useful for rotating content)

Authentication​

For URLs requiring authentication:

  • Use the Request settings to add authorization headers
  • HTTP Basic Authentication is supported through URL format: https://username:password@example.com/file.csv
  • Bearer tokens can be added via custom headers

Error Handling​

The datasource includes built-in error handling for:

  • Network connectivity issues
  • Invalid URLs or missing files (404 errors)
  • File format parsing errors
  • Server timeouts
tip

Enable Ignore Error counter to prevent temporary network issues from disabling your datasource.

Completion​

8. Click Save to create your File from URL datasource.

After saving, the system will:

  • Validate the URL accessibility
  • Perform an initial file fetch
  • Parse and validate the file format
  • Schedule regular updates according to your configuration
info

Monitor the datasource status in the Datasources list to ensure successful operation. Initial setup may take a few minutes for the first file fetch and processing.

For troubleshooting common issues, see the File from URL troubleshooting guide.