Understanding Schedulers: Jobs, Batches, and Flow Execution
The Scheduler is a core component of Wiresk, designed to automate the retrieval and processing of data from connected applications. Users can automate data processing by setting schedules for their Flows, eliminating the need for manual initiation.
Functionality Copy Link
When a Flow is scheduled in Wiresk, the Scheduler performs the following actions:
- Data Retrieval: At the designated time, the Scheduler queries the connected application to fetch relevant data. For instance, if you’re working with a CRM system like Odoo and have a query for all contacts named “John,” the Scheduler will retrieve all matching records.
- Job Creation: Each retrieved data item is treated as an individual job. Continuing with the Odoo example, if 10 contacts named “John” are found, the Scheduler will create 10 separate jobs, each corresponding to one contact.
- Flow Execution: Each job is processed independently by the associated Flow. This means the Flow is executed once for each data item, ensuring isolated and reliable processing.
Data Processing Approach Copy Link
Wiresk uses a dual strategy for data processing to balance performance and reliability.:
- Batch Retrieval: The Scheduler retrieves data in medium-sized batches (e.g., 100 or 1000 records at a time) from the source application. This approach balances efficiency with system resource utilization.
- Individual Processing: Despite batch retrieval, each data item is processed individually. This design ensures that if one item encounters an error during processing, it does not affect the processing of other items. Such isolation enhances the robustness of the automation process.
Mapping and Sample Data Copy Link
When configuring a Flow, users can utilize the “Use (formerly Get Sample)” feature to retrieve a sample data item. This sample represents the structure and content of individual data items that the Flow will process. Users can then map fields based on this sample, and Wiresk will apply the same mapping to all subsequent data items processed by the Flow. This approach optimizes the setup process and ensures consistent data handling across all jobs.
Scheduling a Flow Copy Link
To schedule a Flow in Wiresk:
- Create a Flow: Design a Flow that specifies the sequence of actions to be performed on each data item. Detailed instructions on creating a Flow can be found in the How to Create a Flow in Wiresk guide.
- Set the Schedule: Define the frequency and timing for the Flow execution. This can be done by configuring the Scheduled Trigger settings within the Flow. For guidance on running and scheduling Flows, refer to the How to Run a Flow documentation.
- Monitor Execution: Once scheduled, the Flow will execute at the specified intervals. Users can monitor the execution status, review logs, and handle any errors through Wiresk dashboard.
Error Handling and Optimization Copy Link
Wiresk’s design ensures that each data item is processed independently, which provides several advantages:
- Error Isolation: If an error occurs while processing a specific data item, it does not impact the processing of other items. This isolation prevents a single point of failure from disrupting the entire batch.
- API Compatibility: Many external APIs handle requests individually. By aligning with this behavior, Wiresk ensures smooth integration and reduces the likelihood of encountering API limitations related to batch processing.
- Data Consistency: In scenarios where data consistency is critical (e.g., creating an invoice with multiple line items), Wiresk can handle grouped data appropriately, ensuring that either all related data is processed successfully or none at all, maintaining data integrity.
