Service Overview
In many data-intensive scientific fields, access to large datasets in proximity to the application environment for high-performance access is crucial. Bulk Data Transfer methods provide efficient solutions tailored to the need of transferring data close to execution environment. This service simplifies the migration, automation, and monitoring of data transfer workflows while preserving existing client-side configurations for authentication, access, and firewalls at origin. Powered by CERN’s robust File Transfer Service (FTS) which is responsible for distributing most of the Large Hadron Collider’s data across the Worldwide LHC Computing Grid (WLCG) infrastructure, you know your data is in capable hands.
By leveraging Bulk Data Transfer solutions, you can safely and securely move large data to the EOSC EU Node infrastructure services where applications with high Input/Output Operations Per Second (IOPS) requirements can be executed on the data.
Benefits
| Efficient data transfer | IOPS (Input/Output Operations Per Second) performance | Seamless data handling |
Features
- Ability to orchestrate and monitor the transfer of large datasets in automatic mode using “fire and forget” methodology.
- Ability to transfer large datasets between different endpoints using different protocols.
- Easy and simple (AAI or personal certificates) authentication.
- Easy automation and integration with third-party software via FTS RESTful API.
Service Capabilities
|
Data transfer between different storage endpoints |
Advanced data migration tool to transfer data between storage endpoints like cloud services, on-premises servers, and hybrid environments, ensuring data integrity and minimal downtime |
|
Web-based interface and monitoring |
User-friendly interface with real-time analytics, system health checks, and customisable dashboards for efficient storage management. |
|
RESTfull API for easy integration |
Robust RESTful API that facilitates easy integration with existing systems and applications, enabling automated workflows, seamless data exchange |
How it works
The data transfer process starts with the user specifying source and destination endpoints via a web interface or RESTful API. The system authenticates the user using AAI or personal certificates, then the File Transfer Service (FTS) orchestrates the transfer using advanced protocols like GridFTP. Real-time monitoring and logging are provided throughout the transfer, and upon completion, the system verifies data integrity and notifies the user of any discrepancies. The transferred data is then available at the destination endpoint, ready for high-performance access, seamlessly integrating with existing configurations.