Workflow System External Integrations

This page documents the technical implementation details of integrations with external workflow systems and repositories.

Integration Architecture

LabID's workflow system supports multiple external integrations through a modular architecture:

• Git Repository Integration: Direct import from Git repositories (public and organizational)
• WorkflowHub Integration: Bidirectional workflow exchange using TRS API and RO-Crate format
• Galaxy Integration: Workflow and invocation import using BioBlend API

Common Integration Patterns

All integrations follow similar patterns:

1. Authentication: SSH keys (Git), API keys (WorkflowHub), API tokens (Galaxy)
2. Data Models: PublishedWorkflow model tracks external references
3. Import Process: External metadata → LabID models → Git storage
4. Export Process: LabID models → External format → API submission

Git Repository Integration

Technical Implementation

Git repository integration uses the system Git command with custom SSH configuration:

• Storage: Local Git repositories using UUID-based directory structure
• Authentication: SSH keys managed through DJANGO_GIT_SSH_CONFIG_FILE
• Command Execution: Custom SSH wrapper script for repository access

Supported Repository Types

Based on the codebase implementation:

Type	Protocol	Authentication	Implementation
Public	https://, git+ssh://	None	Direct Git clone
Organizational	git+ssh:// only	SSH keys	Technical account access

API Endpoints

From the actual URL patterns and ViewSet actions:

• Import: /api/v2/workflows/workflows/new-from-remote/ - Git repository import

Request Parameters for Git Import

Based on ImportWorkflowSerializer and new_from_remote action:

{
    "name": "Workflow Name",
    "remote_repository": "git@github.com:org/repo.git",
    "workflow_manager": "SNAKEMAKE|NEXTFLOW|GALAXY|CWL|WDL|AIRFLOW|OTHER",
    "versions": [  # Optional
        {
            "commit_hash": "abc123",
            "name": "Version 1.0",
            "files": [
                {"file_path": "Snakefile", "data_type": "MAIN"},
                {"file_path": "config.yaml", "data_type": "OTHER"}
            ]
        }
    ]
}

WorkflowHub Integration

TRS API Implementation

The WorkflowHub integration implements the GA4GH Tool Registry Service (TRS) specification through the query_trs endpoint.

API Endpoints

From actual URL patterns in workflows/urls.py:

Endpoint	Method	Purpose	Implementation
/query_trs/	GET	Search workflows via TRS API	api.query_trs function
/api/v2/workflows/workflows/import-rocrate/	POST	Import RO-Crate from WorkflowHub	WorkflowViewSet.import_rocrate action
/search_workflowhub_projects/	GET	Search WorkflowHub teams/projects	api.search_workflowhub_projects function

TRS Query Parameters

From query_trs function implementation:

• query: Search query string
• endpoint: WorkflowHub endpoint URL (default from DEFAULT_WORKFLOWHUB setting)
• Standard TRS parameters: toolClass=Workflow, format=json

RO-Crate Import Parameters

From import_rocrate action:

{
    "url": "https://dev.workflowhub.eu/workflows/123/ro_crate?version=1",
    "workflow_id": "existing-workflow-id"  # Optional, for updating existing workflow
}

Data Models

PublishedWorkflow Model

From actual model definition in workflows/models/__init__.py:

class PublishedWorkflow(SoftDeletable, Ownable, NameableAbstractBaseModel):
    url = CustomCharField(max_length=2000)  # URL to published workflow
    workflowrepository = ForeignKey(WorkflowRepository)
    workflowversion = ForeignKey(WorkflowVersion)
    visibility = ChoiceField(choices=["PUBLIC", "INTERNAL", "PRIVATE"])

WorkflowRepository Model

Configuration stored in DEFAULT_PUBLIC_WORKFLOWREPOSITORIES setting:

{
    "https://dev.workflowhub.eu": {
        "label": "WorkflowHub (dev)",
        "url": "https://dev.workflowhub.eu",
        "api_url": "https://dev.workflowhub.eu",
        "trs": {
            "info": "https://dev.workflowhub.eu/ga4gh/trs/v2/service-info",
            "search": "https://dev.workflowhub.eu/ga4gh/trs/v2/tools"
        }
    }
}

WorkflowHub Publishing

From WorkflowVersionViewSet.publish_to_workflowhub action:

Publishing Endpoint

• URL: /api/v2/workflows/workflowversions/{id}/publish_to_workflowhub/
• Method: PUT
• Requirements: WorkflowVersion must be committed (is_committed=True)

Publishing Parameters

{
    "project": {
        "value": {
            "name": "Project Name",
            "url": "project-id-or-url"
        }
    },
    "type": "full|minimal"  # Default: "full"
}

User Credential Management

WorkflowHub credentials are stored encrypted in UserProfile.attributes:

• Storage: UserProfile.attributes['workflowhub'][workflow_hub_url]
• Fields: key, name, email
• Encryption: Cryptographic key generated on first server start

User Configuration Endpoint

From UserViewSet.workflowhub_config action:

• URL: /api/v2/users/{id}/workflowhub_config/
• Methods: GET, PUT, DELETE
• Parameters: url (WorkflowHub instance), key (API key)

Galaxy Integration

BioBlend API Implementation

Galaxy integration uses the BioBlend Python library for API communication with configured Galaxy instances.

API Endpoints

From actual ViewSet actions:

Endpoint	Method	Purpose	Parameters
/api/v2/workflows/workflows/import_from_galaxy/	POST	Import Galaxy workflow	url (Galaxy workflow URL)
/api/v2/workflows/workflowruns/import_from_galaxy/	POST	Import workflow invocation	url, create_datasets

Configuration

Galaxy endpoints configured in settings:

• DJANGO_GALAXY_URL: Primary Galaxy instance
• DJANGO_GALAXY_ENDPOINTS: Additional Galaxy endpoints

From settings/base.py:

GALAXY_ENDPOINTS = {
    "https://usegalaxy.eu": {
        "url": "https://usegalaxy.eu",
        "label": "Galaxy Europe",
        "is_default": False,
        "data_library_sync": False,
        "import_workflows": True,
    },
    "https://usegalaxy.org": {
        "url": "https://usegalaxy.org",
        "label": "Galaxy Main",
        "is_default": False,
        "data_library_sync": False,
        "import_workflows": True,
    },
    # ... other instances
}

Data Models

Workflow Import Process

From WorkflowManager.import_from_galaxy:

1. Workflow URL Validation: Supports Galaxy API URLs and published workflow URLs
2. BioBlend Integration: Uses gi.workflows.show_workflow() and gi.workflows.export_workflow_dict()
3. Model Creation: Creates Workflow, WorkflowVersion, and PublishedWorkflow records

Galaxy Annotation Storage

Galaxy-specific metadata stored in workflow annotations:

• Annotation Type: "galaxy_annotation_object"
• Storage: JSON format containing Galaxy workflow/invocation metadata
• Fields: Galaxy IDs, URLs, version information

Workflow Invocation Import

From WorkflowRunViewSet.import_from_galaxy:

Supported URL Formats

• https://usegalaxy.eu/workflows/invocations/{invocation_id}
• Galaxy WorkflowInvocation API URLs

Import Process

1. Extract invocation ID from URL
2. Fetch invocation details via BioBlend
3. Create or link existing Workflow/WorkflowVersion
4. Create WorkflowRun with Galaxy annotation
5. Optionally create dataset associations

Dataset Association Logic

Input datasets associated when: 1. Dataset exists in LabID with GalaxyAnnotation 2. Galaxy dataset ID matches annotation 3. Dataset was previously synced from same Galaxy instance

Related Documentation

• User Guide: Workflow Integrations - End-user workflows
• Admin Guide: Workflow Configuration - Setup and configuration
• SSH Keys Configuration - SSH key setup for Git access