Come up with a design for plugins

[asa1997]

[bananth2008]

This is the original high level design submitted -

While the diagram captures
a. Separation of Roles: BeSMAn, BLIMan
b. Central Store: Plugin Store (need more clarity on this store @asa1997 )
c. Reusable Tools: e.g. SonarQube and Garak can be used by playbooks across different POI

The point of confusion is
a. Shouldn't gitlab and BeSLighthouse be part of infra - they are The Lab which BLIMan installs.
b. Sonarqube as a plugin is fine as it is a server application and has permamnence but garak and others are command line tools. Shouldn't they be installed in the temporary env using BeSLab and then destroyed when the assessment and the env is destroyed. I mean should they really persist? (permanent infrastructure/services and temporary tools )?
c. are we planning to run multiple assessments in the same environment - or its one envc /assessment and then destroy the env once the assesment is done?

@panickervinod, can you please help bring a clear distinction here between a plugin and the NOT plugin? Can you help clarify points b & c above?

[panickervinod]

@bananth2008 @asa1997 The decision to choose a BeSPlugin is based on the need for reuse and sharing across multiple instances of BeSEnvironment, or for handling resource-intensive tasks that cannot be performed on a security analyst's machine. A BeSPlugin is a type of BeSEnvironment with additional capabilities to publish its endpoints. These endpoints can be invoked from a BeSPlaybook as an API call, and the plugin responds with the execution result, which is then used to prepare the final OSAR (open source assessment report). Environments specific to security activities are suitable candidates for BeSPlugins. Once deployed in a lab, a BeSPlugin will have exposed endpoints listed. If a plugin's endpoint is not accessible, the BeSPlaybook will attempt to install the security tool in the environment.

[bananth2008]

As per the discussion - A plugin is justified if it's either a shared service used across many projects or a proxy for a resource-intensive or licensed tool.
a. As a shared Service -A tool should be a plugin if it's going to be used across multiple projects or models, especially as part of an automated administrative process. The idea is to treat the plugin as a centralized capability of the lab rather than a single-use script.
b. As a proxy - A plugin is ideal for tools that are resource-intensive (e.g., "model bench"), require a special license, or have complex access requirements. The plugin acts as a trusted gateway within the lab. A user can trigger the action with a simple command, and the plugin handles the actual permissioned or heavy execution in the secure lab environment.

[bananth2008]

2. Assessment Workflow Scenarios

The design accommodates different types of assessment workflows, from simple, immediate scans to complex, long-running jobs.

Scenario 1: Standard (Synchronous) Assessment
This is the straightforward workflow used for fast-running tools where results are returned immediately.

Use Case: Best for linters, simple vulnerability scanners, or any tool that completes its job in seconds or a few minutes.

Workflow:

The analyst executes a playbook.

The tool runs and completes its analysis within the same session.

The playbook receives the results directly.

A final OSAR (Open Source Assessment Report) is generated with a "Done" status and the complete results in a single, uninterrupted step.

Scenario 2: Asynchronous Long-Running Assessment

This workflow is essential for resource-intensive or time-consuming tools that cannot return a result immediately.

Use Case: Necessary for deep source code analysis, extensive fuzzing, or security scans on large Language Models (LLMs) with tools like Garak.

Part A: Dispatching the Job
The playbook sends an assessment request to a central plugin service.

The service acknowledges the request, queues the job for execution, and immediately returns a unique Request ID.

The playbook execution finishes, creating an initial OSAR that acts as a receipt. This OSAR contains:

status: In Progress

Request ID: The unique ID from the service.

All relevant metadata (Project Name, User ID, Timestamp).

An empty or null results section.

Part B: Checking the Result (Polling)
With the job running on the central service, the analyst is free to destroy the temporary assessment environment and perform other work.

Later, to check the job's status, the analyst re-instantiates the same environment and runs the same playbook again.

The playbook’s logic is designed to be smart: It detects the existing "In Progress" OSAR for this task. It then uses the stored Request ID to query the central plugin service for the job's current status.

The playbook handles the service's response accordingly:

If the status is still "In Progress": The playbook execution ends, leaving the OSAR unchanged. The report continues to act as a receipt for the ongoing job.

If the status is "Complete": The service also returns the final report data. The playbook updates the OSAR with these results and changes the status to "Done".

If the status is "Failed": The playbook updates the OSAR with the relevant error information and changes the status to "Failed".