Skip to content

DNM - connection/aw_sss - Use Port forwarding session for file transfer instead of S3 bucket #2265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

abikouo
Copy link
Contributor

@abikouo abikouo commented Mar 10, 2025

SUMMARY

We are implementing the file transfer using a Port forwarding session to improve the velocity of the aws_ssm connection plugin.

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

connection/aws_ssm

@abikouo abikouo force-pushed the ssm_file_transfer_20250310 branch from fb41f29 to 08b355f Compare March 10, 2025 17:24
Copy link

github-actions bot commented Mar 10, 2025

Docs Build 📝

Thank you for contribution!✨

The docsite for this PR is available for download as an artifact from this run:
https://github.com/ansible-collections/community.aws/actions/runs/14329623147

You can compare to the docs for the main branch here:
https://ansible-collections.github.io/community.aws/branch/main

File changes:

  • M collections/community/aws/aws_ssm_connection.html
Click to see the diff comparison.

NOTE: only file modifications are shown here. New and deleted files are excluded.
See the file list and check the published docs to see those files.

diff --git a/home/runner/work/community.aws/community.aws/docsbuild/base/collections/community/aws/aws_ssm_connection.html b/home/runner/work/community.aws/community.aws/docsbuild/head/collections/community/aws/aws_ssm_connection.html
index b62a732..94ffd9e 100644
--- a/home/runner/work/community.aws/community.aws/docsbuild/base/collections/community/aws/aws_ssm_connection.html
+++ b/home/runner/work/community.aws/community.aws/docsbuild/head/collections/community/aws/aws_ssm_connection.html
@@ -245,6 +245,22 @@ see <a class="reference internal" href="#ansible-collections-community-aws-aws-s
 </div></td>
 </tr>
 <tr class="row-odd"><td><div class="ansible-option-cell">
+<div class="ansibleOptionAnchor" id="parameter-host_port_number"></div><p class="ansible-option-title" id="ansible-collections-community-aws-aws-ssm-connection-parameter-host-port-number"><strong>host_port_number</strong></p>
+<a class="ansibleOptionLink" href="#parameter-host_port_number" title="Permalink to this option"></a><p class="ansible-option-type-line"><span class="ansible-option-type">integer</span></p>
+<p><em class="ansible-option-versionadded">added in community.aws 10.0.0</em></p>
+</div></td>
+<td><div class="ansible-option-cell"><p>The Port number of the server on the instance when using Port Forwarding Using AWS System Manager Session Manager to transfer files from/to local host to/from remote host.</p>
+<p>The port <code class="ansible-value docutils literal notranslate"><span class="pre">80</span></code> is used if not provided.</p>
+<p>The <code class="docutils literal notranslate"><span class="pre">nc</span></code> command should be installed in the remote host to use this option.</p>
+<p>This is not supported for Windows hosts for now.</p>
+<p class="ansible-option-line"><strong class="ansible-option-default-bold">Default:</strong> <code class="ansible-option-default docutils literal notranslate"><span class="pre">80</span></code></p>
+<p class="ansible-option-line"><strong class="ansible-option-configuration">Configuration:</strong></p>
+<ul class="simple">
+<li><p>Variable: ansible_aws_ssm_host_port_number</p></li>
+</ul>
+</div></td>
+</tr>
+<tr class="row-even"><td><div class="ansible-option-cell">
 <div class="ansibleOptionAnchor" id="parameter-instance_id"></div><p class="ansible-option-title" id="ansible-collections-community-aws-aws-ssm-connection-parameter-instance-id"><strong>instance_id</strong></p>
 <a class="ansibleOptionLink" href="#parameter-instance_id" title="Permalink to this option"></a><p class="ansible-option-type-line"><span class="ansible-option-type">string</span></p>
 </div></td>
@@ -255,6 +271,21 @@ see <a class="reference internal" href="#ansible-collections-community-aws-aws-s
 </ul>
 </div></td>
 </tr>
+<tr class="row-odd"><td><div class="ansible-option-cell">
+<div class="ansibleOptionAnchor" id="parameter-local_port_number"></div><p class="ansible-option-title" id="ansible-collections-community-aws-aws-ssm-connection-parameter-local-port-number"><strong>local_port_number</strong></p>
+<a class="ansibleOptionLink" href="#parameter-local_port_number" title="Permalink to this option"></a><p class="ansible-option-type-line"><span class="ansible-option-type">integer</span></p>
+<p><em class="ansible-option-versionadded">added in community.aws 10.0.0</em></p>
+</div></td>
+<td><div class="ansible-option-cell"><p>Port number on local machine to forward traffic to when using Port Forwarding Using AWS System Manager Session Manager to transfer files from/to local host to/from remote host.</p>
+<p>An open port is chosen at run-time if not provided.</p>
+<p>The <code class="docutils literal notranslate"><span class="pre">nc</span></code> command should be installed in the remote host to use this option.</p>
+<p>This is not supported for Windows hosts for now.</p>
+<p class="ansible-option-line"><strong class="ansible-option-configuration">Configuration:</strong></p>
+<ul class="simple">
+<li><p>Variable: ansible_aws_ssm_local_port_number</p></li>
+</ul>
+</div></td>
+</tr>
 <tr class="row-even"><td><div class="ansible-option-cell">
 <div class="ansibleOptionAnchor" id="parameter-plugin"></div><p class="ansible-option-title" id="ansible-collections-community-aws-aws-ssm-connection-parameter-plugin"><strong>plugin</strong></p>
 <a class="ansibleOptionLink" href="#parameter-plugin" title="Permalink to this option"></a><p class="ansible-option-type-line"><span class="ansible-option-type">string</span></p>

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/f9f3c70568fc4e319e5a616a2c98f54c

✔️ ansible-galaxy-importer SUCCESS in 3m 18s (non-voting)
✔️ build-ansible-collection SUCCESS in 10m 46s
✔️ ansible-test-splitter SUCCESS in 4m 28s
✔️ integration-community.aws-1 SUCCESS in 23m 41s
✔️ integration-community.aws-2 SUCCESS in 14m 24s
✔️ integration-community.aws-3 SUCCESS in 14m 39s
✔️ integration-community.aws-4 SUCCESS in 15m 07s
✔️ integration-community.aws-5 SUCCESS in 15m 05s
✔️ integration-community.aws-6 SUCCESS in 15m 50s
✔️ integration-community.aws-7 SUCCESS in 14m 58s
✔️ integration-community.aws-8 SUCCESS in 15m 11s
✔️ integration-community.aws-9 SUCCESS in 15m 05s
✔️ integration-community.aws-10 SUCCESS in 5m 39s
integration-community.aws-11 RETRY_LIMIT in 1m 55s
Skipped 11 jobs

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/8a77ca98a51740729c63fc79b65e1ef1

ansible-galaxy-importer FAILURE in 5m 08s (non-voting)
✔️ build-ansible-collection SUCCESS in 10m 21s
✔️ ansible-test-splitter SUCCESS in 4m 15s
integration-community.aws-1 FAILURE in 7m 21s
integration-community.aws-2 FAILURE in 27m 22s
integration-community.aws-3 FAILURE in 25m 53s
integration-community.aws-4 FAILURE in 29m 11s
integration-community.aws-5 FAILURE in 30m 52s
integration-community.aws-6 FAILURE in 27m 27s
integration-community.aws-7 FAILURE in 28m 42s
integration-community.aws-8 FAILURE in 27m 06s
integration-community.aws-9 FAILURE in 27m 50s
✔️ integration-community.aws-10 SUCCESS in 4m 11s
integration-community.aws-11 FAILURE in 24m 17s
Skipped 11 jobs

Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/c2052a4271914ac0a003325b934edc39

ansible-galaxy-importer FAILURE in 4m 36s (non-voting)
✔️ build-ansible-collection SUCCESS in 10m 52s
✔️ ansible-test-splitter SUCCESS in 3m 58s
✔️ integration-community.aws-1 SUCCESS in 24m 02s
✔️ integration-community.aws-2 SUCCESS in 13m 31s
✔️ integration-community.aws-3 SUCCESS in 16m 08s
✔️ integration-community.aws-4 SUCCESS in 18m 34s
✔️ integration-community.aws-5 SUCCESS in 14m 31s
✔️ integration-community.aws-6 SUCCESS in 16m 13s
✔️ integration-community.aws-7 SUCCESS in 17m 31s
✔️ integration-community.aws-8 SUCCESS in 17m 48s
✔️ integration-community.aws-9 SUCCESS in 14m 11s
✔️ integration-community.aws-10 SUCCESS in 7m 52s
✔️ integration-community.aws-11 SUCCESS in 15m 03s
Skipped 11 jobs

@abikouo abikouo force-pushed the ssm_file_transfer_20250310 branch 2 times, most recently from ce819ea to e546ce9 Compare March 12, 2025 15:48
Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/b5b6bb970d81417087c0852b634df06f

✔️ ansible-galaxy-importer SUCCESS in 5m 15s (non-voting)
✔️ build-ansible-collection SUCCESS in 10m 33s
✔️ ansible-test-splitter SUCCESS in 3m 58s
✔️ integration-community.aws-1 SUCCESS in 24m 03s
✔️ integration-community.aws-2 SUCCESS in 18m 50s
✔️ integration-community.aws-3 SUCCESS in 16m 21s
✔️ integration-community.aws-4 SUCCESS in 17m 08s
✔️ integration-community.aws-5 SUCCESS in 30m 15s
✔️ integration-community.aws-6 SUCCESS in 14m 11s
✔️ integration-community.aws-7 SUCCESS in 18m 21s
✔️ integration-community.aws-8 SUCCESS in 16m 09s
✔️ integration-community.aws-9 SUCCESS in 13m 56s
✔️ integration-community.aws-10 SUCCESS in 9m 02s
✔️ integration-community.aws-11 SUCCESS in 15m 11s
Skipped 11 jobs

@abikouo abikouo force-pushed the ssm_file_transfer_20250310 branch from a7c7b4f to 5c81cf6 Compare March 13, 2025 14:26
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/fefcb398ffaf40909727b58a610b6149

✔️ ansible-galaxy-importer SUCCESS in 3m 19s (non-voting)
✔️ build-ansible-collection SUCCESS in 10m 24s
✔️ ansible-test-splitter SUCCESS in 4m 36s
✔️ integration-community.aws-1 SUCCESS in 21m 17s
✔️ integration-community.aws-2 SUCCESS in 15m 44s
✔️ integration-community.aws-3 SUCCESS in 14m 24s
integration-community.aws-4 FAILURE in 21m 27s
✔️ integration-community.aws-5 SUCCESS in 12m 23s
✔️ integration-community.aws-6 SUCCESS in 14m 37s
✔️ integration-community.aws-7 SUCCESS in 14m 32s
✔️ integration-community.aws-8 SUCCESS in 13m 30s
✔️ integration-community.aws-9 SUCCESS in 14m 08s
✔️ integration-community.aws-10 SUCCESS in 5m 31s
✔️ integration-community.aws-11 SUCCESS in 15m 09s
✔️ integration-community.aws-12 SUCCESS in 14m 37s
Skipped 10 jobs

@abikouo abikouo force-pushed the ssm_file_transfer_20250310 branch from 5c81cf6 to b399656 Compare April 2, 2025 14:22
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/e8fc8c8d44394eeda34a748768f45591

ansible-galaxy-importer FAILURE in 4m 02s (non-voting)
✔️ build-ansible-collection SUCCESS in 10m 14s
✔️ ansible-test-splitter SUCCESS in 4m 07s
✔️ integration-community.aws-1 SUCCESS in 24m 55s
✔️ integration-community.aws-2 SUCCESS in 14m 28s
✔️ integration-community.aws-3 SUCCESS in 16m 24s
integration-community.aws-4 FAILURE in 21m 23s
✔️ integration-community.aws-5 SUCCESS in 14m 56s
✔️ integration-community.aws-6 SUCCESS in 17m 21s
✔️ integration-community.aws-7 SUCCESS in 15m 08s
✔️ integration-community.aws-8 SUCCESS in 17m 09s
✔️ integration-community.aws-9 SUCCESS in 14m 38s
✔️ integration-community.aws-10 SUCCESS in 5m 27s
✔️ integration-community.aws-11 SUCCESS in 15m 41s
✔️ integration-community.aws-12 SUCCESS in 14m 11s
Skipped 10 jobs

return content
raise TimeoutError(f"Unable to match expr '{expr}' from content")

def flush_stderr(self) -> str:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we handle potential errors that may occur during the read process like OSError or TimeoutError?



class AnsibleAwsSSMSession:
def __init__(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The constructor is quite long, and splitting it into smaller methods could improve readability. How do you feel about separating it into session management and socket handling?

def _socket_write(self, port: int, in_path: str) -> None:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as session:
self._socket_connect(session=session, port=port)
with open(in_path, "rb") as f:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use a try-except block here?

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/1aadb67a799d475e9ccaf544920bd781

✔️ ansible-galaxy-importer SUCCESS in 3m 25s (non-voting)
✔️ build-ansible-collection SUCCESS in 10m 46s
✔️ ansible-test-splitter SUCCESS in 3m 59s
integration-community.aws-1 TIMED_OUT in 1h 00m 30s
integration-community.aws-2 FAILURE in 25m 41s
integration-community.aws-3 FAILURE in 24m 12s
integration-community.aws-4 FAILURE in 20m 38s
integration-community.aws-5 FAILURE in 26m 58s
integration-community.aws-6 FAILURE in 24m 48s
integration-community.aws-7 FAILURE in 26m 55s
integration-community.aws-8 FAILURE in 23m 57s
integration-community.aws-9 FAILURE in 26m 48s
✔️ integration-community.aws-10 SUCCESS in 4m 21s
integration-community.aws-11 FAILURE in 27m 03s
integration-community.aws-12 FAILURE in 22m 06s
Skipped 10 jobs

Copy link
Contributor

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/ansible-collections/community.aws for 2265,f1454c107c1b19cc069205f22f680fd86ac64b70

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants