Docker Volumes in AWS - Write to EBS, EFS, S3

Got EFS, EBS, S3 Plugins with AWS Policy Working

Full answer details at https://github.com/codedellemc/rexray/issues/820#issuecomment-323624765.

• I used RexRay Docker Volume plugin for EBS, EFS, and S3 support
  • The volume allows you to automatically load one of those as a file-system mount inside the container without using ANY AWS API. That is, transparently to the container!!!

Here is the example

docker service create --replicas 10     --name echo-to-efs     --mount type=volume,source=echo_data,target=/test,volume-driver=rexray/efs     alpine /bin/sh -c 'echo "writing from $HOSTNAME" >> \                 /test/echo.log ; cat /test/echo.log ; sleep 30'iy4rzd4imq776abjlqdqf8l10

EFS MOUNTS

EBS MOUNTS

Install Volume Plugin

• Installation process described at https://rexray.readthedocs.io/en/stable/user-guide/docker-plugins/#amazon

Repeat for each of them, using their docker command-line arguments...

[root@ip-10-zz-yy-xx ec2-user]# sudo docker plugin install --grant-all-permissions rexray/s3fslatest: Pulling from rexray/s3fs5a6437c91cc6: Download completeDigest: sha256:6ba105974d5298ff49def3c83e5547d41e573e0a3cb8604f1dcb1a0f90ee070bStatus: Downloaded newer image for rexray/s3fs:latestInstalled plugin rexray/s3fs

Docker Plugin ls

• After setting up the UserData to install the 3 plugins , everything works properly!

[ec2-user@ip-10-82-48-59 ~]$ docker plugin lsID NAME DESCRIPTION ENABLED0eb15d4a9594 rexray/s3fs:latest REX-Ray FUSE Driver for Amazon Simple Stor... true6bae636cb377 rexray/ebs:latest REX-Ray for Amazon EBS true9e231d9177c2 rexray/efs:latest REX-Ray for Amazon EFS true

Docker Containers automatically creating/mounting Volumes

As a rule of thumb, I learned the following:

• EFS = Concurrent Writes (x replicas scenarios)
• EBS = Single Writes (1 replicate scenarios)

So, the example below shows the following:

• Create 10 Docker Containers concurrently writing to the same file

Last login: Mon Aug 21 00:34:46 2017 from 10.82.49.230[ec2-user@ip-10-zz-yy-xx ~]$ docker service create --replicas 10 --name echo-to-efs
--mount type=volume,source=echo_data,target=/test,volume-driver=rexray/efs alpine /bin/sh -c 'echo "writing from $HOSTNAME" >> /test/echo.log ; cat /test/echo.log ; sleep 30'iy4rzd4imq776abjlqdqf8l10

• The docker ps shows some of them writing and failures
  • If someone can explain why some failures in the case please let me know...
  • I have REXRAY_PREEMPT=true so, not sure if that's not related

[ec2-user@ip-10-82-48-59 ~]$ docker service ps echo-to-efsID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS8c8lxqcoc3n5 echo-to-efs.1 alpine:latest ip-10-82-48-59.vpc.internal Running Running 8 seconds ago23meciw5cvad echo-to-efs.2 alpine:latest ip-10-82-49-52.vpc.internal Running Running 2 seconds agosi7lg30kb13b \_ echo-to-efs.2 alpine:latest ip-10-82-48-93.vpc.internal Shutdown Failed about a minute ago "starting container failed: er…"rmj163xu97ai echo-to-efs.3 alpine:latest ip-10-82-48-96.vpc.internal Running Running 7 seconds ago5jtao2yu5udt echo-to-efs.4 alpine:latest ip-10-82-48-95.vpc.internal Running Running 8 seconds agowot9hd5nbjsz echo-to-efs.5 alpine:latest ip-10-82-49-71.vpc.internal Running Running 7 seconds agohcbbgfiuga9c echo-to-efs.6 alpine:latest ip-10-82-48-140.vpc.internal Running Starting 49 seconds agowrnnrsm16usr \_ echo-to-efs.6 alpine:latest ip-10-82-48-25.vpc.internal Shutdown Rejected 58 seconds ago "create echo_data: VolumeDrive…"w3v4x0pf9uri \_ echo-to-efs.6 alpine:latest ip-10-82-48-37.vpc.internal Shutdown Rejected about a minute ago "create echo_data: VolumeDrive…"okgda40hb4hy \_ echo-to-efs.6 alpine:latest ip-10-82-49-107.vpc.internal Shutdown Rejected about a minute ago "create echo_data: VolumeDrive…"bzoafno0gb4d echo-to-efs.7 alpine:latest ip-10-82-48-74.vpc.internal Running Running 7 seconds agork7uebz4cdql echo-to-efs.8 alpine:latest ip-10-82-49-111.vpc.internal Running Running 6 seconds agohy1rfid4zitl echo-to-efs.9 alpine:latest ip-10-82-48-162.vpc.internal Running Running 7 seconds agok5lmmmu44yh3 echo-to-efs.10 alpine:latest ip-10-82-48-187.vpc.internal Running Running 8 seconds ago

• Now we can verify if the file is being written...

[ec2-user@ip-10-82-48-59 ~]$ docker service logs echo-to-efsecho-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing from 3da21bc313f7echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing from 3da21bc313f7echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing from 334b26ce3747echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing from 3da21bc313f7echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing from 334b26ce3747echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing from 334b26ce3747echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing from fcefd6eb5035echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing from 84ffd56fdcfaecho-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing from b7ecbd01debeecho-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing from e5ef2a2821a2echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing from 5179f81f04f9echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing from 5e859dce0c2fecho-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing from fcefd6eb5035echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing from fcefd6eb5035echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing from 84ffd56fdcfaecho-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing from b7ecbd01debeecho-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing from 84ffd56fdcfaecho-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing from b7ecbd01debeecho-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing from e5ef2a2821a2echo-to-efs.4.5jtao2yu5udt@ip-10-82-48-95.vpc.internal | writing from 3da21bc313f7echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing from 5179f81f04f9echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing from e5ef2a2821a2echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing from 5e859dce0c2f......

• Now see the current value in the file.
  • The command here will mount the volume and at the same and then read the file from EFS mount.
  • It does take a long while until the container prepares the mount... Like 2min... Not sure if that's expected.

[ec2-user@ip-10-82-48-59 ~]$ docker run -ti --volume-driver=rexray/efs -v echo_data:/test alpine cat /test/echo.logUnable to find image 'alpine:latest' locallylatest: Pulling from library/alpineDigest: sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebeStatus: Downloaded newer image for alpine:latestwriting from 3da21bc313f7writing from 334b26ce3747writing from fcefd6eb5035writing from 84ffd56fdcfawriting from b7ecbd01debewriting from e5ef2a2821a2writing from 5179f81f04f9writing from 5e859dce0c2fwriting from af94a5a29b4awriting from a3f860caaf03writing from b7b3d30d2f8ewriting from cf65a52760c0writing from a551b529e599writing from 0533784ce9e5writing from d78f781baa4dwriting from 06b84229ec32writing from 4e3974cfb8a9writing from 007cd66062e7writing from 665f76589ae2writing from 68b04985559awriting from 9a2e19e110fbwriting from 1ed7903abf86writing from 763c7eea52fbwriting from 529c7fff9c71writing from 7172d5fb56d9writing from c33294ce8d9dwriting from b95a5b5f1bc1writing from ff269d8a78bawriting from b1299af12f30writing from 4f970a6e234dwriting from a4a1f331b3c9writing from bb3915d9d242writing from 0fbafba12a0cwriting from 017f70a769dawriting from 8d657b23571fwriting from 570e2fa76e16writing from 45a92f3d19b4writing from 3b02cb3675e4writing from e2d49358df6f

Docker Volumes Status

• Docker volumes now maps all the Mounts with the Plugins
  • This is a major win for me!

[ec2-user@ip-10-82-48-59 ~]$ docker volume lsDRIVER VOLUME NAMErexray/s3fs:latest 7820-3357-5600-us-west-2rexray/s3fs:latest cf-templates-8hmsxuf2muaa-us-west-2rexray/efs:latest echo_datarexray/s3fs:latest marcello-12345rexray/s3fs:latest marcello-testing-bucket-1rexray/s3fs:latest marcello-testing-bucket-2rexray/s3fs:latest new-config-publisherrexray/s3fs:latest publisher.intuit-caasrexray/s3fs:latest publisher.intuit-caas-new-updaterexray/ebs:latest quay-registry-configsrexray/ebs:latest quay-registry-images-datastorerexray/ebs:latest quay-registry-mysql-datastorerexray/ebs:latest registry_configsrexray/ebs:latest registry_mysql-datarexray/s3fs:latest swarmbucket-vpc-155e5072