Friday, 23 June 2017

install AWS command line on linux for data movement

I need to move a lot of data quickly, so I’m going to use an S3 bucks and install the aws cli on my linux hosts to be able to put and get.
Let’s begin the dance…
to use aws-cli, you need to install with pip.
to install pip https://pip.pypa.io//en/latest/installing/#get-pip-py-options you need to wget or at least copy a copy of get-pip.py for python > 2.6
remember that you need to set the proxy (more than likely on your server)
export http_proxy=http://moirs:Password\!@proxy:8080
Then test it
[root@ronin0 ~]# wget www.google.com
--2017-06-23 16:05:57--  http://www.google.com/
Resolving proxy... 10.241.10.79
Connecting to proxy|10.241.10.79|:8080... connected.
Proxy request sent, awaiting response... 302 Found
Location:
http://www.google.co.nz/?gfe_rd=cr&ei=pZNMWeKlFqHM8gfo7LDQCg [following]
--2017-06-23 16:05:57-- 
http://www.google.co.nz/?gfe_rd=cr&ei=pZNMWeKlFqHM8gfo7LDQCg
Connecting to proxy|10.241.10.79|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html.1'

    [ <=>                                                                                                                        ] 13,313      --.-K/s   in 0.001s
2017-06-23 16:05:57 (15.6 MB/s) - `index.html.1' saved [13313]
Tidy, proxy is good

Try the following to download
wget https://bootstrap.pypa.io/get-pip.py

Now you need to run the get-pip.py script that you’ve copied in
python get-pip.py --proxy="http://moirs:Password\!@proxy:8080"
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Collecting pip
/tmp/tmpHeNyVB/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
/tmp/tmpHeNyVB/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% |################################| 1.3MB 600kB/s
Collecting setuptools
  Downloading setuptools-36.0.1-py2.py3-none-any.whl (476kB)
    100% |################################| 481kB 1.4MB/s
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% |################################| 71kB 1.3MB/s
Collecting argparse; python_version == "2.6" (from wheel)
  Downloading argparse-1.4.0-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, argparse, wheel
  Found existing installation: argparse 1.2.1
    Uninstalling argparse-1.2.1:
      Successfully uninstalled argparse-1.2.1
Successfully installed argparse-1.4.0 pip-9.0.1 setuptools-36.0.1 wheel-0.29.0
/tmp/tmpHeNyVB/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can
Great, now I can install aws-cli
[root@ronin0 ~]# pip --proxy http://moirs:Password\!@proxy:8080 install awscli
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Collecting awscli
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading awscli-1.11.111-py2.py3-none-any.whl (1.2MB)
    100% |################################| 1.2MB 585kB/s
Collecting botocore==1.5.74 (from awscli)
  Downloading botocore-1.5.74-py2.py3-none-any.whl (3.5MB)
    100% |################################| 3.5MB 282kB/s
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
  Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
    100% |################################| 51kB 5.1MB/s
Collecting s3transfer<0.2.0,>=0.1.9 (from awscli)
  Downloading s3transfer-0.1.10-py2.py3-none-any.whl (54kB)
    100% |################################| 61kB 82kB/s
Requirement already satisfied: argparse>=1.1; python_version == "2.6" in /usr/lib/python2.6/site-packages (from awscli)
Collecting docutils>=0.10 (from awscli)
  Downloading docutils-0.13.1-py2-none-any.whl (537kB)
    100% |################################| 542kB 1.3MB/s
Collecting colorama<=0.3.7,>=0.2.5 (from awscli)
  Downloading colorama-0.3.7-py2.py3-none-any.whl
Collecting PyYAML<=3.12,>=3.10 (from awscli)
  Downloading PyYAML-3.12.tar.gz (253kB)
    100% |################################| 256kB 1.9MB/s
Collecting simplejson==3.3.0; python_version == "2.6" (from botocore==1.5.74->awscli)
  Downloading simplejson-3.3.0.tar.gz (67kB)
    100% |################################| 71kB 4.0MB/s
Collecting ordereddict==1.1; python_version == "2.6" (from botocore==1.5.74->awscli)
  Downloading ordereddict-1.1.tar.gz
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.5.74->awscli)
  Downloading python_dateutil-2.6.0-py2.py3-none-any.whl (194kB)
    100% |################################| 194kB 2.3MB/s
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.5.74->awscli)
  Downloading jmespath-0.9.3-py2.py3-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
  Downloading pyasn1-0.2.3-py2.py3-none-any.whl (53kB)
    100% |################################| 61kB 4.8MB/s
Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.9->awscli)
  Downloading futures-3.1.1-py2-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.5.74->awscli)
  Downloading six-1.10.0-py2.py3-none-any.whl
Building wheels for collected packages: PyYAML, simplejson, ordereddict
  Running setup.py bdist_wheel for PyYAML ... done
  Stored in directory: /root/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
  Running setup.py bdist_wheel for simplejson ... done
  Stored in directory: /root/.cache/pip/wheels/5a/a5/b9/b0c89f0c5c40e2090601173e9b49091d41227c6377020e4e68
  Running setup.py bdist_wheel for ordereddict ... done
  Stored in directory: /root/.cache/pip/wheels/cf/2c/b5/a1bfd8848f7861c1588f1a2dfe88c11cf3ab5073ab7af08bc9
Successfully built PyYAML simplejson ordereddict
Installing collected packages: simplejson, ordereddict, six, python-dateutil, jmespath, docutils, botocore, pyasn1, rsa, futures, s3transfer, colorama, PyYAML, awscli
  Found existing installation: simplejson 2.0.9
    Uninstalling simplejson-2.0.9:
      Successfully uninstalled simplejson-2.0.9
  Found existing installation: ordereddict 1.2
    DEPRECATION: Uninstalling a distutils installed project (ordereddict) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling ordereddict-1.2:
      Successfully uninstalled ordereddict-1.2
Successfully installed PyYAML-3.12 awscli-1.11.111 botocore-1.5.74 colorama-0.3.7 docutils-0.13.1 futures-3.1.1 jmespath-0.9.3 ordereddict-1.1 pyasn1-0.2.3 python-dateutil-2.6.0 rsa-3.4.2 s3transfer-0.1.10 simplejson-3.3.0 six-1.10.

Now,
Now we can run it, but remember https_proxt envrionment variable too
export https_proxy=http://moirs:Password\!@proxy:8080
aws configure
AWS Access Key ID [None]: GFDGHTRHBT
AWS Secret Access Key [None]: GFDHGF
Default region name [None]: ap-southeast-2
Default output format [None]:
Note that you get your secret key and access ID from the AWS console for your username
you are cooking with gas…
aws s3 ls

No comments:

Extending JDE to generative AI