Amazon EC2 SSH Public Keys via IAM Service

There will come a day when we can provision users and provide the necessary authorization to required enterprise resources with easy, but until that day comes there will be hacks like this one.

This is very much Proof of Concept (PoC), I have not tested this beyond my network, and there would be a lot of angles to consider with any type of deployment, security (sanitization and additional logic), IAM SSH Public Keys were designed for CodeCommit, and other unknowns.

My first attempt was to create a PAM module that would authenticate SSH users via aws-sdk-cpp libraries, accessing the API using instance profile / EC2 IAM roles, allowing EC2 instances to obtain credentials to access the information. This seemed like a good route, but trying to hack a PAM module together that would be portable enough for distribution would be a challenge, static versions of aws-sdk-cpp, etc. I will revisit this in the future, but for now, I will move on.

On my second attempt, I create a Bash script that would produce the correct output for SSH AuthorizedKeysCommand directive using SSH public keys from AWS IAM service via awscli. As the EC2 instance can utilize the instance profile role via aws command. This didn’t come without headaches, permission issues, etc.

Check sshd DEBUG logs if having issues with execution.

chmod 0744 /usr/local/bin/aws-iam-authorized-keys
chown root:root /usr/local/bin/aws-iam-authorized-keys

Once I was finally able to get sshd to execute aws-iam-authorized-keys (my aptly named script), it was able to query for the username’s public SSH keys in IAM and output them in the proper format.

For the EC2 instance to access the IAM to list and retrieve keys, it will require an instance profile, that allows ListSSHPublicKeys and GetSSHPublicKey actions, similar to the following IAM role policy I have been using. You can read more in the user guide under IAM Roles for Amazon EC2:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetSSHPublicKey",
                "iam:ListSSHPublicKeys"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

The two /etc/ssh/sshd_config directives

AuthorizedKeysCommand /usr/local/bin/aws-iam-authorized-keys
AuthorizedKeysCommandUser nobody

The daemon sshd will pass the username to the script or application that you supply to AuthorizedKeysCommand as the first argument, and will execute it as the user-defined in AuthorizedKeysCommandUser.

In this scenario, I have a user in IAM with the name jonathan, SSH public key, SSH private key on my notebook (passphrase protected), local EC2 Linux user jonathan. This will allow my notebook SSH client to authenticate, after I provide the passphrase of course.

jonathan$ ssh [email protected]
Enter passphrase for key '/Users/jonathan/.ssh/id_rsa': 
Last login: Thu Jan 26 03:44:03 2017 from

 __| __|_ )
 _| ( / Amazon Linux AMI
 ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.09-release-notes/

[[email protected] ~] $ id uid=501(jonathan) gid=501(jonathan) groups=501(jonathan)
[[email protected] ~] $ ls -la .ssh/ total 8 drwxr-xr-x 2 jonathan jonathan 4096 Jan 26 02:12 . drwx------ 3 jonathan jonathan 4096 Jan 26 03:26 .. 
[[email protected] ~]

Eureka! We have made a successful authentication. I have increased the log level to DEBUG using the sshd_config directive LogLevel. With this, we can see what sshd is doing.

sshd[5457]: debug1: userauth-request for user jonathan service ssh-connection method none [preauth]
sshd[5457]: debug1: attempt 0 failures 0 [preauth]
sshd[5457]: debug1: PAM: initializing for "jonathan"
sshd[5457]: debug1: PAM: setting PAM_RHOST to "X"
sshd[5457]: debug1: PAM: setting PAM_TTY to "ssh"
sshd[5457]: debug1: userauth-request for user jonathan service ssh-connection method publickey [preauth]
sshd[5457]: debug1: attempt 1 failures 0 [preauth]
sshd[5457]: debug1: test whether pkalg/pkblob are acceptable [preauth]
sshd[5457]: debug1: temporarily_use_uid: 99/99 (e=0/0)
sshd[5457]: debug1: restore_uid: 0/0
sshd[5457]: debug1: temporarily_use_uid: 99/99 (e=0/0)
sshd[5457]: Found matching RSA key:
sshd[5457]: debug1: restore_uid: 0/0
sshd[5457]: Postponed publickey for jonathan from X port 53067 ssh2 [preauth]
sshd[5457]: debug1: userauth-request for user jonathan service ssh-connection method publickey [preauth]
sshd[5457]: debug1: attempt 2 failures 0 [preauth]
sshd[5457]: debug1: temporarily_use_uid: 99/99 (e=0/0)
sshd[5457]: debug1: restore_uid: 0/0
sshd[5457]: debug1: temporarily_use_uid: 99/99 (e=0/0)
sshd[5457]: Found matching RSA key:
sshd[5457]: debug1: restore_uid: 0/0
sshd[5457]: debug1: ssh_rsa_verify: signature correct
sshd[5457]: debug1: do_pam_account: called
sshd[5457]: Accepted publickey for jonathan from X port 53067 ssh2: RSA
sshd[5457]: debug1: monitor_child_preauth: jonathan has been authenticated by privileged process
sshd[5457]: debug1: monitor_read_log: child log fd closed
sshd[5457]: debug1: SELinux support disabled
sshd[5457]: debug1: PAM: establishing credentials
sshd[5457]: pam_unix(sshd:session): session opened for user jonathan by (uid=0)
sshd[5457]: User child is on pid 5513
sshd[5513]: debug1: PAM: establishing credentials
sshd[5513]: debug1: permanently_set_uid: 501/501
sshd[5513]: debug1: Entering interactive session for SSH2.

We can verify the output by running aws-iam-authorized-keys from the command line on an EC2 instance. Output was reduced for brevity.

jonathan$ /usr/local/bin/aws-iam-authorized-keys jonathan
ssh-rsa AAAAB3Nza...noc7qee3 jonathan

Although I am authenticating to a local Linux user account, which doesn’t scale that well, the account information could have come from an external directory like Active Directory or RADIUS via PAM modules.

To recap the instance-side tool chain is as follows, sshd -> ams-iam-authorized-keys -> awscli -> IAM.

You can find the script in Downloads. Enjoy.

Jan 26th, 2017 • Filed under Amazon Web Service, Bash, CLI, EC2, IAM, Linux, SSH

Getting Started with Amazon AWS CLI

Like any good service Amazon Web Services (AWS) has a command line tool that interfaces with it’s various services.

Being the geek that I am, I have always preferred the command line (CLI), be it Cisco IOS, Bash, or the earlier MS-DOS. So getting acquainted with this tool only seems natural, on my journey to learn and discover more about AWS. Command line tools, albeit usually a high learning curve, give way to the underpinnings of the services.

Learning how to use it has it’s advantages, for one it gives to script-ability, therefore complex idempotent tasks can be codified. Reducing technical debt, and really get return on investment, by doing work once, and having the entire company benefit from the scripts that come out of that work.

After all, it’s a lot easier explaining how to run a script, than it is to explain steps that needs to be done, some times in specific order, etc.

Getting Command Line Tools

If you’re not working from AMI Linux, lets get the tools installed. I am working from a CentOS machine, I found them in the repository as package awscli. There are other methods of installation for example pip, easy_install, and of course a clone of the git repository.

# yum install awscli
Loaded plugins: fastestmirror
...
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
 Installing : libjpeg-turbo-1.2.90-5.el7.x86_64 1/15 
 Installing : libwebp-0.3.0-3.el7.x86_64 2/15 
 Installing : python2-jmespath-0.9.0-1.el7.noarch 3/15 
 Installing : jbigkit-libs-2.0-11.el7.x86_64 4/15 
 Installing : libtiff-4.0.3-25.el7_2.x86_64 5/15 
 Installing : python-pillow-2.0.0-19.gitd1c6db8.el7.x86_64 6/15 
 Installing : python-docutils-0.11-0.2.20130715svn7687.el7.noarch 7/15 
 Installing : python-colorama-0.3.2-3.el7.noarch 8/15 
 Installing : python2-futures-3.0.5-1.el7.noarch 9/15 
 Installing : python2-pyasn1-0.1.9-7.el7.noarch 10/15 
 Installing : python2-rsa-3.4.1-1.el7.noarch 11/15 
 Installing : python-dateutil-1.5-7.el7.noarch 12/15 
 Installing : python2-botocore-1.4.85-1.el7.noarch 13/15 
 Installing : python2-s3transfer-0.1.9-1.el7.noarch 14/15 
 Installing : awscli-1.11.28-2.el7.noarch 15/15 
 Verifying : python-docutils-0.11-0.2.20130715svn7687.el7.noarch 1/15 
 Verifying : python2-botocore-1.4.85-1.el7.noarch 2/15 
 Verifying : python-dateutil-1.5-7.el7.noarch 3/15 
 Verifying : python2-pyasn1-0.1.9-7.el7.noarch 4/15 
 Verifying : libtiff-4.0.3-25.el7_2.x86_64 5/15 
 Verifying : python2-rsa-3.4.1-1.el7.noarch 6/15 
 Verifying : python-pillow-2.0.0-19.gitd1c6db8.el7.x86_64 7/15 
 Verifying : python2-futures-3.0.5-1.el7.noarch 8/15 
 Verifying : python-colorama-0.3.2-3.el7.noarch 9/15 
 Verifying : python2-s3transfer-0.1.9-1.el7.noarch 10/15 
 Verifying : libjpeg-turbo-1.2.90-5.el7.x86_64 11/15 
 Verifying : jbigkit-libs-2.0-11.el7.x86_64 12/15 
 Verifying : python2-jmespath-0.9.0-1.el7.noarch 13/15 
 Verifying : libwebp-0.3.0-3.el7.x86_64 14/15 
 Verifying : awscli-1.11.28-2.el7.noarch 15/15 

Installed:
 awscli.noarch 0:1.11.28-2.el7 
...
Complete!
#

As you can  see Python 2 was installed as a dependency of awscli. If you are running a different OS, refer to the user guide to learn how to install for the platform of your choosing.

Supported services

The list of supported services can be found in the documentation, under available services.

Your environment, it’s important.

The AWS CLI tool makes use of the following environment variables.

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY hold the Access Key ID and Secret Access Key respectively. AWS_CONFIG_FILE holds the path to a AWS configuration file. Some of the tools make use of additional environment variables. EC2_CERT and EC2_PRIVATE_KEY are used to hold the paths to the public and private key files respectively.

I created the shell script awsenv to display this information and perform some permission validation on the credential files. The script is non-destructive, other than permission changes after prompting the user, running it will do no harm.

How you set the variables is up to you. I usually set them via shell script that isn’t part of a repository and source from .bashrc, or you can just set them temporarily in your shell. Example output from awsenv script.

Running my script awsenv gives an overview of what is configured.

$ awsenv 
[x] Amazon AWS CLI environment
[x] Checking for AWS CLI secrets and credential files...
Checking /root/.aws_secrets...found
Checking /root/.s3curl...not found
[!] AWS_S3_CURL contains path to file that doesn't exist.
Checking /root/.aws/credentials.json...found
Checking /root/.aws/aws_config_file...found

[x] AWS variables
AWS_ACCESS_KEY_ID: ABCDEFGH
AWS_SECRET_ACCESS_KEY: 12345678
AWS_CONFIG_FILE: /root/.aws/aws_config_file
AWS_CREDENTIAL_FILE: 
EC2_CERT: 
EC2_PRIVATE_KEY:

Ground Control to Major Tom!

Once the environment variables have been set and the credential files created with correct content and proper permissions. We can verify our setup with a single command.

# aws iam get-user
{
   "User": {
     "UserName": "jonathan", 
     "Path": "/", 
     "CreateDate": "2016-10-01T18:22:32Z", 
     "UserId": "Y", 
     "Arn": "arn:aws:iam::X:user/jonathan" } 
}

Executing the command aws iam get-user returns your user information out of AWS IAM. The user/jonathan is a member of a group that gives Administrator privileges, if you’re having troubles, verify your account has the correct authorization.

You should now be able to run any necessary AWS CLI-based scripts, given the Access Key ID you’re using with awscli has the correct authorization.

There is a aws-shell program that gives command completion, I have not tried this, one step at a time for me.

Install Command Completion

complete -C aws_completer aws
Jan 18th, 2017 • Filed under Amazon Web Service, CLI, EC2, IAM, S3

Release of aslookup

Recently I have been hacking away at some C code, I had wanted to create a tool to lookup Border Gateway Protocol (BGP) Autonomous System (AS) numbers from the command line with ease. I had some time, so I decided to code a little.

I was able to create this tool for performing BGP AS Number lookup, aslookup. Its concept is similar to nslookup, just pass it an AS number or IP address.

Example:

$ aslookup 8.8.8.8
15169 | US | arin | 2000-03-30 |
$ aslookup AS15169
15169 | US | arin | 2000-03-30 | GOOGLE - Google Inc., US

This tool queries the DNS zones of Team Cymru. The tool has options to query IPv4, IPv6, and ASN for Autonomous System (AS) information. In the future, I may add an option to query AS information from an AS or peers lookup.

More can be found on the aslookup page or just download.

The code is released under the GPL2.

Dec 14th, 2016 • Filed under Bash, Linux, Networking, Routing, Troubleshooting

Update MySQL Server Time Zone Data

I started to get the error message Database returned an invalid value in QuerySet.dates(). Are time zone definitions and pytz installed? from a Django application. These messages presented when I started to utilize date_hierarchy.

From the MySQL server run the following command

mysql_tzinfo_to_sql /usr/share/zoneinfo/ | mysql -u root -p mysql

This will parse the zone info directory and send the SQL output to the mysql command, which will update the time zone information on the server. Once this was done, I could proper utilize the date hierarchy features of Django admin.

Nov 11th, 2016 • Filed under Databases, Django, Linux, MySQL, Python

Quickly Find Active Directory Object Distinguished Name (DN)

A lot of Microsoft command line utilities require information in the form of Distinguished Name (DN), this value can be quickly found using the utility ldp.exe, found on Domain Controllers (DC), and client computers with the Administration Tools.

From the Run prompt or Command Prompt launch ldp.exe. This should present you with a dual pane dialog box utility. The first step is to make a connection to a DC, Connection > Connect…, you can leave the hostname field blank to connect to the DC you are currently on, otherwise type the DC you wish to connect. Select SSL, if your environment uses LDAPS.

connect

Next you need to bind (Connection > Bind…) using credentials, these could be credential you are currently logged in as, or this can be credentials that you provide.

bind

Once you complete the bind, click View and select Tree, select from the combo dropdown box the BaseDN you will like to browse. Click OK, Active Directory objects should be visible in the left pane, and the output of object selection should be in the right pane.

tree-view

In the left pane, navigate to the object, select the object. Double clicking the object will attempt to expand the it further, printing details in the right pane. The DN value is in bold near the top of the object output. Right click the object and click Copy DN to copy the object DN value to the clipboard.

Now with the value on the clipboard, you can paste the value to where ever it is needed, if rdpclip is behaving you can copy between server and client.

Oct 21st, 2016 • Filed under Active Directory, Microsoft, Windows Server